/// article

What to Do After Discovering Your Linux Mail Server Was Compromised

Discovering malware or unauthorized access on a Linux mail server changes the situation immediately. You are no longer troubleshooting an ordinary service failure. You are responding to a security incident. Killing a suspicious process, deleting a malicious file, or removing a bad cron entry may stop what you can currently see. It does not prove […]

By lordfrancs3

Discovering malware or unauthorized access on a Linux mail server changes the situation immediately. You are no longer troubleshooting an ordinary service failure. You are responding to a security incident. Killing a suspicious process, deleting a malicious file, or removing a bad cron entry may stop what you can currently see. It does not prove that the server is clean. The attacker may have installed another persistence mechanism, added SSH keys, stolen administrator credentials, copied TLS private keys, modified application files, created users, installed systemd services, altered mail routing, or moved to another server. This guide explains how to contain, investigate, preserve evidence from, and recover a compromised Linux mail server. The procedures are based partly on lessons from our compromised Zimbra server, where we encountered malicious processes, executable files in temporary and unusual directories, cron-based persistence, suspicious SSH keys, and evidence of successful SSH authentication. The principles apply to most Linux mail servers. Current NIST guidance treats incident response as a cycle involving detection, response, recovery, and continuous improvement. CISA also recommends quarantining compromised hosts, collecting artifacts, reimaging confirmed compromised systems, and issuing new credentials. 1. Do Not Treat Malware Removal as Incident Recovery Suppose you discover this: ps aux | grep suspicious-process You kill it: kill -9 PID Then you delete its executable: rm /path/to/malware The process disappears. That does not mean the system is trustworthy again. An attacker who gained sufficient access may have made changes you have not discovered. For example, they may have: Added another cron job Installed a systemd service Added an SSH authorized key Created another user Changed sudo configuration Modified shell startup files Installed a kernel-level rootkit Modified the mail application Stolen passwords Copied API credentials Copied TLS private keys Copied DKIM signing keys Modified mailbox forwarding rules Installed another payload under a different filename Moved laterally to another server This is the main mindset change required during incident response. You are not asking: “Did I remove the malware?” You are asking: “Can I prove that I understand what the attacker did, what credentials were exposed, what other systems may have been reached, and whether this operating system can still be trusted?” If the answer is no, rebuilding is usually safer than continuing to clean the existing installation. CISA’s guidance for confirmed Zimbra compromises specifically recommends collecting and reviewing artifacts, quarantining affected hosts, reimaging compromised hosts, and provisioning new credentials. 2. Lessons From Our Compromised Zimbra Server Our Zimbra incident demonstrated why visible malware is only one part of a compromise. We encountered suspicious executables and related files under locations including: /dev/shm/ /home/SSL/ Suspicious process names included: idle javab We also discovered persistence through a Zimbra user’s crontab. A malicious command was being executed every minute. The investigation later found suspicious entries in: /opt/zimbra/.ssh/authorized_keys and authentication logs showed successful SSH public-key access to the Zimbra account. This changed the incident significantly. The problem was no longer: Malware process -> kill malware -> problem solved It became: Initial compromise | +--> malicious executable | +--> cron persistence | +--> SSH persistence | +--> possible credential exposure | +--> possible lateral movement | +--> unknown additional modifications That is the correct way to think about a compromised mail server. 3. First Priority: Contain the Server Once you have credible evidence of compromise, limit the attacker’s ability to continue operating. CISA recommends immediately isolating affected systems and collecting relevant logs, data, and artifacts. Possible containment methods include disconnecting the server from the network, placing it in an isolated VLAN, restricting communication using upstream firewall rules, or blocking internet access while allowing only controlled incident-response access. CISA specifically recognizes physical disconnection, network-layer isolation, restricted VLANs, and strict firewall controls as valid isolation approaches depending on the situation. For a mail server, however, containment requires judgment. Immediately unplugging the server may destroy volatile information that exists only in RAM. That information can include: Running malware Process arguments Network sessions Open files Logged-in sessions Temporary credentials Encryption material Command history still held in memory Attacker connections NIST SP 800-86 explains that volatile information can disappear after shutdown or reboot and recommends deciding early whether it needs to be collected. If an attacker is actively using the machine, containment normally takes priority. If the attacker appears inactive and you have incident-response capability available, collect important volatile information before shutting the machine down. 4. Do Not Reboot Immediately One of the first instincts administrators have is: reboot Avoid doing this automatically. A reboot destroys useful volatile evidence. It can remove: Running processes Memory contents Current network connections Temporary files in tmpfs Processes deleted from disk but still running Active attacker sessions It can also trigger persistence mechanisms that only execute at boot. NIST notes that collecting volatile operating-system information is only possible while the affected system remains running. There are situations where immediate shutdown is justified, particularly when continued operation creates severe business or security risk. The key is to make the decision deliberately rather than rebooting as a troubleshooting habit. 5. Start an Incident Timeline Write down what happened from the moment you discovered the issue. Record timestamps in UTC where possible. For example: date -u Record: When the issue was discovered Who discovered it Symptoms observed Suspicious filenames Process IDs User accounts involved Source IP addresses Destination IP addresses Commands executed by administrators Files deleted or moved Firewall changes Accounts disabled Credentials rotated Systems disconnected Services stopped Do not rely on memory. Your own administrative commands change the compromised system. NIST forensic guidance stresses documenting evidence handling and the actions taken during an investigation. For serious incidents, preserve a formal chain of custody if legal action, insurance, contractual investigation, or regulatory reporting may become relevant. 6. Capture Volatile Information Before killing suspicious processes, collect enough information to understand them. Create an evidence destination. Ideally, use external storage or a trusted remote collection system rather than storing everything on the compromised server. For a basic triage collection: date -u hostname uname -a uptime who w last lastb Capture running processes: ps auxww A process tree can reveal parent-child relationships: ps auxf or: pstree -ap Check active network connections: ss -plant and UDP: ss -panu Check listening services: ss -lntup Open files can also be useful: lsof -nP Check deleted files that are still held open by processes: lsof +L1 This is particularly useful when an attacker deletes a malware executable after launching it. You may see something similar to: malware 12345 attacker txt REG ... /tmp/.something (deleted) Do not assume the malware is gone merely because its file is no longer visible. 7. Preserve Suspicious Files Before Deleting Them If you discover something suspicious such as: /tmp/.something /dev/shm/program /home/user/.hiddenfile do not immediately run: rm -rf suspicious-file Preserve it first. Never execute the suspicious file. Record metadata: stat suspicious-file Check its type: file suspicious-file Inspect extended attributes if available: lsattr suspicious-file Calculate cryptographic hashes: sha256sum suspicious-file sha512sum suspicious-file Copy it to protected evidence storage: cp -a suspicious-file /mnt/incident-evidence/ Then hash the preserved copy again: sha256sum /mnt/incident-evidence/suspicious-file Hashes help establish that the file being analyzed later is the same file you originally collected. If you have proper forensic capability, acquiring a disk image is preferable to performing extensive analysis directly on the affected filesystem. NIST SP 800-86 explains that forensic analysis is safer when performed on captured evidence rather than repeatedly modifying the original system. 8. Stop Malicious Processes When Necessary If malicious activity is active and containment cannot sufficiently restrict it, stopping the process may be necessary. First identify it: ps auxww | grep PROCESS Check its executable: readlink -f /proc/PID/exe Inspect its command line: tr '\0' ' ' < /proc/PID/cmdline echo Check its environment if required: tr '\0' '\n' < /proc/PID/environ Check open network connections: lsof -Pan -p PID -i Preserve these results before terminating the process when possible. Then: kill PID If it refuses to terminate: kill -9 PID Be aware that killing the process only terminates that instance. If persistence remains, it may return within seconds or minutes. That is exactly what happened during our Zimbra investigation. Killing malicious processes did not resolve the issue because another mechanism was starting them again. 9. Search for Cron Persistence Cron remains a common persistence mechanism because it is simple and easy to hide among legitimate scheduled tasks. Check the root crontab: crontab -l Check application users: crontab -u zimbra -l or: crontab -u postfix -l Check all user crontabs where appropriate. Also inspect: cat /etc/crontab and: ls -la /etc/cron.d/ ls -la /etc/cron.hourly/ ls -la /etc/cron.daily/ ls -la /etc/cron.weekly/ ls -la /etc/cron.monthly/ Search scheduled-task configuration for suspicious paths: grep -R "/tmp\|/dev/shm\|/var/tmp\|/home" \ /etc/cron* \ /var/spool/cron* \ 2>/dev/null Do not assume a cron line is legitimate because it runs under an application account. In our Zimbra incident, malware persistence existed directly in the Zimbra user’s crontab and ran every minute. That explained why deleting or killing the visible malware did not permanently stop it. Before removing a malicious cron entry, preserve: The complete crontab Modification timestamps The executed command Referenced scripts and binaries Hashes of those files Then remove or disable it. 10. Check systemd Persistence Modern Linux systems provide attackers with another powerful persistence mechanism through systemd. List services: systemctl list-units --type=service --all List installed service definitions: systemctl list-unit-files --type=service Check timers: systemctl list-timers --all systemctl list-timers displays systemd timer units and the services they activate. Review system service locations: /etc/systemd/system/ /run/systemd/system/ /usr/lib/systemd/system/ /lib/systemd/system/ Depending on the distribution, some of these directories may not exist. Look for recently modified unit files: find /etc/systemd/system \ /usr/lib/systemd/system \ /lib/systemd/system \ -type f \ -mtime -30 \ -ls 2>/dev/null Inspect suspicious units: systemctl cat suspicious.service Check their status: systemctl status suspicious.service Also investigate per-user systemd services: ~/.config/systemd/user/ Attackers do not need to name a malicious service: malware.service They may use innocent-looking names such as: system-update.service network-helper.service cache-cleaner.service Judge the unit by what it executes, where that executable came from, who created it, and whether it belongs to a legitimate package. 11. Check Other Persistence Locations Cron and systemd are important, but they are not the only mechanisms. Inspect: /etc/rc.local /etc/init.d/ /etc/profile /etc/profile.d/ ~/.bashrc ~/.profile ~/.bash_profile ~/.ssh/rc /etc/ssh/sshrc Check scheduled at jobs: atq Investigate new or unexpected users: cat /etc/passwd Look for users with UID 0: awk -F: '$3 == 0 {print}' /etc/passwd Normally only root should have UID 0. Inspect sudo configuration: visudo -c and: ls -la /etc/sudoers.d/ Do not limit your investigation to one persistence mechanism simply because you already found one. Attackers can establish several independent access paths. 12. Investigate SSH Access Carefully SSH deserves special attention after a Linux server compromise. Check: /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys Application users should also be inspected. For Zimbra: /opt/zimbra/.ssh/authorized_keys OpenSSH normally obtains user public keys from configured authorized-key files. The standard configuration commonly includes .ssh/authorized_keys, although AuthorizedKeysFile can be changed in sshd_config. Search broadly: find /root /home /opt \ -type f \ -name authorized_keys \ -print 2>/dev/null Do not look only for unknown public keys. OpenSSH authorized-key lines can contain options before the key. For example: command="some-command",no-port-forwarding ssh-ed25519 AAAA... Options such as command= can force execution of a particular command whenever that key authenticates. This makes an unexpected authorized-key entry particularly important during an investigation. OpenSSH documents the authorized-key format and its optional key settings. In our Zimbra investigation, suspicious entries in the Zimbra account’s authorized-key configuration became an important indicator that the compromise involved more than a single malware process. 13. Review SSH Authentication Logs Now determine whether suspicious keys or passwords were actually used. On Ubuntu and Debian: grep sshd /var/log/auth.log Search successful authentication: grep "Accepted" /var/log/auth.log Specifically: grep "Accepted publickey" /var/log/auth.log and: grep "Accepted password" /var/log/auth.log On systems using the journal: journalctl -u ssh or: journalctl -u sshd Search by time: journalctl -u sshd \ --since "2026-09-01 00:00:00" \ --until "2026-09-02 23:59:59" Look for: Unexpected IP addresses Unexpected login times Application-account SSH access Root login Authentication from neighboring servers New public keys Repeated failures followed by success In our incident, successful public-key authentication from an internal address was particularly important. An internal source address does not automatically make authentication legitimate. It can indicate that another internal machine was compromised first. 14. Investigate Lateral Movement Once a server is confirmed compromised, ask: What systems could this server access? Check: Database servers LDAP servers Backup servers Monitoring servers Storage servers DNS servers Web applications Management servers Other mail nodes File servers Hypervisors Cloud services Administrative workstations Look at SSH client configuration: cat ~/.ssh/config Search for private keys: find /root /home /opt \ -type f \ \( -name "id_rsa" \ -o -name "id_ed25519" \ -o -name "*.pem" \ -o -name "*.key" \) \ -ls 2>/dev/null Do not assume those files are all malicious. The goal is to identify credentials the compromised host could use elsewhere. Check known hosts: find /root /home /opt \ -type f \ -name known_hosts \ -print 2>/dev/null These files can help identify systems administrators or applications have previously connected to. Also examine shell history where available: cat ~/.bash_history but remember that attackers can delete, modify, disable, or bypass shell history. CISA recommends assuming possible lateral movement in serious compromises and investigating connected systems rather than limiting analysis to the first discovered host. 15. Rotate Credentials From a Trusted Machine Do not change passwords while assuming the compromised server can safely handle those new credentials. If an attacker installed credential-stealing software, your newly entered password could immediately be captured. Perform credential rotation from a known-good administrative workstation. Potential credentials include: Root password Administrator passwords Application administrator accounts SSH keys Database credentials LDAP credentials SMTP relay credentials Backup credentials Object-storage credentials Monitoring credentials API tokens Cloud credentials DNS credentials Registrar credentials Service-account passwords Automation credentials For a mail system, also consider credentials stored inside application configuration files. CISA recommends provisioning new account credentials after confirmed compromise. Do not simply change the password of the account you know was abused. Assume any credential accessible to the attacker may have been copied. 16. Rotate SSH Keys Where Exposure Is Possible If an attacker obtained root privileges, assume they could read SSH private keys stored on that server. Generate replacement keys on a trusted machine. Remove unauthorized public keys from servers where the old key had access. Then revoke or remove the old legitimate key where compromise cannot be ruled out. Do not forget service accounts. If the compromised mail server used passwordless SSH to: Backup servers Storage systems Automation hosts Monitoring machines Other mail servers those relationships must be reviewed. Otherwise rebuilding the mail server may still leave the attacker with valid credentials to another system. 17. Review TLS Certificates and Private Keys Mail servers contain valuable private keys. These can include: HTTPS certificates SMTP TLS certificates IMAP TLS certificates LDAP certificates Internal CA certificates Client certificates If an attacker had sufficient privilege to read those private keys, consider them exposed. Inspect your certificate files and determine: Which private keys were present Which accounts could read them Whether the attacker reached those accounts Whether those keys protect public services Whether certificates should be revoked and reissued You can inspect a certificate with OpenSSL: openssl x509 \ -in certificate.crt \ -noout \ -subject \ -issuer \ -dates \ -fingerprint \ -sha256 If compromise of the private key is possible, generate a new key pair and reissue the certificate rather than simply reinstalling the same certificate. OpenSSL provides mechanisms for checking relationships between certificates and private keys, but cryptographic consistency does not tell you whether a private key was stolen. 18. Do Not Forget DKIM Keys This is particularly important for mail servers. If your server performs DKIM signing, private DKIM keys may be stored locally. An attacker who copies a DKIM private key could potentially sign messages that validate as originating from your domain until the associated DNS key is changed or revoked. Locate your DKIM configuration and signing keys. The exact locations depend on your mail software. For example, OpenDKIM installations often keep key material under locations such as: /etc/opendkim/ /etc/opendkim/keys/ Zimbra manages DKIM differently. If there is reasonable evidence that DKIM private keys could have been read, create new keys and update the corresponding DNS records. Also inspect: SPF DKIM DMARC MX records Mail relay configuration for unauthorized changes. 19. Review Mail Configuration for Abuse Attackers may compromise a mail server without leaving an obvious malware process running. Inspect mail configuration for unauthorized modifications. Depending on your platform, check: Aliases Forwarders Transport maps Relay configuration Mailbox rules Catch-all addresses Administrator accounts Delegated administration SMTP authentication Virtual domains Mail routing Milter configuration Content filters Webmail plugins Check the outgoing mail queue. Postfix: postqueue -p or: mailq Exim: exim -bp Look for unusual volumes, senders, recipients, or destinations. If the server was used for spam, your public IP address or domain reputation may also have been damaged. 20. Preserve Mail and Authentication Logs Copy relevant logs before rotation or cleanup destroys older evidence. Potential sources include: /var/log/auth.log /var/log/syslog /var/log/messages /var/log/secure /var/log/mail.log /var/log/maillog /var/log/audit/ /var/log/journal/ For Zimbra, also examine relevant logs under: /opt/zimbra/log/ /var/log/zimbra.log Preserve rotated logs as well: auth.log.1 auth.log.2.gz mail.log.1 syslog.1 Copy them without modifying the originals unnecessarily. For example: cp -a /var/log/auth.log* /mnt/incident-evidence/ cp -a /var/log/mail.log* /mnt/incident-evidence/ Hash the evidence: find /mnt/incident-evidence \ -type f \ -exec sha256sum {} \; \ > /mnt/incident-evidence/SHA256SUMS.txt Treat evidence storage as sensitive. Logs may contain: Usernames IP addresses Internal hostnames Tokens Email addresses Authentication information Infrastructure details 21. Check for Package and Application Modifications If your distribution supports package verification, use it as another signal. Debian and Ubuntu: dpkg -V RPM-based distributions: rpm -Va These commands can reveal package-managed files that differ from expected metadata. Do not automatically assume every difference indicates compromise. Configuration files are often legitimately modified. Also inspect recently modified files: find /etc /usr /opt \ -xdev \ -type f \ -mtime -7 \ -ls 2>/dev/null The time range should correspond to the suspected compromise window. Remember that timestamps are evidence, but attackers with sufficient access can alter them. 22. Identify How the Attacker Entered Recovery is incomplete until you understand the likely entry point. Possible causes include: Unpatched mail application vulnerability Exposed administrative interface Stolen SSH key Weak password Credential reuse Compromised administrator workstation Vulnerable webmail component Malicious plugin Application vulnerability Misconfigured firewall Compromised neighboring system This matters because rebuilding without fixing the entry point can produce: Old server compromised | v Server rebuilt | v Same vulnerability remains | v New server compromised CISA has previously documented active exploitation of Zimbra vulnerabilities and recommended patch management, restricted management interfaces, MFA where possible, network segmentation, incident response, and reimaging compromised systems. 23. Check Neighboring Systems Once you identify indicators of compromise, search other systems for them. Useful indicators include: File hashes Filenames IP addresses Domains SSH key fingerprints Usernames Cron commands Process names Systemd unit names Destination ports Timestamps If one mail server connected to another machine using SSH, investigate that machine. If an authentication originated from an internal server, investigate that source. If credentials stored on the mail server allowed access to backups, databases, DNS, or monitoring systems, those systems become part of the investigation. Do not wait until obvious malware appears there. 24. Protect Your Backups A backup becomes critical during recovery. It can also become another infection source. Do not immediately restore the newest backup without investigation. Determine: When the compromise started When the backup was created Whether malicious files exist in the backup Whether persistence exists in configuration backups Whether unauthorized SSH keys are included Whether credentials stored in the backup are still valid CISA recommends securing backups and ensuring they remain protected during incident response. A backup from yesterday is not a clean backup if the attacker gained access three weeks ago. 25. Cleanup or Complete Rebuild? This is one of the most important decisions. Cleanup may be reasonable when all of the following are true: The compromise is narrowly understood. The attacker never obtained privileged access. Persistence mechanisms are fully identified. Affected files are known. Credential exposure is understood. Integrity can be validated against trusted sources. Business requirements justify cleanup. A complete rebuild should be strongly considered when: Root access was obtained. An application account with extensive privileges was compromised. Unknown SSH keys were installed. System files may have been modified. Multiple persistence mechanisms were discovered. The attacker had access for an unknown period. Credentials could have been stolen. Evidence suggests lateral movement. You cannot determine the full scope. You cannot prove system integrity. For a fully compromised server, rebuilding from a known-good source is generally safer. CISA’s Zimbra compromise guidance explicitly recommends reimaging compromised hosts. That is an important distinction. You do not rebuild because: “We found malware.” You rebuild because: “We can no longer establish that the operating system and application stack have maintained integrity.” 26. Why Cleaning the Existing Server May Not Restore Trust Imagine that you discovered five attacker modifications: 1. Malware executable 2. Cron persistence 3. SSH key 4. Hidden script 5. Modified application file You successfully remove all five. The problem is that you cannot necessarily prove there was no sixth modification. Or twentieth. If the attacker had privileged access, the affected operating system itself becomes an unreliable witness. Commands such as: ps ls netstat ss find normally help you investigate. But sophisticated malware can interfere with the operating system or utilities you rely on. NIST forensic guidance specifically warns that a deeply compromised system may return misleading information and recommends using trusted forensic tools when possible. This is the foundation of the rebuild decision. The concern is not simply malware. The concern is loss of trust in system integrity. 27. Build the Replacement Server From Trusted Sources If rebuilding, do not clone the compromised operating system. Install from a trusted installation source. Then: Apply current security updates Install only required software Restore only necessary configuration Restore mail data from a validated backup Generate new credentials Generate new SSH keys where appropriate Generate new private keys when exposure is possible Patch the vulnerability used for entry Restrict management access Review firewall rules Enable monitoring Enable centralized logging Do not copy the entire old filesystem because it is convenient. Configuration should be reviewed before being transferred. For example, instead of copying: cp -a /etc /new-server/ restore specific validated files. A rebuild is also an opportunity to compare the new system against the old server and determine what should legitimately exist. 28. Restore Mail Services Carefully Do not reconnect the rebuilt server to the internet immediately. First test internally. Confirm: SMTP works IMAP/POP works if used Webmail works Authentication works LDAP works if applicable Database services work TLS certificates are correct DKIM signing works SPF is correct DMARC is correct Mail queues behave normally Backups work Monitoring works Logging works Check listening ports: ss -lntup Check processes: ps auxf Test outbound and inbound mail through controlled accounts. Verify TLS. Check that your new server is not attempting connections to any suspicious infrastructure observed during the incident. Only restore public connectivity after validation. 29. Monitor Closely After Recovery Recovery does not end when mail starts flowing again. Increase monitoring after the replacement server goes online. Watch: Authentication failures Successful SSH logins New processes Unexpected outbound connections CPU usage Network traffic Mail volume SMTP authentication New users Cron changes systemd changes Filesystem changes Mail queues DNS changes Administrator logins Also watch the neighboring systems identified during the investigation. If the original compromise resulted from stolen administrator credentials rather than a server vulnerability, rebuilding only the server will not solve the problem. 30. Preserve the Old Server If possible, do not immediately erase the compromised server. Keep it: Powered off Disconnected from production networks Clearly labeled Access controlled Documented If storage permits, retain forensic disk images rather than depending only on the physical server. Evidence may become useful later when: A second compromised machine is discovered A new indicator appears Management requests a report A customer asks for an explanation Law enforcement becomes involved An insurer requests evidence You need to determine the initial compromise date NIST recommends preserving relevant forensic evidence and documenting how it was collected and handled. 31. The Actionable Linux Mail Server Incident-Response Checklist Use this checklist when a Linux mail server compromise is confirmed or strongly suspected. Record the current UTC date and time. Start an incident timeline. Notify the appropriate technical and management personnel. Do not discuss the incident through the compromised mail system if attacker access may still exist. Restrict or isolate the affected host. Avoid rebooting before deciding whether volatile evidence is required. Capture running processes. Capture active network connections. Capture logged-in users and recent logins. Preserve suspicious process command lines and executable paths. Copy suspicious files before deleting them. Calculate SHA-256 hashes of collected files. Preserve authentication logs. Preserve mail logs. Preserve system logs. Preserve application-specific logs. Check root’s crontab. Check mail-application user crontabs. Inspect /etc/cron.* and user cron locations. Check systemd services. Check systemd timers. Check user-level systemd services. Check at jobs. Inspect shell startup files. Inspect /etc/sudoers and /etc/sudoers.d/. Check for unexpected accounts. Check for unexpected UID 0 accounts. Inspect all relevant authorized_keys files. Investigate SSH authentication history. Identify the initial access method if possible. Determine whether privileged access occurred. Identify credentials accessible from the server. Rotate exposed credentials from a trusted machine. Replace potentially exposed SSH keys. Review TLS private keys and reissue certificates where necessary. Review or rotate DKIM signing keys where exposure is possible. Check DNS, MX, SPF, DKIM, and DMARC records. Inspect mail forwarding and transport configuration. Inspect outgoing mail queues. Search neighboring systems for the same indicators. Investigate internal machines that authenticated to the compromised server. Protect and inspect backups. Determine the earliest known compromise time. Patch the vulnerability or access path that enabled entry. Decide whether cleanup can actually establish system integrity. Rebuild if privileged compromise or uncertain scope prevents establishing trust. Restore only validated data and configuration. Generate new credentials on the replacement system. Validate SMTP, webmail, authentication, TLS, DKIM, and DNS before production use. Increase monitoring after recovery. Preserve the original compromised system or forensic image. Produce a final incident report. Document lessons learned and update your incident-response procedures. 32. Evidence Preservation Warning There is an unavoidable tension during incident response. You need to stop the attacker. You also need to preserve evidence. Every command you run on the compromised system can change its state. For example: ls find ps cp kill rm systemctl can alter access times, process state, logs, caches, or other information. This does not mean administrators should refuse to touch a compromised system. It means actions should be deliberate and documented. For a serious compromise involving financial loss, customer information, regulatory requirements, legal proceedings, or a potentially sophisticated attacker, involve qualified digital-forensics personnel before performing extensive cleanup. NIST SP 800-86 specifically warns that live forensic collection changes system state and that organizations must weigh the value of volatile evidence against the risks associated with collecting it. 33. What Our Zimbra Incident Ultimately Teaches Our Zimbra experience provides a useful example. At first, suspicious processes and executables were visible. Those processes could be killed. The files could be disabled or removed. But additional investigation uncovered persistence through cron and suspicious SSH authorization. That changed the security assessment. The lesson is simple: Finding malware tells you that something happened. Removing malware tells you that one artifact is gone. Neither tells you that the server is trustworthy. Trust must be re-established. Sometimes investigation can provide enough confidence to retain a system. When privileged access, multiple persistence mechanisms, stolen credentials, or uncertain scope are involved, a clean rebuild usually provides a much stronger basis for restoring trust. For a mail server, this is particularly important. A mail server often contains: Thousands of user accounts Authentication data Email contents TLS private keys DKIM keys Administrative credentials Internal network access DNS dependencies Backup credentials Customer and business information Treating a confirmed compromise as an ordinary malware infection can leave the attacker with access long after the visible process has disappeared. Treat it as an incident. Contain it. Preserve evidence. Determine the scope. Rotate trust material. Investigate connected systems. Then rebuild when you cannot confidently prove the integrity of the existing installation. Official References NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management NIST SP 800-61 Rev. 3 NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response NIST SP 800-86 CISA, Federal Government Cybersecurity Incident and Vulnerability Response Playbooks CISA Incident and Vulnerability Response Playbooks CISA and MS-ISAC, Threat Actors Exploiting Multiple CVEs Against Zimbra Collaboration Suite CISA Zimbra Security Advisory AA22-228A OpenSSH sshd Documentation OpenSSH sshd Manual systemctl Documentation Linux systemctl Manual SEO Metadata SEO Title: What to Do After Your Linux Mail Server Is Compromised SEO Description: Learn how to respond to a compromised Linux mail server, preserve evidence, investigate SSH and persistence, rotate credentials, check connected systems, and decide when a complete rebuild is required. Social Title: Your Linux Mail Server Was Compromised. What Should You Do Next? Social Description: A practical incident-response guide for compromised Linux mail servers covering containment, evidence preservation, cron and systemd persistence, SSH investigation, credential rotation, recovery, and rebuilding. Focus Keyword: Linux mail server compromised Tags: Linux Security, Incident Response, Mail Server Security, Zimbra Security, Linux Server, Cybersecurity, Digital Forensics, SSH Security, Malware Investigation, System Administration, Zimbra, PinoyLinux