/// article

Zimbra Server Compromise: Incident Timeline, Findings, and Lessons Learned

A compromised mail server rarely presents itself as a clean security alert. Sometimes the first symptom is an unfamiliar process. Sometimes it is unusual CPU usage. Sometimes a service refuses to start. Sometimes administrators spend hours troubleshooting databases, filesystems, or application services before realizing that a security incident is unfoldin...

By lordfrancs3

A compromised mail server rarely presents itself as a clean security alert. Sometimes the first symptom is an unfamiliar process. Sometimes it is unusual CPU usage. Sometimes a service refuses to start. Sometimes administrators spend hours troubleshooting databases, filesystems, or application services before realizing that a security incident is unfolding at the same time. That was the situation in this Zimbra incident. What began as troubleshooting on a failing Linux mail server eventually revealed suspicious executables, recurring malicious processes, cron-based persistence, unexpected SSH authorization entries, and successful SSH public-key authentication to the Zimbra account. At the same time, the server was suffering serious operational problems involving MariaDB/MySQL, filesystem state, Zimbra mailbox services, and application recovery. This case study reconstructs the incident using the evidence we actually collected. All customer-identifying information has been removed or generalized. IP addresses, domains, usernames other than standard service accounts, SSH keys, credentials, remote endpoints, and other sensitive values have been sanitized. The goal is not to identify an attacker. We do not have sufficient evidence to do that. The goal is to document: What happened. What we observed. What worked. What failed. Which conclusions were supported by evidence. Which questions remained unresolved. What we would do differently during the next incident. The incident also demonstrates an important principle: Removing visible malware does not restore trust in a compromised server. 1. Environment and Scope The affected system was a production Linux server running Zimbra Collaboration. It was an important mail system containing: User mailboxes. Zimbra application services. MariaDB/MySQL databases. SMTP services. LDAP. Webmail. TLS certificates and private keys. Service credentials. SSH configuration. Scheduled Zimbra maintenance tasks. Because this was an active business system, the response had two competing requirements. The first was security: Contain the compromise. Preserve evidence. Determine persistence. Understand access. Prevent further attacker activity. The second was availability and data recovery: Recover mail data. Restore database availability. Bring Zimbra services back online. Reduce business interruption. Those goals do not always align. Actions that are useful for restoring a production service can destroy forensic information. Actions that preserve a system perfectly for forensic examination can prolong business downtime. This tension influenced several decisions during the incident. 2. Important Limitation of This Postmortem This report distinguishes among three categories of information. Confirmed These are findings directly supported by files, processes, logs, filesystem output, or commands observed during the investigation. Strong indication These are conclusions strongly supported by several related artifacts but where some details remain unknown. Unknown These are questions for which the available evidence does not support a reliable conclusion. This distinction is particularly important when discussing: Initial access. Attacker identity. Whether an internal system was also compromised. Whether the malware caused the database corruption. Whether filesystem abnormalities were caused by the attacker. Exactly how long unauthorized access existed. We will not fill those gaps with speculation. 3. High-Level Incident Timeline The sanitized timeline can be summarized as follows: August 25 | +--> Zimbra database/service instability observed | +--> Artifacts later preserved carried timestamps from this period | August 28 | +--> MariaDB crash-recovery problems continue | August 29 | +--> Major troubleshooting and recovery work +--> Suspicious files/processes investigated +--> Malware artifacts preserved +--> MySQL force-recovery attempts +--> mailboxd remains unhealthy | August 30 | +--> auth.log records successful public-key SSH authentication to the Zimbra account from an internal address | September 2 | +--> Malicious activity recurs +--> javab process seen again +--> /home/SSL contains suspicious support files +--> cron persistence discovered +--> /home/SSL/.khp executing every minute +--> SSH authorized_keys receives deeper scrutiny | After containment | +--> Wider persistence searches +--> Evidence collection +--> Credential and key review +--> Database/data recovery +--> Decision that malware removal alone could not restore trust The following sections provide the detailed chronology. 4. Phase One: Service Problems Appear The incident did not initially present itself as: SECURITY ALERT: SERVER COMPROMISED Operational problems were among the earliest major symptoms being investigated. Zimbra services were unstable, particularly the database layer. MariaDB/MySQL was not operating reliably. At one point: mysqladmin could not connect to the expected Zimbra MySQL socket: /opt/zimbra/data/tmp/mysql/mysql.sock and there was no functioning mysqld process or expected database listener. Startup troubleshooting showed MariaDB repeatedly entering startup and recovery behavior. This immediately affected dependent services. Zimbra is composed of multiple components. If its database is unavailable, mailboxd cannot simply function normally because the application depends heavily on database state. At this stage, database recovery was a valid operational priority. What was not yet fully understood was that we were also dealing with a security incident. 5. MariaDB Crash-Recovery Problems Between approximately August 25 and August 29, database logs showed repeated instability. Observed symptoms included: InnoDB crash recovery. Log sequence number inconsistencies. MariaDB becoming available temporarily. MariaDB later stopping. mysqld_safe reporting process termination. Automatic or manual restart attempts. Zimbra services remaining unavailable or unstable. During one recovery cycle, MariaDB processed more than 1,500 InnoDB-related files before eventually reporting that it was ready for connections. That success was temporary. The database process later terminated again. This created a dangerous troubleshooting situation. It would have been easy to assume: Database failure = primary cause of everything The security investigation later showed that this assumption would have been incomplete. 6. Suspicious Processes Are Discovered During system inspection, two particularly suspicious process names appeared: javab idle These were not normal Zimbra processes. The names were deliberately plausible enough to avoid immediate attention. For example: javab could easily be mistaken at a glance for something related to Java. Zimbra uses Java extensively, which makes that naming especially deceptive. Similarly: idle sounds like a harmless system process. Process names alone are never sufficient evidence, but investigation of the associated executables and directories made these processes highly suspicious. Files using these names were associated with locations including: /dev/shm/ /home/SSL/ 7. Why /dev/shm Was Important /dev/shm is normally a shared-memory filesystem backed by RAM. Applications legitimately use it. Attackers also like temporary memory-backed locations because they can provide convenient places to stage or execute payloads. Suspicious files found or preserved from this location included names such as: idle javab .khp .khp_ts Evidence preserved from the investigation contained files with timestamps dating back to approximately August 25. For example, preserved copies of .khp and .khp_ts carried timestamps around that period. This was important because it suggested that malicious artifacts existed earlier than some of the later visible recurrence. It did not, however, prove that August 25 was the original compromise date. The compromise may have started earlier. File timestamps alone cannot reliably establish initial access, particularly after a privileged compromise because timestamps can be manipulated. 8. Suspicious /home/SSL Directory Another important location was: /home/SSL/ The directory name appears security-related and therefore might initially escape suspicion. It contained files associated with the suspicious processes and persistence mechanism. Observed names during the investigation included: idle javab .khp .rguard .app_args .khp_ts .java_cache Some of those filenames appeared designed to resemble runtime support files or legitimate Java-related data. This reinforces an important incident-response rule: Do not judge a file by whether its name looks technical. Investigate: file suspicious-file stat suspicious-file sha256sum suspicious-file and determine which processes reference it. 9. Process Termination Once the suspicious processes were identified, they were terminated. Commands used during the response included process termination through mechanisms such as: pkill javab pkill idle or PID-based termination where appropriate. This achieved one immediate objective. The visible processes stopped. At that moment, however, we had not yet eliminated the mechanism responsible for launching them. This became one of the most important lessons from the incident. 10. Removing the Process Did Not Remove the Compromise The initial containment actions were useful, but they were incomplete. The relationship looked like this: Malicious executable | v Malicious process We initially dealt with the bottom of that relationship. But the actual system later proved to look more like: Persistence mechanism | v Launcher | v Malicious executable | v Running process Killing only the final process left everything above it intact. This is why malware frequently returns after apparently successful cleanup. 11. Execution Permission Removed From /home/SSL As part of containment, execution permission was removed from the suspicious directory. A defensive action similar to the following was used: chmod -R a-x /home/SSL The purpose was straightforward. Even if something attempted to launch the known executables from that directory, the operating system would no longer allow normal direct execution of those files. This was useful containment. It was not remediation. An attacker with sufficient privileges could restore execution permissions, copy the payload somewhere else, execute it through an interpreter, or use another persistence mechanism. We therefore treated this as a temporary defensive control. 12. Evidence Was Preserved Instead of deleting every suspicious artifact immediately, copies were retained for investigation. An incident evidence directory was created. Preserved material included suspicious files originating from areas such as: /dev/shm /home/SSL Metadata was retained where possible. This allowed later searches to continue even after files disappeared from live temporary storage. That proved useful because /dev/shm was later found empty during subsequent investigation. Without the earlier evidence copy, some artifacts would have been lost. This was one of the response actions that worked well. 13. MySQL Recovery Continued in Parallel While the security investigation was happening, Zimbra’s database still needed attention. Standard MariaDB startup was not reliable. The next step was InnoDB forced recovery. For a detailed technical discussion, see the companion PinoyLinux article: Recovering a Corrupted Zimbra MySQL Database with innodb_force_recovery The important timeline points for this case study are summarized here. Lower recovery levels were attempted first. Values including: innodb_force_recovery = 3 and: innodb_force_recovery = 4 did not provide a sufficient stable recovery path. Eventually: innodb_force_recovery = 6 allowed MariaDB to start far enough for inspection. At that point: SHOW DATABASES; could return database information. That was valuable because level 6 is an emergency recovery setting. Its purpose is data extraction, not returning the server to production. 14. Database Dump Results Data extraction produced mixed results. A substantial number of databases could be dumped. Recovery output documented: 99 database dumps completed successfully 3 databases failed The failed databases were mailbox-group databases, sanitized here as: mboxgroup-A mboxgroup-B mboxgroup-C The original identifiers have intentionally been removed from this public postmortem. The recovered dump set was approximately a few hundred megabytes. This was an important partial success. It showed that severe database problems did not mean that every database was unrecoverable. However, the failed mailbox-group databases meant that database recovery was not complete. Another complication was that not every generated dump contained the expected completion marker, so the existence of a dump file alone could not automatically be treated as proof that the dump was complete and healthy. 15. zmmyinit and Database State During recovery work, Zimbra’s MySQL initialization tooling was also used. zmmyinit recreated or initialized database-related structures and changed database credentials as part of that process. Later testing encountered authentication problems involving the Zimbra database account. This is an important recovery lesson. Tools that initialize or rebuild application database structures can change system state significantly. During a damaged-system recovery, administrators need to carefully distinguish: Original damaged database state from: State after recovery tooling modified it This does not mean zmmyinit was the wrong command in every circumstance. It means that once such a command is executed, forensic reconstruction of the database environment becomes more complicated. A stronger response process would maintain separate copies for: Forensic preservation and: Operational recovery 16. mailboxd Remained Unavailable Even when MariaDB could be made to start, Zimbra was not automatically restored. mailboxd remained unhealthy or unavailable during significant parts of the recovery. This reinforced another lesson: Database running ≠ Zimbra recovered mailboxd depends on several things. These include: Database connectivity. Correct database credentials. LDAP. Java. Zimbra configuration. Filesystem access. Permissions. Disk availability. Certificates and keystores. Ports. Other Zimbra components. For the detailed diagnostic process, see: Troubleshooting Zimbra Mailboxd When the Service Will Not Start During this incident, certificate verification was included as one possible dependency check, but we did not establish TLS certificates as the root cause of the mailboxd failure. For certificate-specific troubleshooting, see: How to Check and Troubleshoot SSL Certificates in Zimbra 17. Filesystem Problems Complicated Recovery The server also displayed filesystem abnormalities. At one point, the root ext4 filesystem had previously experienced behavior associated with: errors=remount-ro meaning filesystem errors could cause Linux to remount the filesystem read-only. Later inspection showed the root filesystem mounted read-write again. This raised an important question: Were the filesystem problems related to the compromise? The evidence available to us does not support a definitive answer. Filesystem corruption can result from: Storage problems. Sudden shutdown. Hardware problems. Kernel or filesystem faults. Database crashes. Administrative actions. Malicious activity. We therefore did not attribute the filesystem damage to the attacker. 18. An Unusual /tmp Attribute Was Found Another anomaly appeared when /tmp was inspected. Its extended attributes showed: ----ia-------e-- The important flags were: i = immutable a = append-only Finding both attributes on /tmp was unusual and required investigation. Again, this cannot automatically be attributed to the attacker. It could have resulted from: Previous administrative hardening. An accidental command. A recovery action. Malicious modification. Some earlier undocumented configuration. The correct response was to document it and investigate its effect rather than make assumptions. This is an example of the discipline required during a postmortem. Not every strange configuration item is necessarily an indicator of compromise. 19. Suspicious Activity Returns The most important turning point came when suspicious activity returned after the initial cleanup. By September 2, suspicious processes appeared again. A javab process was observed running with arguments resembling: ./javab -o <REDACTED_REMOTE_ENDPOINT>:<REDACTED_PORT> The actual remote address and port have intentionally been removed. The important fact was that the process was communicating or attempting to communicate with a remote endpoint. At this point, recurrence was confirmed. The first cleanup had not removed the persistence mechanism. 20. /dev/shm Was Now Empty During the recurrence investigation: /dev/shm did not contain the previously observed files. That could easily have led to the mistaken conclusion that the system was clean. But the malicious process was active again. This is another critical incident-response lesson. Indicators can move. Attackers can change locations. Temporary files can disappear. A directory that previously contained malware may later be clean while the compromise remains active elsewhere. The search therefore expanded beyond /dev/shm. 21. /home/SSL Still Contained Supporting Artifacts The /home/SSL directory still contained suspicious supporting files. Examples included: .app_args .khp_ts .java_cache Other files associated with the earlier malware activity had also existed there. These artifacts indicated that /home/SSL remained an important location in the persistence chain. Investigation continued into anything referencing those files. That led directly to the critical persistence discovery. 22. The One-Minute Cron Persistence Mechanism The Zimbra user’s crontab was inspected. A suspicious entry was found: * * * * * /home/SSL/.khp This command executed: /home/SSL/.khp every minute. The entry existed outside Zimbra’s normal managed scheduling block. System logs confirmed repeated cron execution. For example, sanitized entries resembled: CRON: (zimbra) CMD (/home/SSL/.khp) every minute. This explained the recurrence. The system behaved approximately like this: Every minute | v cron | v /home/SSL/.khp | v malicious process launched or restored | v javab / idle activity Now the earlier behavior made sense. Killing javab was temporary because .khp was being called again by cron. This was the most important persistence mechanism positively identified during the incident. 23. Why the Cron Discovery Changed the Incident Before finding the cron entry, we knew: Malicious files existed. Malicious processes existed. After finding it, we knew: The compromise included deliberate persistence. That is a much more serious security state. Persistence means someone or something had changed system configuration so that malicious code would continue executing. It also means that deleting a visible process could never have been considered sufficient remediation. At this point, broader persistence hunting became mandatory. 24. Searches Expanded Beyond Cron The response then included checks for other startup and persistence locations. Areas investigated included: cron systemd init scripts shell startup configuration temporary directories service account home directories SSH configuration The purpose was not simply to find another copy of .khp. It was to answer: What other mechanism could restore attacker access after the known cron entry is removed? This is how incident-response searches should expand. Once one persistence mechanism is found, assume another may exist until investigation gives you reasonable confidence otherwise. 25. Lock Files in /tmp Additional suspicious artifacts were found under /tmp. These included lock-like filenames resembling: .rguard_lockd .idle_lockd The names suggested coordination or duplicate-process prevention related to the previously observed components. Their existence provided additional correlation between the suspicious processes and files discovered elsewhere. They were useful indicators. They were not, by themselves, proof of the attacker’s identity, tooling family, or initial access vector. 26. SSH Investigation Begins Because the compromised processes ran under the Zimbra account and persistence had been created under that account, SSH access became a major area of investigation. The file: /opt/zimbra/.ssh/authorized_keys was inspected. Unexpected entries were present. One entry included a forced-command configuration. Other keys also required investigation. OpenSSH allows options such as: command="..." to precede an authorized public key. That feature is legitimate and is sometimes used by applications. However, unexpected forced-command keys on a compromised service account must be carefully investigated. The keys themselves are omitted from this postmortem. 27. Why authorized_keys Required Caution We did not simply declare every unfamiliar Zimbra SSH key malicious. Zimbra can legitimately use SSH mechanisms for internal operations. Instead, we needed to ask: Was this key expected? When was it created? What command did it force? Which system possessed the corresponding private key? Was that source system legitimate? Did authentication logs show the key being used? This is an important difference between incident response and guesswork. Deleting every unfamiliar key without understanding its purpose can damage the application and destroy useful evidence. 28. auth.log Reveals Successful SSH Authentication Authentication logs were then examined. A particularly important finding was an event showing: Accepted publickey for zimbra from <REDACTED_INTERNAL_IP> The original address has been removed. The event occurred on approximately August 30. This confirmed that public-key SSH authentication to the Zimbra account had succeeded from another internal address. What this tells us: A public key was successfully used to access the Zimbra account. What this does not automatically tell us: That this specific login was malicious. That the source machine was compromised. That this was the original access path. That this was the attacker's first login. Those questions require correlation with: SSH key fingerprints. Expected administrative activity. Source-host logs. Command execution records. File timestamps. Network logs. Still, given the surrounding compromise, the event significantly expanded the investigation scope. 29. An Internal Source IP Is Not Automatically Safe One mistake administrators can make is assuming: Internal address = trusted activity That assumption is dangerous. An internal authentication source could represent: A legitimate administrator. An automation server. Another Zimbra node. A management system. A compromised neighboring server. A workstation used by an attacker. A pivot point. The correct response was therefore to investigate the source host independently. This is where the incident stopped being only a mail-server investigation. It became a potential lateral-movement investigation. 30. What We Could Not Prove About Initial Access Despite extensive evidence of compromise, we did not establish the original intrusion method with enough confidence for this postmortem to name one. Possible Zimbra attack paths exist historically, including vulnerabilities in exposed Zimbra services. CISA has documented active exploitation of several Zimbra vulnerabilities in previous campaigns. However, the existence of known Zimbra vulnerabilities does not prove that one of them was used in this incident. Likewise, successful SSH authentication does not prove SSH was the original intrusion method. The following remain unresolved: Exact initial access method Exact compromise date Attacker identity Whether the internal SSH source was itself compromised Whether a Zimbra vulnerability was involved Whether credentials were stolen before the server compromise A responsible incident report should leave those items as unknown rather than replace missing evidence with a plausible story. 31. Evidence Preservation Became Increasingly Important One thing that proved valuable was retaining suspicious files instead of immediately deleting everything. The evidence set allowed later correlation of: File names File timestamps Process names Cron commands Temporary lock files SSH configuration Authentication logs Database events This is consistent with NIST forensic guidance, which recommends preserving relevant evidence and understanding that live investigation changes system state. For future incidents, we would make this process much more formal from the beginning. 32. What Worked Several actions materially improved the investigation. 32.1 Looking Beyond Normal Zimbra Processes Identifying: javab idle as abnormal rather than assuming they belonged to Java or Zimbra was critical. Process names should always be correlated with: Executable path Parent process User Command line Network connection Open files 32.2 Searching Multiple Filesystem Locations The investigation did not remain focused only on /dev/shm. Expanding searches to: /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra helped establish relationships among the artifacts. 32.3 Preserving Suspicious Files Retaining copies of suspicious files allowed investigation to continue after temporary locations changed. This was particularly important once /dev/shm became empty. 32.4 Inspecting User Crontabs Checking the Zimbra user’s cron configuration exposed the one-minute persistence mechanism: * * * * * /home/SSL/.khp This was one of the highest-value investigative steps. 32.5 Correlating Cron With Logs The cron configuration alone was useful. Finding log entries showing repeated execution made the finding stronger. Configuration plus execution evidence is far more useful than configuration alone. 32.6 Checking SSH Authorization Inspecting: /opt/zimbra/.ssh/authorized_keys expanded the incident from malware analysis into access-path investigation. That was necessary. 32.7 Checking Authentication Logs The successful public-key SSH login provided important evidence about how the Zimbra account had been accessed. Again, it did not prove initial intrusion, but it identified another event requiring investigation. 32.8 Using innodb_force_recovery Carefully Escalating recovery values allowed us to obtain database visibility when normal MariaDB recovery was unsuccessful. Recovery level 6 eventually made data inspection and partial extraction possible. For the technical procedure, see: Recovering a Corrupted Zimbra MySQL Database with innodb_force_recovery 32.9 Treating mailboxd as a Dependency Problem Instead of repeatedly issuing: zmmailboxdctl restart we eventually treated the problem as a dependency chain. That is a better diagnostic approach. See: Troubleshooting Zimbra Mailboxd When the Service Will Not Start 33. What Did Not Work The incident also exposed several ineffective approaches. 33.1 Killing the Malicious Process Alone This produced temporary relief. It did not eliminate persistence. The process returned because cron launched the persistence script every minute. 33.2 Assuming an Empty /dev/shm Meant the Threat Was Gone Later, /dev/shm was empty. The malicious activity still existed. Indicators can disappear or move. Never define compromise status using one directory. 33.3 Treating File Removal as Proof of Recovery Removing or disabling known malware files reduced active risk. It did not establish system integrity. Other persistence paths, credentials, SSH keys, application modifications, or unknown files could still exist. 33.4 Repeated Service Restarting During early recovery, repeated attempts to start database and Zimbra services consumed time without resolving the underlying conditions. When a service repeatedly fails, the investigation should move toward: Why is it failing? rather than: How many times can we restart it? 33.5 Lower InnoDB Recovery Levels Recovery levels 3 and 4 did not provide the required database access. Escalating to level 6 was necessary for emergency extraction. This was not a mistake. It was a failed recovery path that provided diagnostic information. 33.6 Mixing Recovery and Forensics This was perhaps the biggest procedural weakness. We were simultaneously trying to: Investigate the compromise Recover the database Restore Zimbra Preserve business operations Those activities changed the same server we were investigating. A stronger response would separate them. 34. What We Would Do Differently If we encountered the same situation again, we would change the response sequence significantly. 34.1 Declare a Security Incident Earlier Once abnormal executables and unexpected processes were confirmed, troubleshooting should formally transition into incident-response mode. That changes priorities. Instead of: Fix service first the sequence becomes: Contain Preserve Collect Investigate Recover 35. Capture Volatile Evidence Before Killing Processes Before terminating javab, idle, or another suspicious process, we would capture: ps auxww pstree -ap ss -plant lsof -nP readlink -f /proc/PID/exe and preserve: /proc/PID/cmdline /proc/PID/environ open file handles network sessions where appropriate. NIST notes that volatile information can disappear when processes terminate or systems reboot. 36. Create a Forensic Snapshot Before Recovery Work Before performing major MySQL repair or Zimbra initialization, we would create a disk or VM snapshot when infrastructure permits it. That creates two paths: Original evidence copy | +--> Forensic investigation Recovery copy | +--> MySQL repair +--> Zimbra recovery +--> configuration changes This prevents recovery work from destroying the only copy of the original system state. 37. Hash Evidence Immediately Every suspicious file should be hashed when collected. For example: sha256sum suspicious-file The evidence record should contain: Original path File size Owner Permissions Timestamps SHA-256 Collection time Collector This creates a much stronger postmortem record. 38. Inspect Persistence Immediately Once malicious code is confirmed, we would immediately inspect: All user crontabs /etc/crontab /etc/cron.d systemd services systemd timers init scripts SSH authorized_keys shell startup files at jobs sudo configuration unexpected users Finding the .khp cron entry earlier would have saved time spent repeatedly dealing with returning processes. 39. Investigate SSH Keys Immediately A compromised application account should trigger immediate inspection of its SSH configuration. For Zimbra: /opt/zimbra/.ssh/ should be reviewed early. Authorized keys should be correlated with fingerprints and expected source systems. Authentication logs should then be searched for those keys. 40. Investigate Internal Source Hosts Immediately The successful SSH login from an internal address should trigger investigation of that source system. The question should be: Why did this host authenticate as zimbra? and then: Was that expected? If not, that host enters the incident scope. 41. Rotate Credentials From a Trusted System Credentials should not be rotated while trusting the compromised server. Potentially exposed secrets include: Zimbra administrator credentials Database passwords LDAP credentials SSH private keys Backup credentials SMTP relay credentials API credentials TLS private keys DKIM signing keys New secrets should be generated from a trusted environment. 42. Review Certificates and Keys Because the server held TLS private keys, certificate review was part of recovery planning. We did not establish certificate compromise. However, if an attacker had sufficient access to read those private keys, reusing them indefinitely would be risky. For deeper certificate validation, see: How to Check and Troubleshoot SSL Certificates in Zimbra 43. Consider Rebuild Earlier Once we had: Malicious processes + Persistent cron execution + Suspicious files + SSH authorization concerns + Successful SSH authentication the question should no longer have been: Can we delete everything malicious? The better question was: Can we still prove this operating system is trustworthy? That is a much harder standard. NIST’s current incident-response guidance emphasizes recovery as part of an organizational incident-response process rather than merely eliminating one observable symptom. CISA’s guidance for confirmed Zimbra compromises has also recommended collecting artifacts, quarantining affected hosts, remediating malicious activity, and reimaging compromised systems where appropriate. 44. Cleanup Versus Rebuild This incident illustrates the difference between those concepts. Cleanup means: Remove known malicious artifacts. A rebuild means: Re-establish trust from a known-good operating system and application base. Cleanup can remove: javab idle .khp .rguard malicious cron entry unexpected SSH key But cleanup cannot automatically prove that you found everything. If the attacker had sufficient privileges, they could have modified: System binaries Libraries Application files SSH configuration PAM configuration Startup scripts Zimbra configuration Database records User accounts Keys Credentials The absence of another known indicator is not proof that none exists. For that reason, once the incident scope included persistence and account access, the safer recovery direction was to treat the existing host as untrusted and separate mail-data recovery from restoration of server trust. 45. Did the Malware Cause the MySQL Corruption? We cannot say. The compromise and the database failure occurred during the same broad incident window. That does not establish causation. Possible explanations include: Malware directly affected database operations Filesystem problems damaged database files Storage problems caused database corruption Unexpected shutdown damaged InnoDB state Database corruption existed independently Several factors occurred simultaneously The evidence collected did not allow us to select one with sufficient confidence. Therefore: The database corruption is a confirmed incident condition, but its direct causal relationship to the compromise remains unproven. This distinction is essential in a postmortem. 46. Did the Attacker Cause the Filesystem Anomalies? Again, unknown. We observed: Prior ext4 read-only/remount behavior. Filesystem recovery concerns. Unusual attributes on /tmp. Those are facts. Attribution is not. A security postmortem should never turn correlation into causation simply because both events occurred on the same compromised server. 47. Was SSH the Initial Entry Point? Unknown. Successful public-key SSH authentication was confirmed. Suspicious authorized_keys contents were investigated. That means SSH was relevant to the incident. It does not establish that SSH was how the attacker first entered. It is equally possible that: An attacker gained application access first | v established SSH persistence later or: SSH access occurred first or: the observed SSH event was legitimate while another mechanism caused the compromise Without stronger correlation, we should not choose among those possibilities. 48. Was a Known Zimbra Vulnerability Used? Unknown. CISA has documented historical exploitation of vulnerabilities in Zimbra Collaboration Suite. That is useful industry context. It is not evidence that one of those vulnerabilities was responsible for this server. We did not identify enough evidence to assign a specific CVE to the incident. Therefore this postmortem does not claim one. 49. The Most Important Technical Finding If one technical finding defined the compromise, it was this: * * * * * /home/SSL/.khp That one line explained why malicious activity returned. It also illustrates why administrators should always inspect persistence mechanisms after terminating suspicious processes. The visible process was only the symptom. Cron was one of the mechanisms keeping the compromise alive. 50. The Most Important Operational Finding The biggest operational lesson was different. We were dealing with two incidents at once: Security compromise and: Application/data recovery Trying to solve both directly on the same production server complicated the investigation. The better architecture for response is: Compromised server | +------------+------------+ | | v v Evidence image Recovery copy | | v v Investigation Data extraction | v Clean replacement This would reduce uncertainty significantly. 51. The Most Important Security Lesson The central lesson from this incident is: A compromised Linux server does not become trustworthy merely because the files and processes you discovered have been removed. Trust is broader than malware presence. You must consider: Persistence Credentials SSH keys TLS keys Application configuration System binaries Scheduled tasks Neighboring systems Backups Administrative workstations Database integrity If you cannot establish the integrity of the installed operating system, rebuilding from a known-good source is safer than repeatedly searching for the next hidden file. For the broader response methodology, see: What to Do After Discovering Your Linux Mail Server Was Compromised 52. Lessons Learned Lesson 1: Service Failure Can Hide a Security Incident A database failure or application outage may be operational. It may also occur alongside a compromise. Do not become so focused on restoring the application that unusual processes and files are ignored. Lesson 2: Process Names Are Meaningless Without Context javab looked Java-related. idle sounded harmless. Neither assumption was useful. Investigate executable paths, parent processes, owners, network connections, and persistence mechanisms. Lesson 3: Killing Malware Is Containment, Not Recovery Stopping a process can be necessary. It does not eliminate the mechanism that created it. Lesson 4: Always Hunt Persistence Once malicious activity is confirmed, immediately inspect: cron systemd SSH startup files application hooks users sudo scheduled jobs The one-minute .khp cron entry was decisive in this incident. Lesson 5: Preserve Before Deleting Suspicious files can disappear. Temporary filesystems can be cleared. Processes can terminate. Preserve evidence before removing it when operational risk permits. Lesson 6: Logs Are Critical auth.log provided information that the filesystem alone could not. Centralized remote logging would have made the investigation stronger because an attacker with privileged access to the host could theoretically alter local logs. Lesson 7: Internal IP Addresses Are Not Proof of Trust Internal systems can be compromised. Unexpected authentication from inside the network should still be investigated. Lesson 8: Separate Recovery From Forensics Do not perform extensive database repair on your only forensic copy when you can avoid it. Create a recovery clone or image. Lesson 9: Backups Must Be Evaluated Against the Compromise Timeline A backup created after compromise may contain: malware persistence modified configuration unauthorized keys Newest does not necessarily mean cleanest. Lesson 10: Rebuild Restores Trust Better Than Endless Cleanup Once privileged compromise and persistence are confirmed, administrators should seriously consider a clean rebuild. The objective is not merely to make malware disappear. The objective is to regain confidence in the platform. 53. Incident-Response Checklist Use this checklist when responding to a similar Linux or Zimbra mail-server compromise. Detection and containment Record the current time. Start an incident timeline. Identify suspicious processes. Record process command lines. Record process executable paths. Capture active network connections. Capture logged-in users. Restrict outbound connectivity where appropriate. Isolate the server when operationally possible. Avoid unnecessary rebooting. Evidence preservation Preserve suspicious files before deletion. Record file ownership and permissions. Record timestamps. Calculate SHA-256 hashes. Copy authentication logs. Copy mail logs. Copy Zimbra logs. Preserve rotated logs. Preserve cron configuration. Preserve SSH configuration. Create a filesystem, disk, or VM snapshot where possible. Persistence investigation Check root crontab. Check Zimbra crontab. Check other user crontabs. Inspect /etc/crontab. Inspect /etc/cron.d. Check systemd services. Check systemd timers. Inspect init scripts. Inspect shell startup files. Check at jobs. Search /tmp. Search /var/tmp. Search /dev/shm. Search service-account home directories. SSH investigation Inspect root authorized_keys. Inspect user authorized_keys. Inspect Zimbra authorized_keys. Identify unexpected keys. Record key fingerprints. Search for forced-command keys. Review successful SSH logins. Review failed SSH logins. Investigate unexpected source hosts. Review SSH private keys stored on the server. Credential exposure Rotate administrator passwords. Rotate database credentials. Rotate LDAP credentials where appropriate. Replace exposed SSH keys. Review backup credentials. Review SMTP relay credentials. Review API tokens. Review TLS private keys. Review DKIM signing keys. Perform credential rotation from a trusted system. Zimbra recovery Check MariaDB health. Preserve the original database state. Use forced recovery only for emergency extraction. Validate SQL dumps. Identify failed mailbox groups. Check LDAP. Check mailboxd. Check proxy. Check MTA. Verify certificates. Check disk and filesystem health. Review permissions. Neighboring systems Investigate internal SSH source hosts. Search other systems for the same filenames. Search for the same hashes. Search for the same SSH keys. Search for the same cron commands. Search network logs for suspicious endpoints. Review administrator workstations if warranted. Recovery Determine whether the existing OS can still be trusted. Prefer a clean rebuild after privileged compromise when trust cannot be established. Patch the replacement system. Restore only validated data. Restore only reviewed configuration. Generate new secrets. Test mail flow. Test authentication. Test TLS. Test DKIM. Test backups. Enable increased monitoring. Preserve the old host or forensic image. Post-incident Document confirmed findings. Separate assumptions from evidence. Record unresolved questions. Determine the earliest confirmed compromise activity. Document what worked. Document what failed. Update incident-response procedures. Improve centralized logging. Review patch procedures. Review access controls. Conduct a final lessons-learned review. 54. Final Assessment The strongest conclusions supported by the evidence were: The Zimbra server was compromised. Malicious or unauthorized processes named javab and idle were present. Suspicious files existed under /dev/shm and /home/SSL. The malicious activity returned after process termination. A persistence script named .khp was executed through the Zimbra user's cron every minute. SSH authorized_keys required investigation. Successful SSH public-key authentication to the Zimbra account from an internal system was recorded. The server simultaneously suffered serious MariaDB recovery problems. mailboxd remained unavailable during significant parts of recovery. Filesystem anomalies complicated the investigation. Known malware removal alone was insufficient to re-establish trust. The evidence did not allow us to reliably conclude: Who the attacker was. Exactly when the initial compromise occurred. Exactly how initial access was obtained. Whether SSH was the original entry point. Whether a specific Zimbra CVE was used. Whether the attacker caused the MySQL corruption. Whether the attacker caused every filesystem anomaly. Whether every unknown SSH entry was malicious. Those unanswered questions belong in the report. Leaving a question unanswered is better than writing an unsupported conclusion. NIST’s incident-response guidance treats recovery and lessons learned as part of a broader security-risk process, and its forensic guidance emphasizes preserving and analyzing evidence carefully. For Zimbra specifically, CISA has previously warned that exploited Zimbra systems require investigation for indicators of compromise and remediation beyond simply patching the affected application. This incident reinforced that principle directly. The real endpoint of incident response is not: Malware process no longer running. It is: Threat contained. Scope understood as far as evidence permits. Credentials and trust material addressed. Data recovered safely. Affected systems replaced or validated. Monitoring restored. Lessons documented. Only then can recovery be considered complete. Related PinoyLinux Articles This postmortem is part of a broader Zimbra troubleshooting and recovery series. For deeper technical procedures, see: What to Do After Discovering Your Linux Mail Server Was Compromised A broader incident-response guide covering containment, evidence preservation, credential rotation, lateral-movement investigation, and rebuild decisions. Recovering a Corrupted Zimbra MySQL Database with innodb_force_recovery A detailed recovery procedure for situations where Zimbra MariaDB cannot start normally and emergency InnoDB recovery is required. Troubleshooting Zimbra Mailboxd When the Service Will Not Start A systematic diagnostic approach for mailboxd failures involving databases, Java, ports, configuration, filesystems, certificates, and dependent services. How to Check and Troubleshoot SSL Certificates in Zimbra A guide to inspecting deployed Zimbra certificates, expiration, certificate chains, private-key relationships, and live TLS connections. References NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management NIST SP 800-61 Rev. 3 NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response NIST SP 800-86 CISA and MS-ISAC, Threat Actors Exploiting Multiple CVEs Against Zimbra Collaboration Suite CISA Advisory AA22-228A