/// article

Zimbra Server Compromise Indicators: Suspicious Processes, Cron Jobs, SSH Keys, and Malware Artifacts

By lordfrancs3

A strange process called javab does not prove that your Zimbra server has been compromised. Neither does a file named .khp, an unfamiliar object under /dev/shm, an SSH key you do not immediately recognize, or an unexpected cron job. Indicators become meaningful when they connect. During a real PinoyLinux investigation of a compromised Zimbra Collaboration server running on Ubuntu, several pieces of evidence eventually formed a consistent picture: zimbra account | +---- suspicious processes | | | +---- javab | +---- idle | +---- suspicious files | | | +---- /dev/shm | +---- /home/SSL | +---- /tmp | +---- /home/SSL/.khp | | | +---- executed by zimbra crontab | +---- SSH authorization data | | | +---- additional keys | +---- forced-command entry | +---- authentication logs | +---- successful public-key login The importance was not any single filename. It was the relationship among processes, files, ownership, cron execution, SSH configuration, and logs. The original PinoyLinux investigation confirmed unauthorized processes and files operating under the zimbra account, an unauthorized cron job executing /home/SSL/.khp every minute, and SSH-related activity that required further investigation. This page organizes those indicators into one investigation reference. It does not replace the detailed incident articles. Use it to understand what was actually found, what each indicator meant in context, and where to continue the investigation. Start With an Important Rule: An Indicator Is Not Proof Incident response becomes unreliable when filenames are treated as verdicts. Consider these names: javab idle ksmd .khp .rguard Linux allows executables and processes to use almost any filename. An attacker can choose a legitimate-looking name. A legitimate administrator or application can also create a strange-looking name. The correct question is therefore not: Is javab malware? The better questions are: Which user is running it? Where is the executable stored? What is its parent process? What command-line arguments are present? What files does it open? Does it communicate over the network? When did it appear? What starts it? Does it return after termination? Does another configuration file reference it? During the PinoyLinux Zimbra incident, javab and idle became significant because they were associated with the zimbra account, related artifacts appeared in unusual locations, and the activity returned after the processes were terminated. That correlation turned suspicious names into incident evidence. Confirmed Incident Indicators at a Glance Indicator Status in the PinoyLinux incident Why it mattered javab Confirmed suspicious process/artifact Ran under the zimbra account and was associated with suspicious files and recurring activity idle Confirmed suspicious process/artifact Appeared with the same broader unauthorized activity /home/SSL/.khp Confirmed suspicious artifact and persistence component Referenced by the unauthorized zimbra cron entry .khp_ts Confirmed related artifact Found with other suspicious files during the investigation .rguard Confirmed related artifact Found among related suspicious artifacts /dev/shm artifacts Confirmed incident location Contained files associated with unauthorized activity /home/SSL Confirmed suspicious artifact location Contained .khp, javab, idle, and other related files /tmp artifacts Confirmed related evidence Lock-related objects connected with names already being investigated * * * * * /home/SSL/.khp Confirmed unauthorized cron persistence Executed the suspicious file every minute /var/log/syslog cron records Confirmed execution evidence Showed cron actually running /home/SSL/.khp as zimbra /opt/zimbra/.ssh/authorized_keys Confirmed area requiring investigation Contained multiple keys and at least one forced-command entry Successful Accepted publickey event Confirmed SSH authentication evidence Showed successful authentication to zimbra from an internal IP during the incident period ksmd Search indicator, not clearly confirmed in the published incident evidence Included in targeted filename searches, but the current articles do not establish it as a confirmed artifact That last distinction matters. ksmd appeared in the filename search used during the investigation: find \ /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \ -xdev \ \( \ -name 'javab' -o \ -name 'idle' -o \ -name '.khp' -o \ -name '.khp_ts' -o \ -name '.rguard' -o \ -name 'ksmd' \ \) \ -ls 2>/dev/null But the published PinoyLinux incident articles clearly identify javab, idle, .khp, .khp_ts, .rguard, /dev/shm, /home/SSL, and related /tmp artifacts as findings. They do not provide the same confirmation for ksmd. So: ksmd found in your environment != proof of this malware Treat it as something to investigate. 1. javab: The Process That Started the Investigation One of the clearest incident indicators was a process named: javab A suspicious process used a command resembling: ./javab -o <REMOTE_HOST>:<REMOTE_PORT> The actual destination was removed from the published incident report. The important details were: it was not expected Zimbra software it ran under the zimbra account related files existed in suspicious locations it appeared together with other unauthorized activity terminating it did not permanently stop the activity Those facts made it an incident indicator. The filename alone did not. Safe first checks pgrep -af 'javab' Then inspect a specific PID: ps -o pid,ppid,user,lstart,etime,args -p <PID> Check what executable Linux says is running: readlink -f /proc/<PID>/exe Inspect the command line: tr '\0' ' ' < /proc/<PID>/cmdline echo Linux exposes /proc/<PID>/exe as a symbolic link to the executed program. /proc/<PID>/cmdline exposes the process command-line data, although a process can alter what appears there, so it should be treated as one source of evidence rather than absolute truth. For the full process investigation, read: How to Investigate Suspicious Processes and Files on a Linux Server 2. idle: Significant Because of the Context Around It The second recurring process name was: idle Again, idle by itself proves nothing. What made it relevant was that it appeared in the same environment as: javab .khp /dev/shm artifacts /home/SSL artifacts cron persistence The process also returned after containment attempts, which helped push the investigation toward persistence rather than simple process termination. Check it the same way: pgrep -af 'idle' Then: ps -o pid,ppid,user,lstart,etime,args -p <PID> readlink -f /proc/<PID>/exe Do not use a rule such as: process name = idle therefore malware Process names can be misleading. 3. /dev/shm: Suspicious in Context, Normal by Design Several suspicious artifacts appeared under: /dev/shm This directory deserves careful explanation. /dev/shm is not a malware directory. On Linux, POSIX shared-memory objects are normally backed by a tmpfs filesystem mounted under /dev/shm. Legitimate applications use it every day. During this incident, files under /dev/shm included names associated with the unauthorized activity, including: javab idle .khp .khp_ts .rguard That relationship made the location relevant. Inspect without deleting ls -lah /dev/shm For an individual object: stat /dev/shm/<FILE> file /dev/shm/<FILE> sha256sum /dev/shm/<FILE> You can also inspect recently modified files: find /dev/shm \ -xdev \ -type f \ -mmin -120 \ -ls A result means: investigate It does not automatically mean: delete 4. /home/SSL: A Legitimate-Looking Name Hiding Suspicious Artifacts Another major incident location was: /home/SSL The name looks plausible. An administrator seeing SSL may initially assume it contains certificates or TLS-related material. During the incident, however, it contained suspicious components including: .khp .khp_ts javab idle The broader investigation also encountered related names such as .app_args and .java_cache at different stages. The lesson is simple: Do not trust a directory because its name sounds legitimate. Inspect it. ls -lah /home/SSL Record directory metadata: stat /home/SSL Then inspect individual files: stat /home/SSL/.khp file /home/SSL/.khp sha256sum /home/SSL/.khp Do not execute an unknown file merely to discover what it does. 5. .khp: The Indicator That Connected Files to Persistence The file: /home/SSL/.khp became one of the most important indicators in the entire investigation. It was not important simply because its name began with a dot. Hidden files are normal on Linux. Its importance came from another discovery: * * * * * /home/SSL/.khp That line was present in the zimbra user’s crontab. The crontab therefore established a persistence relationship: zimbra crontab | v /home/SSL/.khp | v runs every minute | v suspicious activity returns Linux crontabs contain scheduling fields followed by a command. Asterisks in all five time fields cause the command to be eligible to run every minute. This was a confirmed persistence mechanism in the incident. For the full persistence investigation, read: Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server 6. The Unauthorized zimbra Crontab Entry The critical cron entry was: * * * * * /home/SSL/.khp You can inspect the Zimbra account’s scheduled jobs with: crontab -u zimbra -l Do not assume that every zimbra cron entry is malicious. Zimbra legitimately uses scheduled jobs. The problem in this incident was a specific unauthorized entry pointing to a file that was already connected to suspicious activity. Zimbra’s own compromise-check guidance explicitly recommends checking the zimbra and root crontabs for unauthorized entries. This distinction matters: zimbra has a crontab -> normal zimbra crontab contains an unexpected command executing a suspicious artifact every minute -> investigate immediately 7. /var/log/syslog: Turning a Cron Hypothesis Into Execution Evidence Finding a cron entry tells you what is configured. Logs can tell you whether it actually ran. During the PinoyLinux incident, /var/log/syslog contained records showing cron executing: /home/SSL/.khp as the zimbra user. A useful search is: grep -hE 'CRON.*zimbra' \ /var/log/syslog \ /var/log/syslog.1 \ 2>/dev/null Once you know the suspicious path: grep -R '/home/SSL/.khp' /var/log 2>/dev/null You can also search the known process names: grep -R -E 'javab|idle|\.khp' /var/log 2>/dev/null Be careful with large log trees. Recursive searches can consume substantial disk I/O. The evidence chain became: crontab says .khp should execute | v syslog shows .khp executing | v suspicious processes appear That is much stronger evidence than a filename alone. 8. .khp_ts, .rguard, and Related Files Other confirmed related artifacts included: .khp_ts .rguard These were found in the same broader group of suspicious files during the investigation. Their names alone should not become universal detection signatures. Instead of creating a rule like: find .rguard -> server hacked use them as pivots: find \ /home /tmp /var/tmp /dev/shm /run /opt/zimbra \ -xdev \ \( \ -name '.khp' -o \ -name '.khp_ts' -o \ -name '.rguard' \ \) \ -ls 2>/dev/null Then investigate: owner group permissions hashes timestamps parent directories references from cron or services open processes network connections neighboring files created at similar times GNU find supports searching directory hierarchies using criteria such as name, type, ownership, time, and permissions. 9. What About ksmd? ksmd deserves special treatment because it can easily become an overstated IOC. The PinoyLinux investigation included ksmd in a targeted filename search: -name 'ksmd' However, the current published incident articles do not identify ksmd with the same level of confirmation as: javab idle .khp .khp_ts .rguard Therefore, this article does not classify ksmd as a confirmed artifact from the incident. If you find something called ksmd, investigate: pgrep -af 'ksmd' If it is a process: readlink -f /proc/<PID>/exe ps -o pid,ppid,user,lstart,etime,args -p <PID> If it is a file: stat <PATH> file <PATH> sha256sum <PATH> Then correlate it with the rest of the system. The correct label is: candidate indicator until evidence establishes what it is. 10. Suspicious SSH Authorization Data The investigation later expanded into: /opt/zimbra/.ssh/authorized_keys The file contained: multiple SSH public-key entries requiring investigation at least one entry using a forced command additional keys whose legitimacy needed verification The incident also contained successful public-key authentication involving the zimbra account. OpenSSH supports AuthorizedKeysFile for defining the files containing public keys allowed to authenticate users. The default commonly includes .ssh/authorized_keys, although administrators can configure another path. Preserve before editing cp -a /opt/zimbra/.ssh/authorized_keys \ /root/incident-evidence/authorized_keys.before-change Then inspect: nl -ba /opt/zimbra/.ssh/authorized_keys Do not publish the complete key material. Generate fingerprints instead: ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys A fingerprint is easier and safer to compare with administrator inventories. For the detailed SSH investigation, read: Investigating Suspicious SSH Access on a Compromised Linux Server 11. Successful SSH Authentication Was Evidence, Not Root-Cause Proof The incident’s /var/log/auth.log contained a successful public-key authentication involving the zimbra account from an internal IP address during the broader incident period. A sanitized event looks like: <DATE> <HOST> sshd[<PID>]: Accepted publickey for zimbra from <INTERNAL_IP> port <SOURCE_PORT> ssh2 Search: grep -E 'Accepted publickey for zimbra' \ /var/log/auth.log Also inspect rotated logs where available: grep -hE 'Accepted publickey for zimbra' \ /var/log/auth.log \ /var/log/auth.log.1 \ 2>/dev/null This record establishes that OpenSSH recorded successful public-key authentication. It does not prove: that SSH was the original compromise vector that the internal IP belonged to an attacker that every SSH key was malicious that the login created .khp that the same person controlled every suspicious component The PinoyLinux investigation deliberately avoided claiming SSH as the original intrusion vector because the available evidence did not establish that conclusion. This is the correct incident-response discipline. Evidence, Hypothesis, Containment, Remediation, and Root Cause Are Different These terms should not be mixed. Evidence Evidence is something actually observed. Examples from this incident: javab process exists /home/SSL/.khp exists zimbra crontab contains: * * * * * /home/SSL/.khp syslog records cron executing .khp authorized_keys contains entries requiring investigation auth.log records successful public-key authentication Hypothesis A hypothesis is an explanation that may fit the evidence. For example: The SSH key may have provided persistence. or: An attacker may have obtained control of the zimbra account. Those require further proof. Containment Containment tries to stop active damage. Examples can include: isolating the host terminating confirmed unauthorized processes disabling confirmed persistence blocking malicious destinations restricting compromised accounts Containment changes the system and can destroy evidence, so collect what you can first. Remediation Remediation removes the cause and restores the environment safely. It can include: patching vulnerable software rotating credentials replacing SSH keys restoring trusted configuration rebuilding systems reviewing neighboring hosts Root-cause attribution Root cause answers: How did the compromise begin? The published PinoyLinux incident established that the server was compromised. It did not conclusively establish the original attack vector or attacker identity. That distinction should remain intact. A Safe Evidence-Collection Workflow When you find similar indicators on a Zimbra server, avoid turning the investigation into an immediate cleanup session. Start by recording state. Record running processes ps auxwwf Then target known indicators: pgrep -af 'javab|idle|\.khp' pgrep searches currently running processes and can match process attributes and command lines. Inspect process details ps -o pid,ppid,user,lstart,etime,args -p <PID> readlink -f /proc/<PID>/exe readlink -f /proc/<PID>/cwd tr '\0' ' ' < /proc/<PID>/cmdline echo Record suspicious-file information ls -lah <PATH> stat <PATH> file <PATH> sha256sum <PATH> Search targeted directories find \ /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \ -xdev \ \( \ -name 'javab' -o \ -name 'idle' -o \ -name '.khp' -o \ -name '.khp_ts' -o \ -name '.rguard' -o \ -name 'ksmd' \ \) \ -ls 2>/dev/null Remember that ksmd in this command is a search pivot, not a confirmed finding from the published case. Preserve scheduled tasks crontab -u zimbra -l \ > /root/incident-evidence/zimbra-crontab.txt Also inspect: crontab -u root -l and: ls -lah /etc/cron.d/ ls -lah /etc/cron.hourly/ ls -lah /etc/cron.daily/ Zimbra’s own compromise checklist recommends checking newly added cron entries for both zimbra and root. Preserve SSH authorization data cp -a /opt/zimbra/.ssh/authorized_keys \ /root/incident-evidence/ Then: ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys Review authentication activity grep -E \ 'Accepted|Failed|Invalid user' \ /var/log/auth.log Narrow the time window when possible. Review cron activity grep -hE 'CRON.*zimbra' \ /var/log/syslog \ /var/log/syslog.1 \ 2>/dev/null Check Zimbra service state su - zimbra -c 'zmcontrol status' Zimbra documents zmcontrol status as displaying the running or stopped state of Zimbra components. A healthy service list does not prove the operating system is clean. That was one of the important lessons from the incident. Additional Checks When You Find Similar Evidence The following are investigation recommendations. They are not additional confirmed findings from the PinoyLinux incident. Check recently changed Zimbra files Zimbra’s compromise guidance recommends looking for unknown and recently modified files, including files beneath the web application directories and elsewhere under /opt/zimbra. Example: find /opt/zimbra/jetty/webapps/ \ -type f \ -newerct "-30 days" \ -ls Do not delete everything returned by this command. Software updates also create new files. Review listening ports ss -lntup Compare the listeners with the services you expect. Inspect systemd persistence systemctl list-timers --all systemctl list-unit-files --type=service The PinoyLinux incident confirmed cron persistence. It did not establish a malicious systemd service or timer. Search for other SSH keys find /root /home /opt/zimbra \ -type f \ -name authorized_keys \ -ls 2>/dev/null Zimbra’s current compromise checklist recommends validating SSH configuration, checking which authorized-keys file is in use, and reviewing authorized keys for users. Correlate everything by time Compare: SSH authentication time file creation/change time process start time cron execution time Zimbra log events system log events network-device logs One event may mean little. Five related events within minutes can tell a very different story. Why You Should Not Immediately Delete the Indicators Imagine you find: /home/SSL/.khp and immediately run: rm -f /home/SSL/.khp You may lose: timestamps ownership permissions hash information file contents metadata a sample needed for later analysis The same applies to an unknown SSH key. Deleting it may help containment, but it also modifies the original evidence. Where practical: collect | v preserve | v hash | v document | v contain For serious production incidents, evidence should ideally be preserved on trusted storage separate from the compromised host. Zimbra’s Own Compromise Guidance Supports This Approach Zimbra’s official compromise checklist recommends administrators inspect: recently changed and unknown files web application directories unexpected files elsewhere on the server zimbra and root cron entries unknown administrator accounts unknown Zimlets SSH daemon configuration authorized-key configuration and contents open network ports Zimbra also states that if evidence shows a server has been compromised, rebuilding the system is an appropriate response path rather than assuming removal of the visible files has restored trust. That aligns closely with what the PinoyLinux incident demonstrated. Cleaning the Indicators Does Not Restore Trust Suppose you: kill javab kill idle remove the bad cron entry delete .khp remove an unauthorized SSH key You have removed visible components. You have not proven that: no other persistence exists no additional credentials were stolen no Zimbra files were modified no web shell exists no privileged account was added no neighboring system was affected no unknown binaries remain no application data was accessed no attacker-controlled configuration remains A critical production mail server should therefore be treated as untrusted after a serious compromise until you have a defensible recovery plan. Zimbra publishes rebuild procedures specifically for compromised ZCS servers. A safer recovery approach may involve: Preserve required evidence. Determine the affected time window. Rotate passwords, SSH keys, API credentials, and other secrets that may have been exposed. Build a clean replacement host. Install a supported and fully patched Zimbra release. Restore only trusted mail data and configuration. Validate administrator accounts and SSH keys. Review firewall exposure. Monitor the replacement system closely. Retain the old system only as required for forensic investigation. Containment stops activity. Recovery restores service. Rebuilding trust requires more. What We Can Say About the PinoyLinux Incident Based on the current published evidence, we can say: The Zimbra server was compromised. Unauthorized processes named javab and idle ran under the zimbra account. Suspicious files existed under locations including /dev/shm, /home/SSL, and /tmp. The file /home/SSL/.khp was connected to an unauthorized zimbra crontab entry. That entry ran every minute. System logs confirmed cron executing /home/SSL/.khp. SSH authorization information under /opt/zimbra/.ssh/authorized_keys required investigation. At least one key entry used a forced command. Successful public-key authentication to the zimbra account was found in /var/log/auth.log during the broader incident period. We should not claim from the published evidence that: SSH definitely caused the compromise. A specific Zimbra vulnerability definitely caused the compromise. The internal SSH source IP was definitely the attacker. Every unfamiliar SSH key was malicious. ksmd was definitely part of the malware. A filename alone identifies the malware family. Removing the visible files made the server trustworthy again. Those statements go beyond the evidence currently documented. Related PinoyLinux Incident Articles Full incident investigation How We Investigated a Compromised Zimbra Mail Server on Ubuntu Start here for the complete overview of the original compromise investigation. Cron persistence investigation Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server Read this for the detailed connection between .khp, the zimbra crontab, recurring processes, and /var/log/syslog. Process and filesystem investigation How to Investigate Suspicious Processes and Files on a Linux Server Read this for process inspection, /proc, /dev/shm, /tmp, /home/SSL, file metadata, hashing, and targeted filesystem searching. SSH investigation Investigating Suspicious SSH Access on a Compromised Linux Server Read this for authorized_keys, forced commands, key fingerprints, Accepted publickey events, and the limits of SSH evidence. Key Takeaways A Zimbra compromise should not be diagnosed from a filename. javab was significant because it ran under the affected zimbra account, appeared with related artifacts, and formed part of recurring unauthorized activity. idle mattered for the same reason. /dev/shm is a legitimate Linux shared-memory location. Its presence in an incident becomes meaningful only when the files inside correlate with other evidence. /home/SSL looked legitimate by name but contained suspicious artifacts. .khp became especially important when the zimbra user’s crontab showed: * * * * * /home/SSL/.khp and /var/log/syslog confirmed that cron actually executed the command. SSH evidence added another layer. Unknown or questionable keys and a successful public-key login demanded investigation, but they did not prove that SSH was the initial compromise vector. ksmd should not be promoted into a confirmed IOC merely because it appeared in a search expression. The strongest incident findings come from relationships: process + file + user + persistence + logs + authentication + timeline That is the difference between finding something strange and understanding a compromise. And once a critical Zimbra server has been seriously compromised, removing the visible malware is not the same as restoring trust.

[object Object]