/// article

Investigating Suspicious SSH Access on a Compromised Linux Server

When investigating a compromised Linux server, finding malicious processes or suspicious files is only part of the job. You also need to determine how accounts on the server were accessed. During our investigation of a compromised Zimbra mail server running on Ubuntu, we examined the zimbra account’s SSH configuration and discovered entries in: /opt/zimbra/.ssh/authorized_keys that required investigation. Among them were additional SSH public keys and an entry containing a forced command. We then examined: /var/log/auth.log and found evidence of successful SSH public-key authentication to the zimbra account from an internal IP address during the incident period. That raised several important questions: Who owned the SSH keys? Were the keys legitimately installed? When were they added? What did the forced command do? Which IP addresses successfully authenticated? Could those logins be matched to known administrators? Did the SSH activity occur before or after other indicators of compromise? Was SSH the original entry point or simply another persistence mechanism? This article explains how we approached those questions and how Linux administrators can investigate suspicious SSH access without immediately destroying useful evidence. Customer information, domains, actual IP addresses, SSH public keys, key fingerprints, hostnames, credentials, and other identifying information have been removed. Important: Finding an unknown SSH key or an Accepted publickey log entry does not by itself prove how a server was originally compromised. Treat each finding as evidence that must be correlated with other evidence. What We Actually Found Before discussing general SSH investigation techniques, it is important to distinguish our confirmed findings from the additional checks recommended later in this guide. During our Zimbra incident, we inspected: /opt/zimbra/.ssh/authorized_keys The file contained SSH public-key entries that required investigation. One entry included a forced-command configuration. There were also additional keys whose legitimacy needed to be verified. We then examined: /var/log/auth.log and found a successful public-key authentication event involving the zimbra account from an internal IP address during the incident period. A sanitized example of the type of event we investigated looks like: <DATE> <HOST> sshd[<PID>]: Accepted publickey for zimbra from <INTERNAL_IP> port <SOURCE_PORT> ssh2 Depending on the OpenSSH version and logging configuration, an authentication record may also include the key type and fingerprint. These findings were significant because we were already investigating suspicious processes and files running under the same zimbra account. However, we did not establish from this evidence alone that SSH was the original attack vector. That distinction is important. Why /opt/zimbra/.ssh/authorized_keys Matters OpenSSH normally uses an authorized_keys file to determine which public keys are allowed to authenticate as a particular user. The OpenSSH sshd documentation describes the authorized-keys format as containing, for each key: options key-type base64-key comment The options field is optional. For a normal Linux user, the file is commonly found under: ~/.ssh/authorized_keys In our case, the zimbra account’s SSH files were under: /opt/zimbra/.ssh/ because that was the account’s environment on the affected server. You can verify an account’s configured home directory with: getent passwd zimbra A typical result may resemble: zimbra:x:<UID>:<GID>:Zimbra:/opt/zimbra:/bin/bash Do not assume that every server uses the same home directory. Verify it. You should also verify which authorized-key files OpenSSH is actually configured to use. A useful command is: sshd -T | grep -i authorizedkeysfile You may see something similar to: authorizedkeysfile .ssh/authorized_keys .ssh/authorized_keys2 The effective configuration is more useful than assuming the default. OpenSSH documents the AuthorizedKeysFile setting in sshd_config. Preserve the SSH Configuration Before Changing It If you suspect an SSH key is unauthorized, do not immediately delete the line. First preserve the file. For example: mkdir -p /root/incident-evidence chmod 700 /root/incident-evidence Copy the file while preserving metadata: cp -a /opt/zimbra/.ssh/authorized_keys \ /root/incident-evidence/authorized_keys.zimbra Record its hash: sha256sum /root/incident-evidence/authorized_keys.zimbra \ > /root/incident-evidence/authorized_keys.zimbra.sha256 Record metadata from the original: stat /opt/zimbra/.ssh/authorized_keys \ > /root/incident-evidence/authorized_keys.zimbra.stat You can also record the collection time: date -Is \ > /root/incident-evidence/collection-time.txt This creates a simple record of what existed before you modify the account. For incidents involving legal, regulatory, or disciplinary investigation, follow your organization’s forensic evidence procedures instead of relying only on an operational copy. Inspect Ownership and Permissions Start by examining the SSH directory and key file: ls -ld /opt/zimbra/.ssh ls -l /opt/zimbra/.ssh/authorized_keys For more metadata: stat /opt/zimbra/.ssh stat /opt/zimbra/.ssh/authorized_keys Pay attention to: owner group permissions modification time metadata-change time file size For example, unexpected ownership would deserve investigation: root:root when you expected: zimbra:zimbra Likewise, a recent timestamp during the known incident window may help build your timeline. A timestamp alone does not prove when an attacker added a key. Files may be copied, restored, edited, or deliberately timestamped. Treat filesystem timestamps as evidence to correlate with other information. Inspect authorized_keys Carefully To view the file locally: cat /opt/zimbra/.ssh/authorized_keys For easier review with line numbers: nl -ba /opt/zimbra/.ssh/authorized_keys A normal key may resemble: ssh-ed25519 AAAA...REDACTED... administrator@example An entry containing options may look more complex: from="<TRUSTED_NETWORK>",restrict ssh-ed25519 AAAA...REDACTED... backup-key Or: restrict,command="<FORCED_COMMAND>" ssh-ed25519 AAAA...REDACTED... automation-key Do not publish the actual key contents when documenting an incident. The long base64-encoded portion is the public key. Public keys are not secret in the same way private keys are, but publishing production authorization data unnecessarily provides useful information about the environment. Understanding Forced Commands One of the entries we found during our incident contained a forced command. OpenSSH supports this syntax legitimately: command="<COMMAND>" ssh-ed25519 AAAA... When that key successfully authenticates, OpenSSH executes the specified command instead of the command requested by the SSH client. Forced commands have legitimate uses. Examples include: backup accounts automated deployment accounts Git access restricted file transfer monitoring synchronization services administrative automation Therefore: command="..." does not automatically mean malware. The correct question is: Should this key and this forced command exist on this account? In our case, the forced-command entry required investigation because it appeared on an account already associated with suspicious activity and its legitimacy was not immediately established. Other authorized_keys Options to Look For OpenSSH allows several options before a public key. Some common ones include: command="..." from="..." restrict no-pty no-port-forwarding no-agent-forwarding no-X11-forwarding no-user-rc permitopen="host:port" permitlisten="host:port" environment="NAME=value" OpenSSH documents these options as part of the authorized-keys format. Several deserve special attention during an investigation. command="..." Forces a particular command whenever the key authenticates. Investigate unexpected commands. from="..." Restricts where the key may authenticate from. For example: from="192.0.2.0/24" ssh-ed25519 AAAA... A from= restriction can be a legitimate security measure. OpenSSH supports hostnames, addresses, and CIDR notation for this option. permitopen= Limits SSH local port forwarding. For example: permitopen="127.0.0.1:3306" This may be completely legitimate for a tunnel account. But an unexpected forwarding configuration on a compromised account deserves investigation. environment= Can associate environment variables with the key when permitted by the SSH server configuration. Again, context matters. Search for Unusual Key Options A quick way to identify key lines containing options is: grep -nE \ 'command=|from=|restrict|no-pty|no-port-forwarding|no-agent-forwarding|no-X11-forwarding|permitopen=|permitlisten=|environment=' \ /opt/zimbra/.ssh/authorized_keys The -n option displays line numbers. Do not paste this output into public forums because the command prints the complete matching key lines. For incident documentation, redact the key material. For example: Line 3: restrict,command="<REDACTED_COMMAND>" ssh-ed25519 <REDACTED_KEY> <COMMENT> This provides enough information for your report without exposing the actual authorization material. Generate Fingerprints for the Authorized Keys SSH key fingerprints provide a convenient way to identify keys without repeatedly handling the full public-key text. OpenSSH’s ssh-keygen -l option displays the fingerprint of a public key. For the entire authorized-keys file, you can try: ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys The result may resemble: 256 SHA256:<FINGERPRINT_1> <COMMENT> (ED25519) 3072 SHA256:<FINGERPRINT_2> <COMMENT> (RSA) The fingerprint is much easier to compare than a long public-key string. You can use fingerprints to answer questions such as: Does this key belong to one of our administrators? Does it match a key in our configuration-management repository? Was this key deployed to another server? Does an SSH authentication log identify the same fingerprint? Is this key present on multiple compromised hosts? Do not rely on the key’s comment. A comment such as: admin@laptop is simply text attached to the public key. OpenSSH does not use the comment to authenticate the user. An attacker could choose a convincing comment. The fingerprint identifies the actual key. Create a Key Inventory For an incident investigation, create a simple inventory. For example: Line Key Type Fingerprint Comment Known Owner Status 1 ED25519 SHA256:<REDACTED> admin-key Administrator A Verified 2 RSA SHA256:<REDACTED> backup Unknown Investigate 3 ED25519 SHA256:<REDACTED> automation Unknown Investigate Do not include full keys in a public incident report. Then ask the administrators individually whether they recognize each fingerprint. Do not ask: Is this comment yours? Ask: Does this SHA-256 SSH key fingerprint belong to you? That is a much stronger verification method. Investigating /var/log/auth.log Ubuntu documents: /var/log/auth.log as containing authorization-related activity, including remote login activity. During our incident, this log provided an important clue. We found successful public-key authentication involving the zimbra account. To search for successful SSH authentication: grep -E 'sshd.*Accepted' /var/log/auth.log To focus on the zimbra account: grep -E 'sshd.*Accepted.*zimbra' \ /var/log/auth.log For public-key authentication specifically: grep -E 'Accepted publickey for zimbra' \ /var/log/auth.log A sanitized record may resemble: <DATE> <HOST> sshd[<PID>]: Accepted publickey for zimbra from <SOURCE_IP> port <SOURCE_PORT> ssh2 On systems that log the key fingerprint, it may look similar to: <DATE> <HOST> sshd[<PID>]: Accepted publickey for zimbra from <SOURCE_IP> port <SOURCE_PORT> ssh2: ED25519 SHA256:<FINGERPRINT> That gives you several useful fields: timestamp username source IP source port authentication method key type key fingerprint, when logged Understand the Source Port Consider: from <SOURCE_IP> port 54321 That port is normally the client’s source port for that SSH connection. It does not mean your SSH server was necessarily listening on port 54321. The server may still have been listening on the normal SSH port or whatever port was configured for sshd. Do not confuse the remote source port with the local SSH listening port. Search Rotated Authentication Logs The activity you need may no longer be in the current: /var/log/auth.log Check: /var/log/auth.log.1 as well. For example: grep -hE 'Accepted publickey for zimbra' \ /var/log/auth.log \ /var/log/auth.log.1 \ 2>/dev/null Older logs may be compressed. Search them with: zgrep -hE 'Accepted publickey for zimbra' \ /var/log/auth.log.*.gz \ 2>/dev/null This can provide a much longer authentication history. Search for Failed SSH Authentication Successful authentication is only part of the picture. Also inspect failed attempts: grep -E \ 'Failed password|Failed publickey|Invalid user|authentication failure' \ /var/log/auth.log For rotated logs: grep -hE \ 'Failed password|Failed publickey|Invalid user|authentication failure' \ /var/log/auth.log \ /var/log/auth.log.1 \ 2>/dev/null and: zgrep -hE \ 'Failed password|Failed publickey|Invalid user|authentication failure' \ /var/log/auth.log.*.gz \ 2>/dev/null Ubuntu’s security documentation specifically notes that auth.log can be useful for determining whether unauthorized remote authentication may have occurred and shows failed SSH authentication records as examples. Failed Attempts and Successful Logins Tell Different Stories Imagine you find: Failed publickey for zimbra from <IP_A> Failed publickey for zimbra from <IP_A> Failed publickey for zimbra from <IP_A> Accepted publickey for zimbra from <IP_A> That pattern deserves investigation. But it still does not automatically prove malicious access. Perhaps an administrator initially offered the wrong key and then used the correct one. Now consider: Accepted publickey for zimbra from <UNKNOWN_IP> using a key fingerprint that nobody on the administration team recognizes. That is much stronger evidence of unauthorized access. Context changes the meaning. Correlate Source IP Addresses In our incident, an Accepted publickey record came from an internal IP address. An internal source does not automatically make the authentication legitimate. You need to determine what device owned that address at the time. Ask: Was this an administrator workstation? Was it another server? Was it a VPN address? Was it a jump host? Was DHCP involved? Could the address have belonged to a different machine earlier? Was that internal system itself compromised? An IP address tells you where the connection appeared to originate from at the network layer. It does not automatically identify the human responsible. Extract Source IP Addresses From Successful SSH Events For quick analysis: grep 'Accepted publickey for zimbra' /var/log/auth.log You can extract the source address from typical OpenSSH records with: grep 'Accepted publickey for zimbra' /var/log/auth.log | sed -n 's/.* from \([^ ]*\) port .*/\1/p' | sort | uniq -c | sort -nr Example output: 12 <INTERNAL_IP_A> 3 <VPN_IP_B> 1 <UNKNOWN_IP_C> This gives you a frequency count. Do not treat the highest count as the most suspicious. A legitimate automation system may authenticate hundreds of times. A single unknown successful login may be more important. Correlate Authentication With Time Timestamps become much more useful when compared across evidence sources. Suppose you observe: 10:14 SSH public-key authentication 10:16 suspicious file created 10:17 suspicious process started 10:18 cron modified That sequence deserves close investigation. Your actual evidence sources may include: /var/log/auth.log /var/log/syslog filesystem timestamps process information Zimbra logs cron entries shell history network-device logs During our Zimbra incident, we had already discovered suspicious processes and cron persistence. The SSH authentication records became another piece of the same timeline. Search a Specific Time Window Instead of reviewing an entire large log file, narrow the search. For example: grep 'Sep 2 09:' /var/log/auth.log Or search for SSH activity: grep -E 'Sep 2 09:.*sshd' \ /var/log/auth.log For publication or shared incident reports, replace the actual date and address with sanitized values unless they are necessary. A report can instead say: At <TIME_A>, the zimbra account successfully authenticated using a public key from <INTERNAL_IP>. Use the systemd Journal When Appropriate On systems using the systemd journal, SSH records may also be available through journalctl. For example: journalctl -u ssh For a defined period: journalctl -u ssh \ --since "YYYY-MM-DD HH:MM:SS" \ --until "YYYY-MM-DD HH:MM:SS" Depending on the Ubuntu release and logging configuration, useful SSH information may exist in /var/log/auth.log, the journal, or both. Our confirmed incident finding came from /var/log/auth.log. The journalctl checks are additional recommendations. Compare Log Fingerprints With Authorized Keys If your SSH authentication logs include the key fingerprint, compare it directly with: ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys Suppose your log contains: Accepted publickey for zimbra from <SOURCE_IP> ... ED25519 SHA256:<ABC123> and your authorized-key inventory contains: 256 SHA256:<ABC123> admin@example (ED25519) You have now identified which key authenticated. Then verify who actually owns that fingerprint. Do not assume the comment identifies the owner correctly. Check the Effective SSH Configuration Another useful investigation step is determining how sshd is configured. Instead of reading only: /etc/ssh/sshd_config use: sshd -T This displays the effective configuration. OpenSSH documents -T as a configuration test mode that prints the effective server configuration. Useful settings to extract include: sshd -T | grep -E \ 'authorizedkeysfile|pubkeyauthentication|passwordauthentication|permitrootlogin|loglevel|allowusers|allowgroups' This can help answer: Is public-key authentication enabled? Which authorized_keys files are used? Is password authentication enabled? Can root log in? Are SSH users restricted? What logging level is configured? Do not modify the SSH configuration during evidence collection unless containment requires it. Check for Configuration Fragments Modern Ubuntu configurations may also use: /etc/ssh/sshd_config.d/ Inspect the files: ls -lah /etc/ssh/sshd_config.d/ Search SSH authorization-related configuration: grep -RniE \ 'AuthorizedKeysFile|AuthorizedKeysCommand|PubkeyAuthentication|PasswordAuthentication|PermitRootLogin|AllowUsers|AllowGroups|ForceCommand' \ /etc/ssh/sshd_config \ /etc/ssh/sshd_config.d \ 2>/dev/null An unexpected: AuthorizedKeysCommand or: ForceCommand could materially change how SSH authentication behaves. This is an additional recommended check. We did not establish such a configuration change as part of the original Zimbra incident. Check Whether Other Accounts Have SSH Keys Once one account contains suspicious authorization data, inspect other relevant accounts. A targeted search can locate authorized_keys files: find /root /home /opt/zimbra \ -type f \ -name authorized_keys \ -ls 2>/dev/null Then inspect each result. For broader investigations, be careful with filesystem scope and mounted storage. Zimbra’s current compromise-check guidance specifically recommends validating SSH configuration, determining which authorized-keys file is in use, and checking the contents of authorized-key files for all users. Be Careful With Key Removal Once you have confirmed that a key is unauthorized, disabling it may be appropriate. But preserve the original file first. Rather than immediately deleting evidence, you can make a preserved copy and then edit: cp -a /opt/zimbra/.ssh/authorized_keys \ /root/incident-evidence/authorized_keys.before-removal Then: nano /opt/zimbra/.ssh/authorized_keys or: vi /opt/zimbra/.ssh/authorized_keys Remove only the confirmed unauthorized entry. Afterward: ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys to inventory the remaining keys. If the account should not accept interactive SSH access at all, the correct long-term configuration may involve stronger account or SSH restrictions. That should be planned carefully so legitimate Zimbra operations are not broken. Do Not Lock Yourself Out During Containment SSH configuration changes can disconnect your administrative path to the server. Before changing: sshd_config authorized_keys firewall rules AllowUsers AllowGroups PubkeyAuthentication PasswordAuthentication make sure you have a tested recovery path. If you modify SSH server configuration, validate it before restarting: sshd -t OpenSSH documents -t as a configuration and key sanity check. If that returns no error, you can then plan the service reload or restart according to your environment. Do not blindly restart SSH on a remote production server after editing its configuration. Does Accepted Publickey Prove Unauthorized Access? No. This is one of the most important points in the investigation. Consider: Accepted publickey for zimbra from <INTERNAL_IP> This proves that OpenSSH recorded a successful public-key authentication for that account. It does not, by itself, tell you: whether the key was authorized by your organization who physically used the corresponding private key whether the originating computer was compromised whether SSH was the original intrusion vector whether the session performed malicious actions You need additional evidence. A stronger case emerges when several facts align. For example: The key is unknown to administrators. The key appeared during the incident period. The source IP is unexplained. The same account owns suspicious files. Suspicious processes run under the account. Persistence was installed under the account. Authentication occurred shortly before those changes. That combination is much more meaningful than a single log entry. Why an Internal IP Address Is Not Automatically Safe Our incident included public-key authentication from an internal address. It is tempting to conclude: It came from inside the network, so it must have been legitimate. That would be unsafe. Possible explanations include: legitimate administrator access a management server a monitoring system a deployment system a VPN client a compromised workstation another compromised server unauthorized lateral movement The source address must be mapped to the system that owned it at the incident time. Depending on your environment, useful sources may include: DHCP logs VPN logs firewall logs switch records endpoint inventory asset-management records administrator records That network investigation was beyond the specific SSH findings described here, but it is an important next step. Examine Login History, But Know Its Limits Another useful check is: last -ai This can provide historical login information from the system’s login records. You can also focus on one account: last -ai zimbra Treat this as supporting evidence. Not every type of SSH command execution produces the same interactive-login artifacts. A forced command, non-interactive session, automation task, or tampered accounting database may make login-history evidence incomplete. Do not conclude that SSH access did not occur simply because last does not show a matching interactive session. The SSH authentication logs are more important for determining whether authentication succeeded. Look for SSH Sessions Around the Same Period You can search authentication logs for the SSH daemon’s process ID. Suppose the accepted login contains: sshd[28471] Search: grep 'sshd\[28471\]' /var/log/auth.log Related records may show additional activity associated with the connection. This technique depends on the OpenSSH version and logging configuration, but it can sometimes help connect authentication to session creation and termination. Increase SSH Logging for Future Investigations Ubuntu documentation notes that OpenSSH normally logs through the authentication logging facility and that increasing the SSH LogLevel to VERBOSE records additional authentication information. Check the current effective setting: sshd -T | grep loglevel If your security policy requires more SSH authentication visibility, consider: LogLevel VERBOSE in the SSH server configuration. Do not make this change in the middle of evidence collection without documenting it. Changing logging during an incident modifies the system and may complicate timeline analysis. This is a future hardening recommendation, not a confirmed configuration from our incident. What We Actually Confirmed The following findings came directly from our Zimbra investigation: We inspected /opt/zimbra/.ssh/authorized_keys. The file contained multiple SSH key entries requiring investigation. At least one entry included a forced command. We reviewed /var/log/auth.log. We found successful public-key authentication involving the zimbra account. The source was an internal IP address. The SSH activity occurred during the broader incident period. The same zimbra account was already associated with suspicious processes, files, and cron persistence. Those are the confirmed findings relevant to this article. What We Did Not Prove We did not establish from the SSH evidence alone: that SSH was definitely the original compromise vector that the internal source IP belonged to the attacker that every unfamiliar key was malicious that the forced-command syntax itself was malicious that a particular person controlled the private key that the successful login directly created every suspicious file These distinctions are important when writing an incident report. A good report separates: observations evidence reasonable conclusions unverified hypotheses Additional Checks I Recommend If I were repeating the investigation, I would perform and document the following checks systematically. 1. Preserve the SSH authorization file cp -a /opt/zimbra/.ssh/authorized_keys \ /root/incident-evidence/ 2. Record metadata stat /opt/zimbra/.ssh stat /opt/zimbra/.ssh/authorized_keys 3. Generate fingerprints ssh-keygen -E sha256 -lf \ /opt/zimbra/.ssh/authorized_keys 4. Verify the effective key location sshd -T | grep authorizedkeysfile 5. Review successful authentication grep -E 'sshd.*Accepted' \ /var/log/auth.log 6. Review failed authentication grep -E \ 'Failed password|Failed publickey|Invalid user|authentication failure' \ /var/log/auth.log 7. Search rotated logs grep -hE 'sshd.*Accepted' \ /var/log/auth.log \ /var/log/auth.log.1 \ 2>/dev/null zgrep -hE 'sshd.*Accepted' \ /var/log/auth.log.*.gz \ 2>/dev/null 8. Inspect effective SSH configuration sshd -T | grep -E \ 'authorizedkeysfile|pubkeyauthentication|passwordauthentication|permitrootlogin|loglevel' 9. Search other accounts for authorized keys find /root /home /opt/zimbra \ -type f \ -name authorized_keys \ -ls 2>/dev/null 10. Correlate authentication times with other incident evidence Compare SSH events against: cron modifications file timestamps suspicious process start times Zimbra logs system logs firewall logs VPN logs These are recommended investigation steps. They should not be interpreted as additional malicious findings from our original Zimbra incident. A Practical SSH Investigation Workflow If you discover questionable SSH access on a Linux server, a useful workflow is: 1. Preserve the authorized_keys file. 2. Record its ownership, permissions, and timestamps. 3. Generate SHA-256 fingerprints for every key. 4. Ask administrators to verify fingerprints, not comments. 5. Identify forced commands and unusual key restrictions. 6. Verify which AuthorizedKeysFile sshd actually uses. 7. Search auth.log for successful SSH authentication. 8. Search for failed attempts. 9. Review rotated logs. 10. Identify source IP addresses. 11. Determine which device owned each source address at that time. 12. Correlate authentication timestamps with suspicious filesystem and process activity. 13. Disable only keys confirmed to be unauthorized. 14. Rotate affected SSH keys and credentials when necessary. 15. Continue investigating the account for other persistence mechanisms. Do not treat this as a checklist that automatically proves compromise. Each result needs interpretation. The Most Important Lesson From Our Incident The suspicious SSH evidence did not exist in isolation. By the time we investigated /opt/zimbra/.ssh/authorized_keys, we had already found suspicious activity associated with the zimbra account. That included: javab idle .khp /home/SSL cron persistence Then we found questionable SSH authorization data. Then we found successful public-key authentication in: /var/log/auth.log That made SSH account access an important part of the incident. But we still avoided making the unsupported conclusion that: The attacker definitely entered through SSH. The evidence did not prove that. The server may have been compromised through another mechanism first, followed by the installation of an SSH key for persistence. Or an existing SSH key may have been used. Or some of the authentication could have been legitimate. A security investigation should distinguish what the evidence shows from what we think may have happened. Zimbra-Specific Guidance Zimbra’s security documentation explicitly recommends checking SSH when investigating a compromised Zimbra system. Its current compromise-check guidance includes: checking the SSH daemon configuration validating which authorized-keys file is being used validating the contents of authorized-key files checking users for unknown SSH authorization data Zimbra also recommends replacing SSH keys when there is reason to believe files or authentication material may have been accessed. That aligns closely with what we learned during our investigation. Lessons Learned Several lessons from this SSH investigation apply to Linux servers in general. Do not trust SSH key comments. A comment is descriptive text. Verify the fingerprint. A forced command is not automatically malicious. OpenSSH supports command="..." for legitimate restricted-access use cases. Investigate whether the command and key are authorized. An internal source IP does not automatically mean legitimate access. Identify the actual device that owned the address during the incident. Successful authentication does not prove the original attack vector. It proves authentication occurred. Check rotated logs. The event you need may no longer exist in the current auth.log. Correlate timestamps. SSH authentication becomes much more meaningful when compared with suspicious process starts, file creation, cron modifications, and other activity. Fingerprint keys. A SHA-256 fingerprint is a better identifier than a comment or filename. Preserve evidence before deleting keys. Once the authorization file is changed, part of the original system state is gone. Investigate application accounts too. Service accounts such as zimbra can be valuable targets because they may have access to application data, configuration, and operational processes. Do not stop after removing the unauthorized key. If an attacker had enough access to install an SSH key, determine what other changes may have been made. References OpenSSH sshd(8) Manual The official OpenSSH documentation for SSH authentication and the authorized_keys file format, including command=, from=, restrict, forwarding controls, and other key options. OpenSSH sshd(8) manual OpenSSH sshd_config(5) Manual Documents AuthorizedKeysFile, AuthorizedKeysCommand, authentication configuration, and other SSH server settings. OpenSSH sshd_config(5) manual OpenSSH ssh-keygen(1) Manual Documents ssh-keygen -l for displaying public-key fingerprints and related key-management functions. OpenSSH ssh-keygen(1) manual Ubuntu, Viewing and Monitoring Log Files Ubuntu documentation describing /var/log/auth.log and its use for authorization activity and remote logins. Ubuntu Viewing and Monitoring Log Files Ubuntu, Basic Security: Did I Just Get Owned? Ubuntu security guidance covering the use of auth.log when investigating unauthorized access and SSH authentication attempts. Ubuntu Basic Security investigation guide Zimbra, Investigating and Securing Systems Zimbra’s incident investigation guidance covering suspicious processes, SSH access, modified cron jobs, compromised accounts, and replacement of SSH keys when compromise is suspected. Zimbra Investigating and Securing Systems Zimbra, 10 Steps to Check a Zimbra Server for Compromise Zimbra’s compromise checklist specifically recommends checking sshd configuration, determining the authorized-keys location, and validating authorized-key contents for users. Zimbra 10 Steps to Check a Server for Compromise