/// article
Linux File Attributes Explained: Why lsattr Showed “ia” on /tmp
Linux file permissions are usually the first thing administrators check when they encounter a file or directory they cannot modify. We run: ls -l and look for familiar permissions such as: rwxr-xr-x Sometimes that is not enough. During our troubleshooting of a compromised Zimbra mail server, we investigated /tmp because some filesystem behavior did not […]
Linux file permissions are usually the first thing administrators check when they encounter a file or directory they cannot modify. We run: ls -l and look for familiar permissions such as: rwxr-xr-x Sometimes that is not enough. During our troubleshooting of a compromised Zimbra mail server, we investigated /tmp because some filesystem behavior did not make sense. The normal permissions appeared writable, but another command revealed something much more interesting: lsattr -d /tmp The result was: ----ia-------e-- /tmp The important characters were: i a e The i attribute means immutable. The a attribute means append-only. The e attribute indicates that extents are being used to map the object’s blocks on an ext4 filesystem. Finding both i and a on /tmp was unusual and important because /tmp is expected to be available to programs that need temporary files. The Filesystem Hierarchy Standard specifically defines /tmp as a location made available to programs for temporary files. This article explains what those attributes mean, how they differ from normal Linux permissions, how lsattr and chattr work, and why you should never remove filesystem attributes blindly from an important system directory. What We Actually Observed During our Zimbra troubleshooting, we checked /tmp using several commands. The standard permission information showed that /tmp had the expected general writable permissions and sticky bit. We also checked its filesystem attributes: lsattr -d /tmp The output was: ----ia-------e-- /tmp The -d option is important. It tells lsattr to display the attributes of the directory itself instead of listing attributes for files contained inside it. We also confirmed that the root filesystem was ext4. The unusual part was therefore not the e attribute. The unusual part was that /tmp itself had both: i a set. This article does not claim that we established exactly who set those attributes or whether they were directly caused by the compromise. They were an abnormal filesystem condition discovered while troubleshooting the affected server. Normal Linux Permissions Versus File Attributes Linux administrators are familiar with traditional permission bits. For example: ls -ld /tmp might display: drwxrwxrwt 10 root root 4096 Sep 13 10:30 /tmp The important portion is: drwxrwxrwt These permissions control normal read, write, and execute access for: owner group others The final t is the sticky bit. On /tmp, the sticky bit allows multiple users to create files while generally preventing users from deleting files owned by somebody else. Linux file attributes operate at another layer. They can change how the filesystem treats a file or directory even when normal Unix permissions appear to allow an operation. This explains an important troubleshooting situation: Permissions say: writable Filesystem attributes say: immutable Result: writing or modifying can still fail The Linux kernel exposes these as inode flags. lsattr reads them and chattr modifies supported flags. A Terminology Note: These Are Not the Same as Linux xattrs You will sometimes hear lsattr attributes described informally as “extended attributes.” Technically, that can cause confusion. The flags manipulated by: lsattr chattr are Linux file attributes or inode flags. Linux also has a separate feature called extended attributes, commonly shortened to xattrs. Those can contain named values such as: user.example security.selinux system.posix_acl_access and are commonly inspected with tools such as: getfattr setfattr The systemd documentation explicitly notes that Linux file attributes such as those manipulated by chattr are a different concept from extended attributes. So throughout this article, when we discuss: i a e we are talking about filesystem inode flags displayed by lsattr. Understanding lsattr The command: lsattr lists supported file attributes. For example: lsattr /etc/example.conf might return: --------------e------- /etc/example.conf For a directory itself, use: lsattr -d /tmp Without -d, lsattr normally examines the directory’s contents instead. The Linux lsattr manual documents -d specifically for treating directories like normal files rather than listing their contents. You can also use: lsattr -a /tmp to include hidden entries. For recursive inspection: lsattr -R /some/path Be careful with recursive commands on large production filesystems. What the i Attribute Means The most important attribute in our /tmp result was: i The i flag means immutable. For a regular file, immutable means the file cannot normally be modified, deleted, renamed, linked to, or opened for writing. The Linux chattr documentation states that most metadata cannot be modified either. Only a privileged process with CAP_LINUX_IMMUTABLE can set or clear this attribute. For example: touch /tmp/example chattr +i /tmp/example After that: echo "test" >> /tmp/example should fail. So should: rm /tmp/example even if the ordinary permission bits would otherwise allow the operation. The immutable flag therefore provides restrictions beyond: rwx permissions. Immutable on a Directory Is More Serious Our incident involved: /tmp itself, not simply one file inside /tmp. A directory is essentially a structure containing directory entries that map names to filesystem objects. Changing those entries requires modifying the directory. The Linux kernel’s ext4 documentation describes directories as structures containing these directory entries. If the directory itself is immutable, normal attempts to alter its directory contents can fail. That can affect operations such as: creating files removing files renaming entries creating subdirectories removing subdirectories This makes an immutable /tmp particularly problematic. A large number of programs expect to be able to create temporary files there. What the a Attribute Means The second important flag was: a This means append-only. For a regular file, append-only means data can be added to the end of the file, but existing contents cannot normally be overwritten. For example: chattr +a example.log allows: echo "new entry" >> example.log but prevents ordinary operations that would overwrite or truncate the existing data. The Linux manual states that only a privileged process with CAP_LINUX_IMMUTABLE can set or clear the append-only flag. Append-Only Behaves Differently on Directories The behavior becomes especially interesting when a is applied to a directory. Linux documentation states that the append-only flag on a directory prevents files from being removed from that directory through operations such as unlink() and rename(). That means something like: /tmp being append-only can interfere with normal temporary-file cleanup. Programs might create temporary files but later be unable to remove or rename them. System cleanup tools can also be affected. This is important because systemd-tmpfiles normally performs creation, removal, and cleanup operations involving temporary and volatile locations including /tmp and /var/tmp. What Happens When i and a Are Both Set? Our actual output was: ----ia-------e-- /tmp So /tmp had both: i = immutable a = append-only The immutable flag is the stronger restriction. If the directory cannot be modified because it is immutable, the append-only behavior does not suddenly make normal modification possible. The combination is therefore highly restrictive. For a directory whose purpose is providing temporary storage to applications, that condition deserves immediate investigation. What the e Attribute Means Our output also contained: e This was: ----ia-------e-- The e flag means the object uses extents for mapping filesystem blocks. The Linux chattr documentation states that this flag indicates extent-based block mapping and that it cannot be removed using chattr. The ext4 documentation also lists: e = extents format as an ext4 filesystem attribute. Therefore, on an ext4 filesystem: e by itself is not an indication of compromise. This is an important distinction. When reading: ----ia-------e-- we should not treat every letter as suspicious. In our troubleshooting case: i required investigation a required investigation e expected ext4-related filesystem information Other Useful lsattr Attributes Linux supports several other file attributes, although availability depends on the filesystem. Some useful ones administrators may encounter include: Attribute Meaning i Immutable a Append-only A Do not update access time d Exclude from dump backups D Synchronous directory updates e Extent format j Data journaling on ext3/ext4 S Synchronous updates I Indexed directory V fs-verity enabled Not every filesystem supports every flag. The chattr documentation explicitly warns that filesystem support varies. You should therefore identify the filesystem before interpreting attributes. For example: findmnt -T /tmp or: df -T /tmp can help determine what filesystem contains /tmp. How chattr Changes Attributes chattr changes supported Linux file attributes. The basic syntax is: chattr OPERATOR ATTRIBUTE PATH The three primary operators are: + add an attribute - remove an attribute = replace the selected attributes For example: chattr +i important.conf adds the immutable flag. This: chattr -i important.conf removes it. And: chattr +a application.log adds append-only behavior. The chattr manual documents these +, -, and = operations. Why You Should Not Immediately Run chattr -ia /tmp After seeing: ----ia-------e-- /tmp the obvious reaction might be: chattr -ia /tmp That would remove both: i a from /tmp. But that should not be your first step. Before changing filesystem attributes on an important production directory, determine why they are there. Record the existing state first: lsattr -d /tmp Then record permissions: ls -ld /tmp Check ACLs: getfacl -p /tmp Check metadata: stat /tmp Identify the filesystem: findmnt -T /tmp Check the mount state: mount | grep ' on / ' or: findmnt / If the system is part of an active security incident, preserve these results before modifying anything. Do Not Confuse Immutable Attributes With a Read-Only Filesystem This distinction is important. Suppose a command fails with: Operation not permitted You might assume the filesystem has become read-only. Check: findmnt / or: mount | grep ' on / ' A filesystem mounted as: rw can still contain an immutable file or directory. For example: Filesystem: read-write /tmp permissions: rwxrwxrwt /tmp attribute: immutable can still result in operations failing against /tmp. Normal permissions, filesystem mount state, ACLs, and inode flags are separate controls. You need to check each one. Why /tmp Is Important to Applications The Filesystem Hierarchy Standard defines /tmp as the directory for temporary files and states that it must be available to programs that require temporary files. Applications may use temporary storage for activities such as: extracting temporary data building intermediate files creating sockets or lock files generating temporary configuration performing software installation processing uploads sorting or transforming data This means abnormal restrictions on /tmp can produce failures that seem unrelated to the filesystem. You might see an application report: Cannot create temporary file or: Operation not permitted while /tmp still appears to have: drwxrwxrwt permissions. That is exactly why lsattr is worth checking when normal permission troubleshooting does not explain the behavior. How It Can Affect Package Operations Software installation and maintenance tools may use temporary files during package extraction, script execution, configuration generation, or other intermediate operations. If /tmp cannot be modified normally, a package operation may fail even when: df -h shows available disk space and: ls -ld /tmp shows normal permissions. Do not immediately conclude: APT is broken. or: The package database is corrupt. Check the underlying filesystem conditions too. A useful troubleshooting sequence is: ls -ld /tmp lsattr -d /tmp getfacl -p /tmp df -h /tmp df -T /tmp findmnt -T /tmp These commands examine different parts of the problem. How It Can Affect Databases Databases and supporting tools can also use temporary storage. The exact location depends on the database software and configuration. If a database or maintenance process is configured to use /tmp, restrictive attributes on the directory may prevent it from: creating temporary files renaming temporary files removing temporary files creating sockets performing maintenance operations This can lead administrators down the wrong troubleshooting path. A database service may fail, but the database engine itself may not be the root cause. Always inspect the error logs before changing database configuration. If the error refers to: /tmp temporary file permission denied operation not permitted cannot create cannot unlink cannot rename check both normal permissions and filesystem attributes. How It Can Affect Cleanup Temporary files are normally expected to come and go. On systemd-based Linux systems, systemd-tmpfiles can create, remove, and clean temporary or volatile files and directories. Its documentation specifically mentions locations such as: /run /tmp /var/tmp among the areas commonly managed by the mechanism. Now consider an append-only directory. Linux states that an append-only directory prevents file removal through operations such as unlink() and rename(). Cleanup operations can therefore fail. An immutable directory is even more restrictive. Why Root May Still See “Operation Not Permitted” One of the confusing aspects of immutable files is that normal root privileges do not make the flag irrelevant. The kernel documentation states that immutable restrictions apply even to the superuser during ordinary modification attempts. A privileged process with the required CAP_LINUX_IMMUTABLE capability can set or clear the relevant flag. So this: rm protected-file may fail as root. But root can normally remove the immutable flag first: chattr -i protected-file and then remove the file. That difference is important. The immutable flag is not saying: root can never change this It means normal file-modification operations are blocked until an appropriately privileged process removes the flag. A Safe Test on a Lab File If you want to understand these attributes, do not experiment on /tmp or another system directory. Create a lab directory: mkdir ~/attr-test cd ~/attr-test Create a file: echo "original" > example.txt Inspect it: lsattr example.txt Set immutable: sudo chattr +i example.txt Check again: lsattr example.txt You should now see an i. Try: echo "changed" >> example.txt or: rm example.txt The operation should fail. Remove the attribute: sudo chattr -i example.txt Then verify: lsattr example.txt This is a much safer way to learn chattr than experimenting with: /etc /var /tmp /opt on a production server. Never Use Recursive chattr Blindly chattr supports: -R for recursive operations. For example: chattr -R -i /some/directory can remove immutable attributes recursively. That can be dangerous. You may unintentionally change attributes on: application data security-sensitive files database files configuration files backup repositories system-managed files The original attribute may have been intentionally set as a security or operational control. Always inspect first. For example: lsattr -R /some/directory and record the state before changing anything. Do Not Use chattr = Unless You Understand the Consequences This syntax: chattr =i file does not simply add immutable. The = operator sets the selected file attributes as the resulting attribute set supported by that operation. That is different from: chattr +i file which only adds i. When troubleshooting production systems, targeted changes are safer than broad attribute replacement. The e Flag Should Usually Be Left Alone Because our /tmp output contained: e an administrator might attempt: chattr -e /tmp That is not appropriate. Linux documents e as indicating extent-based block mapping and specifically says it may not be removed using chattr. On our ext4 filesystem, e was not the problematic part of: ----ia-------e-- The attributes requiring investigation were: i a A Better Troubleshooting Workflow If a file or directory behaves as though it is not writable even though normal permissions look correct, check several layers before modifying anything. Start with: ls -ld /path Then: lsattr -d /path Check ACLs: getfacl -p /path Inspect metadata: stat /path Identify the filesystem: findmnt -T /path Check capacity: df -h /path Check inode availability: df -i /path For /tmp, you can also perform a non-destructive creation test when appropriate: touch /tmp/test-file-$$ and then: rm /tmp/test-file-$$ Do not perform this test if you are preserving a compromised system for forensic analysis, because even a simple test modifies filesystem state. File Attributes Can Also Be Used Legitimately The presence of: i or: a does not automatically mean malware. An administrator might intentionally use immutable on a sensitive configuration file: chattr +i important.conf Likewise, append-only behavior can be useful when you want existing file contents protected from ordinary overwrite or truncation. The security issue is context. Ask: Who set this? Why was it set? Is it documented? Does the application expect it? When was it changed? Does it correspond with the incident timeline? In our Zimbra case, finding i and a on /tmp was relevant because the server was already under security and filesystem investigation. File Attributes Can Also Be Abused The same properties that make immutable attributes useful for protection can make them useful to an attacker. For example, malicious files can be marked immutable to make ordinary deletion fail. An administrator might try: rm malicious-file and receive: Operation not permitted even as root. Before assuming filesystem corruption, inspect: lsattr malicious-file If it shows: ----i----------- then the immutable flag may explain the behavior. But do not immediately run: chattr -i malicious-file rm malicious-file during an incident. Preserve the file, its metadata, its hash, and relevant process information first. Our Incident: What the Output Told Us The important command in our troubleshooting was: lsattr -d /tmp which returned: ----ia-------e-- /tmp We could interpret that as: Attribute Meaning Assessment in Our Case i Immutable Abnormal for /tmp, required investigation a Append-only Abnormal for /tmp, required investigation e ext4 extent format Expected filesystem-related attribute This explained why simply looking at: ls -ld /tmp was not enough. Normal Unix permissions and ACLs are only part of Linux access control. The inode flags shown by lsattr can impose additional restrictions. What We Confirmed and What We Did Not From our actual troubleshooting, we confirmed that: /tmp was on the affected Linux server. /tmp had normal Unix-style access permissions. /tmp was on the ext4 root filesystem. lsattr -d /tmp returned attributes containing i, a, and e. The server was already being investigated for compromise and filesystem problems. We did not establish from the lsattr output alone: who set the i and a flags exactly when they were set whether malware set them whether an administrator set them earlier whether they directly caused every Zimbra or MySQL problem we observed That distinction matters. The correct conclusion was: /tmp had abnormal filesystem attributes that could interfere with normal operations and required further investigation. It would have been incorrect to state: The malware definitely set /tmp immutable. unless other evidence proved it. Lessons Learned The biggest lesson from this troubleshooting case was that Linux permissions do not tell the entire story. A directory can appear writable through: ls -ld while filesystem attributes still prevent modifications. The i attribute can make a file or directory immutable. The a attribute can restrict it to append-oriented behavior and, on directories, prevent existing entries from being removed or renamed. The e attribute on ext4 indicates extent-based storage and is not itself suspicious. When Linux reports unexplained: Operation not permitted or applications suddenly fail to create, delete, or rename files, check: lsattr before changing permissions repeatedly. And if you discover unusual attributes on an important path such as: /tmp /etc /var /opt do not blindly remove them. Document the state, understand why the flags exist, correlate the timestamps and incident evidence, and only then decide whether changing them is appropriate. Sometimes the problem is not: chmod at all. It is: chattr References Linux lsattr(1) Manual Documents lsattr, including the -d, -R, and -a options and its purpose for displaying ext2, ext3, and ext4 file attributes. Linux lsattr(1) manual Linux chattr(1) Manual Documents immutable, append-only, extents, journaling, no-atime, synchronous updates, and other Linux file attributes. Linux chattr(1) manual Linux FS_IOC_GETFLAGS and FS_IOC_SETFLAGS Manual Documents the underlying Linux inode flags, including the exact semantics of immutable and append-only attributes on files and directories. Linux inode flag documentation Linux ext4 Documentation Documents ext4 file attributes, including append-only, immutable, journaling, and extent format. Linux ext4(5) manual Linux Kernel ext4 Inode Documentation Documents ext4 inode flags, including EXT4_IMMUTABLE_FL, EXT4_APPEND_FL, and other filesystem-level flags. Linux kernel ext4 inode documentation systemd tmpfiles.d Documentation Explains the management of temporary and volatile files and explicitly distinguishes Linux file attributes from extended attributes. Linux tmpfiles.d documentation Filesystem Hierarchy Standard, /tmp Defines /tmp as a directory that must be available to programs requiring temporary files. Filesystem Hierarchy Standard 3.0