/// article

Recovering a Corrupted Zimbra MySQL Database with innodb_force_recovery

By lordfrancs3

A Zimbra mail server can fail even when the mailbox data itself is still present on disk. One possible cause is corruption in the database used by the Zimbra mailbox service. That was the situation we encountered while recovering a damaged Zimbra installation. The database service would not start normally. Lower innodb_force_recovery settings did not provide enough recovery to make the database usable. Eventually, setting: innodb_force_recovery = 6 allowed the database server to start far enough for us to inspect the available databases. That did not mean the database was repaired. It meant we had reached an emergency state where some information became readable. That distinction is critical. MySQL explicitly states that innodb_force_recovery should be used only in emergencies to start InnoDB long enough to dump data. Values of 4 or greater can permanently corrupt data files, and level 6 is described as drastic because redo-log roll-forward is disabled and pages can remain obsolete. This article documents what happened during our Zimbra recovery, explains levels 1 through 6, and presents a safer recovery workflow for extracting data, rebuilding the database, and restoring Zimbra. Critical warning: Do not treat this article as a command sequence to paste blindly into production. Database recovery is destructive if performed incorrectly. Preserve a complete copy of the database files before changing recovery levels, deleting files, running zmmyinit, dropping databases, or rebuilding InnoDB. What Happened in Our Zimbra Recovery Our Zimbra database would not start normally. The server had already experienced broader filesystem and security problems, so we could not assume that the database failure was an isolated MySQL issue. We attempted to start the database with forced InnoDB recovery. Lower recovery levels did not get the database into a sufficiently usable state. Recovery settings including: innodb_force_recovery = 3 and: innodb_force_recovery = 4 did not solve the startup problem sufficiently for recovery. Eventually: innodb_force_recovery = 6 allowed the database server to start. At that point we could run commands such as: SHOW DATABASES; and see the Zimbra-related databases. That was encouraging, but the database was still severely damaged. Dump attempts did not complete successfully for all of the data we needed. We later performed additional database initialization work using Zimbra’s zmmyinit, and we encountered authentication problems involving the Zimbra database user afterward. That experience reinforced an important lesson: Getting MySQL to start is not the same as recovering the database. What innodb_force_recovery Actually Does InnoDB normally performs several recovery operations when MySQL starts. These can include: redo-log processing rollback of incomplete transactions purge processing background processing insert-buffer merging recovery of transactional state When database structures are damaged, one of those normal recovery operations can itself cause MySQL to crash. innodb_force_recovery progressively disables portions of normal InnoDB recovery so the database engine has a better chance of starting. The purpose is primarily: START DATABASE | v READ WHATEVER DATA IS STILL ACCESSIBLE | v DUMP THAT DATA | v REBUILD A CLEAN DATABASE The intended workflow is not: START AT LEVEL 6 | v DATABASE STARTS | v RETURN SERVER TO PRODUCTION MySQL’s current documentation specifically says that forced recovery should be used to get InnoDB running so tables can be dumped. It recommends starting at level 1 and increasing the value only as necessary. Always Start at the Lowest Recovery Level The normal setting is: innodb_force_recovery = 0 which means normal InnoDB operation. If corruption prevents startup, MySQL recommends beginning with: innodb_force_recovery = 1 If that does not work, try: 1 2 3 incrementally. Levels: 4 5 6 are increasingly dangerous. MySQL warns that values 4 or greater can permanently corrupt database files and recommends testing those values against a separate physical copy of the database rather than the only production copy. That is one reason a filesystem-level backup should come before experimentation. Understanding Levels 1 Through 6 The following table summarizes the current MySQL behavior. Level Internal Name What It Does Risk 1 SRV_FORCE_IGNORE_CORRUPT Attempts to continue even when corrupt pages are detected Lowest forced-recovery level 2 SRV_FORCE_NO_BACKGROUND Stops background and purge threads Increased risk 3 SRV_FORCE_NO_TRX_UNDO Prevents rollback of transactions after crash recovery Significant consistency risk 4 SRV_FORCE_NO_IBUF_MERGE Disables insert-buffer merging and table-statistics calculation Dangerous, can permanently corrupt data 5 SRV_FORCE_NO_UNDO_LOG_SCAN Does not scan undo logs and treats incomplete transactions as committed Very dangerous 6 SRV_FORCE_NO_LOG_REDO Skips redo-log roll-forward Drastic, highest risk A higher level includes the behavior of the lower levels. For example: innodb_force_recovery = 3 includes the effects of levels: 1 2 3 MySQL also prevents normal INSERT, UPDATE, and DELETE operations whenever innodb_force_recovery is greater than zero. At level 4 and above, InnoDB is placed in read-only mode. Level 1: Ignore Corrupt Pages Configuration: innodb_force_recovery = 1 Internal mode: SRV_FORCE_IGNORE_CORRUPT This tells InnoDB to attempt to continue even when it detects corrupt pages. The idea is to allow queries to skip damaged areas where possible so that readable records can still be extracted. This is the safest forced recovery level. If level 1 lets the server start and your dumps complete, there is usually no reason to increase the recovery level. Level 2: Stop Background Processing Configuration: innodb_force_recovery = 2 Internal mode: SRV_FORCE_NO_BACKGROUND This prevents InnoDB’s master and purge threads from running. It can help when corruption causes crashes during background or purge operations. Again, the goal remains extraction. If this level lets you dump the database, stop increasing the recovery value. Level 3: Skip Transaction Rollback Configuration: innodb_force_recovery = 3 Internal mode: SRV_FORCE_NO_TRX_UNDO This prevents InnoDB from rolling back incomplete transactions during crash recovery. This can allow startup when rollback itself triggers the failure. But transactional consistency becomes less reliable. A database that starts at level 3 should therefore be viewed as a source of recoverable records, not as a database ready to resume production. Level 4: Disable Insert-Buffer Merge Configuration: innodb_force_recovery = 4 Internal mode: SRV_FORCE_NO_IBUF_MERGE At this level, MySQL stops insert-buffer merging and table-statistics calculation. This is where the risk increases sharply. MySQL warns that level 4 can permanently corrupt data files and advises being prepared to rebuild secondary indexes after using it. InnoDB also enters read-only mode at this level. Do not casually move to level 4 because levels 1 through 3 were inconvenient. Level 5: Skip Undo-Log Scanning Configuration: innodb_force_recovery = 5 Internal mode: SRV_FORCE_NO_UNDO_LOG_SCAN At level 5, InnoDB does not examine undo logs during startup. Incomplete transactions can effectively be treated as committed. That means the resulting database view may contain data that would normally have been rolled back. MySQL warns that this level can permanently corrupt database files. This is a salvage mode. Level 6: Skip Redo Recovery Configuration: innodb_force_recovery = 6 Internal mode: SRV_FORCE_NO_LOG_REDO This is the highest recovery level. InnoDB does not perform redo-log roll-forward. MySQL describes this level as drastic because database pages may remain in an obsolete state, and further corruption can appear in B-trees and other structures. This was the level that finally allowed our damaged database to start sufficiently for inspection. Again: Level 6 did not repair our database. It allowed us to access enough of the database engine to determine what data might still be recoverable. Why Level 6 Starting Was Both Good and Bad News Once MySQL started, we could run: SHOW DATABASES; and see databases such as the Zimbra database and mailbox-group databases. That meant the server could still read at least some database metadata. The temptation at this point is to think: Great. MySQL is working again. That is exactly the wrong conclusion. A database requiring: innodb_force_recovery = 6 to start should be considered severely damaged. MySQL warns that level 6 skips redo-log recovery and may expose outdated pages and damaged B-tree structures. The next question is therefore not: Can we restart Zimbra? It is: How much data can we extract before rebuilding the database? Step 1: Check the Underlying Server First Before assuming InnoDB is the only problem, inspect the server. Check filesystem capacity: df -h Check inode availability: df -i Check the Zimbra filesystem: df -h /opt/zimbra Inspect mounts: findmnt Check whether the root or Zimbra filesystem unexpectedly became read-only: findmnt / Look for kernel filesystem errors: dmesg -T | grep -Ei \ 'ext4|xfs|I/O error|filesystem|read-only|remount' If the filesystem itself is damaged or being remounted read-only, database recovery attempts may fail regardless of the InnoDB setting. This was particularly relevant to our recovery because the server had previously experienced filesystem problems. Step 2: Check Zimbra and Database Status Switch to the Zimbra account: su - zimbra Check the overall Zimbra state: zmcontrol status Then check the database service using the tools available in your Zimbra version. For older and classic Zimbra installations, this may include: mysql.server status Attempting a normal start may be useful before forced recovery: mysql.server start If it fails, do not repeatedly restart it without reading the error log. Step 3: Read the MySQL Error Log For classic Zimbra installations, an important log is: /opt/zimbra/log/mysql_error.log Inspect the latest entries: tail -100 /opt/zimbra/log/mysql_error.log Or search for InnoDB problems: grep -iE \ 'innodb|corrupt|error|crash|assert|tablespace|page' \ /opt/zimbra/log/mysql_error.log | tail -100 Zimbra’s troubleshooting documentation identifies mysql_error.log as the log containing database health information, startup messages, and InnoDB corruption errors. Do not begin deleting database files simply because you see an InnoDB error. First establish what is actually failing. Step 4: Make a Physical Copy Before Forced Recovery This is one of the most important steps. WARNING Before experimenting with recovery levels, especially: 4 5 6 make a copy of the original database files. MySQL explicitly recommends ensuring that you have a database backup before using forced recovery. Zimbra’s own crash-recovery procedure also instructs administrators to preserve a copy of: /opt/zimbra/db/data before destructive database operations. Ideally, stop the database before creating a filesystem copy. For example: su - zimbra mysql.server stop Then, as root, preserve the database directory. A simple approach might be: cp -a /opt/zimbra/db/data \ /recovery/zimbra-db-data-original For large installations, an LVM snapshot, storage snapshot, VM snapshot, or block-level image may be more practical. The important point is this: DO NOT experiment on your only copy. If the database is highly valuable, create: Original damaged copy | +---- untouched evidence copy | +---- working recovery copy Perform forced-recovery experiments against the working copy where practical. Step 5: Identify the Correct Zimbra Database Configuration Zimbra’s Tech Center recovery procedure documents: /opt/zimbra/conf/my.cnf as the configuration location used for the bundled database in the classic recovery workflow. Before editing anything: grep -n 'innodb_force_recovery' \ /opt/zimbra/conf/my.cnf Preserve the configuration: cp -a /opt/zimbra/conf/my.cnf \ /root/my.cnf.before-recovery Then edit the configuration. Under: [mysqld] begin with: innodb_force_recovery = 1 Do not start at 6. Step 6: Increase Recovery Incrementally Start the database: su - zimbra mysql.server start If startup still fails, inspect: tail -100 /opt/zimbra/log/mysql_error.log Then stop the service before changing the setting again. Try: innodb_force_recovery = 2 Then: innodb_force_recovery = 3 and only proceed higher if necessary. The correct pattern is: 1 | failed v 2 | failed v 3 | failed v consider 4 only with a preserved copy MySQL explicitly recommends increasing recovery levels incrementally. Before Using Levels 4, 5, or 6 Stop and confirm that you have: a physical copy of the original database enough disk space for database dumps a recovery destination outside volatile storage the Zimbra version documented database configuration backed up relevant passwords and configuration available a rebuild or restore plan Do not use: innodb_force_recovery = 6 as an experimental first step. Levels 4 through 6 are emergency salvage modes. Step 7: See What Is Still Readable Once the database starts, do not immediately start Zimbra mailbox services. First determine what can be read. Load the Zimbra database environment where appropriate: su - zimbra source ~/bin/zmshutil zmsetvars Then: mysql -e "SHOW DATABASES;" During our recovery, getting this command to work at level 6 was an important milestone. It confirmed that database metadata was still accessible. You might see databases including: zimbra mboxgroup1 mboxgroup2 mboxgroup3 ... The exact names depend on the installation. Inspect Before Running Complex Queries At high recovery levels, simple queries may work while complex queries fail. MySQL warns that if a high innodb_force_recovery value is needed, damaged internal structures may cause queries using clauses such as WHERE or ORDER BY to fail, while simple table reads may still work. Start conservatively. For example: SHOW DATABASES; Then: USE zimbra; SHOW TABLES; For an individual table: SELECT * FROM mailbox LIMIT 10; Do not assume success with one table means every mailbox database is healthy. Step 8: Attempt Logical Dumps The objective of forced recovery is to extract data. Zimbra’s recovery documentation follows this same general pattern: start the database in recovery mode dump relevant databases rebuild clean databases import the dumps test start Zimbra services Create a dump destination on reliable storage. Avoid relying exclusively on /tmp. For example: mkdir -p /recovery/mysql-dumps chown zimbra:zimbra /recovery/mysql-dumps chmod 700 /recovery/mysql-dumps Then: su - zimbra source ~/bin/zmshutil zmsetvars Confirm which dump utility exists: command -v mysqldump mysqldump --version Zimbra’s Tech Center notes that installations from ZCS 8.7 onward moved bundled utilities such as mysqldump under: /opt/zimbra/common/bin/ in that documented generation. Always verify your installed release rather than assuming a path. Create a Database List You can inspect databases with: mysql --batch --skip-column-names \ -e "SHOW DATABASES;" For a classic Zimbra installation, relevant logical databases often include: zimbra mboxgroup* Additional Zimbra modules may have their own databases. Do not blindly use a list from another server or another Zimbra version. Dump the Zimbra Database First A command pattern may look like: mysqldump \ -S "$mysql_socket" \ -u root \ --password="$mysql_root_password" \ zimbra \ > /recovery/mysql-dumps/zimbra.sql Then mailbox databases can be attempted individually. For example: mysqldump \ -S "$mysql_socket" \ -u root \ --password="$mysql_root_password" \ mboxgroup1 \ > /recovery/mysql-dumps/mboxgroup1.sql Individual dumps are useful because corruption in one mailbox group may not prevent you from recovering another. MySQL’s mysqldump utility generates logical SQL output that can later be loaded into another server. Why Individual Dumps Are Better During Corruption Imagine you have: mboxgroup1 good mboxgroup2 good mboxgroup3 corrupt mboxgroup4 good A single large dump can stop when it hits: mboxgroup3 and leave you uncertain about later databases. Dumping separately lets you determine exactly which databases are readable. You could maintain a simple recovery table: Database Dump Result zimbra Success mboxgroup1 Success mboxgroup2 Success mboxgroup3 Failed mboxgroup4 Success This is much easier to reason about during recovery. Verify That Dumps Actually Completed Do not assume that a large .sql file means the dump succeeded. Check the command’s exit status: echo $? A value of: 0 normally means success. Anything else needs investigation. Zimbra’s recovery page also recommends checking dump files for the normal completion marker written by mysqldump. You can inspect the final lines: tail -20 /recovery/mysql-dumps/mboxgroup1.sql Also record hashes: sha256sum /recovery/mysql-dumps/*.sql \ > /recovery/mysql-dumps/SHA256SUMS What Happened With Our Dump Attempts In our case, level 6 allowed database inspection. However, our dump attempts still encountered failures. That was another reminder that: MySQL started does not mean: all tables are readable Level 6 suppresses important recovery mechanisms to keep the server alive. It does not reconstruct damaged pages. It does not guarantee that B-trees are consistent. It does not guarantee that every table can be scanned. If a Full Table Dump Fails MySQL’s current forced-recovery documentation notes that if corruption prevents reading an entire table, reversing traversal by primary key may sometimes recover rows located after the corrupted portion. For example, a normal query: SELECT * FROM affected_table; might crash or fail. Depending on the table structure, an administrator may attempt targeted extraction around the damaged area. This is advanced salvage work. Do not begin experimenting with complex queries against your only database copy. Use a recovery copy where possible. Recovery Mode Is Not Production Mode This deserves its own section. Do not leave: innodb_force_recovery = 6 enabled and restart normal Zimbra services. At forced-recovery levels above zero, normal InnoDB write behavior is restricted. At level 4 and above, InnoDB is read-only. Even if Zimbra appears to start, the system is not operating normally. Do not allow: users SMTP delivery IMAP webmail mailbox writes scheduled maintenance to resume against a badly damaged database simply because MySQL accepts connections. The correct next step is extraction and rebuild. Step 9: Build a Clean Recovery Environment Once you have extracted as much data as possible, rebuild the database in normal mode. The safest approach is often: Damaged database | v Extract SQL | v Clean database instance | v Import recovered SQL | v Validate | v Start Zimbra For a server that was also compromised at the operating-system level, I strongly prefer performing the restoration on a clean Zimbra installation rather than trusting the original host. Database recovery and security recovery are separate problems. A repaired database does not make a compromised operating system trustworthy. About zmmyinit Zimbra includes database initialization mechanisms used to create a clean database environment. Zimbra’s crash-recovery documentation describes a recovery scenario where the old database directory is moved aside and: /opt/zimbra/libexec/zmmyinit is used to initialize a new database structure. This is a high-risk step. WARNING Do not run zmmyinit casually against your only database installation. Before using it: Stop Zimbra. Preserve the entire existing database directory. Verify your Zimbra release. Verify the correct zmmyinit syntax for that release. Record the database credentials and configuration. Make sure you have usable logical dumps or another restore source. During our recovery work, zmmyinit successfully created database structures, but we later had credential mismatches involving the zimbra database user. That complicated the recovery. This is another reason initialization must be treated as part of a planned rebuild, not as a generic “fix MySQL” command. Never Delete InnoDB Files Before Preserving Them Some older recovery procedures include operations against files such as: ibdata* ib_logfile* or entire database directories. These actions can permanently eliminate your remaining recovery options. Zimbra’s classic crash-recovery article itself instructs administrators to take a copy of: /opt/zimbra/db/data before dropping databases and removing InnoDB files. Treat any command resembling: rm -rf /opt/zimbra/db/data/ib* as destructive. DO NOT RUN THIS AGAINST YOUR ONLY COPY A better pattern is: original data directory | v preserve it untouched | v create clean database elsewhere If disk capacity permits, moving damaged data aside is safer than deleting it. Step 10: Remove Forced Recovery Before Rebuilding Before starting a newly initialized or restored database normally, remove: innodb_force_recovery = 6 or whichever recovery value you used. Normal operation requires: innodb_force_recovery = 0 which normally means the option is simply absent. Confirm: grep -n 'innodb_force_recovery' \ /opt/zimbra/conf/my.cnf The recovery option should not remain enabled after the clean rebuild. Step 11: Restore Logical Dumps The exact database creation and restoration process depends on the Zimbra release. In the classic Zimbra recovery sequence, the central: zimbra database is restored before the: mboxgroup* databases because mailbox-group databases depend on metadata stored in the main Zimbra database. A restoration pattern can resemble: mysql zimbra \ < /recovery/mysql-dumps/zimbra.sql Then individual mailbox-group databases: mysql mboxgroup1 \ < /recovery/mysql-dumps/mboxgroup1.sql Repeat for each successfully recovered mailbox database. Do not restore damaged dump files without first determining whether they completed correctly. Step 12: Test the Restored Database Before Starting Zimbra Do not immediately run: zmcontrol start after importing the dumps. Test database access first. List the databases: mysql -e "SHOW DATABASES;" Check tables: mysql zimbra -e "SHOW TABLES;" Run a small query: mysql zimbra \ -e "SELECT id, account_id FROM mailbox LIMIT 5;" The exact columns available depend on your Zimbra version, so inspect schema first if necessary. Review the MySQL error log: tail -100 /opt/zimbra/log/mysql_error.log Check for: InnoDB errors page corruption table errors authentication failures startup failures Only when the database behaves normally should you begin starting dependent Zimbra services. Step 13: Start Zimbra and Watch the Logs Start services: su - zimbra zmcontrol start Then: zmcontrol status Watch: tail -f /opt/zimbra/log/mysql_error.log and: tail -f /opt/zimbra/log/mailbox.log Zimbra’s crash-recovery procedure specifically recommends checking both the MySQL error log and mailbox log after recovery. Do not judge recovery success only by: mysql Running or: mailbox Running Test actual mailbox functions. Step 14: Validate Zimbra Functionality After services start, test: administrator login user login mailbox listing message access message delivery SMTP reception IMAP access if used mailbox searches sending mail receiving mail mailbox creation if appropriate database error logs mailbox logs Also check whether specific recovered mailbox groups generate errors. The recovery may be partially successful even if one damaged mailbox or mailbox group remains problematic. The Difference Between Backup and Salvage There are two very different situations. Normal Restore You have: known-good backup and restore it. This is the preferred situation. Database Salvage You have: damaged live database and attempt to extract whatever remains readable. innodb_force_recovery belongs mainly to the second category. It is not a replacement for backups. MySQL’s backup documentation emphasizes having an established backup and recovery strategy rather than depending on database-file salvage after failure. A Better Backup Strategy for Zimbra A recovery incident should lead to a review of the backup process. At minimum, your strategy should answer: What data is backed up? How often? Where is the backup stored? Is the backup on a different system? How many restore points exist? Are backups immutable or protected? When was the last restore test? How long does a complete restore take? For the database specifically, you want at least one recovery method that does not depend on the damaged production data directory. Possible mechanisms depend heavily on your Zimbra edition and release. These may include: supported Zimbra backup features storage-level snapshots VM snapshots coordinated with application state filesystem snapshots logical database exports tested disaster-recovery copies The most important characteristic of a backup is not that it exists. It is that you have successfully restored it. Store Recovery Dumps Outside /tmp Zimbra’s older recovery documentation frequently uses: /tmp/mysql.sql for dump storage. The same documentation warns that some operating systems clear /tmp during reboot. For modern recovery work, I prefer a dedicated directory on persistent storage: /recovery/mysql-dumps or mounted recovery storage such as: /mnt/recovery/zimbra/ This reduces the risk of losing dumps due to: reboot tmp cleanup capacity limits accidental cleanup Keep the Original Damaged Database Even after a successful restoration, do not immediately delete the original damaged files. Keep them until: mailboxes have been validated required messages are accessible restored database integrity has been checked management has accepted the recovery retention requirements have been satisfied security investigation is complete The corrupted copy may still contain data that was not successfully dumped during the first recovery attempt. What If Even Level 6 Does Not Start? If: innodb_force_recovery = 6 still cannot start the database, forced recovery has reached its practical limit. Do not invent: innodb_force_recovery = 7 There is no supported level 7. The valid forced-recovery values are: 0 through 6 MySQL’s documentation states that if recovery still fails, recovery from a backup may be required. Zimbra also documents scenarios where a completely unusable database must be replaced with a newly initialized database and restored from available backups. At that stage, your realistic options are: restore from backup restore from snapshot rebuild from Zimbra backup specialist database salvage recover individual readable table data What We Actually Did During our Zimbra recovery, we confirmed the following: The database did not start normally. We attempted InnoDB forced recovery. Lower recovery levels did not provide a usable database. Recovery level 6 eventually allowed the database server to start. SHOW DATABASES returned database information. The Zimbra databases were still visible. Logical dump attempts encountered failures. Mailboxd was not operating normally during the broader recovery. We later ran Zimbra database initialization procedures. Database credentials became another issue during that stage of recovery. Those are observations from our incident. What This Article Adds as Recommended Practice The following are recommendations based on current MySQL guidance and what we would do more systematically in another recovery: Create an untouched physical copy before forced recovery. Perform high-level recovery tests on a separate copy where possible. Increase recovery levels sequentially. Keep levels 4 through 6 away from normal production use. Store dumps on persistent recovery storage. Dump databases individually. Record dump exit codes and hashes. Keep a recovery worksheet showing which databases succeeded. Rebuild a clean database instead of operating from forced recovery. Validate restored databases before starting Zimbra services. Keep the original damaged database until recovery is fully accepted. These should not be confused with steps we necessarily executed in exactly this order during the original incident. A Practical Recovery Flow A safer overall workflow looks like this: MySQL fails to start | v Inspect mysql_error.log | v Check disk and filesystem | v Stop Zimbra/database | v Create physical backup | v Try innodb_force_recovery=1 | +---- starts ---> attempt dumps | v Try 2 | +---- starts ---> attempt dumps | v Try 3 | +---- starts ---> attempt dumps | v HIGH-RISK AREA | v 4 -> 5 -> 6 only if necessary | v Extract everything readable | v Preserve original damaged DB | v Build clean database | v Remove innodb_force_recovery | v Import recovered dumps | v Validate database | v Start Zimbra | v Monitor logs and test mailboxes The important point is the direction of travel. The goal is always to move: damaged database toward: clean database reconstructed from recovered data not to keep the damaged database alive indefinitely. Commands Worth Keeping in Your Recovery Notes Check the server: df -h df -i findmnt Check Zimbra: su - zimbra zmcontrol status Check database errors: tail -100 /opt/zimbra/log/mysql_error.log Preserve database data before destructive work: cp -a /opt/zimbra/db/data \ /recovery/zimbra-db-data-original Check the forced recovery setting: grep -n 'innodb_force_recovery' \ /opt/zimbra/conf/my.cnf Load Zimbra’s database environment on applicable releases: source ~/bin/zmshutil zmsetvars Check databases: mysql -e "SHOW DATABASES;" Identify the dump binary: command -v mysqldump mysqldump --version Dump one database: mysqldump \ -S "$mysql_socket" \ -u root \ --password="$mysql_root_password" \ zimbra \ > /recovery/mysql-dumps/zimbra.sql Verify the dump: echo $? tail -20 /recovery/mysql-dumps/zimbra.sql sha256sum /recovery/mysql-dumps/zimbra.sql Inspect recovery logs: tail -f /opt/zimbra/log/mysql_error.log tail -f /opt/zimbra/log/mailbox.log These commands are tools for a recovery process. They are not a single script that should be executed automatically. Lessons Learned From Our Recovery The first lesson was that a running MySQL process does not mean a healthy database. At recovery level 6, we could finally inspect databases, but significant corruption still existed. The second lesson was that forced recovery is an extraction tool. Its job is to give you a chance to rescue data. The third lesson was that higher recovery values are not better recovery values. A higher number means that InnoDB is disabling more of its normal safety and consistency behavior. The fourth lesson was that level 6 is a last resort. When redo recovery is disabled, the database should be treated as a salvage source. The fifth lesson was that reinitialization must be planned. Running zmmyinit changes the database environment. If configuration and credentials are not tracked carefully, you can replace one problem with authentication and schema problems. The sixth lesson was that filesystem health matters. Database corruption and startup failures can be symptoms of underlying storage, filesystem, or operating-system problems. Finally, a Zimbra server that was also compromised should not simply be repaired and trusted again. Recover the business data. Then restore that data onto a clean and trusted system. That separates the problem into two manageable goals: recover the mail data and: restore trust in the server They are not the same task. References MySQL 8.4, Forcing InnoDB Recovery Current MySQL documentation explaining innodb_force_recovery, levels 1 through 6, the dangers of levels 4 through 6, read-only behavior, and the intended use of forced recovery for dumping damaged tables. MySQL 8.4: Forcing InnoDB Recovery MySQL 8.4, Troubleshooting Recovery Failures Current MySQL guidance for cases where normal InnoDB crash recovery fails and forced recovery or restoration from backup may be required. MySQL 8.4: Troubleshooting Recovery Failures MySQL 8.4, mysqldump Current MySQL documentation for creating logical database dumps using mysqldump. MySQL 8.4: mysqldump Documentation MySQL 8.4, Backup and Recovery Current MySQL documentation covering backup methods, recovery planning, logical dumps, and recovery strategies. MySQL 8.4: Backup and Recovery Zimbra Tech Center, MySQL Crash Recovery Zimbra’s database crash-recovery procedure covering forced recovery, logical dumps, preservation of /opt/zimbra/db/data, database reconstruction, import order, and post-recovery testing. The article reflects classic Zimbra database layouts, so verify paths and commands against your installed Zimbra release before using them. Zimbra: MySQL Crash Recovery Zimbra Tech Center, Troubleshooting Performance Issues Documents Zimbra logs including mysql_error.log, which contains database health information and MySQL errors. Zimbra: Troubleshooting Performance Issues

[object Object]