Published on

My Hermes backup said DONE, but a tar exclude dropped every database

My Hermes backup said DONE, but a tar exclude dropped every database
Authors
On this page

The script printed DONE. Both uploads finished. Both files were 346 MB. Everything looked perfect.

Then I restored one into a temp directory, and db/ was empty.

I'd backed up my Hermes setup that night: config, skills, cron jobs, profiles, and the SQLite files that hold session history. One archive went to my homelab over SSH, and a copy without secrets went to Google Drive.

The trap in the exclude list

The script took consistent snapshots with sqlite3 .backup, then built the archive with a long list of excludes. One of them was --exclude='*.db'. It was there to skip the live database files and their -wal and -shm companions, which are unsafe to copy while the agent is running.

But the snapshots were named state.db, kanban.db and so on. The exclude matched them too.

tar skipped them without a warning. The archive was valid, just missing the part I cared about most. state.db alone is 985 MB.

Nothing in the script's output could have told me. The exit code was 0 and the file sizes looked reasonable.

The exclude pattern *.db matched both the live databases and the snapshots, so the 346 MB archive had an empty db folder; renaming snapshots to .snap gave a 670 MB archive with the databases in it.

The fix

Name the snapshots so the exclude can't match them:

sqlite3 "$H/$db" ".backup '$W/db/$db.snap'"

The new archive is 670 MB, and the databases are in it.

Check the restore

I extracted the new archive into a scratch directory on the homelab and ran SQLite's quick_check on each restored file. Four small ones passed.

The 985 MB state.db timed out twice on the homelab, which was busy with other work. A check that times out has told you nothing, so I split the question in two:

  • A SHA-256 of the restored file matched the snapshot I took, byte for byte.
  • quick_check on the snapshot itself, on a machine with a quiet disk, came back ok with 3,946 sessions and 203,548 messages.

Together that covers it: the file I backed up is a valid database, and the file I restored is the same file.

The lesson

Here's the habit I'm taking from this:

  1. List the archive and grep for the files you can't lose, before trusting it.
  2. Restore into a scratch directory, never over the live one.
  3. Open the restored database and count rows in a table you know.
  4. Check permissions. My first archives were world-readable and contained auth tokens.

A backup you haven't restored is a guess. Mine looked fine by every measure except the one that mattered.