Published on

Did you back up your Hermes? Mine "succeeded" with no databases in it

backupsqlitehermesdevops
Authors
On this page

I backed up my Hermes setup tonight. Config, skills, cron jobs, profiles, and the SQLite files that hold session history. One archive went to my homelab over SSH, and a copy without secrets went to Google Drive.

The script printed DONE. Both uploads finished. Both files were 346 MB.

Then I restored one into a temp directory, and db/ was empty.

What went wrong

The script took consistent snapshots with sqlite3 .backup, then built the archive with a long list of excludes. One of them was --exclude='*.db', there to skip live database files and their -wal and -shm companions, which are unsafe to copy while the agent is running.

The snapshots were named state.db, kanban.db and so on. The exclude matched them too. tar skipped them without a warning, and the archive was valid, just missing the part I cared about most. state.db alone is 985 MB.

Nothing in the script's output could have told me. The exit code was 0 and the file sizes looked reasonable.

The fix

Name the snapshots so the exclude can't match them:

sqlite3 "$H/$db" ".backup '$W/db/$db.snap'"

The new archive is 670 MB, and the databases are in it.

Check the restore

I extracted the new archive into a scratch directory on the homelab and ran SQLite's quick_check on each restored file. Four small ones passed.

The 985 MB state.db timed out twice on the homelab, which was busy with other work, and a check that times out has told you nothing. So I split the question in two. A SHA-256 of the restored file matched the snapshot I took, byte for byte. Then quick_check on the snapshot itself, on a machine with a quiet disk, came back ok with 3,946 sessions and 203,548 messages.

Together that covers it: the file I backed up is a valid database, and the file I restored is the same file.

The habit I'm taking from this:

  1. List the archive and grep for the files you can't lose, before trusting it.
  2. Restore into a scratch directory, never over the live one.
  3. Open the restored database and count rows in a table you know.
  4. Check permissions. My first archives were world-readable and contained auth tokens.

A backup you haven't restored is a guess. Mine looked fine by every measure except the one that mattered.

Discuss on X