Posted on 2020-04-28.

It's said that if you aren't testing your backups periodically, you don't really have backups. But testing backups is a tedious chore. It needs to be automated as much as possible.

Fully automating it would defeat the purpose. For example, if I have a script that validates the backup every day and emails me when there's an error, and I never get any emails, maybe my backup works - or maybe the scheduler failed to run the script. Having the script email me on success too would be better. But still, if the email just says "no problems!", there's the possibility that the script is buggy and overlooking major issues in the data.¹ Given the high frequency of bugs in software in general, that's a non-neglible risk.

Ideally, the computer should collect all the information I want to see to have reasonable confidence that the backup is working, and present it in a compact form. Two types of information that can help are metrics and thumbnails.

Metrics

I think the most helpful metrics are ones that:

1. Are related to the specific data domain. For example, "number of tasks stored in my Todoist export".
2. Can be intuitively evaluated as plausible or suspicious. If the number of tasks in my Todoist export dropped from dozens to 1, that would be cause for concern.
3. Would fail to calculate if the data is corrupt, or would be unlikely to give a plausible result. If the code to count the number of tasks works by parsing the JSON and finding the number of elements in a field named "items", then the fact that it even runs without error is some evidence that you have a valid backup file.

My tool for managing personal data exports tracks user-defined metrics for each dataset and generates reports comparing them over time, like this:

Metrics for todoist-fullsync:

name   1 days ago  8 days ago
------------------------------
files  1           1
bytes  82363       86661
items  85          87

Metrics for instagram:

name    16 days ago  117 days ago  303 days ago
------------------------------------------------
files   1            1             1
bytes   136144015    131003691     77073582
photos  481          461           253
videos  10           9             5

Thumbnails

When I want to manually check that a backup archive is good, one of the things I do is glance at some of the files in it to make sure there aren't glaring problems. Doing this one at a time is boring. I wrote a tool that rercursively checks files in an archive/directory and builds a report with thumbnails of all the files, so all I have to do is scroll through it.

1. I'm mainly concerned with cases where the backup is an export from some app/service that I don't control, and thus I can't simply do a byte-for-byte check against the original data to verify that the backup matches it.