As my little monitoring project has outgrown what I at first thought it would be, some time ago I tightened some screws on its backup process -- or, actually, the restore part.
The classic backup dilemma
How do you know if you really have backups of your Zabbix? No matter how often you backup your database or to how many destinations you store your backups, it's all for nothing in case you are not able to restore your backups. Even being able to restore your backups is useless if you are unable to use your software after the restore or if the data is all messed up.
To test all this... well... Zabbix to the rescue! This time, as the topic is very broad, I'm not giving too much technical details, but ideas for you.
"I don't need backups, I have multiple database servers"
This seems to be something you occasionally see on online discussion boards. For me, being able to have a MySQL replica or delayed replication or multi-master setup means that it's comparable to RAID setup on storage: it lessens the risk of something going wrong when the underlying hardware says bye. In case of a master database failing, a replica can take its role, or another master in multi-master setup can continue just fine, or SQL Proxy can stop directing queries to failed server, or whatever is the case for you for automatic remediation.
However, all this can be rendered useless by us humans, let that be a bad error from admin or a visit from a miscreant. Separate physical backups are very much needed to make sure you can return to well-known good status even if something happens to your data.
Taking the physical backups
Depending on your database software and database size, the way you should take your backups can and will vary. For me at home, just the good old mysqldump is good enough as my What's up, home? database is not very big.
For bigger instances, something like Percona xtrabackup or mariabackup would be better options if you are using any MySQL version.
Restoring and testing the backups
Some time ago I did setup a separate virtual machine running on my Linux laptop, purely dedicated for testing my Zabbix backups. How does it work? The flow for my local Jenkins job is something like this:
- Every night 3am Jenkins triggers a script on my Raspberry Pi and backups my What's up, home? database with mysqldump, piping it to gzip so I have compressed archives. The filename contains the current date in format YYYY-MM-DD -- that way I have a backlog of backups and also I can be sure that my scripts are working with a fresh copy of the database, and not some old version that's forgotten to disk.
- As I don't want to transfer cleartext backups of a database to any cloud service, next the backups are copied into a Zulucrypt container. Jenkins moves on to this step only if the database backup itself was successful.
- The encrypted container is uploaded to multiple locations, one of them being the dedicated virtual machine.
- Next my Jenkins calls a script on my dedicated backup tester. The scripts opens the Zulucrypt container and imports the database backup, finally closing the container.
- My Zabbix also monitors this dedicated VM, but during the duration of this test, Zabbix puts it to daily maintenance mode so I don't get alerts. The daily maintenance is active for this host only between 3am-5am.
- Finally, it tests the availability of the guinea pig Zabbix through a web scenario. For now, the test is very simple: if my web scenario fails to load Zabbix front page, something is wrong. Maybe I could go for more granular checks, but every time I have seen that Zabbix has had issues connecting to database, it has complained about a total outage between the communication or about some database version difference if a migration has not been OK, so in theory my test should be good enough for home use.
- For actual production use, one could select rows from database itself or do some API calls to check for any values that should be there. One day, I'll do that, too, but even this simple solution gives me some piece of mind about the health of my backups.
Benefits of this kind of setup
I can be sure that whatever happens to my database, I have a known good configuration which I can restore. The effort of manual testing is gone, I'll get to know easily if my database backups suddenly stop working instead of me only thinking they work. And, if things go south, I can return my setup online quickly. Losing at worst a day of history and possible config changes is far better option than losing everything I have done since March 2022 with this project.
Add new comment