And one day something terrible happens.

The server which this site was running on (and a few others) had a few problems with the mainboard power module and started shutting down every now and then. It is a mini-ATX system using some kind of proprietary power supply (which was replaced promptly for an identical one – since the problem persists, I can now guarantee it’s related to the board). It was a matter of hours till I found a similar system on eBay: no hard drive, no DVD, no processor and no memory, just the board and the box. Dirt cheap. Which is nice.

I’m still able to power up the system and get it working for an hour or two in a row, but this was the perfect excuse to test my backup strategy – and disaster recovery procedure.

So every two weeks, I do filesystem backups (plain tar.gz files) consisting of:

• user directories (/home/* and /root/* – this includes the ftp site root at /home/ftp)
• website files (/var/www/* and /sites/*)
• webserver configuration (/etc/apache/* and /etc/php5/*)
• ftp server configuration (/etc/proftpd.conf)
• samba configuration (/etc/samba/*)
• mail server configuration (/etc/postfix/*)
• mt-daapd configuration (/etc/mt-daapd.conf – mp3 files come from CDs I own – cof!)
• cacti configuration (/etc/cacti/*)
• crontab configuration (including users crontabs – all in /etc/cron*)
• installed package configuration (the list is somewhat simple, but for reference sake, I save a copy of a dpkg –list output just in case, along with a copy of /etc/apt/sources.list. Should anything fail, it’s just a matter of looking at the diff of installed packages – with a little perl-fu).
• mySQL configuration (/etc/mysql/*)
• mySQL databases dump (mysqldump -ubackup_user -p –all-databases)

These backups usually fit a single DVD-R and there’s always a blank inside the recorder. I rely on a crontab job creating the backup and burning it. When the recorder tray is open, this usually means the backup was successful and all I need to do is label the DVD and insert a new one.

Then there are those things where running a bimonthly backup won’t do you any good if the information keeps changing and you just can’t recover if you’re stuck to a 20 something day old backup. MySQL is a pretty good example. User home directories are too, but most of their contents are usually found scattered in a number of places.
MySQL databases are usually very small or could achieve small sizes if compressed. And it has been proved that the best way to keep backups at hand is… away from the server.

I found an ideal way to achieve this: three lines of shell script which create a daily database dump, compress it, encrypt it, compress it and email it to a Gmail account.

mysqldump -u [user] -p [password] --all-databases > mysqldump.sql
gpg -r [myemail] -q -t --yes --trust-model always --output mysqldump.sql.gpg --encrypt mysqldump.sql
date | mutt -s "MySQL backup" -a mysqldump.sql.pgp [gmail_account]

There was a spare Dell laptop around with a few keyboard and touchpad problems which was being kept aside for dog knows what. It took me about 30 minutes to slam an Ubuntu live+install CD and install a base system, get the last DVD backup, tar xvfz back the whole relevant files, install a few packages (actually, perform an apt-get dist-upgrade), get the latest MySQL backup from Gmail and put the system back on. There was some downtime meanwhile, but I had to leave home several times during the process (and I do have a life).

Most important question here: do you backup? do you have a backup plan? a disaster recovery plan? anything?