2d ago

Backups: Am I doing this right?

I'm in the process of setting up backups for my home server, and I feel like I'm swimming upstream. It makes me think I'm just taking the wrong approach.

I'm on a shoestring budget at the moment, so I won't really be able to implement a 3-2-1 strategy just yet. I figure the most bang for my buck right now is to set up off-site backups to a cloud provider. I first decided to do a full-system backup in the hopes I could just restore it and immediately be up and running again. I've seen a lot of comments saying this is the wrong approach, although I haven't seen anyone outline exactly why.

I then decided I would instead cherry-pick my backup locations instead. Then I started reading about backing up databases, and it seems you can't just back up the data directory (or file in the case of SQLite) and call it good. You need to dump them first and backup the dumps.

So, now I'm configuring a docker-db-backup container to back each one of them up, finding database containers and SQLite databases and configuring a backup job for each one. Then, I hope to drop all of those dumps into a single location and back that up to the cloud. This means that, if I need to rebuild, I'll have to restore the containers' volumes, restore the backups, bring up new containers, and then restore each container's backup into the new database. It's pretty far from my initial hope of being able to restore all the files and start using the newly restored system.

Am I going down the wrong path here, or is this just the best way to do it?

23 comments

I figure the most bang for my buck right now is to set up off-site backups to a cloud provider.
Check out Borgbase, it's very cheap and it's an actual backup solution, so it offers some features you won't get from Google drive or whatever you were considering using e.g. deduplication, recover data at different points in time and have the data be encrypted so there's no way for them to access it.
I first decided to do a full-system backup in the hopes I could just restore it and immediately be up and running again. I've seen a lot of comments saying this is the wrong approach, although I haven't seen anyone outline exactly why.
The vast majority of your system is the same as it would be if you install fresh, so you're wasting backup space in storing data you can easily recover in other ways. You would only need to store changes you made to the system, e.g. which packages are installed (just get the list of packages then run an install on them, no need to backup the binaries) and which config changes you made. Plus if you're using docker for services (which you really should) the services too are very easy to recover. So if you backup the compose file and config folders for those services (and obviously the data itself) you can get back in almost no time. Also even if you do a full system backup you would need to chroot into that system to install a bootloader, so it's not as straightforward as you think (unless your backup is a dd of the disk, which is a bad idea for many other reasons).
I then decided I would instead cherry-pick my backup locations instead. Then I started reading about backing up databases, and it seems you can't just back up the data directory (or file in the case of SQLite) and call it good. You need to dump them first and backup the dumps.
Yes and no. You can backup the file completely, but it's not a good practice. The reason is that if the file gets corrupted you will lose all data, whereas if you dumped the database contents and backed that up is much less likely to corrupt. But in actuality there's no reason why backing up the files themselves shouldn't work (in fact when you launch a docker container it's always an entirely new database pointed to the same data folder)
So, now I'm configuring a docker-db-backup container to back each one of them up, finding database containers and SQLite databases and configuring a backup job for each one. Then, I hope to drop all of those dumps into a single location and back that up to the cloud. This means that, if I need to rebuild, I'll have to restore the containers' volumes, restore the backups, bring up new containers, and then restore each container's backup into the new database. It's pretty far from my initial hope of being able to restore all the files and start using the newly restored system.
Am I going down the wrong path here, or is this just the best way to do it?
That seems like the safest approach. If you're concerned about it being too much work I recommend you write a script to automate the process, or even better an Ansible playbook.

I don't do a full system backup, just data I need. I use restic to backup to B2 storage. As for docker data and databases I have a script that brings down the containers, runs a sqldump (or db type equivalent) into a tar.bz file and that gets backed up before bringing the containers back up. I used to run a Borg backup to a local drive and then rclone that to B2 so I have live data, local data, and remote. But these days I just back up remote and suck up the egress costs if needed.

I figure the most bang for my buck right now is to set up off-site backups to a cloud provider.
If you don't have the budget for on-premises backup, you almost certainly can't afford to restore the cloud backup if anything goes wrong.
Then I started reading about backing up databases
Go read the instructions for your database in particular. They are completely different from each other. Ignore generic instructions.
now I’m configuring a docker-db-backup container
What is perfectly fine. But I'd first look how this interferes with the budget you talked about earlier and if it wouldn't be better to keep things simpler and put the money on data replication.
Either way, if your budget is low, I'd focus a lot on making sure you have the data when you need to restore, and less on streamlining the restore procedure. (That seems to be the direction you are going, so yeah, I'd say it's good.) Just make sure to test the restore procedure once in a while.

Just remember any backup is better than nothing. Even if the backup is done wrong (this includes untested!) odds are you can read it and extract at least some data, it just may take a lot of time. Backups that are done right just mean that when (not if!) your computers break you are quickly back up and running.
There are several reasons to backup data only and not the full system. First you may be unable to find a computer exactly/enough like the one that broke, and so the old system backup won't even run. Second, even if you can find an identical enough system, do you want to, or maybe it is time to upgrade anyway - there are pros and cons of arm (raspberry pi) vs x86 servers (there are other obscure options you might want but those are the main ones), and you may want to switch anyway since you have. Third, odds are some of the services need to be upgraded and so you may as well use this forced computer time to apply the upgrade. Last, you may change how many servers you have, should you split services to different computers, or maybe consolidate the services on the system that died to some other server you already have.
The only advantage of a full system backup is when they work they are the fastest way to get going again.

Some clarifications :
The 3 2 1 rule applies only for the data. Not the backup, in my case I have the real/live data, then a daily snapshot in the same volume /pool and a external off-site backup
For the databases you got misleading information, you can copy the files as they are BUT you need to be sure that the database is not running (you could copy the data and n the middle of a transaction leading to some future problems) AND when you restore it, you need to restore to the exact same database version.
Using the export functionality you ensure that the data is not corrupted (the database ensure the correctness of the data) and the possibility to restore to another database version.
My suggestion, use borgbackup or any other backup system with de duplication, stop the docker to ensure no corruptions and save everything. Having a downtime of a minute every day is usually not a deal breaker for home users
- Much easier than what I was trying to do. Thank you!

In regards to full system backups, there's no real need to back up the OS itself. Canonical will give you a clean Ubuntu install if you ask then nice enough, after all. Personally, the risk of having to spend an afternoon reconfiguring my system isn't that big a deal compared to the storage and time needed to back up an entire image.
I know systems generate a lot of "cruft" in terms of instslled programs and tweaked configurations over time which can be hard to keep track of and remember. But imo that should be avoided at all costs because it leads to compatibility and security issues.
For backing up databases, there's scripts like automysqlbackup and pg_dump which will export a database to an sql file which can be easily backed up without worrying about copying a broken file.
I actually recently set up borgmatic earlier today and I'd recommend it except for the fact that you seem to be using Docker, and I'm not sure how best to backup containers.
- I usually also backup the etc directory so if I had an issue I would at least have the config files from the old setup. This has already saved me a few times when I have really messed up configuration files.
- Yeah, I keep everything as simple as possible. Everything is containerized, and all the configs live in one directory and they store their data on my RAID. I don't need to go track down configs across the system, and adding a new service doesn't require any backup config so no risk of forgetting something.
  Docker is simple. You map directories in the container to directories on your host, so you put the important data where it'll get backed up and the less important data (e.g. logs) where it won't.

I first decided to do a full-system backup in the hopes I could just restore it and immediately be up and running again. I’ve seen a lot of comments saying this is the wrong approach, although I haven’t seen anyone outline exactly why.
The main downside is the size of the backup, since you're backing up the entire OS with cache files, log files, other junk, and so on. Otherwise it's fine.
Then I started reading about backing up databases, and it seems you can’t just back up the data directory (or file in the case of SQLite) and call it good. You need to dump them first and backup the dumps.
You can back up the data directory, that works fine for selfhosted stuff generally because we don't have tons of users writing to the database constantly.
If you back up /var/lib/docker/volumes, your docker-compose.yaml files for each service, and any other bind mount directories you use in the compose files, then restoring is as easy as pulling all the data back to the new system and running docker compose up -d on each service.
I highly recommend Backrest which uses Restic for backups, very easy to configure and supports Healthchecks integration for easy notifications if backups fail for some reason.
- Second rest and backrest!
- If that's the main downside to a full-system backup, I might go ahead and try it. I'll check out Backrest too. Looks great!
  
  Yeah there are plenty of advantages of a full system backup, like not having to worry that you're backing up all the specific directories needed, and super easy restores since the whole bootable system is saved.
  Personally I do both, I have a full system backup to local storage using Proxmox Backup Server, and then to Backblaze B2 using Restic I backup only the really important stuff.

Some things you should determine first:
Total amount of data you will be backing up
Frequency of backups
Number of copies to keep
Plug these numbers into cost calculators for whatever cloud service you're hoping to use, because this is honestly not going to be the cheapest route to store off-site if there are ingress charges like with S3.
I know Cloudflare's R2 service doesn't charge for ingress or egress (for now), but you might be able to find something even cheaper if you're only backing up certain types of data that can be easily compressed.
I'd also investigate cheap ways to maybe just store an off-site drive with your data: office/work, family house, friends house...etc. Storage devices are way cheaper than monthly cloud costs.
- Had considered a device with some storage at a family member's house, but then I'd have to maintain that, fix it if it goes down, replace it if it breaks, etc. I think I'd prefer a small monthly fee for now, even if it may work out more expensive in the long run.
  Good call on the cost calculation. I'll take another look at those factors...
  
  There's also the option of just leaving an offline disk at someone's and visiting them regularly to update the backup.
  Having an entirely offline copy also protects you/mitigates against a few additional hazards.

If you're using docker (like your DBs run in docker), then I think you're overthinking it personally. Just back up the volume that the container uses, then you can just plop it back and it will carry on carefree.
I usually did a simple tar cvf /path/to/compressed.tar.gz /my/docker/volume for each of my volumes, then backed up the tar. Kept symlinks and everything nice and happy. If you do that for each of your volumes, and you also have your config for running your containers like a docker-compose, congrats that's all you need.
I don't know who said you can't just back up the volume, to me that's kind of the point of docker. It's extreme portability.
- OK, cool. That’s helpful. Thank you!
  I know in general you can just grab a docker volume and then point at it with a new container later, but I was under the impression that backing up a database in particular in this way could leave you with a database in a bad state after restoring. Fingers crossed that was just bad info. 😅
  
  In theory the database can end up in an invalid state when you leave the database container running. What I do for most containers is to temporarily stop them, backup the Docker volume and then restart the container.

23 comments