Anniversary Refinements to my Backup Routine

[RJM_Web_Design]

It’s now been roughly a year since I’ve been using Virtualmin, and I decided to celebrate by refining my backup routine. I’d been doing full daily backups and copying them over to Backblaze using rclone with 14-day retention. (Back then, Backblaze wasn’t S3-compatible. It is now, but I still prefer rclone.)

The thing is that I have several sites with huge amounts of static image and video files (and I built another over the weekend), so the daily full backups were getting ridiculous. I’d done them at first because I was new to Virtualmin and wanted as much backup as I could get. Today I decided to clean things up.

I started by deleting all the backups locally and remotely. Then I set up weekly full backups and daily incrementals to /backup/weekly and backup/daily respectively (along with the existing /backup/server for the server configuration). I also reduced the retention from 14 days to 7 to save space because image and video files don’t compress especially well.

I added the following as a post-backup script to the weekly and daily backups:

rclone sync -v /backup backblaze:{my-bucket-name} --transfers 8

Previously I’d been using

rclone copy /backup backblaze:{my-bucket-name} --transfers 8

which would be wasteful in this instance because it would transfer the full backup every day the sync command skips unchanged files.

Then I manually ran both the full and incremental backups, and they ran and synced perfectly.

The advantages are less load on the server, less bandwidth used for transfer, less storage used on Backblaze, and seven days of incremental restore points stored both locally and remotely. I’ve never needed more than one on any server I ever managed, so seven should be fine.

Richard

[dimitrist]

after many years fooling around with virtualmin-tar/rsync/rdiff/others, i chose borgbackup and never looked back.
less time, less load, less storage, less bandwidth…

[RJM_Web_Design]

I’ve never tried borgbackup, mainly because I’ve never had a problem using the native backup routine and rclone. But it looks interesting. I’ll look into it. Thanks.

I started using rclone when I wanted to back up cPanel accounts to Backblaze. At the time, they had no native functionality for that; and even when they did implement it, it only allowed for transferring full backups. rclone made it easy to get around that by disabling the native remote transfer altogether and letting rclone transfer the contents of /backup as a separate process. The only times it failed were when Backblaze had some connectivity issues early on. rclone itself never failed.

The only thing to be aware of with rclone sync is that it works both ways; so if there are a ton of existing backups in the remote, they have to be moved or deleted before switching from copy to sync on the local machine. Else it will attempt to copy all the remote backups back to the local machine (and of course fail due to lack of space).

Thanks again for the heads up on borgbackup.

Richard