A couple of years ago @Joe mentioned creating a Golden Image of the OS. This seems a tantalizing idea. Joe mentions a few directories but I’m left wanting a comprelhensive techique or tool. I was able to find some descriptions on how one might do this with Windows but not much about Linux (Ubuntu).
Does anyone use this Golden Image idea? How do you accomplish it? Do you store the Golden Image on a separate server? How do you restore the Golden Image to a fresh server?
The most popular open source tool for that purpose for bare metal hardware is probably Clonezilla. https://clonezilla.org/
There are other ways to do it. I’ve often just used dd in the past, though things have gotten weirder with UEFI booting and partitioning and volumes and such.
These days (at my day job I maintain a large fleet of devices, but not web servers and they have a somewhat more complicated bringup process as they’re tied to hardware that has to be calibrated to the specific devices attached), I prefer to make a custom installer. I was using this tool, with some customizations: GitHub - linuxrrze/ubuntu-24.04-autoinstall-generator: Generate a fully-automated Ubuntu ISO for unattended installations.
But, Ubuntu has, once again, broken bundling custom packages into the image and the newer autoinstall features in Ubuntu suck. So, I’m not really happy with my options. I don’t consider Clonezilla a suitable option (though it is for many people), as it requires you to install and configure the OS, and then make a clone of it, all a manual process, subject to user error. Also time-consuming. I need something I can check into git, so I can track changes, and never rely on a human doing the work. You could write an postinstall script that you run after OS installation that removes most of the variables, but that’s still pretty clunky, IMHO. But, if you only need to do it every few months or whatever, it’s no big deal.
Note that cloud-init also allows customizations at first boot time (and subsequent boots), so that may also be a suitable option for getting a fresh OS virtual machine image into a suitable state.
For virtual machines and containers, I’ve used Hashicorp Packer. That is, IMHO, the most reasonable way to do things (works from a config file, can be checked into git, reproducible without human error), but it doesn’t support making a bootable installer ISO for bare metal, which is a requirement for my use case (and maybe not yours, you didn’t mention if you’re dealing with hardware or virtual machines, the latter is much simpler to contend with and there are better tools for the job).
And, to be clear, the idea of this image is to standardize what your base installation looks like, reducing variables. The goal is to make it so you only need to backup the user data, and everything else is the same across all systems. Smaller backups, quicker disaster recovery, and more predictable systems. When you have a problem it will be the same problem across all servers with the same solution, when updates are needed for security or bugfixes it’ll be the same update at the same time across all systems, and even when you’re tired and burned out you’re more likely to know where everything is and how everything works because it’s the same across all systems.
Reducing variables reduces human errors and reduces surface area for security issues. For an individual administrator or a small team, you probably can’t maintain five different distros and versions without making serious security mistakes, and you probably can’t remember where everything is. One of our biggest problems supporting so many OSes is that we just can’t possibly remember where everything is and how everything works, there are too many variables (at any given time we support three distros in two or three versions each, with a few subdistros that have minor differences on the EL side, as well). All of us have been doing this for decades on all of these distros, and we still have to look stuff up all the time. A smart person reduces the complexity they have to manage.
And, even if you have to support more than one OS, having a standard installation of it, at least, reduces the variables on each of those OSes. So, you have one Ubuntu and one EL, rather than a dozen of each.
Thank you for your thoughtful reply, Joe.
Sorry I omitted context. I’m on a VPS, so bare-metal from my point of view, and have just one server. I’d like to backup remotely regularly and keep versions.
VirtualMin’s backup is probably the best server backup system I’ve encountered. Really awesome and has saved my neck a couple of times.
I guess what I’d like is something like that but for the OS configuration. Now that you’ve clarified your Golden Image concept, I think it’s not applicable to me, but still interesting.
Thanks for mentioning Clonezilla. It looks powerful but with involved setup to save an image to, say, S3. And to recover, the VPS provider has to support “booting VPS from a recovery ISO or custom ISO image.
Again, I’m thinking more along the lines of VirtualMin’s backup where it just backs up its config files rather than all of its packages.
TimeShift looks promising but lacks an interface to remote storage.
I welcome your further thoughts, Joe.
@SteveSteinitz I am using clone zilla - this works perfect but also I use virtualmin backups as well as I am a bit paranoid (all my git repos and websites are selfhosted on ssds and as we know there is limited life span) however with one 8tb spinning drive and rsync - its all worked and proved - 10 years by now. Also you can run rsync to your other online locations.
Thanks for your endorsement of Clonezilla, @unborn, and for sharing your seriousness about backup. Me too.
The thing that keeps me from Clonezilla is lack of native support for S3. That’s another area where VirtualMin’s Backup & Restore shines: they did the hard work of supporting various backup targets.
I have snapshot on my vps that has the same OS as my current server plus Virtualmin installed and webmin configs I have use on my current server as well. Only thing missing are the virtual servers. I install it every so often to update the packages (do any updated configs from webmin) and then do another snapshot. I use it for testing and (it hasn’t happen yet) disaster recovery then I can restore my last usable backup to get it back up and running, a very quick process.
With vultr I use a Reserved IP on my current server that I can then attached to the recovered server so no need to update the DNS settings.
I guess thats a “goldern image”
Hi @stefan1959 – thanks for your reply. It sounds like you really know what you’re doing.
So many questions:
I’ll stop there… 
One question at a time
Vultr does, I would of thought most do, I would shutdown the server before doing it.
I do have Virtualmin included as i thought I said.
I should said restore it as a new server. When I create a new sever it allows me to use the snapshot to restore.
I do it to keep the snapshot have the latest packages, so I don’t need to update much.
Snapshots are cheap. A live system is $30-40 per month, compare to $1 dollars per month.
sometimes called images … and other monikers but effectively the same thing
prices can vary quite a lot (but often come with obvious conditions) like being in the same data centre/region - nearly always cheaper than backups - which I find are better managed locally.
Thanks for your replies @stefan1959. I think I understand your system. Seems pretty cool – and comprehensive. The missing piece in my understanding was that you restore your snapshot to another server for playing around.
Thanks for sharing.
That’s a nice separation @Stegan: images/snapshots managed by the vendor; backups managed by the user. Makes sense.
