today I tested the migration of a 11gb Virtualmin site from an older server to a newer one, and was my first attempt to use Transfer Virtual Server. As many will recall, internally that function uses the VM backup/restore functions.
I am proposing an enhancement, or a new variation, on both this item and the more general Backup function, to directly pipe the TAR/GZ output to the remote server.
Presently, as my site is 11gb, I was able to observe how a temporary file was created on the older server FIRST, then it appeared that file was moved to the new server and then deleted from the /tmp/.webmin/22825-/ temporary directory on the original server — obviously I needed to have an extra 11gb of space available (well not actually that much, see my comments below on GZ compression).
I did a crude manual test of my idea:
tar zcf - \. | ssh vmin-backups@vhost55.wvnet.edu "cat > backup.tar.gz"
and after a prompt for the remote user vmin-backups password, the TAR function did pipe directly to the target system and did not use any space on the original system.
If this variation is already present in Virtualmin, my apologies for missing it, and would love to have pointers to the correct settings for its use.
a general observation – the tar.gz size will vary greatly depending on the input … in my case the 11gb footprint on the original server only compresses down to 9.5gb – of course hindsight is always clearer so in my case I could have saved some time by not using GZ at the “expense” of the extra 1.5gb of space it would have consumed. Naturally I would have not known that until AFTER I did the first backup/transfer attempt
There isn’t a built-in “pipe over SSH” mode in Virtualmin today. Nor yet I’m not sure if it would be hard to implement or useful to many users. @Jamie, what are your thoughts?
Though, without looking deeply into the problem, a quick solution I would try is to point Webmin’s temp directory to a remote mount on the new server, create a one-time backup for download, and then find it on the remote new server to restore using as a local file.
Service providers force us to opt for the next bigger plan because they deliberately give us less disk space. This feature of piped use of workspace on a remote sever for backups could help many of us Virtualmin admins save money by letting us use the lower priced plan for our Virtualmin systems because we will then not have to worry about keeping room for large backups. On smaller Virtualmin systems, our savings from this could be as much as 50% per month.
On larger Virtualmin systems (where mail itself could take up about quarter or half TB of space), if the temp space for backups is used on the remote system, we could profit from this by provisioning an additional client or few on the Virtualmin system .
Do you mean to create a remote mount or pipe directly to a remote?
We are talking about migration, right? Most hosting providers charge by the hour, so even getting a more expensive plan for a couple of hours during migration shouldn’t be a problem. This alone isn’t a good reason to justify such a change, I don’t think.
I agree this would be useful to implement, but last time I looked into it, the work needed would be non-trivial. So it’s currently on my long-term TODO list.
