I’m running a Virtualmin server on a Xeon E5-2630 processor (64Go of RAM, more than enough storage space).
I have a schedule to do a full weekly backup, and one incremental the other days. Backups are transferred to an S3 compatible storage.
Full backups are around 7/8To on average.
If I run backups process without encryption, it takes one day and a half to do the full backup and transfer it outside which is fine. During that process, the processor reaches 30 in term of load but it has not really any impact on performance of machine.
If I run backups with encryption, it takes 5 days minimum and processor is just 5% more busy than normal during the process.
Clearly the encryption process is terrible. Is it due at the shitty intel processor that don’t handle well encryption algorythms ? Is there something that can be done in Linux to improve that ? or a way to run GPG in a lot more efficient way than it is by default ?
interest only - most of my VM are on Linode “computes” of varying sizes and global locations. They offer encryption and storage I know no more but have been wondering if this might be a step in that direction
you need to check other encrypted backup solutions that offer extra feautures (eg deduplication) and are much faster. like borgbackup, restic, etc.
can’t do that from virtualmin backups (iirc), but you can disable virtualmin backups, and instead use some system solution that would be much faster and using less resources.
Thanks for suggestion but it’s a total no go for me as I don’t want to loose the efficiency of Virtualmin backup system that always backup everything properly and easily
@Jamie, there is an issue with our backup_encryption_command because, by default, GPG compresses data before encrypting it in a single-threaded process. We are essentially doing this twice, so we should fix it by adding the --compress-algo none flag.
Also, I don’t see why we use the --sign flag, which can be expensive since we never call --verify?
@vincen Find the bkeys-lib.pl file, replace the following function, restart Webmin, and try again:
sub backup_encryption_command
{
local ($key) = @_;
&foreign_require("webmin");
local @uinfo = getpwnam("root");
$ENV{'HOME'} = $uinfo[7]; # In case HOME was reset by another module
return $webmin::gpgpath.
(&gpg_supports_no_tty() ? " --no-tty" : "")." --encrypt".
" --recipient ".quotemeta($key->{'key'}).
" --default-key ".quotemeta($key->{'key'}).
" --always-trust".
" --cipher-algo AES256".
" --compress-algo none";
}
No, I don’t think we need it, not really… because even without the --sign flag, we still get both confidentiality and integrity (SEIPD, i.e., tag 18 in all modern GPG versions).
… half an hour later: ah, we do check for exit codes when decrypting… and, even though using the --sign flag adds a 30-40% overhead and increases the backup time by many many hours for large backups, it is useful for authenticity, not integrity, now I get it—we need to make sure a user cannot sign an altered backup with an admin’s public key, and an admin later restores a malicious backup. Makes sense. I have re-added the --sign flag back.
Yet, I still think it isn’t the most optimal approach as we already don’t allow restoring backups outside the backup logs for the domain owner. This means we can simply check for sha256sum, which takes seconds to minutes to create and store in our backup log metadata, instead of taking extra hours or days on large backups when using --sign.
That would only work when restoring on the same system though.
I suppose we could add an option to only encrypt but not signed backups, for systems where performance matters but the admin doesn’t care about backups being tampered with.
If you’re the only user on the system, then yes. Otherwise, use that patch and add --sign next to the --encrypt flag, so the whole function looks like:
sub backup_encryption_command
{
local ($key) = @_;
&foreign_require("webmin");
local @uinfo = getpwnam("root");
$ENV{'HOME'} = $uinfo[7]; # In case HOME was reset by another module
return $webmin::gpgpath.
(&gpg_supports_no_tty() ? " --no-tty" : "")." --sign --encrypt".
" --recipient ".quotemeta($key->{'key'}).
" --default-key ".quotemeta($key->{'key'}).
" --always-trust".
" --cipher-algo AES256".
" --compress-algo none";
}
Sorry for the late but upgraded to 2.610 but see no changes in Backup configuration page. Am I missing something obvious ? Regarding the format of the archive is there one more efficient than others in term of time ? so far I’m on tar.
Thanks and best for christmas time
