I often have this backup error :
Creating TAR file of home directory …
… TAR failed! cat: memory exhausted
gzip: stdout: Broken pipe
/bin/tar: -: Wrote only 4096 of 10240 bytes
/bin/tar: Error is not recoverable: exiting now
Is there a way to avoid this ?
How big is your backup data?
I would assume its trying to compress everything in memory and the TAR file output is bigger than the amount of memory (RAM) you actually have available in your machine. You should try testing the compression manually first or trace it.
Try to run a tar manually on the same data and see if it fails or not first. I suspect it could be that the command Virtualmin is using has the -S option or another one that can lead to memory exhaustion problems with directories that contain a lot of files or it can be indeed a problem with low memory on your system where you are just trying to backup something too big.
Its hard to guess without more information.