Sure, you can do that.
You would just need to make sure your private key was in the .ssh dir in the home directory of whatever user is sending the backups to the backup server.
So, if you’re having Virtualmin send the backups to your server, you’d need to make sure the private key was in the root users home directory.
To test that – log into your server as root, and make sure you’re then able to SSH into your backup server without a key.
-Eric