I think you should create a A record only on your Virtualmin server and forget the Google website. Then you can create a CNAME for www that will point the real ggole name of your webserver
example.com. IN A 34.12...
mail.example.com. IN A 34.12...
example.com. IN MX 5 mail.example.com.
www.example.com. IN CNAME yourstaticsite.google.realhostname.
If you want example.com to serve the same site, you could make a website redirect to www in Virtualmin. In that case you need to have a webserver for this domain in your Virtualmin server but it will only manage the redirection.
Another solution is to keep all DNS info on the Google side and add there information about your new mailserver
example.com. IN MX 5 my.virtualmin.host.
This should allow you to manage mail in your virtualmin server.