domain.tld points to IP address of web optimized Virtualmin
www.domain.tld points to IP address of web optimized Virtualmin
mail.domain.tld points to IP address of mail optimized Virtualmin
And this is the most important thing to wrap yoor head around: on mail optimised Virtualmin as well as on web optimised Virtualmin, create a virtual server for domain.tld (yes! Same domain name on both Virtualmins).
Then everything will work perfectly. When creating a virtual server on mail optimised Virtualmin, uncheck the boxes for web services and vice versa.
Oh, if you manually configure the DNS record of the web optimised Virtualmin to point to the MX of the mail optimised Virtualmin, you will attain nirvana forthwith.