On my Virtualmin system, the hostname is vps02.indiax.com and when I had installed Virtualmin, I had created a virtual server with the same domain name as the hostname. I had then applied for SSL certificates for the virtual server via Virtualmin → Server Configuration → SSL Certificates
This got SSL enabled on the website at vps02.indiax.com and I was then able to get IMAP and SMTP working via SSL by using the tab captioned “Service Certificates” in the screenshot above and then using the buttons there to apply the same certificate to services such as Postfix (for SMTP) and Dovecot (for IMAP) amongst other services listed on the page.
My clients / mail users on the Virtualmin system who have hosted domains (such as abc.tld) use in their email clients the Virtualmin hostname vps02.indiax.com as incoming server and outgoing server for IMAP and SMTP (and not mail.abc.tld, please note).
I hope this answers your question and gets you started in the right direction. I have used a real (and live, in-production) hostname so that you - and others who are looking for this information - could use tools to dig further and see how things have been set up for the domain to work well as a hostname in Virtualmin in order to replicate the same on your Virtualmin systems.