So, um…dumb question, but are you absolutely sure you’re connecting to your Virtualmin server? i.e. is DNS right (or are you hitting it with the IP address)? Even without STARTTLS working, your SMTP conversation over telnet above looks like it’s missing a bunch of stuff that a Virtualmin+Postfix system would say. Makes me think you’re talking to some other server or a weird proxy at your ISP or something.