Setting up a robust mailserver for a shared hosting environment in 2023

Virtualmin
1

Hi all,

I guess this is standard scenario for many people using Virtualmin: we have a Virtualmin machine, a couple of domains hosted on that machine with their own emails. By default Virtualmin does a good job of wiring most things so it basically works, but, nowadays it seems to be really hard to make this shared environment secure and make sure the emails that we send are reaching INBOXes not Junk folders.

Is there a guide, good tutorial that tackles everything that is involved to basically make sure as a sysadmin that emails really work?

Background:
I’ve used Virtualmin for more than 10 years and sent emails through this setup ever since. I currently have an old Debian 8 setup (upgraded OS in the meantime), but I imported into that an older Virtualmin configuration file (as I didn’t want to redo everything, too much work). I host maybe tens of domains for some friends/clients. They usually have WordPress sites. Most domains use email. Had many incidents during the years with hacked websites (unfortunately my friends/clients don’t really know or understand the importance of maintaining a website), but also hacked email passwords so my server sent SPAM and I had to clean/fix/unlist from blacklists lots of times. I also have a second IP and at some point, for really important domains I tried to do do some Postfix sender transport mapping to use this one and avoid really important domains (like personal ones) being affected by negligence of other domain owners. However, this is really really painful and I still don’t have a really robust system. Most of the time people complain that emails sent from my server reaches Junk folders. And no, I wasn’t on blacklists or having obvious configuration issues. I know SPF, DKIM, DMARC, how this works and I configured this in Virtualmin. I lost clients because of these recurring issues. Recently I’ve sent again a personal email to someone from a bank and it told me it found my email in Spam… I’m really tired and frustrated, checking with mail-test the only important think that it complains is about rDNS but that’s setup for my second IP, unfortunately though, being a single Postfix instance using same HELO for both IPs with both having the same rDNS, the name can only be resolved back to one of the IPs so maybe this is again a flag for some providers? Etc. etc. etc.

I also setup at some point for a project of mine, a SendGrid account so that I make sure that the emails sent by my project really get to people’s Inboxes. Guess what. They go to promotions. And no, they are not promotions, are emails like account management. Facepalm.

I really need to do a reliable setup with nowadays standards. Where can I really find useful information? For me, by looking around the internet it feels like there are only small parts of the solution here and there. And it really feels that Virtualmin’s default setup is just to wire things up to make email delivery going but it’s by far of no use if you want a reliable shared hosting environment. Or, should I try reinstalling and reconfiguring lates Virtualmin from scratch? Is it better in providing a good default than my really old Virtualmin installation that has its roots like 10y ago? But that would be a ton of work.

I’m thinking of all kinds of ideas also:

  • Revise the standards: SPF, DKIM, DMARC - is there a way to simply regenerate all configuration here?
  • Cut internet access to websites, so that hacked websites can’t send spam anymore by standard php mail() - how could this be setup? On the other hand, this is not good at all for WordPress auto-update functions
  • Have best scanning in place not only for incoming email but also outgoing and some good throttling so that I avoid that hacked email accounts are used for sending spam
  • Considering also to use some external providers that I can hook globally somehow into the entire system (not per domain)

Do you guys have any ideas? What is your experience with this? How do you make sure that most emails really get where they should? This entire email delivery thing is such a bad frustrating chaos and it seems to me sometimes that the big guys almost purposely leave you out of their Inboxes with different excuses.

2

Really?

There are millions of servers running Virtualmin, cPanel etc. which offer shared hosting but are secure, so I am unable to agree with your assessment that it is hard to make a shared environment secure.

It is quite the opposite, in fact. With the advent of things like php-fpm it is now easier than ever to keep a shared environment secure.

Yes, there are tutorials and the issue of configuring Virtualmin as a mail server has been discussed afresh less than a month ago. Why don’t you scroll up and find the thread?

Learn about Virtualmin’s mail rate limiting feature in order to limit the number of messages a spammer can send via a Virtualmin system when an email account is compromised. This too was recently discussed in the set of topics related to configuring Virtualmin as a mail server.

This issue is unrelated to Virtualmin.

Well, I can confirm that Virtualmin works fine for me and my clients, some of whom have to send mail to about 30,000 subscribers.

If you have the attitude that Virtualmin is no use to run a reliable shared hosting environment then why are we having this conversation at all in Virtualmin’s forums?

I have no comment about your decision to keep using your 10 year old system and jumping to the conclusion that it is Virtualmin’s fault that your system is not secure and your mail does not get delivered. I have absolutely no comment at all.

1 Like
3

You got it wrong. Sorry if I confused you by not being more clear. I’m not claiming Virtualmin in itself is not secure and not even saying that the setup that it does account or server-wise is unsecure. Everything is strictly mail related. And I’m talking from my experience over those 10y+.

Maybe today a brand new Virtualmin setup does things differently out of the box. Can’t know, that’s why I’m asking. But the legacy setup that I have carried for years is just awful to continue managing it like that, so many things to think of and configure to try to make it work.

In the end what I’m trying to find out from people with (more recent) experience:

  • Is it worth it to simply reinstall and reconfigure everything on a new and fresh system? It’s a ton of work especially to migrate tens of domains and make them work, so can’t just do it and test
  • Are there complete guides tackling all the issues in detail? If yes, just point me to them. I’m trusting people with experience to guide me in the right direction, especially mail delivery experts.
4

That is entirely on you, not them. If your server was configured properly, they wouldn’t be able to do that to begin with.

Odd how you blame everybody else. There are over 150,000 people using Virtualmin around the globe. You’re one if the very, very few I’ve ever seen show up and start blaming the app rather than the person using it, which is of course you.

Of all the “ideas” you listed, you forgot one; and it’s probably the single best idea you need to act on:

Hire someone that knows what they’re doing to set up your mail server.

5

Let’s review:

That was end of life two years ago. You talk about lazy “clients”? Really?

And Virtualmin always alerts you to every update it has and it will do it if you tell it to. So the idea that you never even bothered to update Virtualmin at all is very telling indeed.

That aside, yes. It is very easy to upgrade to a new OS. Just use Vitualmin to back up your Virtual servers, install your new Grade A OS, install Virtualmin then restore your virtual servers.

It’s very simple and takes no time at all.

Edit to add: Here are the grade A OS you can choose from:

  • Rocky, Alma, and RHEL 8 and 9 on x86_64
  • Ubuntu 20.04 LTS and 22.04 LTS on i386 and amd64
  • Debian 10 and 11 on i386 and amd64
6

yep, its called google :slight_smile:

1 Like
7

Guys. Again. Not blaming Virtualmin. It’s great software. Probably wrote too much and didn’t choose the words carefully out of frustration. I really look for email delivery experts here that could guide me into what to do to harden the mail handling part of a Virtualmin-driven server. Probably things not directly related to Virtualmin, maybe related more to Postfix, I don’t know. I don’t blame anyone, I just state how wild things can be in a shared hosting environment and you know it. The problem is that even if Virtualmin does great job in providing isolated virtual servers / accounts, when talking about mail, if one account is hacked then everybody suffers due to blacklisting. What can we do to prevent this? Can we SPAM scan every outgoing mail even mail sent via php mail()? Are there good public lists or services that we can use for that? Is it a good idea to restrict internet access to accounts (helping also when hacked accounts may do DDoS attacks)? What can we do to harden security even further in a more hostile environment?

I’m sorry I’m not up to speed with all topics on the forums. But there are thousands and thousands and most of them are about bugs and basic things like SPF, DKIM not topics more advanced. If I’m not seeing them, just link them here if you want to help.

Thanks.

8

You did three times.

Again, it has nothing at all to do with a shared hosting environment.

There are basically two ways you get hacked:

  • You set up a basic server with no advanced password requirements and/or short, silly, easy passwords.
  • You got hit by a pro.

In your case, I can guarantee you it’s the former. When you set up Virtualmin, did you require advanced passwords?

Of course you didn’t.

So some of your clients probably used “password1234” as their password. That’s what got their site hacked.

Did you limit the number of emails any client can send?

Of course you didn’t. There’s a flood control setting I’m sure you never touched at all, which is what allowed the spam to occur to being with after the simple password policy got you hacked.

None of this is rocket science. At all. It’s basic stuff.

9

I already wrote that the clean install was a Debian 8, but OS was upgraded. I keep Virtualmin and OS packages up to date. I’m not complaining about any vulnerabilities or bugs in the software, my questions are about more advanced setups and strategies to deal well with hacked email accounts and websites. Probably I should have written my initial message totally different. Sorry, my mistake.

1 Like
10

Ummm…ahem:

https://www.debian.org/News/2020/20200709

The Debian Long Term Support (LTS) Team hereby announces that Debian 8 jessie support has reached its end-of-life on June 30, 2020, five years after its initial release on April 26, 2015.

Debian will not provide further security updates for Debian 8.

11

Thanks but you are still making assumptions. Accounts get hacked even with complicated passwords, because the passwords may be stolen. Also, I’ve seen too many WordPress sites hacked due to code vulnerabilities. This is what I am talking about.

The email rate limiting I tried years ago. It didn’t work for me, I had to turn that off but excuse me if I can’t remember now the precise details. Maybe it got improved so I should try that again. But this is only a part of the story.

12

OS upgrade means upgrade not update. Currently running Debian 10.

13

You know how I can tell you’re not telling the truth?

You have no idea where to find those settings or you would have mentioned it.

14

Now you’re just making it up as you go along.

Good luck. You’re going to need it. I’m out.

15

I think this was a clear enough statement. The point was the initial start was during Debian 8 era. Maybe that information is important to a reader. But fine, have a good day.

16

Take it easy. No need to argue about it. I’m always fine with suggestions to run a currently maintained distro, as it is absolutely mandatory to even begin to be safe on the internet. But, beyond that recommendation, no need to keep going back and forth about it. (And, besides, very old distros can’t even get updates of Virtualmin packages without jumping through some hoops. There will be more reminders that what they’re doing is a bad idea, when running a long EOL OS.)

1 Like
17

@jazzman,

Okay, let’s try to hit this on the head… Though this will not be my “complete” post as I’ve gotta run in a moment.

Email security is like any security, despite the tone @calport and @Gomez_Adams did raise a few good points on the topic, however let me paraphrase and summarize a few things.

Security (email inclusive) is only as good as the system is setup. By default, Virtualmin does a GREAT job of setting up a variety of good security settings including but not limited to:

  • Firewall
    this restricts access to the server by port and/or service (fail2ban also writes rules to it based on it’s own rules)

  • SASL Authentication
    this offers a nice and secure way of handling password authentication especially for email authentication

  • Fail2Ban
    this offers your system “intrusion detection” and handling of things like password guess attempts by analyzing key log files based on rules and blocking potentially malicious activity.

  • Encrypted Password
    the native system does this by default, but this extra feature means that the passwords won’t be exposed in the GUI (there are pros and cons to this ofcourse)

Once installed, most systems out of the box are pretty good to go.

A few things to consider in order to further “harden” your installation…

  1. make sure passwords are strong – as mentioned earlier a weak password is one of the most common ways people hack into a system.

  2. make sure the OS and scripts used (like WordPress) are kept up to date (especially modules which are often the most vulnerable)

  3. conduct a “pen-test” against your server to check for any weak points.

  4. implement a blocklist based on known hackers and spammers (there are some lists that are maintained dynamically and can be monitored and updated regularly)

  5. monitor your system as often as possible, and perhaps write scripts to identify abnormal behavior (perhaps hire someone for this if you don’t know where to start)

  6. make sure you enable SPF, DKIM, and DMARC against your domains and set things up properly

  7. wash, rince, repeat :slight_smile: (managing an email server can become a full-time job)

Will any of this “guarantee” you are protected? ABSOLUTELY NOT.

The fact is, hackers are getting more and more creative each day, so you need to keep on your toes and watch for new trends, and work out ways to address them as they come. By monitoring how many emails leave your server from a given user, along with other key vitals of your system, you’ll become a better Sys Admin as time goes on.

I’m a full-time Sys Admin and responsible for a few dozen servers on any given day. One of the biggest projects I’ve take on to date includes designing, and managing a large “Email Cluster” for a Mailing List host who delivers nearly a quarter million emails each month. Downtime, and problems are not acceptable and when things go bad (yes they do), fast, swift action is required to get things back online as quickly as possible while learning something from the situation to prevent a repeat offence.

Anyways, gotta head out now for a few hours!

Final note: Does installing a new version of an OS and/or Virtualmin help? Absolutely, as both the OS and Virtualmin will have learned a few new tricks to help you, help yourself!

1 Like
18

Your initial post was complaining that a lot of your outgoing emails end up in recipients Junk or Promotional folders.
I would think that is more an issue about content, not the system that handled the email.

As a suggestion I run all incoming and outgoing email through separate spam/virus filter servers. That allows me to monitor what is going on and deal with problems quickly and easily, and so far I haven’t had any of my IP’s get into blocklists - eg Spamhaus etc. You would need extra IPs to do this.

Good luck.

19

I’m actually doing most of things you’re trying to do. I try to keep everything documented so I can easily find them later. And most of my documentations are public. You may find these helpful,

Virtualmin Setup with LEMP
Virtualmin Post-Installation Configuration and Server Optimization Guide
Configure Emails with Virtualmin
Secure Virtualmin and other services with UFW and Fail2Ban - Auto ban bad actors
Web Application Firewall for Virtualmin & Nginx
Real-time Malware scanning with optional VM blocking with Virtualmin

3 Likes
20

Thank you guys, finally useful information.

@Joe @tpnsolutions: So my hunch was right, re-installing from scratch today (probably on a new server) and migrating virtual servers from old host is a better way to go than simply continuing to do OS and software upgrades (including updates of course) on the old server. I will need to handle somehow the IP changes and slave DNS servers updates but I’ll see how I can do that… This is why I mentioned that the last time I installed from scratch was Debian 8 times and just upgraded since, also used even older Virtualmin Configuration imported. Probably best to reconfigure everything on clean, I assume tons of things changed in the meantime. And just upgrading and keeping old configuration simply can’t cope with it in terms of current recommendations and best practices.

@Randomz: Also though about scanning everything including outgoing email, but how to do it properly? Will a new Virtualmin installation help with that? Are there options? Because I didn’t find any settings or any easy way to just enable outgoing email scanning years ago… I also think that using some public/shared big databases (like pyzor? - I recently heard about that) would be really best as they are constantly updated? I would love some documentation on the topic.

@vpsfix: Thanks for sharing this! I remember stumbling on some of these articles when searching around. There are tons of articles and forum posts around, but as I was saying, very few focus on hardening the mailserver setup and strategies to prevent but also mitigate hacks and many may be obsolete because software, strategies and best practices may change during time. Really helpful to have someone dealing with this mess recommending at what exactly to look.

Disclaimer: I’m not a dedicated sysadmin. Hosting isn’t even a main business, besides family I actually do software architecture, engineering, programming and build and operate my own distributed software products. Sysadmin part is just the glue that I need of course to deploy and operate my products. So I have some experience and I’ve seen a couple of things during the years but I don’t have the resources to keep myself up-to-date daily so your help is really invaluable. And Virtualmin/Webmin are really great products helping me for years to more easily manage my infrastructure. Thumbs up for these products!