Just happened again and I tried to find out exactly which service caused the trouble and it is Apache2. So by restarting Apache, the websites became available once again. I should mention that Apache was running all the time (it didn’t crash and it showed as running in the Virtualmin panel), but the websites still became unavailable to me and everyone else online.
Now I am trying to find out exactly what has happened and what caused this to happen, but so far with no luck.
The warnings I see coming before the crash are coming all the time in the error log file, even after restarting the server, and I see approximately one such error per second. I am not sure what it is and whether this is causing the problem or influencing it, so if anyone can help, I would be grateful about that.
The graceful shutdown coming after the errors is me restarting the apache service.
Anywhere else I should look to find out what’s going on or what caused the error?
In auth.log I see there are lots of login attempts, but this seems to be quite constant… is this what might be causing the problem?
I’m not convinced. Too many @praguepraha.com (CloudFlare) why not being picked up by fail2ban. You said that you have no mail server running - so why even bother with postfix server?
you are suggesting an Apache problem - some conflict in the configuration perhaps.
Postfix is running because (based on my experience), if I disable it, the contact forms on the websites running on the server stop sending the emails as people fill in the forms.Once I enable Postfix, the emails are sent, and since that is one of the most important part of the business with contact forms, I cannot risk those emails not arriving. If I am wrong in this and it should work even with Postfix disabled, I would be really grateful to hear about that.
disable on a per domain basis, shouldn’t that allow forms to still work? and firewall incoming mail port maybe. But I can’t see how postfix would effect your websites.
You say apache is still running, and not crashing. whats the error when you try to connect to the website in the browser?
Is this on a VPS or on actual hardware? If so do a disk check as well.
So the program on the website is sending with an account on the server - so the server’s account is publicly exposed by that program? is that where all the “attacks” originate? I would be looking into the code of the contact form. Why does it even need an email exposed client side? A contact form presumably has only four fields:
the client’s email/ phone/ reply to;
the type of contact being made;
the contact message;
(hidden) an encrypted code to assure validity of the form.
All should be handled server-side as appropriate. no server user email exposed.
But I still do not understand why you need postfix. It should have nothing to do with contact forms unless they have been poorly setup on the website. Are you using an account server-side to send out emails? I thought you said not.
I know you would really like to know what process is taking the server down - it is just that I think the problem has more to do with the website you are running (or Apache config) than something to do with Virtualmin - otherwise we would all be seeing this problem.
I don’t set up a lot of software on sites but I know many require you to log into the mail server instead of having their own mail scripts. Seems safer in the long run.
and in fact it’s off topic the point of this thread is to workout why apache and/or php fails, we could take the point that these are wordpress sites (php) and not some other ‘professional’ mumbo jumbo software installation. You may find php relies on an mta to send, mail hence the requirement for an mta.
So do we know that both apache & ,if required, php are not being killed by the oom killer ?
If the web sites are, if required, running the correct version (and perhaps mode) of php, yes stuff may work in php fpm mode but not in php fcgi mode. so from this it would be good to know if the sites are php based or something else so php can be ruled out.
Completely unnecessary and far from being safer as it exposes the mail to the client which is about as good as a mailto: in any website an invitation to spam. A contact form which is client side should simply submit to server-side any data in the form.
But as it seems to be “off topic” I’ll leave it at that. I hope someone resolves this.
The errors in the the error log “Network is unreachable” indicates whatever is being proxied to isn’t responding. If it’s a PHP app, then it’d be PHP-FPM or the FCGId server that’s failing to respond in a timely manner.
Why it is failing to respond can have infinite answers, but probably one of a few common problems. Could be resource contention (memory, disk, whatever), which would maybe indicate database isn’t properly tuned for the workload/system. Note that a lot of folks have a tendency to just randomly start increasing limits (such as on number of PHP processes or size of database buffers), which can make things worse if it hits physical limits of the system.
So it is probably not Apache (it almost never is, serving web pages fast is a solved problem). It is probably an application problem, maybe database, maybe PHP tuning/configuration, maybe some ugly queries in your app that need table indexes or need to be rewritten to be more efficient.
Just guessing, but you need to look at that side of things. Poking at Apache isn’t going to solve it, because Apache isn’t the one misbehaving, I don’t think.
Thanks for all the help and advice that has been given here. I am working hard at trying to solve the problems, and I just enabled the Fail2Ban log file and as I compare it with a different server I run with Virtualmin it looks completely different. All I can see in the Fail2Ban log is as follows:
Also, the jail seems to be empty, meaning that Fail2Ban doesn’t seem to really ban users who attempt logging in or using the mail server to often. At my other serve, the log looks normal and I can also see lots of IP addresses in the Fail2Ban jail.
So, might there be some other thing like this actually causing the problem, that IPs aren’t banned, and that the server simply cracks down by too many attempts? Anyone able to give any advice on how to solve those Fail2Ban errors seen in the first screenshot?
There was an issue raised a little while back. If your firewall gets restarted then fail2ban needs restarted. It also must start in the proper order.
I think the reload behavior can be fixed in /etc/firewalld/firewalld.conf
# FlushAllOnReload
# Flush all runtime rules on a reload. In previous releases some runtime
# configuration was retained during a reload, namely; interface to zone
# assignment, and direct rules. This was confusing to users. To get the old
# behavior set this to "no".
# Default: yes
FlushAllOnReload=yes
