How to check what caused Apache2 to completely crash? And possible to auto-restart if it happens again?

I’ve been running Virtualmin on my VPS for almost a week now and I think I have it set up properly and all that, I followed a few tutorials here on the site and so on.

But yesterday I noticed how all my sites (including Virtualmin) suddenly stopped working. It would instantly give me an error when I go to the web site URL. Logging in at the VPS Control panel I could see that only 20mb out of the 256mb memory was used, and usually it’s around 50% when apache, mysql etc. runs.

So something caused Apache (and MySQL?) to just CRASH and don’t come back again by itself. Obviously this is a HUGE problem if it happens right after I go to bed and my sites will be offline for 6-8 hours.

How can I check (what log files?) what caused apache to break down - and even better, can I do something in Virtualmin to make sure everything automatically comes back up? Is there a cronjob or something (?) that can check every 5 minutes if a local URL works and if not, it will restart a bunch of services?

Thanks in advance :slight_smile:

Well, I have a suspicion you’re running into RAM issues (or lack-thereof :slight_smile: 256MB is a bit on the low-end…

If you run the “free” command on your server, what output do you get?

What I’d start off by doing, is removing any services you know you don’t need.

However, in order to see what the problems are, you’ll want to check out various error logs (which ones depend on what distro you’re using). Looking in /var/log/messages and the Apache log, /var/log/apache2/error_log or /var/log/httpd/error_log would be good places to start.

I’d also check out the dmesg output, especially towards the end, as it may contain any kernel errors you’re seeing.

Virtualmin has a “Status monitoring” feature in System Settings -> Features and Plugins. You can use that to have it email you when a site is down. I don’t recall off the top of my head whether it auto-restarts services… but if not, I’ve had great luck with tools like Monit for doing that.

However, my main suggestion in your case is to free up whatever memory you can, and to make sure you have plenty of swap available.

-Eric

Plesk Power Panels (VPS controlpanel) says 110MB used, Virtualmin says 140MB used and “free” says 112MB used:

(EDIT: Okay the copy/paste didn’t look too good in here, so I took a screenshot instead and attached).

I reloaded them all at the same time, so I guess Virtualmin displays more than what is actually being used?

I also tried to disable all the services I don’t need. I’ve attached a screenshot of my bootup & bootdown list within Virtualmin, to this post.

I’ll try to keep an eye on /var/log/apache2/error.log and and /var/log/messages next time it crashes.

My dmesg files are all empty. (the ones in /var/log).

I looked for “Status monitoring” in System Settings > Features and Plugins, but it’s not there. Maybe it’s only for PRO users?

Regarding memory, I tried to free up as much as possible, but if you have suggestions on what else I can do, I would love to hear them. However, I’m just running a single site on it now with a few hundred pageviews a day, so I would be surprised if 256MB isn’t enough for apache, mysql and virtualmin?

Okay, looking at your free output, you don’t appear to have swap – so it looks like you’re using an OpenVZ-based VPS.

The problem is that with no swap space, there’s absolutely no margin for error… no ability to have additional resources if/when your main RAM gets used up.

If there’s any kind of usage spike – whether it be several folks accessing your site at the same time, a few emails being spam checked, cron jobs running at night that require a lot of RAM, or maybe all those at the same time – your system can be without RAM and things will crash.

To try and help you track down what’s going on, you could try installing a monitoring tool such as Monit, which could help you keep an eye on your system. You can configure it to let you know if memory is running low, for example… as well as to restart Apache if it dies.

For pretty graphs and stuff and better usage information, you might also look into Munin.

However, I’ll offer that the forums here are filled with folks who’ve tried to setup OpenVZ-based servers with no swap, and ran into problems when their RAM ran out.

It does look like you did a good job disabling unneeded services.

The best I can offer from here, barring a way to get some additional RAM, would be to use a tool like Monit to restart daemons if something goes awry.

-Eric

I might be using OpenVZ-based VPS, I don’t know, sorry :slight_smile: Is it not possible to have a swap file, then?

I’ll take a look at Monit and see if it can help me anyway, but I’m not good at this linux stuff so if the installation/usage is too complicated, then I’ll have no clue how to use it.

Thanks for the hint about OpenVZ though, I’ll search the forum for more information and tips for setting it up.

Monit actually isn’t terribly bad.
When you have it installed and done basic changes on the .conf.

Throw this in and adjust as per needed

check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/sshd start"
   stop program  "/etc/init.d/sshd stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group mysql
   start program = "/etc/init.d/mysqld start"
   stop program = "/etc/init.d/mysqld stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

check process apache with pidfile /var/run/httpd.pid
   group apache
   start program = "/etc/init.d/httpd start"
   stop program  = "/etc/init.d/httpd stop"
   if failed host www.daworm.net port 80 protocol http
      and request "/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

The main things will be the locations of your PID files and the group/user names of your applications.
Where I got this from for example apache was in as “www” for the user/group. I have to change it to “apache”.

In addition, throw this into a file chmod +x the file and then run it.

#!/bin/bash
echo "This is intended as a guideline only!"
if [ -e /etc/debian_version ]; then
    APACHE="apache2"
elif [ -e /etc/redhat-release ]; then
    APACHE="httpd"
fi
RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1`
RSS=`expr $RSS / 1024`
echo "Stopping $APACHE to calculate free memory"
/etc/init.d/$APACHE stop &> /dev/null
MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'`
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "MaxClients should be around" `expr $MEM / $RSS`

The above will give you a simple output based on your system resources.

After which you adjust your httpd.conf file appropriately for the MaxClients etc.

Here’s what I changed my settings to based on the recommended above and Apache stablised for myself.

<IfModule prefork.c>
StartServers       4
MinSpareServers    3
MaxSpareServers   5
ServerLimit      19
MaxClients       19
MaxRequestsPerChild  4000
</IfModule>

You will have to adjust the MaxClients and ServerLimit to be the same and drop your StartServers down a few (I think default is like 8?.

If you are low traffic this would be more than enough.
AND again, adjust your MinSpareServers and MaxSpareServers.

Yours will be lower than 19. 19 is based on my 2GB ram in the server I have.

And LAST last thing

Look in /var/log/virtualmin/xxx.xxx_error_log

Where xxx.xxx is your domain names.
You may notice some php errors or similar showing up in there. I identified a script from that (which was a few years old and seriously out of date) and since disabling it, i’ve only had to worry about the server doing graceful restarts on log rotates.

:::EDIT:::
Just noticed an entry in my apache portion for Monit…
the “if mem greater than xxx” then restart.
You may want to adjust that down to say 200? Seeing as you only have 256mb.
5 cycles is basically 5 checks, in my case I check every minute.