I seem to be having an issue with my server.
Apache seems to crash on me randomly but doesn’t appear to be showing in the logs. All I see is a graceful restart request log around the times that apache dies, but nothing saying it didn’t restart.
In googling I stumbled accross someone mentioning that their SSL key seemed to be butchered and then generated a new key for their admin tool (their were using ISPConfig or something).
I do know that I cannot access my server the that HTTPS: address, I get an error when attempting to do so and as a result I have disabled the https: access requirement to login to the admin panel.
How/where do I go within VirtualMin/Webmin to configure a new SSL Key to see if that will alleviate my issue?
Yeah, trying to figure out what’s wrong when there aren’t any error messages can be a toughy!
I’ve seen times when Apache – upon being told to restart – doesn’t fully exit. For whatever reason, one of it’s processes gets hung up in memory. From there, as Apache attempts to restart, it can’t, since the old processes that wouldn’t die are still bound to ports 80 and/or 443.
I don’t have a particularly good fix for this, other than trying to kill any stray apache2/httpd processes after telling Apache to to “stop” or “graceful-stop”.
As far as SSL goes – if you want to try that route, for any Virtual Server that has SSL enabled, you can select Server Configuration -> Manage SSL Certificate.
From there, go into “Signing Request”, and you’ll see a button on the bottom labeled “Generate new Self-signed key”.
Ok - keeping my original thread. Still not sorted this.
I have only seen it happen once when I was online at the time and since then another person I know has seen it several times.
The server seems to crash but I can SSH into it without an issue (at least I could when I noticed it happen). Trying to get more information from my friend.
Additionally this is the most recent log
[Sun Jan 10 04:02:12 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 10 04:02:12 2010] [notice] Digest: done
[Sun Jan 10 04:02:12 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Jan 10 04:02:13 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Jan 10 04:02:13 2010] [notice] Graceful restart requested, doing restart
[Sun Jan 10 04:02:13 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Sun Jan 10 04:02:13 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Sun Jan 10 04:02:16 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 10 04:02:16 2010] [notice] Digest: done
[Sun Jan 10 04:02:16 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Jan 10 04:02:16 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
Please note though the fault happened earlier today sometime, not 5 days ago so this doesn’t appear to match up… anything else anyone can suggest?
Just been advised that trying to access the e-mail was not working either, via HTTP or email client (POP3) so I’m wondering if there’s a DNS issue?
I’ve setup a ping test now to check every 15 minutes for the next time it goes down.
Well, you might consider setting up a monitoring system that watches the individual services on your server. If DNS were to stop functioning, for example, a monitoring system should be able to alert you to that fact.
Virtualmin has status monitoring built in, you could always use that. Personally, I like “monit” – and a lot of other folks use Nagios – those two are good as well.
Another thing about tools like Monit and Nagios – they can watch that a particular website is working. If it doesn’t, you can configure it to restart Apache (and then send you an alert that Apache was restarted).
However, the thing with monitoring – you’ll know exactly what is, and isn’t, working at any given moment… and if Apache goes down, you’ll know whether it’s just Apache having trouble, or whether other services are too.
Apache and Mysql are definately crashing. When I got it all going shows uptime of 2 hours and 9 minutes for apache and 2 hours and 16min for mysql - proftpd and postfix are running at 16d and 36d uptime.
Now to find out WHY it crashes I guess…
Monitoring service will kick it back into gear within 1 minute anyway
Resource limit succeeded Service apache
Date: Sat, 16 Jan 2010 14:19:55 +1100
Action: alert
Host: server.daworm.net
Description: 'apache' total mem amount check succeeded [current total mem amount=85132kB]
Your faithful employee,
monit
Resource limit succeeded Service apache
Date: Sat, 16 Jan 2010 14:19:55 +1100
Action: alert
Host: server.daworm.net
Description: 'apache' total mem amount check succeeded [current total mem amount=85132kB]
Your faithful employee,
monit
Looking at this more. I cannot seem to locate a httpd.pid file… wondering if that’s the cause of some things.
If I do a manual stop/start of Apache - the httpd.pid file gets generated… but after a crash (at this stage) it does not generate a httpd.pid in /var/run/
Yet Apache works… this might be the cause of my monit issues in that regards.
I have just altered my MaxClients settings to 19 and lowered starting server threads etc. as appropriate.
From some reason (it might be normal I don’t know) the mem use slowly creeps up and when a process hits around 50MB mem usage it spawns a new thread…
Well - after adjusting MaxClients to 19, and starting 3 by default. I’m only up to 6 clients running at the moment and Apache (this time) is now up to 1Day 6Hours run time (this last week it’s been struggling to get to 20Hours).
BUT! On the same note it has had uptimes of 6-7Days before it randomly dies on me… so time will tell.
On that note though: Can I safely re-install Apache from the VirtualMin Repo’s without having to do anything extra on my server? I mean if I backup my httpd.conf and stop Apache, would a
Well, if your goal is to verify that nothing is corrupt with Apache – you can run this to verify the contents of a given package:
rpm -Vv PACKAGE_NAME
That is, in your case, you could use:
rpm -Vv httpd
That will have RPM check each file that was included in the httpd installation. Doing that checks the file size, MD5 sum, permissions, type, owner and group of each file.
If that doesn’t show anything unusual, you won’t gain anything by reinstalling Apache
