Apache crashing

Virtualmin
1

Hello all,

I have a sever with Virtualmin, and Apache is crashing every several days, close to 4:00AM.
When I try to restart it using virtualmin I get the message:

(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs [FAILED]

Checking the httpd error log, I have:

[Sun Jul 24 01:21:05 2011] [warn] mod_fcgid: process 22904 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:05 2011] [warn] mod_fcgid: process 22902 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:05 2011] [warn] mod_fcgid: process 22900 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:05 2011] [warn] mod_fcgid: process 22897 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:05 2011] [warn] mod_fcgid: process 22896 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:11 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22904) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:11 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22902) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:11 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22900) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:11 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22897) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:11 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22896) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:11 2011] [warn] mod_fcgid: process 22931 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:11 2011] [warn] mod_fcgid: process 22929 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:11 2011] [warn] mod_fcgid: process 22922 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:17 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22931) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:17 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22929) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:17 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22922) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:17 2011] [warn] mod_fcgid: process 22936 graceful kill fail, sending SIGKILL [Sun Jul 24 01:21:23 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22936) exit(communication error), get stop signal 9 [Sun Jul 24 01:21:35 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(11383) exit(lifetime expired), terminated by calling exit(), return code: 0 [Sun Jul 24 01:21:35 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(20167) exit(lifetime expired), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(13817) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22920) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22905) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22901) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22914) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22895) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22899) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:25:41 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22926) exit(idle timeout), terminated by calling exit(), return code: 0 [Sun Jul 24 01:26:48 2011] [notice] mod_fcgid: call /home/97fmfrutalmaster/public_html/templates/yoo_chrome/lib/js/template.js.php with wrapper /home/97fmfrutalmaster/fcgi-bin/php5.fcgi [Sun Jul 24 01:48:16 2011] [notice] mod_fcgid: call /home/magazinehazimemaster/public_html/administrator/__DxS__UPLOAD__Ax5__asd.php with wrapper /home/magazinehazimemaster/fcgi-bin/php5.fcgi [Sun Jul 24 01:48:16 2011] [notice] mod_fcgid: call /home/magazinehazimemaster/public_html/administrator/__DxS__UPLOAD__Ax5__asd.php with wrapper /home/magazinehazimemaster/fcgi-bin/php5.fcgi [Sun Jul 24 01:48:16 2011] [notice] mod_fcgid: call /home/magazinehazimemaster/public_html/administrator/__DxS__UPLOAD__Ax5__asd.php with wrapper /home/magazinehazimemaster/fcgi-bin/php5.fcgi [Sun Jul 24 01:50:13 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(15667) exit(lifetime expired), terminated by calling exit(), return code: 255 sh: sysctl: command not found sh: sysctl: command not found [Sun Jul 24 02:03:27 2011] [error] [client 127.0.0.1] script '/var/www/html/version.php' not found or unable to stat chmod: cannot access `bds': No such file or directory sh: ./tmp/bds: No such file or directory [Sun Jul 24 02:04:55 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(29260) exit(communication error), terminated by calling exit(), return code: 0 [Sun Jul 24 02:05:17 2011] [notice] mod_fcgid: call /home/magazinehazimemaster/public_html/administrator/__DxS__UPLOAD__Ax5__asd.php with wrapper /home/magazinehazimemaster/fcgi-bin/php5.fcgi [Sun Jul 24 02:05:17 2011] [notice] mod_fcgid: call /home/magazinehazimemaster/public_html/administrator/__DxS__UPLOAD__Ax5__asd.php with wrapper /home/magazinehazimemaster/fcgi-bin/php5.fcgi [Sun Jul 24 02:09:19 2011] [notice] mod_fcgid: call /home/revistaespressomaster/public_html/index.php with wrapper /home/revistaespressomaster/fcgi-bin/php5.fcgi [Sun Jul 24 02:10:44 2011] [notice] mod_fcgid: call /home/intercambiowebmaster/public_html/index.php with wrapper /home/intercambiowebmaster/fcgi-bin/php5.fcgi sh: uname-a: command not found sh: uname-a: command not found sh: uname-a: command not found [Sun Jul 24 02:12:38 2011] [notice] mod_fcgid: process /home/intercambiowebmaster/public_html/index.php(8293) exit(lifetime expired), terminated by calling exit(), return code: 0 [Sun Jul 24 02:12:38 2011] [notice] mod_fcgid: process /home/intercambiowebmaster/public_html/index.php(20577) exit(lifetime expired), terminated by calling exit(), return code: 0 [Sun Jul 24 02:12:38 2011] [notice] mod_fcgid: process /home/intercambiowebmaster/public_html/index.php(20578) exit(lifetime expired), terminated by calling exit(), return code: 0 [Sun Jul 24 02:14:05 2011] [notice] mod_fcgid: call /home/frutalmaster/public_html/index.php with wrapper /home/frutalmaster/fcgi-bin/php5.fcgi [Sun Jul 24 02:19:20 2011] [warn] mod_fcgid: process 29262 graceful kill fail, sending SIGKILL [Sun Jul 24 02:19:26 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(29262) exit(communication error), get stop signal 9 [Sun Jul 24 02:21:38 2011] [warn] mod_fcgid: process 29261 graceful kill fail, sending SIGKILL [Sun Jul 24 02:21:44 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(29261) exit(communication error), get stop signal 9 [Sun Jul 24 02:22:00 2011] [warn] mod_fcgid: process 1415 graceful kill fail, sending SIGKILL [Sun Jul 24 02:22:06 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(1415) exit(communication error), get stop signal 9 [Sun Jul 24 02:34:10 2011] [notice] mod_fcgid: call /home/intercambiowebmaster/public_html/index.php with wrapper /home/intercambiowebmaster/fcgi-bin/php5.fcgi [Sun Jul 24 04:02:52 2011] [notice] SIGHUP received. Attempting to restart [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/rogeriobritomaster/public_html/index.php(2912) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/ncastromaster/public_html/index.php(21883) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/97fmfrutalmaster/public_html/index.php(24836) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/patrociniomaster/public_html/index.php(10110) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/corymaster/public_html/docesmomentos/index.php(32138) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22903) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/santaadeliamaster/public_html/index.php(14571) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/biomedicinmaster/public_html/index.php(7467) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/biomedicinmaster/public_html/index.php(8387) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/cartescosmaster/public_html/index.php(19723) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/ghtengenhariamaster/public_html/index.php(23045) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/citrikmaster/public_html/index.php(29538) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/babbardelmaster/public_html/gemelo/index.php(32422) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/ghtengenhariamaster/public_html/index.php(22820) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/convencaodgcmaster/public_html/index.php(18462) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/almamaster/public_html/olk/index.php(19518) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(2728) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/eptvmaster/public_html/index.php(26415) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/frutalmaster/public_html/index.php(8942) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/inscaintercodemaster/public_html/index.php(20388) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/pwcaubrasilmaster/public_html/index.php(12233) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/intercambiowebmaster/public_html/index.php(2867) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/ituramamaster/public_html/index.php(7733) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/magazinehazimemaster/public_html/index.php(1413) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/biomedicinmaster/public_html/index.php(8390) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/cartescosmaster/public_html/index.php(19707) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/frutalmaster/public_html/index.php(30262) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/santaadeliamaster/public_html/index.php(14570) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/inscaintercodemaster/public_html/index.php(28085) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/pwcaluminimaster/public_html/v2/index.php(30930) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/clientesintercodemaster/public_html/casamentojundiai/administrator/index.php(15351) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/souloucoporicekissmaster/public_html/fortaleza/index.php(15971) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/souloucoporicekissmaster/public_html/fortaleza/index.php(15972) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/rogeriobritomaster/public_html/index.php(32709) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/gtogintercambiomaster/public_html/index.php(6866) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/clientesintercodemaster/public_html/casamentojundiai/administrator/index.php(13978) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/97fmfrutalmaster/public_html/index.php(2597) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/inscaintercodemaster/public_html/index.php(20395) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/frutalmaster/public_html/index.php(3755) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/intercambiowebmaster/public_html/index.php(12045) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/ituramamaster/public_html/index.php(7732) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/eptvmaster/public_html/index.php(3571) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/cartescosmaster/public_html/index.php(19651) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/santaadeliamaster/public_html/index.php(14569) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/welcometopwcmaster/public_html/index.php(20994) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/revistaespressomaster/public_html/index.php(22933) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/plantaoodontomaster/public_html/index.php(24845) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/pwcaluminimaster/public_html/v2/index.php(21454) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/plantaoodontomaster/public_html/index.php(1692) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/welcometopwcmaster/public_html/index.php(32445) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/osfigurasicekissmaster/public_html/index.php(6059) exit(shutting down), get stop signal 1 [Sun Jul 24 04:02:53 2011] [notice] mod_fcgid: process /home/welcometopwcmaster/public_html/index.php(32410) exit(shutting down), get stop signal 1 [Sun Jul 24 10:36:16 2011] [warn] Duplicate command 'CBandSpeed' for canaldaelite.com.br:1599 [Sun Jul 24 10:36:16 2011] [warn] VirtualHost 200.170.216.250:80 overlaps with VirtualHost 200.170.216.250:80, the first has precedence, perhaps you need a NameVirtualHost directive [root@linux03 httpd]#

Restarting the server resolves the problem.

Does anyone knows what is happening? How can I stop apache from crashing?

Thank you

  • Rogerio
2

Can you please put that screen output listing in [code][/code] tags? It is close to unreadable like it is now. :slight_smile:

3

Hmm, a first wild guess: Are you running on a virtual machine, e.g. OpenVZ? Maybe out-of-memory issues? Can you check what top/atop is reporting when or before the problem occurs? What does it report otherwise?

Does the crash always occur around 4 AM? Is any script running then? Backup maybe? Something that causes Apache to restart?

You seem to be running a great lot of domains through FCGId. Note that for every FCGI domain, a separate PHP process is created which uses up memory. You’d need quite a bit of RAM for this number of domains.

Do you have some kind of debug logging enabled in Apache? I don’t think it normally logs when it kills FCGI processes due to expired lifetime (which is not an error by the way, but expected behavior).

4

there is some bug in CentOs where the restart of logrotate messes up Apache
I have this in several VPS machines, but not all of them.
There is the main httpd proces running as root which doesnt want to ‘graceful restart’

Until now I have to do a : ‘killall httpd’ and then ‘service httpd restart’ which makes apache run again, without the need to restart the whole box.

5

Can confirm that Apache doesnt want to ‘graceful restart’. Mine is also a CentOs box, and happens once in a while. I thought it had something to do with my Sunday night server backup, as the last couple of times httpd has to be killed and then started on a Monday morning.

Didn’t really know why, so good to note that ronald mentions a bug.

6

Oh, that’s indeed nice to know. (On my Ubuntu the log rotation does not constitute any problems I’m aware of.)

7

Thanks for the comments guys.

My box is also a CentOS 5.5. So is it a bug?

I had another, more serious problem a few days back.
Apache stopped working and wouldn’t restart. On the error log I got a message saying that the logfile for each domain could not be found. Checking the filesystem I’ve found that the /var/log/virtualmin dir was not there! I created it manually and then Apache started. Scary.

Could the log rotation erase the /var/log/virtualmin dir?
Any thoughts?

Thanks

  • Rogerio
8

The log file rotation should not delete any directories… all it should do is gzip files and remove old logs. So, I’m not sure why that might have happened.

My only suggestion would be to keep an eye on it… if it happens again, we might have to do some detective work to figure out what happened.

-Eric

9

Ok Eric, I’m monitoring it.
I’ll let you know if it happens again.

Thank you

  • Rogerio