Logs for server... server sometimes unresponsive - where to look

Help! (Home for newbies)
1

Hi,

I am having intermittent issues with one server. Sometimes it just seems to freeze up for a minute or two, then all seems fine, and it repeats… I am trying to figure out what is causing it…

Are there specific logs i should be looking at besides apache error log and mail log?

Thanks

I am seeing these a lot: (i know the mcrypt is from phpmyadmin, but not sure on the others)

[Mon Jan 23 08:37:38 2012] [warn] mod_fcgid: process 12422 graceful kill fail, sending SIGKILL
[Mon Jan 23 08:37:38 2012] [warn] mod_fcgid: process 12611 graceful kill fail, sending SIGKILL
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jan 23 11:01:36 2012] [warn] mod_fcgid: process 21792 graceful kill fail, sending SIGKILL
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jan 23 11:19:54 2012] [notice] caught SIGTERM, shutting down
[Mon Jan 23 11:19:55 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Mon Jan 23 11:19:55 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jan 23 11:19:55 2012] [notice] Digest: generating secret for digest authentication …
[Mon Jan 23 11:19:55 2012] [notice] Digest: done
[Mon Jan 23 11:19:55 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.6 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured – resuming normal operations
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jan 23 11:51:56 2012] [warn] mod_fcgid: process 24839 graceful kill fail, sending SIGKILL
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jan 23 12:16:39 2012] [warn] mod_fcgid: process 25540 graceful kill fail, sending SIGKILL
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mcrypt.so’ - /usr/lib64/php/modules/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jan 23 12:41:12 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/server-status
[Mon Jan 23 12:42:21 2012] [notice] caught SIGTERM, shutting down
[Mon Jan 23 12:42:21 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Mon Jan 23 12:42:21 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jan 23 12:42:21 2012] [notice] Digest: generating secret for digest authentication …
[Mon Jan 23 12:42:21 2012] [notice] Digest: done
[Mon Jan 23 12:42:21 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.6 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured – resuming normal operations

2

Howdy,

Is it just Apache that’s unavailable? Or is it your entire server?

Can you ping it during that time?

If you can’t even ping it, you may be seeing a network problem.

If you can ping it, you may need to look at the end of the dmesg output, and in /var/log/messages and /var/log/secure.

You may also want to look at your load average in “uptime”, as well as see how many messages are in your mail queue (with “mailq | tail -1”).

-Eric

3

hmmm… i can ping it pretty consistently and dont see any other network issues.
I dont see anything in the logs either… (im no expert though) anything jump out at you here? (attached as files)

[root@host01 ~]# uptime 14:04:04 up 1 day, 17:16, 1 user, load average: 0.04, 0.02, 0.03 [root@host01 ~]#

I am logged into vmin via the IP too. virtualmin itself becomes unresponsive when it happens…

i did also check intodns.com and it shows: (but i dont think this will cause the server to freeze)
Missing nameservers reported by your nameservers ERROR: One or more of the nameservers listed at the parent servers are not listed as NS records at your nameservers. The problem NS records are:
ns2.dealbent.net
This is listed as an ERROR because there are some cases where nasty problems can occur (if the TTLs vary from the NS records at the root servers and the NS records point to your own domain, for example).

Thanks a lot.

4

Howdy,

Well, there’s two hours of logs there, but you had mentioned that the problem only lasts a few minutes… what time exactly did the problem occur?

-Eric

5

On and off… no specific time… ha ha… (i just cant remember)
Hold for a bit - I am on with the server manufacturer support - we may have a raid driver issue. Awaiting their reply…

Thx.

6

Well, i gave it some time but it looks like it was in-fact a raid issue.
Hopefully we are good now…

There should be no issue with any resources as it has 16 proc cores and 24GB of RAM…

Thx