Hi,
i’ve got a few websites on my server, and today, apache reacts so slow most pages time out. Not on the webmin/virtuamin apache, but the webserver one. ‘top’ shows no strange things and normally, if apache is (a bit) slow, restarting it, or, in bad cases restarting the machine solves the problem, but not now.
My question: what are logical steps (with virtualmin and webmin or over ssh) in trying to find the cause?
Thanks!
Howdy,
Well, a couple of thoughts:
I might look in the apache error logs, to see if there’s any clues there (/var/log/httpd/error_log or /var/log/apache2/error_log)
You may want to look at your network – try pinging your box for awhile from another server, and make sure that you aren’t loosing any packets. And try doing a traceroute to make sure nothing in between the systems is causing trouble.
If it’s not network related, and no clear Apache issues, you might consider installing a tool like “mytop” in order to see what’s going on with the database. A “harsh” database query could really slow things down, and mytop will help clarify what all your database is doing.
That’s just a few things to get you started – feel free to discuss your findings, and hopefully we can figure out what’s going on!
-Eric
Eric mentioned network, in addition I would first check DNS on the domains (www.intodns.com)
I would also look to how much RAM is on the system
is it a VPS?
What kind of webpages are slow? static, php?
If php, are they a cms like Joomla! or custom made scripts?
in TOP you see no oddities…so apache doesn’t use a lot of ram nor cpu?
Mytop:
It seems mytop shows nothing weird, but this is done when the server is already hardly responding.
MySQL on localhost (5.0.67-0ubuntu6) up 0+00:47:12 [10:14:32] Queries: 18.1k qps: 7 Slow: 0.0 Se/In/Up/De(%): 54/00/00/00 qps now: 0 Slow qps: 0.0 Threads: 2 ( 1/ 6) 00/00/00/00 Key Efficiency: 98.8% Bps in/out: 0.1/ 9.5 Now in/out: 8.4/ 1.3k
Id User Host/IP DB Time Cmd Query or State -- ---- ------- -- ---- --- ---------- 1020 root localhost 0 Query show full processlist 984 gci localhost gci 164 Sleep
Mysqltuner:
-------- General Statistics --------------------------------------------------
[–] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.67-0ubuntu6
[OK] Operating on 32-bit architecture with less than 2GB RAM
-------- Storage Engine Statistics -------------------------------------------
[–] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[–] Data in MyISAM tables: 46M (Tables: 635)
[!!] InnoDB is enabled but isn’t being used
[!!] Total fragmented tables: 8
-------- Performance Metrics -------------------------------------------------
[–] Up for: 2m 40s (3K q [19.944 qps], 702 conn, TX: 4M, RX: 288K)
[–] Reads / Writes: 97% / 3%
[–] Total buffers: 58.0M global + 2.6M per thread (100 max threads)
[OK] Maximum possible memory usage: 320.5M (15% of installed RAM)
[OK] Slow queries: 0% (0/3K)
[OK] Highest usage of available connections: 8% (8/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/8.2M
[OK] Key buffer hit rate: 98.3% (45K cached / 756 reads)
[OK] Query cache efficiency: 28.0% (647 cached / 2K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 75 sorts)
[!!] Temporary tables created on disk: 28% (51 on disk / 178 total)
[OK] Thread cache hit rate: 98% (8 created / 702 connections)
[!!] Table cache hit rate: 1% (64 open / 5K opened)
[OK] Open file limit used: 12% (128/1K)
[OK] Table locks acquired immediately: 99% (1K immediate / 1K locks)
-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
tmp_table_size (> 32M)
max_heap_table_size (> 16M)
table_cache (> 64)
NSs have same SOA serial
Looks like your nameservers do not agree on the SOA serial. Ths SOA records as reported by your nameservers:
88.208.232.11 -> 2
213.171.223.34 -> 5
This can cause some serious problems that is why you should fix this asap.
I can’t find the error log of apache, so maybe I’ll have to set it in a config file. I’ll look into that now.
Pinging the network works fine, no lost packages.
The DNS error, I remember I’ve already looked into this, and it was kind of impossible for me that is, to set it right on both servers. The other server comes with an admin panel by fasthosts.co.uk. Maybe I should install virtualmin on that server too… But: although I’ve had this dns problem for quite a lot months, the server has been working fine.
I’ll start with the apache error log as the server has stopped responding again.
Thanks for your help people!
the apache error log says:
root@server2:~# tail -f /var/log/apache2/error.log
[Tue Aug 11 10:03:25 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Aug 11 10:05:07 2009] [notice] caught SIGTERM, shutting down
[Tue Aug 11 10:05:09 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured – resuming normal operations
[Tue Aug 11 10:05:14 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Aug 11 10:18:56 2009] [notice] caught SIGTERM, shutting down
[Tue Aug 11 10:20:31 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured – resuming normal operations
[Tue Aug 11 10:20:37 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Aug 11 10:37:23 2009] [notice] caught SIGTERM, shutting down
[Tue Aug 11 10:37:26 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured – resuming normal operations
[Tue Aug 11 10:37:31 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
so I guess I’ll look at maxclients first and will then try to find what sigterm means
I changed the max clients from 150 to 500 and after i did that, the server has been running fine. However, I’m not convinced yet. Might be that if someone was trying to attack the server or something (what do i know…) that they’ve simply stopped doing that at more or less the same I changed the setting.
I’ve no clue either how to optimise apache or mysql, so I guess there’s a learning task too for me.
Well, knowing that I’m just sort of speculating here, the two thoughts that initially come to mind are:
You have a popular website! Nice work 
If bumping that up to 500 works, and you have enough CPU and RAM, that’s great.
It could be a DoS attack of some kind. There’s been a few of those going around lately. I’d use a tool like netstat to try and diagnose that, running “netstat -an”. It’s hard to say what to look for, as it’d be different things depending on the attack. A lot of outstanding “SYN” requests, or perhaps “TIME_WAIT” connections, could indicate one kind of DoS.
You might want to look in System Settings -> Bandwidth Monitoring -> Show Usage Graph, you can see if any hosts are using a particularly large amount of bandwidth – which might help you figure out a little more about what’s going on.
-Eric
do you know anything about the popularity of your website? If it is not an attack, then it must be popular. That would indeed be a good thing.
the thing is, when something goes slow, the lot goes slow. One major site is a joomla site (so php) and one other major site is custom made scripts. However, that one creates static pages and only becomes ‘dynamic’ again when mod rewrite cant find the static files, which only happens when a page has been added to the site. As soon as page is dynamically served then, it is written away as a static file until a new page gets added.
Those two sites get more or less 4000 visitors daily, nothing that should bring a server down I guess.
There are 4 other sites on there (wordpress, joomla). Those only get a few hundred visitors daily.
server (ubuntu 8) has 2 gb ram.
apache doesn’t seem to be using a lot of cpu or ram…
Off puzzling again.
(thought my latest post would end up down here, but it didn’t, so here’s the important bit:)
Those two sites get more or less 4000 visitors daily, nothing that should bring a server down I guess.
There are 4 other sites on there (wordpress, joomla). Those only get a few hundred visitors daily.
thanks, i’ve now enabled bandwidth monitoring.
will have a look at netstat now.
but far from perfect. Sometimes it runs fast, but now for example, I’ve got to wait ten seconds before a new page loads.
(Just to let you know it’s not really solved.)
it’s still no good.
In the morning and early evening, top shows speeds from around 1.00 going down and down while nothing seems to happen. Apache on the normal webserver has to be restarted for the sites to be seen again. Within a few minutes, it can happen again.
Apache on webmin/virtualminruns fine at the same time however, as do ssh and ftp etc.
Can you verify in Webalizer (ie, in http://domain.com/stats/) that there hasn’t been an unusual amount of traffic recently?
Also, can you take the results of a “netstat -an”, and attach it to a forum post here so we can take a look?
-Eric
Hi Eric, yes i did check the stats of the different domains on the server: nothing weird. (Luckily!)
Attached is the netstat file.
Hmm… well, first off, you don’t have a particularly large amount of odd traffic there.
But it is unusual how many outstanding requests there are from 145.50.39.11.
Do you recognize that IP?
If you run netstat again, do you still see that IP with a lot of “SYN_RECV” entries?
Also, in Apache, I often change the “Timeout” parameter from the default of 300 down to something like 30 to make sure Apache closes up shop a little quicker and doesn’t hang on to slow connections.
-Eric
I’ll have a look at the timeout thing. In fact, I’ll have to learn how to (fine)tune apache and mysql.
didn’t see the 145… ip just now, I’ll attach it for you to see.
Hrm, yeah, that all looks fairly normal now.
So, the problem may be elsewhere
-Eric
indeed. So what would your next step be?
Right now, the server goes down within a minute of restarting apache (and/or mysql and/or the whole machine). So I did your netstat thing again, and it’s full of these listings of one ip address:
IP address 77.165.42.1
Hostname ip4da52a01.direct-adsl.nl
ISP ADSL41
Country Netherlands
I’ll keep checking when the server has problems, but it does seem a bit dodgy, doesn’t it?