Users “disneysc”, “halloween”, “fashions” and “nikki” now also exhibit very high write rates. You might wanna check their web pages too.
Well, your box has 32 GB of memory, why not install a PHP cache on several levels, like an instruction cache (xcache), and “memcached” which Wordpress should support.
If you have very fast HDDs that can cope with those high write rates, e.g. RAID-0 arrays with multiple disks, or SSDs, it is okay to use extensive disk caching, but since you’re experiencing problems with system load going through the roof, you should try memory caches instead of disk caches.
This does not solve the spam issue, but that’s another thing. Finding out where those come from is separate from the high load issue. Maybe let’s first try to fix the load issue.
OK - so you think I should try turning off disk caches? I’ve installed APC a few weeks ago, does that work with memcache or xcache too, do you know?
I can see CPU Wait values going up a lot, which implies that your processes are bogged down by waiting for probably Disk I/O. Another indication that your disk caches are excessive.
The Apache logs seem to indicate that the Wordpress sites in question host photos or other images? It’s not overly suggested to cache those on the disk. That’s basically like copying them on every request. Caching is useful for compiled PHP code or dynamic pages that contain mostly text and don’t change very often. Try disabling any disk caching for starters.
Please reload, I edited my previous post, simultaneously when you sent yours.
Also: Do those Wordpress sites have any kind of contact form or other way of sending you emails? That might be what is (ab)used for sending the spam to you.
You might try disabling any contact forms while you test that theory.
OK, I’ve turned off page disc caching for ‘adriana’ and changed it to APC instead - where would I go to see if that’s making a difference?
Also I’m not sure if Wordpress is the right software to host a bunch of high-traffic photo sites, performance-wise. There’s probably better suited software for that, though I don’t know off the bat what you could try.
About disk cache: You should turn it off for ALL your WP sites, and then take a look at atop if the disk activity dies down and CPL goes down too.
Ah ok sorry, I’ve reloaded now. Yes, they are all image galleries - 95% of the content is images. OK, I’ll try disabling page disk caching altogether, is that better than switching it to APC? And I’ll take down the forms too just to see if that helps. Fingers crossed!
Also, take a look at the “cache” value in the MEM row. That indicates how much of your physical memory is used for file system caches. As you can see, it’s at about 2.3 GB already, which means that probably most of the photos that users request are in the memory cache.
So it’s possible you don’t even need xcache or memcached (considering you have an 8-core CPU which is bored out of its mind for the most part), but using those 32 GB RAM as memory cache (which Linux does automatically) might suffice.
Yes, you don’t need ANY page disk cache if you host mostly galleries. As you can see, it’s on the contrary rather counter-productive. Your system spends most of its time writing to the cache, and the processes have to wait for the disk to write that cache.
Reading from the disk cache won’t bring an advantage anyway, since that 8-core CPU and the 32 GB memory, when properly used as file system cache, deliver the gallery pages much quicker than reading the stuff from your disk cache.
Well, I have to say I think you’ve hit the nail on the head!! I switched off all my disk caching last night, and when I checked my server again this morning, it was 1) still online!! and 2) the loads were all around 1.5 which is SO much more reasonable than 200! 
Thank you so much for your help - I can do a little further tweaking now, and I might try installing xcache or memcache also just to speed things up, but it really seems like the disk writing speed was the bottleneck, and without that, the server seems happy enough. Wasn’t a SPAM issue after all then! Thanks again for all your help!!
OK - so here’s a quick update! I installed and configured Memcache, alongside APC which was already installed. I made sure all of my sites are using W3 Total Cache as a Wordpress plugin, with both page caching and minify set to APC, and object & database caching set to Memcache, so nothing is writing caches to disk. The sites that had WP Super Cache installed, have been switched to W3 Total Cache, with Supercache deleted completely.
This seems to have helped a lot, my loads are a lot less and the server is less prone to crashing.
I’ve been checking atop, and although things look better, the rows for DSK are still in the red often enough, with busy listed around 80-90% most of the time, only sometimes falling to 60% or less, and sometimes spiking to 100% (when that happens, all my sites immediately start throwing 500 Internal Server Errors). Where should I look next to see which users are writing to disk so much, or how could I troubleshoot this next?
The list below the generic statistics show which process is doing what. When you press g to switch to “generic data”, then shift+d to order the list by disk activity, you should be able to see who’s the culprit.
Hmm, it seems to be the ‘apache’ process owned by root that does a lit of writing? Could you take a look and confirm I’m reading it right?
I’m watching atop now for about a few minutes, but disk write rates are well below 1 MB/s, with an occasional “spike” of 2 MB/s caused by MySQL. Nothing anywhere near the red zone.
Just now a process named “bw.pl”, possibly triggered by cron, had a quick CPU spike and read a few hundred MBs.
Nothing out of the ordinary so far.
Ok thank you! I actually made a little tweak to my APC settings after my last post, changing the apc.mmap_file_mask to /dev/zero as that was supposed to lower disk activity, and it really has, everything looks pretty good at the moment. Thanks again for all your help - I’ll keep monitoring it over the next few days to see how I go!
Rogerroger, and you’re welcome!
(Oh, and don’t mind the little Trojan I left somewhere on your server. 
)
LOL! Thanks, I always enjoy little surprises like that.
Back with another quick question!! My server’s going so far so good:
uptime
22:15:47 up 1 day, 6:09, 2 users, load average: 0.14, 0.29, 0.35
I’ve been monitoring things via atop, which all seems good, the disk writes are all around 20-30%. However, because I’m now actually using more of my RAM, I’ve seen that creep up and up over the past 2 days. At the moment, the MEM row looks like this:
MEM | tot 31.3G | free 521.1M | cache 22.6G | dirty 7.9M | buff 3.0G | slab 1.5G | | |The value for free has gone from 25+ GB to 500MB over the past 48 hours. Obviously I want to use all of the RAM I have, but is this something to worry about? Do I need to make some changes to make sure the cache doesn’t fill up completely, or, will Linux/APC/Memcache all be smart enough to automatically clear some of the cache when it needs more space?
Nothing to worry about there. RAM used as cache IS basically free and can be reassigned to applications as soon as they require it. Linux follows the philosophy “free ram is wasted ram” and uses as much of it as cache as it can get.