Is there any issue with system resources in the current (3.17) version of VirtualminPro? Over the last couple weeks performance has decayed and anything webserver associated nearly shuts down the entire box.
When I check the system resources I’m using over 800Meg of 1 GIg of RAM. Pretty steep for a dual PIII 933M. There seems to be a cascade of writelog failures that I assume create more process, leading to a ton of serverdown monitor emails.
The only thing coincident with this is using the AWBS CRON job, that process seems to really load up resources but I’m not running it as an active cron.
Hard to say without something more concrete to go on. I’m unaware of any issues, but that’s not to say something can’t be going wrong on your system.
When you say “I’m using over 800Meg of 1 GIg of RAM”, it’s impossible to know where that’s all going without seeing the details. How about sending along the output of this command:
top -b -n 1
This will show me the details of what exactly is using memory, whether it’s in active use or swapped out, etc. We can then figure out which processes we need to start looking at for a cuase and solution to your problems.
Actually, I don’t think the writelogs.pl is a problem, though I could be wrong. It is expected to have one writelogs process per domain. The process sizes you’re looking at include all of the shared libraries–they all share the same libraries, and so the total usage of all of the writelogs.pl processes is probably pretty close to the size of one process.
There do appear to be more miniserv processes than I would expect, though. It shouldn’t increase memory usage by a huge amount (again, lots of shared libs reflected in the process size), but at idle, there will be two miniserv processes (one for Webmin and one for Usermin)…when in use, or when one of the crons that relies on miniserv is running, there would be an extra. I think it would take several simultaneous logged in Webmin/Usermin users to lead to as many miniservs as you have.
Try restarting Webmin and Usermin and see if they go away, temporarily. Could be something broken in the System and Server Status monitors that’s leading to hung miniserv processes (maybe…I think that’s how such problems exhibit themselves).
I had a similar problem a while back. writelogs.pl processes where hogging resources and causing the system to slow to a crawl. I no longer use writelogs.pl and the problem has gone away, although i may end up with issues if users start deleting directories.
1GB is probably a bit tight for a dozen sites, but it may very well be that writelogs.pl is doing something nasty. It’s not immediately apparent what, but it could happen. It’s not running away (at least not obviously so), as the process size is about 30 MB in all cases on your box, with the majority of that being shared libs. If one or more were running away we’d see high and growing memory usage from one or more of them. There’s just so little in writelogs.pl (have a look, it’s only 32 lines, including comments and whitespace…about 25 lines of actual code), it’s hard to see where a problem could sneak in, though each one does take some memory and on a box with low memory to start with it could make the difference.
Try turning off the writelogs feature and see if it makes a big difference. If it does, I’ll see about converting the perl writelogs into a C process with a somewhat smaller memory footprint. Perl can be a wee bit heavy these days.
I should note that shutting down apache gives you back about 500Meg RAM
Which brings me back to concerns when I changed from Slack to centOS. I have a slack 10 machine with webmin and virtualmin and only 515Meg RAM with 375 Meg free with apache running and 8 domain hosts, one domain acting as a very active sendmail server.
As an aside this nagging problem is still around when stop/starting apache:
Error
Failed to start apache : Apache does not appear to be running :
Starting httpd: [[ OK ]]
I turned off writelog and gained significant resources, like system resources graphic showing only about 300 meg used now. I was checking out some other posts and it appears that writelog was used for damage comtrol if someone deleted the log directories or some such follishness.
You or Jamie pointed out in a post that adding a hidden file to the directories could pretty much resolve that issue. Was that implemented? If so, that would pretty much preclude having to rewrite the perl into C.
One more thing to consider, which may be related to this issue (aside from the write logs)…
I also had the issue of my swap space becoming fully utilized recently to the point of the kernel killing off processes to free up space. So, I did some research and discovered a confirmed kernel bug in the current release of Red Hat Enterprise Linux and CentOS. This bug causes the kernel to swap memory too aggressively. For more information, see these bug reports:
Unfortunately, Red Hat has indicated this bug will not be fixed until the next release of RHEL (U4), which is currently in beta test. It may be another month or two before it is released, so my interim solution is to increase the amount of swap space allocated on the server.
You or Jamie pointed out in a post that adding a hidden file to the directories could pretty much resolve that issue. Was that implemented? If so, that would pretty much preclude having to rewrite the perl into C.
It was me. It solves one problem, and it is probably in the current release or will be in the next. But it doesn’t solve all problems…which means the issue still exists, but the window of accidental opportunity is smaller. Deleting the directory isn’t possible with the hidden file solution, but renaming it is–same DoS result, different method.
If you don’t have malicious or stupid users, this shouldn’t be a problem, but if you’ve got a lot of users…the odds of at least one of them being malicious or stupid get pretty high.
I think we’ll need to begin looking for alternative solutions to the problem, and a rewrite or packaging up one of the existing logging utilities seems the likeliest course of action. Another alternative is to take the logs completely out of the users control. This is also probably an unworkable solution, since anyone developing CGI/PHP/etc. scripts will want to see those logs in realtime.
It’s a hard life. But we’ll get it all figured out.