Virtualmin updates crashed my server

Received an email with the following updates.

An update to wbm-virtual-server 3.73-1 is available : New version released

An update to wbm-virtualmin-mailman 5.7-1 is available : New version released

Ran the updates. My websites are now down and I cannot login to virtualmin. Any suggestions on how to get my websites and virtualmin back?

CentOS 5.2

-Jeff

Well, a Virtualmin update shouldn’t effect your websites.

Can you offer more details?

What symptoms are you seeing?

What errors are in the Apache error log, /var/log/httpd/error_log?

If you run “/etc/init.d/httpd restart”, does that help? And if not, what errors show up in the Apache error log now.

-Eric

The websites are all down, I cannot login to virtualmin, SSH is not responding to logins. Tried rebooting server, it will not come up. It seemed to wake up for a moment and gave some email, but went back down right away.

-Jeff

It’s tough to offer a suggestion if you can’t even log in, unfortunately…

Can you log in from the console?

It sounds like you may need to check out:

  1. The various system logfiles, including /var/log/messages and /var/log/secure, and see if they specify any errors or problems.

  2. Run “uptime” to determine the system load average

  3. Run “top”, and see if some programs are taking up a large amount of resources

  4. You might also want to run the “dmesg” command to see if there’s any kernel output for whatever is going on.

-Eric

I managed to get it running again. BIND and MySQL were not running. It is up now, but running very slow and the ram and swap are maxed out. I watched the console and rebooted. The only thing I noticed during boot was that the HAL daemon failed to start. I don’t know exactly what it does and why it might fail.

top shows that spamassassin and clamav are running pretty heavy.

Any suggestions?

-Jeff

It sounds like you’re processing a lot of mail – which can indeed cause your server to run slow.

If you type:

mailq | tail -1

What output do you receive.

Also, what does this show:

uptime

There were about 1100 messages in the mail queue. I deleted several hundred and restarted postfix, but they don’t seem to be leaving the queue. clamscan seems to be the highest process in top.

What’s the best way to turn off clam and spamassassin?

-Jeff

Having that many messages in your mail queue could mean someone is taking advantage of a security hole somewhere in your system – this frequently occurs with older web apps.

A lot of web applications contain vulnerabilities that could allow someone to send massive amounts of spam.

What I would recommend doing is:

  • Verify that all web apps installed on your server are at their absolute latest version

  • Make sure all your system software is up to date

  • Look for any unusual processes that are running

  • Use tools like chkrootkit and rkhunter to assist in finding any problems problems

You can disable clam and spamassassin if you like (in System Settings -> Features and Plugins), but the real problem here is in trying to determine how all that email ended up in your mail queue :slight_smile:

-Eric

All of the email was legitimate mail. The problem was clamscan no longer worked after the last system update, so all the mail in the queue was unable to leave because clamscan would not let go of it.

I disabled the anti-virus & spam scanning in each of my virtual servers, then turned off virus & spam scanning in the system settings->features and plugins, then disabled the boot action for clam & spam.

All is now working perfectly and the mail left the queue immediately.

I had to disable all the servers, i.e. dovecot, httpd, mailman, postfix, proftpd, etc, in order to get virtualmin to work. When everything was on the RAM and swap were completely maxed out and virtualmin could not get a command in sideways. It was like having a denial of service attack.

-Jeff

well ive been having the same exact problem on my two vps slices as well.

after the update spamassasin (more then clam) is eating up memory and cpu grinding the system to a halt.

ive tried limiting spamassasin by selecting to only allow a single spamassasin instance. same happens but the system is slightly more responsive as swap is not totally not exhausted but still basically unsable.

when i kill postfix but leave the spamassasin instances running it finishes in about 5min and with no new instances coming up system relaxes.

but turnning of spamassasin and clam really IS not a solution. when were talking about ~200mails (mailq) and a 512MB system running centos 5.3. same load was ok before the update.

what exactly changed???

-not aok