Webmin Memory Leak?

Webmin
1

I disregarded the first time this happened, but this is the 2nd time this happens and I decided to report it. Webmin uses up all the available RAM, seemingly out of nowhere. The last time anyone logged or made changes to webmin was days ago. Below is the output of “ps -e -orss=,args= | sort -b -k1,1n | tail -n 20” at the time free RAM goes below 600mb (on a 16gb system that usually runs at 40% of RAM capacity)

The first time it happened was on august 10th, and I usually run pretty tight updates, so on that date I was most likely running the most current version available.

This time I am running:
CentOS 6.4 (2.6.32.60-40.art.x86_64)
ASL 3.2.14-31.el6.art
Webmin 1.6.5
Virtualmin 4.02.gpl
Apache 2.2.15
PHP 5.3.3 (mod_fcgid/2.3.7)

Look at the last line:

11660 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf
16332 /usr/bin/newrelic-daemon -A -s -l /var/log/newrelic/newrelic-daemon.log
16760 /usr/bin/php-cgi
22556 /usr/bin/perl /usr/libexec/webmin/status/monitor.pl
45788 /usr/bin/php-cgi
70944 /var/asl/usr/sbin/tortixd
71360 /usr/bin/php-cgi
71436 /usr/bin/php-cgi
73748 /var/asl/usr/sbin/tortixd
76824 /usr/bin/php-cgi
78668 /usr/sbin/httpd
84604 /usr/sbin/httpd
84636 /usr/sbin/httpd
85208 /usr/sbin/httpd
85476 /usr/sbin/httpd
85652 /usr/sbin/httpd
90844 /usr/sbin/httpd
245472 clamd
587480 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
11789388 /usr/libexec/webmin/shell/index.cgi
7380 /usr/sbin/named -u named
8828 /usr/bin/perl -w /usr/bin/psmon --daemon --cron
9360 /var/ossec/bin/ossec-analysisd
9536 /var/ossec/bin/ossec-dbd
16756 /usr/bin/php-cgi
17908 /usr/bin/newrelic-daemon -A -s -l /var/log/newrelic/newrelic-daemon.log
22860 /usr/bin/perl /usr/libexec/webmin/status/monitor.pl
66144 /usr/bin/php-cgi
70356 /var/asl/usr/sbin/tortixd
73328 /var/asl/usr/sbin/tortixd
78648 /usr/sbin/httpd
84456 /usr/sbin/httpd
84484 /usr/sbin/httpd
84488 /usr/sbin/httpd
84512 /usr/sbin/httpd
84588 /usr/sbin/httpd
91184 /usr/sbin/httpd
256496 clamd
617892 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
11601980 /usr/libexec/webmin/shell/index.cgi

2

Howdy,

I’ve asked Jamie for some input on the issue you’re seeing.

However, it appears as if the component that’s using the most memory is the Webmin command shell.

Is there a particular command that is being run from there that seems to trigger the issue you’re seeing?

-Eric

3

nothing running manually. could it be a cron job? there is one cron job that runs every sunday at 0:50 (GMT):
/etc/webmin/virtual-server/backup.pl --id 134137784716897

the last time this memory condition happened was also on a weekend. but the times of the alert are prior to the scheduled cron job:

  1. Monitor on server.com for ‘Free Memory’ has detected that the service has gone down at 10/Aug/2013 14:03
  2. Monitor on server.com for ‘Free Memory’ has detected that the service has gone down at 25/Aug/2013 00:18
4

One way this could happen is if someone used the Command Shell module in Webmin to run a command that outputs hundreds of MB of data. Because Webmin reads this into memory, it could certainly cause that process to explode in size like you saw.

The next Webmin release (version 1.660) will limit the output of commands to 100k by default.

5

how long could such command run for? because in my case nothing had been run via webmin cmd shell for days prior to the incident.

thx jamie

6

If someone ran a command like :

tail -f /var/log/messages

it could potentially run forever, and generate potentially unlimited output.

7

that has not been run. in fact, the only cmd ran a couple days before each time were different and of finite output. are positive i dont need to check the crontab

thx

8

Based on the command that was using all the RAM, it must have been some command run via the Command Shell module that was responsible … and the only way I can see for it to be so RAM intensive would be for it to have produced a large volume of output.

9