Webmin Virtual Memory Issue: MySQL Processes Dying & Website Downtime

Hello,

I’m experiencing an issue with Webmin’s virtual memory becoming full, even though ample physical RAM remains available. This leads to MySQL processes being killed, causing all my websites to go down.

I’ve already taken steps to address the issue:

  • Tripled virtual memory: I significantly increased the virtual memory allocation, but the problem persists.
  • Doubled RAM: I also upgraded the server’s RAM capacity, but it hasn’t resolved the issue.

This indicates that the root cause might lie beyond simply increasing memory resources. I’m looking for guidance on potential solutions.

Any insights or suggestions on how to diagnose and resolve this persistent virtual memory issue would be greatly appreciated.

Thanks in advance for your help!

SYSTEM INFORMATION
Webmin version 2.111
Usermin version 2.010
Virtualmin version 7.10.0
Kernel and CPU Linux 5.4.0-182-generic on x86_64
Processor information Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, 4 cores
Real memory 8.22 GiB used / 6.91 GiB cached / 15.59 GiB total
Virtual memory 1.45 GiB used / 5.99 GiB total
All installed packages are up to date

Bleow my .cnf configuration:

[mysqld]

user		= mysql

bind-address = 127.0.0.1
mysqlx-bind-address	= 127.0.0.1

myisam-recover-options  = BACKUP

log_error = /var/log/mysql/error.log
max_allowed_packet = 32M
max_connections = 200
table_open_cache = 2000
key_buffer_size = 4096M
innodb_buffer_pool_size = 2048M

myisam_sort_buffer_size = 32M
myisam_use_mmap = ON

Below is the Webmin error log:

[18/Apr/2024:06:33:11 +0100] [192.168.1.26] /mysql/save_cnf.cgi : Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details. 
[18/Apr/2024:06:51:49 +0100] [192.168.1.26] /mysql/ : SQL show index from `strflttrck_wc_admin_note_actions` failed : Lost connection to MySQL server during query 
timeout at /usr/share/webmin/miniserv.pl line 4755.
Failed to list unit files: Connection timed out
Failed to list unit files: Connection timed out
timeout at /usr/share/webmin/miniserv.pl line 4755.
timeout at /usr/share/webmin/miniserv.pl line 4755.
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Failed to list unit files: Connection timed out
[18/Apr/2024:09:19:35 +0100] [192.168.1.26] /virtual-server/enable_domain.cgi?subservers=0&dom=16802178755937&confirm=Yes, Enable It : Virtual server is not disabled
[19/Apr/2024:19:30:20 +0100] Shutting down
[19/Apr/2024:19:30:20 +0100] Shutting down
[19/Apr/2024:19:31:16 +0100] miniserv.pl started
[19/Apr/2024:19:31:16 +0100] IPv6 support enabled
[19/Apr/2024:19:31:16 +0100] Using MD5 module Digest::MD5
[19/Apr/2024:19:31:16 +0100] Using SHA512 module Crypt::SHA
[19/Apr/2024:19:31:16 +0100] PAM authentication enabled
[19/Apr/2024:20:17:49 +0100] Shutting down
[19/Apr/2024:20:17:49 +0100] Shutting down
[19/Apr/2024:20:18:21 +0100] miniserv.pl started
[19/Apr/2024:20:18:21 +0100] IPv6 support enabled
[19/Apr/2024:20:18:21 +0100] Using MD5 module Digest::MD5
[19/Apr/2024:20:18:21 +0100] Using SHA512 module Crypt::SHA
[19/Apr/2024:20:18:21 +0100] PAM authentication enabled
[19/Apr/2024:21:53:11 +0100] Shutting down
[19/Apr/2024:21:53:11 +0100] Shutting down
[19/Apr/2024:21:53:36 +0100] miniserv.pl started
[19/Apr/2024:21:53:36 +0100] IPv6 support enabled
[19/Apr/2024:21:53:36 +0100] Using MD5 module Digest::MD5
[19/Apr/2024:21:53:36 +0100] Using SHA512 module Crypt::SHA
[19/Apr/2024:21:53:36 +0100] PAM authentication enabled
Failed to list unit files: Connection timed out
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Failed to list unit files: Connection timed out
timeout at /usr/share/webmin/miniserv.pl line 4755.
Failed to list unit files: Connection timed out
timeout at /usr/share/webmin/miniserv.pl line 4755.
Failed to list unit files: Connection timed out
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
/usr/bin/mysqlshow: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
[22/Apr/2024:19:38:53 +0100] Reloading configuration
[22/Apr/2024:19:39:15 +0100] Reloading configuration
[22/Apr/2024:19:39:16 +0100] Restarting
[22/Apr/2024:19:39:18 +0100] miniserv.pl started
[22/Apr/2024:19:39:18 +0100] IPv6 support enabled
[22/Apr/2024:19:39:18 +0100] Using MD5 module Digest::MD5
[22/Apr/2024:19:39:18 +0100] Using SHA512 module Crypt::SHA
[22/Apr/2024:19:39:18 +0100] PAM authentication enabled
[22/Apr/2024:20:26:09 +0100] Shutting down
[22/Apr/2024:20:26:45 +0100] miniserv.pl started
[22/Apr/2024:20:26:45 +0100] IPv6 support enabled
[22/Apr/2024:20:26:45 +0100] Using MD5 module Digest::MD5
[22/Apr/2024:20:26:45 +0100] Using SHA512 module Crypt::SHA
[22/Apr/2024:20:26:45 +0100] PAM authentication enabled
[22/Apr/2024:21:10:39 +0100] Shutting down
[22/Apr/2024:21:10:39 +0100] Shutting down
[22/Apr/2024:21:12:48 +0100] miniserv.pl started
[22/Apr/2024:21:12:48 +0100] IPv6 support enabled
[22/Apr/2024:21:12:48 +0100] Using MD5 module Digest::MD5
[22/Apr/2024:21:12:48 +0100] Using SHA512 module Crypt::SHA
[22/Apr/2024:21:12:48 +0100] PAM authentication enabled
[24/Apr/2024:00:04:16 +0100] Reloading configuration
[24/Apr/2024:00:05:51 +0100] [192.168.1.26] /virtual-server/domain_setup.cgi : A MySQL database named teltonikatrackers is already owned by virtual server najmchaarani.com 
[24/Apr/2024:00:07:26 +0100] Reloading configuration
[24/Apr/2024:00:09:08 +0100] Reloading configuration
[24/Apr/2024:01:09:42 +0100] Reloading configuration
[24/Apr/2024:01:10:20 +0100] Reloading configuration
[25/Apr/2024:01:33:54 +0100] Reloading configuration
[25/Apr/2024:01:35:27 +0100] Reloading configuration
Failed to list unit files: Connection timed out
timeout at /usr/share/webmin/miniserv.pl line 4755.
Failed to list unit files: Connection timed out
[29/Apr/2024:06:28:03 +0100] Shutting down
[29/Apr/2024:06:28:48 +0100] miniserv.pl started
[29/Apr/2024:06:28:48 +0100] IPv6 support enabled
[29/Apr/2024:06:28:48 +0100] Using MD5 module Digest::MD5
[29/Apr/2024:06:28:48 +0100] Using SHA512 module Crypt::SHA
[29/Apr/2024:06:28:48 +0100] PAM authentication enabled
[01/May/2024:04:15:11 +0100] Reloading configuration
[01/May/2024:04:16:13 +0100] Reloading configuration
[08/May/2024:07:28:16 +0100] Shutting down
[08/May/2024:07:28:17 +0100] Shutting down
[08/May/2024:07:28:42 +0100] miniserv.pl started
[08/May/2024:07:28:42 +0100] IPv6 support enabled
[08/May/2024:07:28:42 +0100] Using MD5 module Digest::MD5
[08/May/2024:07:28:42 +0100] Using SHA512 module Crypt::SHA
[08/May/2024:07:28:42 +0100] PAM authentication enabled
Failed to list unit files: Connection timed out
[15/May/2024:00:14:55 +0100] Shutting down
[15/May/2024:00:15:29 +0100] miniserv.pl started
[15/May/2024:00:15:29 +0100] IPv6 support enabled
[15/May/2024:00:15:29 +0100] Using MD5 module Digest::MD5
[15/May/2024:00:15:29 +0100] Using SHA512 module Crypt::SHA
[15/May/2024:00:15:29 +0100] PAM authentication enabled

I’m posting the 2nd screenshot here be cause of the post attachment restrictions.

Webmin does not have virtual memory. Your operating system does. The Webmin log is irrelevant, and what Webmin is doing is irrelevant (it’s certainly not what’s eating up all of your memory).

That’s crazy for a system with only 8GB of real memory. Running out of memory can only make MariaDB slower.

But, you need to look at what’s using memory. top or htop is probably a good start for that.

Thanks, Joe for your prompt reply!

As on the screenshot, the system had only used 65% of the RAM and it started killing the MySQL processes due to “out of memory”.

key_buffer_size = 4096M
innodb_buffer_pool_size = 2048M

What do you suggest for 16GB system in this case, running 5 wordpress websites?

I also noticed that this happens only when there are write operations on the databases, and using the TOP command most of the memory is taken by MySQL.

And thanks again for your time.

top - 06:27:51 up 1 day,  6:12,  1 user,  load average: 0.27, 0.39, 0.65
Tasks: 465 total,   2 running, 463 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.9 us, 17.7 sy,  0.0 ni, 74.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  15966.4 total,   5219.2 free,   4752.9 used,   5994.3 buff/cache
MiB Swap:   6144.0 total,   5977.6 free,    166.4 used.   9592.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 178679 mysql     20   0 4289416 872992  39056 S   0.3   5.3   0:20.67 mysqld
  61326 contact+  20   0  386580 209904 143820 S   0.0   1.3  10:38.64 php-fpm7.4
 185778 teltoni+  20   0  361652 204288 146960 S   0.0   1.2   0:03.38 php-cgi8.3
 185863 teltoni+  20   0  287804 199676 142292 S   0.0   1.2   0:03.31 php-cgi8.3
   4951 contact+  20   0  389948 196680 130520 S   0.0   1.2  14:58.52 php-fpm7.4
   2228 contact+  20   0  368512 193956 132532 S   0.0   1.2  15:11.86 php-fpm7.4
   1806 contact+  20   0  365248 191100 131472 S   0.0   1.2  15:12.17 php-fpm7.4
    971 contact+  20   0  380340 190804 133044 S   0.0   1.2  15:21.16 php-fpm7.4
 185825 teltoni+  20   0  353184 188096 139292 S   0.0   1.2   0:01.15 php-cgi8.3
 185861 teltoni+  20   0  351124 182252 136148 S   0.0   1.1   0:00.95 php-cgi8.3
 185780 teltoni+  20   0  279288 179952 131204 S   0.0   1.1   0:00.94 php-cgi8.3
 185783 teltoni+  20   0  279288 179952 131204 S   0.0   1.1   0:00.93 php-cgi8.3
 185827 teltoni+  20   0  278572 179364 131444 S   0.0   1.1   0:00.92 php-cgi8.3
 185784 teltoni+  20   0  278572 179232 131204 S   0.0   1.1   0:00.90 php-cgi8.3
 185830 teltoni+  20   0  279288 179196 131508 S   0.0   1.1   0:00.94 php-cgi8.3
 185828 teltoni+  20   0  278508 178416 131508 S   0.0   1.1   0:00.90 php-cgi8.3
 185860 teltoni+  20   0  277284 177004 131420 S   0.0   1.1   0:00.94 php-cgi8.3
 185865 teltoni+  20   0  277276 176996 131420 S   0.0   1.1   0:00.93 php-cgi8.3
 181052 teltoni+  20   0  338944 171464 136504 S   0.0   1.0   0:02.64 php-cgi8.3
 181056 teltoni+  20   0  347884 165956 136536 S   0.0   1.0   0:00.91 php-cgi8.3
 185824 teltoni+  20   0  264020 162348 128924 S   0.0   1.0   0:02.52 php-cgi8.3
 181097 teltoni+  20   0  264020 162080 128480 S   0.0   1.0   0:02.53 php-cgi8.3
 181100 teltoni+  20   0  332548 159636 131420 S   0.0   1.0   0:00.49 php-cgi8.3
 180183 gpstrac+  20   0  266800 158224 121180 S   0.0   1.0   0:02.54 php-cgi8.2
 181102 teltoni+  20   0  330496 156060 129572 S   0.0   1.0   0:00.38 php-cgi8.3
 181105 teltoni+  20   0  330496 156060 129572 S   0.0   1.0   0:00.37 php-cgi8.3
 181101 teltoni+  20   0  330520 153468 127064 S   0.0   0.9   0:00.37 php-cgi8.3
 181054 teltoni+  20   0  330560 152248 126688 S   0.0   0.9   0:00.39 php-cgi8.3

Oops, I misread. But, that’s still too much. You’re not getting any value out those two specific caches being outrageously huge.

Number of sites isn’t relevant, it’s what they’re doing and how heavily loaded they are.

That’s not how it happens. If you’d had only 65% of RAM used, the kernel would not have killed any processes. But, you’ve told Mysql to allocate 6GB (plus probably another GB or two of the usual stuff) right out of the gate.

The OOM killer kicked in because you ran out of memory, not because 65% of memory was used. Processes wanted 100% of your memory, and the kernel did the best thing it knew how to do to keep as much as possible running. When you run out of memory, it is a catastrophe, something has to break.

I’m sure it’s not just write operations in general, but one (or more) very demanding write operations. You might turn on slowquery logging to see what the big consumers of resources are. Whether the slow ones are the big ones may not be directly correlated, but I wouldn’t be surprised if they are…large writes take a long time, and if there’s a lot of little queries involved in a huge transaction it might take a lot of memory, too, during that time. A badly programmed WordPress plugin might be a huge consumer of resources for what seems like little actual result. It’s easy to be sloppy.

I have

key_buffer_size = 128M
max_connections = 140
innodb_buffer_pool_size = 1200M

My server: 16 GB RAM. Ten WordPress.

And I have: Memcached, Opcache activated on the server with PHP-FPM. Also, WP super cache in WordPress and Cloudflare.

Try to test your database with mysqlturner. You can use apache2buddy to check Apache parameters and ps_mem to see the core memory usage for the programs in your server.

I hope this help to you.

What stick out if 3691 Running Processes, that’s crazy. I have 14 virtualservers and 9 have wordpress and I only have 244 running process. The sites don’t have much traffic.
Also 4Gig of Ram and its using 2 or 3% CPU normally but it does raise depending if I jump on in webmin.
RAM is around 50-60%

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.