MariaDB silently crashing. Why? No Notifications

SYSTEM INFORMATION
OS type and version Ubuntu 22.04
Webmin version 2.111
Virtualmin version 7.10.0 Pro
Related packages not sure

@staff

Hi there, I have a problem that MariaDB is suddenly crashing all the time, and the logs don’t reveal why…
I’ve had LAMP servers for a decade where this never happened, so I’m not sure why it’s a) crashing and b) not being restarted by some Cron job of Virtualmin.

This is the default setting, should I change anything?

The latest log is:

MariaDB error message
The full MariaDB error message was : DBI connect failed : Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (111) ?

Once I click on “Start Maria DB”, it’s all fine again.
Now, from the syslog I understand it’s got to do with memory. The machine has 6GB RAM and shouldn’t really be consuming all RAM for the DB server… I’ve disabled CLAMAV as well to reduce some memory usage… What do you suggest I do?

May  7 23:35:17 vm systemd[1]: mariadb.service: Consumed 6min 22.339s CPU time.
May  7 23:35:17 vm systemd[1]: mariadb.service: Failed with result 'oom-kill'.
May  7 23:35:17 vm systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
May  7 23:35:17 vm systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
May  7 23:35:17 vm kernel: [1520127.379198] Out of memory: Killed process 756980 (mariadbd) total-vm:2129984kB, anon-rss:332100kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:944kB oom_score_adj:0
May  7 23:35:17 vm kernel: [1520127.378577] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=fail2ban.service,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=756980,uid=113
May  7 23:35:17 vm kernel: [1520127.378091] [ 756980]   113 756980   532496    83025   966656        0             0 mariadbd
May  5 18:31:13 vm mariadbd[756980]: 2024-05-05 18:31:13 0 [Note] InnoDB: Buffer pool(s) load completed at 240505 18:31:13

Search for MariaDB you will find a number of topics with this problem but it is important to note that this is not common and most Webmin/Virtualmin users do not experience it.

Something is crashing your database server by sending it OOM, it is being restarted but that is not resolving the cause. Whatever is driving it OOM needs to be fixed.

You already know “why”. Your system is running out of memory.

You need to look at memory usage to figure out which processes are consuming the most memory. Just looking at top or htop sorted by memory usage is probably sufficient, but it may be that the big memory user is a cronjob or only happens when visiting one page on a website you’re hosting (though I think you’d have to have altered the PHP memory limits to allow one page to consume a huge amount of memory).

1 Like

How many sites are you hosting?

the number of sites is probably irrelevant (though any additional information can’t be a bad thing) - it is what is on those sites that is consuming memory that is the problem.

Webmin/Virtualmin runs perfectly well for nearly everyone with MariaDB.

1 Like

While we may have been behind the curve in php management, one of the big reasons we switched all our sites to php-fpm a year or two ago is fpm has the option to control how many child processes will be spawned — before this some of my servers would have 10s to 100s of child php processes that triggered OOM operations.

Not sure if this will help any, but wanted to just throw it out there :slight_smile:

that is a good point. But who is NOT using the recommended php-fpm execution mode. (if that is what is triggering these OOM issues then perhaps there should be a stronger health warning when the option is taken to not use php-fpm)

Around 15, but I don’t think the amount is the issue

exactly. Confirm I’m using FPM and I don’t think that’s the issue. Have reduced some of the PHP Variables around memory usage and am monitoring… Thanks!

Thanks for the suggestion. I use MEMCACHED but with the default 65M so I don’t think that’s it (it appears in HTOP). I’ve reduced some of the PHP FPM memory limits and time-outs to be a bit more restrictive.

Observing now and will revert once system is stable again. Thanks to everyone for their suggestions, much appreciated!

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.