Cmd to restart database through the service monitor?

christophera · January 18, 2023, 4:35pm

Hi,

I need to automatically restart my database if mysql is bumped off for some reason.

I have the service being monitored and I receive the email if it goes offline.

I have the command to run if it goes down set to:

systemctl start mariadb

If I run that command through the Virtualmin command shell, it restarts the database.

But for some reason, the system monitor is not restarting the database when I am sent the email.

Any thoughts on where I’m going wrong here?

Thanks,

Chris

Operating system Ubuntu Linux 18.04.6
Webmin version 2.011 Usermin version 1.860
Virtualmin version 7.5

GENLTD · January 20, 2023, 8:44am

IMO Mariadb shouldn’t be quitting on you, and you should really look into the cause rather than just restart it.

In its default configuration (and there’s no guarantee that’s what you’re using) MariaDB should write an error log to /var/log/mariadb/mariadb.err so go look in that and see if there’s any indicators? It could be out of ram (configuration) or hitting a disk space issue, both of which are easy fixes.

If you’ve not got mariadb logging its errors, it will probably produce something useful in
journalctl -xe so go take a look when it falls over.

There are lots of other tools for debugging MariaDB, but if its abnormally terminating then go check these first, in most cases it’ll be an easy fix and then you won’t need to worry about restarting it, but, if you must then you can use systemd to do it. Check the configuration file for MariaDB which is probably something like

/etc/systemd/system/multi-user.target.wants/mariadb.service

and add Restart=always in [service]

then reload with systemctl daemon-reload and you should be sorted.

Randomz · January 20, 2023, 1:22pm

I had a play and it appears that you should have the email address in the setup and then don’t use the “also send email to”.

christophera · January 27, 2023, 8:42pm

Thank you both.

Agreed, I’d love to identify the problem. I have one wordpress site on the server, I don’t know if the problem is with the theme or what - here’s the error log from when it failed today. Is this a memory issue?

(mysql, not mariadb, sorry, mariadb is on my centos server)

2023-01-27T20:23:46.093942Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-01-27T20:23:46.123571Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.41-0ubuntu0.18.04.1) starting as process 32361 ...
2023-01-27T20:23:46.192838Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-01-27T20:23:46.192881Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-01-27T20:23:46.192887Z 0 [Note] InnoDB: Uses event mutexes
2023-01-27T20:23:46.192892Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-01-27T20:23:46.192898Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-01-27T20:23:46.192908Z 0 [Note] InnoDB: Using Linux native AIO
2023-01-27T20:23:46.195018Z 0 [Note] InnoDB: Number of pools: 1
2023-01-27T20:23:46.202990Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-01-27T20:23:46.208232Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-01-27T20:23:46.209413Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2023-01-27T20:23:46.209434Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2023-01-27T20:23:46.209441Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-01-27T20:23:46.209449Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-01-27T20:23:46.209455Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-01-27T20:23:46.209460Z 0 [ERROR] Failed to initialize builtin plugins.
2023-01-27T20:23:46.209465Z 0 [ERROR] Aborting

2023-01-27T20:23:46.213475Z 0 [Note] Binlog end
2023-01-27T20:23:46.214232Z 0 [Note] Shutting down plugin 'CSV'
2023-01-27T20:23:46.215255Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2023-01-27T20:23:46.974134Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-01-27T20:23:46.976532Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.41-0ubuntu0.18.04.1) starting as process 32399 ...
2023-01-27T20:23:47.036938Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-01-27T20:23:47.036980Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-01-27T20:23:47.036985Z 0 [Note] InnoDB: Uses event mutexes
2023-01-27T20:23:47.036989Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-01-27T20:23:47.036994Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-01-27T20:23:47.037002Z 0 [Note] InnoDB: Using Linux native AIO
2023-01-27T20:23:47.038917Z 0 [Note] InnoDB: Number of pools: 1
2023-01-27T20:23:47.044662Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-01-27T20:23:47.046187Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-01-27T20:23:47.046798Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2023-01-27T20:23:47.046816Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2023-01-27T20:23:47.046823Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-01-27T20:23:47.046841Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-01-27T20:23:47.046847Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-01-27T20:23:47.046852Z 0 [ERROR] Failed to initialize builtin plugins.
2023-01-27T20:23:47.046855Z 0 [ERROR] Aborting

2023-01-27T20:23:47.053043Z 0 [Note] Binlog end
2023-01-27T20:23:47.053129Z 0 [Note] Shutting down plugin 'CSV'
2023-01-27T20:23:47.053500Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2023-01-27T20:23:47.748847Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-01-27T20:23:47.759660Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.41-0ubuntu0.18.04.1) starting as process 32439 ...
2023-01-27T20:23:47.818603Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-01-27T20:23:47.818638Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-01-27T20:23:47.818660Z 0 [Note] InnoDB: Uses event mutexes
2023-01-27T20:23:47.818665Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-01-27T20:23:47.818669Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-01-27T20:23:47.818678Z 0 [Note] InnoDB: Using Linux native AIO
2023-01-27T20:23:47.821962Z 0 [Note] InnoDB: Number of pools: 1
2023-01-27T20:23:47.828890Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-01-27T20:23:47.830352Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-01-27T20:23:47.830942Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2023-01-27T20:23:47.830957Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2023-01-27T20:23:47.830962Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-01-27T20:23:47.830970Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-01-27T20:23:47.830975Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-01-27T20:23:47.832810Z 0 [ERROR] Failed to initialize builtin plugins.
2023-01-27T20:23:47.832830Z 0 [ERROR] Aborting

2023-01-27T20:23:47.837121Z 0 [Note] Binlog end
2023-01-27T20:23:47.839067Z 0 [Note] Shutting down plugin 'CSV'
2023-01-27T20:23:47.841351Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2023-01-27T20:23:48.534128Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-01-27T20:23:48.555204Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.41-0ubuntu0.18.04.1) starting as process 32476 ...
2023-01-27T20:23:48.650610Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-01-27T20:23:48.650659Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-01-27T20:23:48.650666Z 0 [Note] InnoDB: Uses event mutexes
2023-01-27T20:23:48.650673Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-01-27T20:23:48.650679Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-01-27T20:23:48.650689Z 0 [Note] InnoDB: Using Linux native AIO
2023-01-27T20:23:48.656890Z 0 [Note] InnoDB: Number of pools: 1
2023-01-27T20:23:48.659788Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-01-27T20:23:48.668163Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-01-27T20:23:48.670388Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2023-01-27T20:23:48.670416Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2023-01-27T20:23:48.670424Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-01-27T20:23:48.670434Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-01-27T20:23:48.671034Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-01-27T20:23:48.671278Z 0 [ERROR] Failed to initialize builtin plugins.
2023-01-27T20:23:48.671290Z 0 [ERROR] Aborting

2023-01-27T20:23:48.671691Z 0 [Note] Binlog end
2023-01-27T20:23:48.672390Z 0 [Note] Shutting down plugin 'CSV'
2023-01-27T20:23:48.673250Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

GENLTD · February 2, 2023, 3:08pm

[Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M

2023-01-27T20:23:46.209434Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool

So you’re asking mysql to allocate 128M to the buffer pool, which is fine, but then it fails to secure that amount of buffer.

This would suggest a memory low condition. Check /proc/meminfo for available memory, or use a tool like smem.

You can change the amount of buffer in /etc/my.cnf.d/server.cnf (or similar depending on distribution). with innodb_buffer_pool_size=XXX but you should look to secure more memory rather than set it lower imo.

Hope that helps,
Rich

Joe · February 2, 2023, 4:35pm

Restarting a database that never even successfully starts is a pointless waste of time.

Presumably when it stops (on the occasions where it does start), it is because the OOM killer killed it. Which guarantees data loss and database corruption at some point. You’re not addressing the right problems here by trying to restart the database automatically.

system · April 3, 2023, 4:35pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.