URGENT | MariaDB crashed after hitting "set as default services certificate" button

[azbce]

SYSTEM INFORMATION
OS type and version	Ubuntu Linux 20.04.6
Webmin version	2.021
Virtualmin version	7.7

Hello!

I have 5 different virtual servers with 5 different domains on my server.

On the Virtualmin > Server Configuration > SSL Certificate page, under one of my domains, I hit the “set as default services certificate” button and my mariaDB failed to restart and made the following error:

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

Running the command systemctl status mariadb.service I get the following return:

● mariadb.service - MariaDB 10.3.38 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-04-27 22:19:41 MST; 11s ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 12243 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 12244 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 12256 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
    Process: 12315 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 12315 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"

When attempting to restart the mariaDB from Webmin > Server > MariaDB Server I got the following error:

DBI connect failed : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

There appears to be no /var/run/mysqld/mysqld.sock in the file system anywhere.

My mysql/my.cnf

# The MariaDB configuration file
#
# The MariaDB/MySQL tools read configuration files in the following order:
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.

#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

Please help. I have multiple high traffic sites on this server and they are all offline.

Thank you!
Kevin

[jimr1]

have you ran

service mysql start && service mysql status

as root ?
it should be the same but it’s worth a go …
I have the same setup as you and dont see

@azbce

$MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 12315 (code=exited, status=1/FAILURE)

on the command line what do they do ?

[azbce]

Same exact return.

[jimr1]

from a working system

as you can see there is a total difference to the startup commands … have you altered anything at all in the maria configuration files ?

[azbce]

No, I have not altered anything. I just hit the button and everything went offline.

[jimr1]

ok how much ram do you have ?

[azbce]

I have 5 gigabytes of RAM and have never had an issue. Until I pressed the button, that is.

[jimr1]

so what does the command

free -h

show ? you should see something like this

[azbce]

              total        used        free      shared  buff/cache   available
Mem:          4.9Gi       1.9Gi       990Mi        19Mi       2.0Gi       2.7Gi
Swap:          89Mi          0B        89Mi

[jimr1]

look enough …
but look at this post MariaDB randomly tuned off and wont turn back on cauisng my wordpress site to fail to lod it turned out to be the deletion of 2 files in the end … It could be these files are corrupt

/var/lib/mysql/ib_logfile0

and

/var/lib/mysql/ib_logfile1
but if you do delete these files make sure you have copied the whole directory elsewhere first

[azbce]

I’m not sure how that would happen. Seems completely unrelated to what I did. I sure hope I didn’t delete my database.

How do you suggest I move forward?

[jimr1]

you could have ran out of ram …
have a look in /var/lib/mysql and see if your databases are there
if everything looks good
do

cp -r /var/lib/mysql /var/lib/mysql.old

this will save a copy of the mysql data so it is safe
then from /var/lib/mysql delete ib_logfile0 then run

service mysql start && service mysql status

again and see if maria starts up

[azbce]

Ok databases look good! Whew!

I saved a copy.

Deleted the file, ran the command, and it didn’t start.

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

[jimr1]

was it the same error ?
running service mysql status should have given more results

[azbce]

@jimr1

service mysql status

root@host:~# service mysql status
● mariadb.service - MariaDB 10.3.38 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-04-28 00:41:42 MST; 1min 50s ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 51338 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 51351 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 51353 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
    Process: 51412 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 51412 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"

Apr 28 00:41:40 host.azbackroads.com systemd[1]: Starting MariaDB 10.3.38 database server...
Apr 28 00:41:42 host.azbackroads.com systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Apr 28 00:41:42 host.azbackroads.com systemd[1]: mariadb.service: Failed with result 'exit-code'.
Apr 28 00:41:42 host.azbackroads.com systemd[1]: Failed to start MariaDB 10.3.38 database server.

[jimr1]

ok now look at your config files in /etc/mysql and /etc/mysql/mariadb.conf.d the latter being more important

[azbce]

[azbce]

Thank you so much for helping me. I really appreciate you!

[jimr1]

no worries, show the contents of 50-server.cnf

[azbce]

[server]

[mysqld]

user                    = mysql
pid-file                = /run/mysqld/mysqld.pid
socket                  = /run/mysqld/mysqld.sock

basedir                 = /usr
datadir                 = /var/lib/mysql
tmpdir                  = /tmp
lc-messages-dir         = /usr/share/mysql

bind-address            = 127.0.0.1

log_error = /var/log/mysql/error.log

expire_logs_days        = 10

character-set-server  = utf8mb4
collation-server      = utf8mb4_general_ci
innodb_file_per_table = 1
max_allowed_packet = 64M
ssl_ca = /etc/mysql/mysql-ssl.ca
ssl_key = /etc/mysql/mysql-ssl.key
ssl_cert = /etc/mysql/mysql-ssl.cert

[embedded]

[mariadb]

[mariadb-10.3]