Mariadb fails to start

SYSTEM INFORMATION
OS type and version CentOS Linux 7.9.2009
Virtualmin version 7.1

My MariaDB is not starting up. On manually trying to start, the error message is:
Job for mariadb.service failed because a timeout was exceeded. See “systemctl status mariadb.service” and “journalctl -xe” for details.

The mariadb log file just shows these two lines on every attempt to start service or reboot server:

220729 05:07:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
220729 05:07:26 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

Is there any way to recover my databases so if I have to reinstall my VPS I should be able to restore the databases at least.

Hi Neorosis,

Super frustrating that this happend. Do you have any idea what happend before the database server broke down? Did you install updates? was there an outage at the provider? any info like that helps in determining where too look faster.

In the meantime, could you login on the console of this server ( SSH ) and provide the output you get from the two commands below. This will help us determine why the server is not starting.

systemctl status mariadb.service
journalctl -xe

Steven

The hosts suspended the VPS for exceeding bandwidth. Then unsuspended as it was disputed.

First one:

mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Fri 2022-07-29 05:12:27 EDT; 2h 8min ago
  Process: 17942 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 17941 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 17907 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 17941 (code=exited, status=0/SUCCESS)

Jul 29 05:07:25 mresmps.com systemd[1]: Starting MariaDB database server...
Jul 29 05:07:25 mresmps.com mariadb-prepare-db-dir[17907]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Jul 29 05:07:25 mresmps.com mysqld_safe[17941]: 220729 05:07:25 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jul 29 05:07:26 mresmps.com mysqld_safe[17941]: 220729 05:07:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jul 29 05:07:26 mresmps.com systemd[1]: mariadb.service: control process exited, code=exited status=1
Jul 29 05:12:27 mresmps.com systemd[1]: mariadb.service stop-sigterm timed out. Killing.
Jul 29 05:12:27 mresmps.com systemd[1]: Failed to start MariaDB database server.
Jul 29 05:12:27 mresmps.com systemd[1]: Unit mariadb.service entered failed state.
Jul 29 05:12:27 mresmps.com systemd[1]: mariadb.service failed.

2nd:

Jul 29 07:51:45 mresmps.com postfix/smtpd[3014]: warning: unknown[5.34.207.81]: SASL LOGIN authentication failed: authentication failure
Jul 29 07:51:45 mresmps.com postfix/smtpd[3242]: connect from unknown[5.34.207.81]
Jul 29 07:51:45 mresmps.com postfix/smtpd[3014]: disconnect from unknown[5.34.207.81]
Jul 29 07:51:46 mresmps.com saslauthd[468]: pam_unix(smtp:auth): check pass; user unknown
Jul 29 07:51:46 mresmps.com saslauthd[468]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jul 29 07:51:48 mresmps.com sshd[9824]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.173.56  user=root
Jul 29 07:51:48 mresmps.com sshd[9824]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 29 07:51:48 mresmps.com postfix/smtpd[9190]: connect from unknown[5.34.207.209]
Jul 29 07:51:49 mresmps.com saslauthd[468]: DEBUG: auth_pam: pam_authenticate failed: User not known to the underlying authentication module
Jul 29 07:51:49 mresmps.com saslauthd[468]: do_auth         : auth failure: [user=jm123@mresmps.com] [service=smtp] [realm=mresmps.com] [mech=pam] [reason=PAM auth error]
Jul 29 07:51:49 mresmps.com postfix/smtpd[3246]: warning: unknown[5.34.207.81]: SASL LOGIN authentication failed: authentication failure
Jul 29 07:51:49 mresmps.com postfix/smtpd[9199]: connect from unknown[5.34.207.81]
Jul 29 07:51:49 mresmps.com sshd[9829]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=20.194.60.135  user=root
Jul 29 07:51:49 mresmps.com sshd[9829]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 29 07:51:49 mresmps.com sshd[9824]: Failed password for root from 61.177.173.56 port 30972 ssh2
Jul 29 07:51:50 mresmps.com saslauthd[467]: pam_unix(smtp:auth): check pass; user unknown
Jul 29 07:51:50 mresmps.com saslauthd[467]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jul 29 07:51:50 mresmps.com saslauthd[466]: pam_unix(smtp:auth): check pass; user unknown
Jul 29 07:51:50 mresmps.com saslauthd[466]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jul 29 07:51:51 mresmps.com sshd[9824]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 29 07:51:51 mresmps.com sshd[9829]: Failed password for root from 20.194.60.135 port 51820 ssh2
Jul 29 07:51:51 mresmps.com sshd[9829]: Received disconnect from 20.194.60.135 port 51820:11: Bye Bye [preauth]
Jul 29 07:51:51 mresmps.com sshd[9829]: Disconnected from 20.194.60.135 port 51820 [preauth]
Jul 29 07:51:51 mresmps.com postfix/smtpd[3246]: disconnect from unknown[5.34.207.81]
Jul 29 07:51:52 mresmps.com postfix/smtpd[3495]: connect from unknown[5.34.207.209]
Jul 29 07:51:52 mresmps.com saslauthd[467]: DEBUG: auth_pam: pam_authenticate failed: User not known to the underlying authentication module
Jul 29 07:51:52 mresmps.com saslauthd[467]: do_auth         : auth failure: [user=dw@mresmps.com] [service=smtp] [realm=mresmps.com] [mech=pam] [reason=PAM auth error]
Jul 29 07:51:52 mresmps.com postfix/smtpd[9789]: warning: unknown[5.34.207.209]: SASL LOGIN authentication failed: authentication failure
Jul 29 07:51:53 mresmps.com postfix/smtpd[3014]: connect from unknown[5.34.207.81]
Jul 29 07:51:53 mresmps.com saslauthd[466]: DEBUG: auth_pam: pam_authenticate failed: User not known to the underlying authentication module
Jul 29 07:51:53 mresmps.com saslauthd[466]: do_auth         : auth failure: [user=cos@mresmps.com] [service=smtp] [realm=mresmps.com] [mech=pam] [reason=PAM auth error]
Jul 29 07:51:53 mresmps.com postfix/smtpd[9002]: warning: unknown[5.34.207.81]: SASL LOGIN authentication failed: authentication failure
Jul 29 07:51:53 mresmps.com sshd[9824]: Failed password for root from 61.177.173.56 port 30972 ssh2
Jul 29 07:51:54 mresmps.com postfix/smtpd[9789]: disconnect from unknown[5.34.207.209]

Try running:

pkill -9 mysql
systemctl restart mariadb

… which helps almost everytime, unless MySQL library directory got corrupted somehow.

1 Like

Nope didn’t work.

Is it possible to recover all databases at least and I’ll import them after reinstalling everything.

[Solved]

Could not repair mariadb to start normally.

Got all files from /var/lib/mysql folder. Used xampp to read the databases and export to /sql files.
Reinstalled OS and restored all virtual servers

Is there a good guide you could point to for how you did this?

I ran into this same issue a couple days ago. I’ve been able to restore everything other than MySQL on a new VPS and hoping I can follow your lead to getting back from there.

I installed XAMPP portable on my windows machine.
From /var/lib/mysql/data from my VPS I copied the folder that had same name as my database and all the files (ibdata and iblog) into the corresponding /data folder in xampp on my windows machine.
Then ran phpmyadmin on xampp and exported the database as a .sql file. PHPmyadmin had errors in some of it’s preinstalled databases but did not interfere in accessing the database that I needed.

I’m not sure if I’m digging in the wrong spot or pulling the wrong information. I went to the downed VPS and pulled everything in /var/lib/mysql but it didn’t have a data folder in it. It did have folders for each individual database I had though. I was able to find the ibdata1, ib_logfile0 and ib_logfile1 in the downed VPS /var/lib/mysql folder though.

I copied the ibdata and ib_logfiles into xampp/mysql/ data and then dropped the database folders in that same folder. When I try to run that though it fails every time. If I check the logs in xampp I get:

 2022-08-02 11:52:58 0 [ERROR] InnoDB: Invalid flags 0x4800 in C:\xampp\mysql\data\ibdata1
 2022-08-02 11:52:58 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
 2022-08-02 11:52:58 0 [Note] InnoDB: Starting shutdown...
 2022-08-02 11:52:59 0 [ERROR] Plugin 'InnoDB' init function returned error.
 2022-08-02 11:52:59 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 2022-08-02 11:52:59 0 [Note] Plugin 'FEEDBACK' is disabled.
 2022-08-02 11:52:59 0 [ERROR] Unknown/unsupported storage engine: InnoDB
 2022-08-02 11:52:59 0 [ERROR] Aborting

Copy only the Database folder and 5 single files. Do not copy ALL folders. See image what I copied. Stop the Server on XAMPP as you copy the files.

image