MySQL / MariaDB stopped overnight, and now will not start

SYSTEM INFORMATION
OS type and version Rocky Linux 9.4
Virtualmin version 7.10.0 Pro

I have 8 sites on my server that back up overnight on Thursday night / Friday morning, running either WordPress or Joomla. This morning when I checked my messages, I had notifications that the first 6 backed up without issue, then a mySQL server down message from Virtualmin itself. That I know of, nothing changed overnight, and there are no package updates or other notices that would indicate a change.

I tried restarting the service, and got an error (full details below). I rebooted the VM that is running virtualmin, no change. I rebooted the host for the VM (as noted in a Stack Exchange post), still nothing.

What can I do to get MariaDB back up and running? Full error messages are below.

(I am leaving for the weekend in a few hours, so I may not get back to this until Monday, possibly Sunday night.)

Error from Dashboard / MariaDB Module:

Error!  MariaDB is not running on your system - database list could not be retrieved.

MariaDB error message
The full MariaDB error message was : DBI connect failed : Can't connect to local server through socket '/var/lib/mysql/mysql.sock' (111) ?

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.

Results from running systemctl status mariadb.service:

root@www:~# systemctl status mariadb.service
Γ— mariadb.service - MariaDB 10.5 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Fri 2024-07-05 11:51:08 EDT; 9s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 8859 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
    Process: 8885 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
    Process: 8920 ExecStart=/usr/libexec/mariadbd --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE)
   Main PID: 8920 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 8.452s

Jul 05 11:51:06 www.aiskon.net systemd[1]: Starting MariaDB 10.5 database server...
Jul 05 11:51:06 www.aiskon.net mariadb-check-socket[8859]: Socket file /var/lib/mysql/mysql.sock exists.
Jul 05 11:51:06 www.aiskon.net mariadb-check-socket[8859]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, so it will be removed autom>
Jul 05 11:51:06 www.aiskon.net mariadb-prepare-db-dir[8885]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Jul 05 11:51:06 www.aiskon.net mariadb-prepare-db-dir[8885]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Jul 05 11:51:08 www.aiskon.net systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 11:51:08 www.aiskon.net systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jul 05 11:51:08 www.aiskon.net systemd[1]: Failed to start MariaDB 10.5 database server.
Jul 05 11:51:08 www.aiskon.net systemd[1]: mariadb.service: Consumed 8.452s CPU time.
lines 1-21/21 (END)

root@www:/var/lib/mysql# 

Results from running journalctl -xeu mariadb.service:

root@www:/var/lib/mysql# journalctl -xeu mariadb.service
β–‘β–‘ 
β–‘β–‘ The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
Jul 05 11:51:08 www.aiskon.net systemd[1]: Failed to start MariaDB 10.5 database server.
β–‘β–‘ Subject: A start job for unit mariadb.service has failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ A start job for unit mariadb.service has finished with a failure.
β–‘β–‘ 
β–‘β–‘ The job identifier is 8469 and the job result is failed.
Jul 05 11:51:08 www.aiskon.net systemd[1]: mariadb.service: Consumed 8.452s CPU time.
β–‘β–‘ Subject: Resources consumed by unit runtime
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ The unit mariadb.service completed and consumed the indicated resources.
Jul 05 11:53:24 www.aiskon.net systemd[1]: Starting MariaDB 10.5 database server...
β–‘β–‘ Subject: A start job for unit mariadb.service has begun execution
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ A start job for unit mariadb.service has begun execution.
β–‘β–‘ 
β–‘β–‘ The job identifier is 9057.
Jul 05 11:53:24 www.aiskon.net mariadb-prepare-db-dir[10269]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Jul 05 11:53:24 www.aiskon.net mariadb-prepare-db-dir[10269]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Jul 05 11:53:25 www.aiskon.net systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
β–‘β–‘ Subject: Unit process exited
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ An ExecStart= process belonging to unit mariadb.service has exited.
β–‘β–‘ 
β–‘β–‘ The process' exit code is 'exited' and its exit status is 1.
Jul 05 11:53:25 www.aiskon.net systemd[1]: mariadb.service: Failed with result 'exit-code'.
β–‘β–‘ Subject: Unit failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
Jul 05 11:53:25 www.aiskon.net systemd[1]: Failed to start MariaDB 10.5 database server.
β–‘β–‘ Subject: A start job for unit mariadb.service has failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://wiki.rockylinux.org/rocky/support
β–‘β–‘ 
β–‘β–‘ A start job for unit mariadb.service has finished with a failure.
β–‘β–‘ 
β–‘β–‘ The job identifier is 9057 and the job result is failed.

root@www:/var/lib/mysql# 

Anything interesting in /var/log/mariadb/mariadb.log?

1 Like

Hello,
I am also having problems with my MariaDB. Maybe it might be related. The problem started recently out of nowhere despite several months of stability. However, I am able to restart MariaDB from the Virtualmin dashboard.

Jul 05 19:31:44  systemd[1]: Starting MariaDB 10.6.18 database server...
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] Starting MariaDB 10.6.18-MariaDB-0ubuntu0.22.04.1 source revision 887bb3f73555ff8a50138a580ca8308b9b5c069c as process 1166146
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Number of pools: 1
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Completed initialization of buffer pool
Jul 05 19:31:44  mariadbd[1166146]: 2024-07-05 19:31:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=4414292227,4419129689
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: To recover: 312 pages
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: 128 rollback segments are active.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: 10.6.18 started; log sequence number 4432007518; transaction id 2201408
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] Plugin 'FEEDBACK' is disabled.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] Server socket created on IP: '127.0.0.1'.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] /usr/sbin/mariadbd: ready for connections.
Jul 05 19:31:45  mariadbd[1166146]: Version: '10.6.18-MariaDB-0ubuntu0.22.04.1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Ubuntu 22.04
Jul 05 19:31:45  systemd[1]: Started MariaDB 10.6.18 database server.
Jul 05 19:31:45  /etc/mysql/debian-start[1166177]: Upgrading MySQL tables if necessary.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 3 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 0 [Note] InnoDB: Buffer pool(s) load completed at 240705 19:31:45
Jul 05 19:31:45  /etc/mysql/debian-start[1166181]: Looking for 'mariadb' as: /usr/bin/mariadb
Jul 05 19:31:45  /etc/mysql/debian-start[1166181]: Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
Jul 05 19:31:45  /etc/mysql/debian-start[1166181]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Jul 05 19:31:45  /etc/mysql/debian-start[1166181]: FATAL ERROR: Upgrade failed
Jul 05 19:31:45  /etc/mysql/debian-start[1166189]: Checking for insecure root accounts.
Jul 05 19:31:45  mariadbd[1166146]: 2024-07-05 19:31:45 4 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Jul 05 19:31:45  debian-start[1166192]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)```

I don’t have that log in my setup or at least not in that directory. Strange right?

No, it’s not strange. Different distros and versions have different log locations and configurations.

Good call - here’s the content, I see repeating itself. Looks like there’s a file I need to clean up.

2024-07-07  0:01:08 0 [Note] Starting MariaDB 10.5.22-MariaDB source revision 7e650253dc488debcb0898ebe6d385bf6dfa3656 as process 420559
2024-07-07  0:01:08 0 [Note] InnoDB: Uses event mutexes
2024-07-07  0:01:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-07-07  0:01:08 0 [Note] InnoDB: Number of pools: 1
2024-07-07  0:01:08 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-07-07  0:01:08 0 [Note] InnoDB: Using Linux native AIO
2024-07-07  0:01:08 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2024-07-07  0:01:08 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-07  0:01:08 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=60737718699,60737718699
2024-07-07  0:01:08 0 [Note] InnoDB: Ignoring data file './starchive/#sql-ib19664.ibd' with space ID 19654. Another data file called ./starchive/staj_session.ibd exists with the same space ID.
2024-07-07  0:01:08 0 [Note] InnoDB: Ignoring data file './starchive/staj_session.ibd' with space ID 19658. Another data file called ./starchive/#sql-alter-557-3a85.ibd exists with the same space ID.
2024-07-07  0:01:09 0 [Note] InnoDB: Starting final batch to recover 452 pages from redo log.
2024-07-07  0:01:09 0 [ERROR] InnoDB: Cannot replay rename of tablespace 19658 from './starchive/#sql-alter-557-3a85.ibd' to './starchive/staj_session.ibd' because the target file exists
2024-07-07  0:01:09 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2024-07-07  0:01:09 0 [Note] InnoDB: Starting shutdown...
2024-07-07  0:01:12 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2024-07-07  0:01:12 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-07  0:01:12 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2024-07-07  0:01:12 0 [ERROR] Aborting

@Joe - so I seem to be in a Catch-22. The DB can’t be restored / deleted while MariaDB is not running, but I can’t start MariaDB while that DB is damaged. How do I manually drop that database, so I can restore it?

If you have PHPMyAdmin installed it is easy.
otherwise "DROP TABLE your db.your table name`" (assuming you have permission)

Unfortunately, I can’t connect to PHPMyAdmin because the MariaDB server is not running.

Sorry for that, I mis-understood that and thought you had MariaDb running.
Obviously if it is not running you do need to fix the reason why.

Maybe concentrate on Access denied for user 'root'@'localhost' (using password: NO)

and perhaps
Plugin initialization aborted with error Data structure corruption...registration as a STORAGE ENGINE failed.

Edit: given that the other 6 are working fine - is it worth concentrating on a separate fresh development system. something seems to have gone wrong with this backup->restore

I was able to get around this by rolling back the VM that Virtualmin is hosted on to a date before the issue occurred. Luckily, data is not modified a lot on this server, so nothing was lost between the snapshot and the SQL failure.