My MySQL server stopped and can't restart

afdhaal · November 6, 2022, 6:18pm

SYSTEM INFORMATION
OS type and version	Ubuntu Linux 20.04.2
Virtualmin version	6.16
Webmin version	1.973
VPS	2GB ram, 50GB memory

The virtualmin was working fine and suddenly I couldn’t connect to database.
When I try to start MySQL server I get the following error.

Job for mysql.service failed because the control process exited with error code. 
See "systemctl status mysql.service" and "journalctl -xe" for details.

Results for “systemctl status mysql.service”

● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-11-06 16:21:09 UTC; 7s ago
Process: 10731 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 10739 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 10739 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"

Nov 06 16:21:08 dev systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 16:21:08 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 06 16:21:08 dev systemd[1]: Failed to start MySQL Community Server.
Nov 06 16:21:09 dev systemd[1]: MySQL.service: Scheduled restart job, restart counter is at 5.
Nov 06 16:21:09 dev systemd[1]: Stopped MySQL Community Server.
Nov 06 16:21:09 dev systemd[1]: mysql.service: Start request repeated too quickly.
Nov 06 16:21:09 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 06 16:21:09 dev systemd[1]: Failed to start MySQL Community Server.

Results for “journalctl -xe”

● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-11-06 16:21:09 UTC; 7s ago
Process: 10731 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 10739 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 10739 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"

Nov 06 16:21:08 dev systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 16:21:08 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 06 16:21:08 dev systemd[1]: Failed to start MySQL Community Server.
Nov 06 16:21:09 dev systemd[1]: MySQL.service: Scheduled restart job, restart counter is at 5.
Nov 06 16:21:09 dev systemd[1]: Stopped MySQL Community Server.
Nov 06 16:21:09 dev systemd[1]: mysql.service: Start request repeated too quickly.
Nov 06 16:21:09 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 06 16:21:09 dev systemd[1]: Failed to start MySQL Community Server.

- Journal -xe
    
     The job identifier is 24444 and the job result is done.
    Nov 06 16:22:07 dev systemd[1]: Starting MySQL Community Server...
    -- Subject: A start job for unit mysql.service has begun execution
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - A start job for unit mysql.service has begun execution.
    
    - The job identifier is 24444.
    Nov 06 16:22:09 dev systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
    -- Subject: Unit process exited
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - An ExecStart= process belonging to unit mysql.service has exited.
    
    - The process' exit code is 'exited' and its exit status is 1.
    Nov 06 16:22:09 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
    -- Subject: Unit failed
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - The unit mysql.service has entered the 'failed' state with result 'exit-code'.
    Nov 06 16:22:09 dev systemd[1]: Failed to start MySQL Community Server.
    -- Subject: A start job for unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - A start job for unit mysql.service has finished with a failure.
    
    - The job identifier is 24444 and the job result is failed.
    Nov 06 16:22:09 dev systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
    -- Subject: Automatic restarting of a unit has been scheduled
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - Automatic restarting of the unit mysql.service has been scheduled, as the result for
    -- the configured Restart= setting for the unit.
    Nov 06 16:22:09 dev systemd[1]: Stopped MySQL Community Server.
    -- Subject: A stop job for unit mysql.service has finished
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - A stop job for unit mysql.service has finished.
    
    - The job identifier is 24526 and the job result is done.
    Nov 06 16:22:09 dev systemd[1]: mysql.service: Start request repeated too quickly.
    Nov 06 16:22:09 dev systemd[1]: mysql.service: Failed with result 'exit-code'.
    -- Subject: Unit failed
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - The unit mysql.service has entered the 'failed' state with result 'exit-code'.
    Nov 06 16:22:09 dev systemd[1]: Failed to start MySQL Community Server.
    -- Subject: A start job for unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: [http://www.ubuntu.com/support](http://www.ubuntu.com/support)
    
    - A start job for unit mysql.service has finished with a failure.
    
    - - The job identifier is 24526 and the job result is failed.
    Nov 06 16:22:11 dev named[780]: client @0x7fb2d809b9f0 86.19.33.185#647 (sl): query (cache) 'sl/ANY/IN' denied

Following is found in /etc/log/mysql/error.log

2022-11-06T18:04:02.526491Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31-0ubuntu0.20.04.1) starting as process 27882
2022-11-06T18:04:02.573393Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-11-06T18:04:03.350796Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.29, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-11-06T18:04:03.350857Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-11-06T18:04:03.849798Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-11-06T18:04:03.850075Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-11-06T18:04:03.850104Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-06T18:04:03.850921Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31-0ubuntu0.20.04.1)  (Ubuntu).
2022-11-06T18:04:04.891502Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31-0ubuntu0.20.04.1) starting as process 27933
2022-11-06T18:04:04.962733Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-11-06T18:04:05.855687Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.29, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-11-06T18:04:05.855734Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-11-06T18:04:06.355552Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-11-06T18:04:06.355871Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-11-06T18:04:06.355904Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-11-06T18:04:06.356749Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31-0ubuntu0.20.04.1)  (Ubuntu).
2022-11-06T18:04:07.397534Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31-0ubuntu0.20.04.1) starting as process 27971
2022-11-06T18:04:07.429282Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

Kindly suggest me a lead on this. Spent whole day and couldn’t figure it out.

Thank you,
Regards

dimitrist · November 8, 2022, 11:40am

did you try removing innodb_fast_shutdown = 2 from config ?

afdhaal · November 8, 2022, 12:45pm

Thank you for your reply!

I checked the MySQL config file (/etc/mysql/mysql.cnf). This is all I have in that file.

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

dimitrist · November 8, 2022, 2:23pm

look for any .cnf files in /etc/mysql and also inside those 2 directories you see in my.cnf.

inside some .cnf file you might find the above parameter which you should try removing before restarting mysql.

afdhaal · November 8, 2022, 6:00pm

Again, thank you very much for the reply.

I checked all the files under /etc/mysql and I couldn’t find any such entrance.

These are what I found.

file: my.cnf.fallback

!includedir /etc/mysql/conf.d/

File: debain.cnf

[client]
host     = localhost
user     = debian-sys-maint
password = rMT3qhIRrRyfCNC5
socket   = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host     = localhost
user     = debian-sys-maint
password = ##################
socket   = /var/run/mysqld/mysqld.sock

File: mysql.conf.d/mysql.cnf

[mysql]

File: mysql.conf.d/mysql.cnf

[mysqld]
user		= mysql
bind-address = 127.0.0.1
mysqlx-bind-address	= 127.0.0.1
key_buffer_size		= 16M
myisam-recover-options  = BACKUP
log_error = /var/log/mysql/error.log
max_binlog_size   = 100M
innodb_file_per_table = 1

There is nothing significant in the etc/mysql/conf.d folder.

I really wonder, is there any other way that I can recover my database? I only can find the .idb files in /var/lib/mysql. I couldn’t get the .frm.

Any help to restart the MySQL or recover my database would be appreciated.

Thank you,
regards

dimitrist · November 8, 2022, 10:16pm

try adding ```innodb_fast_shutdown = 0 in file :

and restarting.
maybe you have a full ram? do you have enough system resources (eg. ram) to run mysql?

p.s. don’t post passwords on public forums, even random ones like these…

afdhaal · November 9, 2022, 12:27am

Hello Dimitrist,

Thank you for the reminder. I have removed the password.

A:
I have the VPS in digitalocean and just to make sure the ram is the issue, I upgraded it to 4Gb ram from 2Gb ram. Even after upgrading, the same error persists.

B:
I tried to add “innodb_fast_shutdown = 0” to /etc/mysql/mysql.conf.d/mysql.cnf. This is the error I got.

The MySQL client program
/usr/bin/mysql
does not appear to be the correct version. Webmin only supports MySQL versions 3 and above.
The command /usr/bin/mysql -V returned :

mysql: [ERROR] unknown variable 'innodb_fast_shutdown=0'.

C:
Then I tried to add the “innodb_fast_shutdown = 0” to /etc/mysql/mysql.cnf , this is the error I got.

mysql: [ERROR] Found option without preceding group in config file /etc/mysql/my.cnf at line 23.
mysql: [ERROR] Fatal error in defaults handling. Program aborted!

Any ideas on how to proceed?

Thank you,
Regards

dimitrist · November 9, 2022, 12:35pm

no ideas from here. can help professionally (for a fee) , but can’t fix it from forum. i think mysql is not really supported anymore , only mariadb…

system · January 8, 2023, 12:35pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.