Last week (thursday) I discovered that one my servers (named vps45) had mySQL down, impossible to restart. After doing all I could to restart, and contacting datacenter engineers, they said that mySQL seemed to have been corrupted and best option was to reinstall (I had backups).
I had 11 virtual hosts on that machine, I moved 8 away on another new machine vps51 and left 3 on vps45, I reinstalled vps45 with the 3 remaining websites.
2 days later vps45 crashed again same symptoms … I then reinstalled these 3 remaining sites on 3 differents servers, each site alone on its server. At this stage, datacenter engineers did some more researching, they were thinking that maybe neither the server nor the websites were the cause of the problem but maybe webmin (they say they are not specialists regarding this panel but …)
So up to now I was in this situation, everything up and running:
vps51 (new server, datacenter2 , freshly installed) 8 vhosts
vps52 (new server, datacenter2, freshly installed) 1 vhost
vps53 (new server, datacenter1, freshly installed) 1 vhost
And this morning, vps51 is down with with the same symptoms (mySQL crashed) and 8 vhosts down …
The only thing I can think of is:
pbm is not with one of the vhosts because different servers did crash with different vhosts
pbm is not with the server as different servers did crash (from different providers and different datacenters)
That leaves the following options:
Ubuntu 20.04 ?
Webmin ? Virtualmin ?
Virtualmin backups (beacause crash seems to happen during the nigh around backup time) ?
my settings (but I have been applying the same settings for a long time, maybe some pbm with new Virtualmin versions ?)
I will add in a following message errors collected … if anybody has a good idea … I already checked a lot of threads regarding mysql crashing errors, none seems to have clear and evident solutions.
The full MySQL error message was : DBI connect failed : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
--
-- Automatic restarting of the unit mysql.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
May 11 07:25:39 vps51.zedd.fr systemd[1]: Stopped MySQL Community Server.
-- Subject: A stop job for unit mysql.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit mysql.service has finished.
--
-- The job identifier is 5191 and the job result is done.
May 11 07:25:39 vps51.zedd.fr systemd[1]: mysql.service: Start request repeated too quickly.
May 11 07:25:39 vps51.zedd.fr systemd[1]: mysql.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit mysql.service has entered the 'failed' state with result 'exit-code'.
May 11 07:25:39 vps51.zedd.fr systemd[1]: Failed to start MySQL Community Server.
-- Subject: A start job for unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit mysql.service has finished with a failure.
--
-- The job identifier is 5191 and the job result is failed.
â—Ź mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2022-05-11 07:19:04 UTC; 4min 27s ago
Process: 3430 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 3438 ExecStart=/usr/sbin/mysqld (code=exited, status=2)
Main PID: 3438 (code=exited, status=2)
Status: "Server startup in progress"
May 11 07:19:04 vps51.zedd.fr systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
May 11 07:19:04 vps51.zedd.fr systemd[1]: Stopped MySQL Community Server.
May 11 07:19:04 vps51.zedd.fr systemd[1]: mysql.service: Start request repeated too quickly.
May 11 07:19:04 vps51.zedd.fr systemd[1]: mysql.service: Failed with result 'exit-code'.
May 11 07:19:04 vps51.zedd.fr systemd[1]: Failed to start MySQL Community Server.
Ok my vps53 with only one site crashed a few minutes ago. Everything is pointing toward a pbm with virtualmin, or virtualmin+Ubuntu because crashes occur on different servers, different providers, different websites but all with the same setup Ubuntu20.04+Virtuamin7
Following up on my message this morning, I reinstalled the one named vps51 with Debian10 instead of Ubuntu and restored 6 websites … we’ll see.
I have this problem on all servers that were upgraded to VM7 in the recent days, it seems to be happening faster on server with more traffic, vps51 has only small sites it took 6 days, vps53 has one site with quite a lot of traffic, it’s happening every 2 days.
PC
Yes but problem is that these server do have enough RAM but yes again, they do crash because of an exponential memory consumption at one time … problem is to find the root cause …
At this time we made a few deductions:
crash do happen only on Ubuntu 20.04 servers not on Debian 10 (and we have way more Debian than Ubuntu)
crash don’t seem to be related to servers/datacenters (because they are with 2 different provides/datacenters)
they might be related with the conjunction of U20/V7 or a specific CMS/U20 because it seems that crashes do follow a specific CMS so my last mods were to isolate specific CMS’s on separate servers, some with U20 and some opn DE10, I’m now waiting for the next crash to see …
These crashes started after the V7 upgrade but at almost the same time we also had the mySQL 8.0.29 update … I don’t think V7 is the culprit (because of so many installations worldwide it would be weird to be the only one impacted), my specific CMS might be a better candidate, it’s mostly used in France and not that much compared to Wordpress, so might be that I am one of the only people having that CMS+U20 …
Yes that is exactly what I was thinking, the main difference between Ubuntu and Debian is mySQL vs MariaDB …
Regarding upcoming V7, from a mail sent IIRC by Joe, I understand that Ubuntu’s mySQL will be replaced by MariaDB on a fresh install, but what about existing installs ? if I have a U20 with mySQL and V6, some have been recently updated to V7 and kept mySQL … mySQL will be replaced only if I run the install again (which I wouldn’t do of course) ?
This is a good question and important one to understand to anyone running Ubuntu 20.04 with Virtualmin 6.
The short answer is no, meaning that we will not replace MySQL with MariaDB on existing installs, even if you upgrade to Virtualmin 7 as long as you stay on /vm/6 repos.
However, if you change to Virtualmin 7 repos for existing installs, for example, by running a new version of Virtualmin 7 install script with flag -s (to setup repositories), then it will eventually break things, as a stack package would be upgraded, forcing MariaDB instead.
That will most definitely break database service and would require a manual fix.
I’m wondering: If we back up our sites, do a fresh install of Virtualmin 7 when it comes out on Ubuntu 22.04, will our MySQL databases be able to be converted to MariaDB and imported to the new install?
Most certainly, yes! MySQL and MariaDB are back-compatible, and we use mysql_dump command to backup and restore databases. I don’t see how it may not work.
Although, there might be issues popping up on application side itself but this is not something we can fix. Also, historically MySQL had much more application incompatibility issues, compared to MariaDB. Joe definitely made a right choice using MariaDB on Ubuntu with upcoming Virtualmin 7.
Mostly is my experience while for example Newer MARIADB version has other default settings for example for “strict” there you have to change some if those are giving trouble in the settings / conf.
So take care of good database backups, but also to read the logs while such errors you can see mostly in log files and solve them.
I assumed that many people would face this issue, so I have added a protection against upgrading existing installations’ repos, even when running Virtualmin 7 install.sh script.