| SYSTEM INFORMATION | |
|---|---|
| OS type and version | Debian 12 |
| Virtualmin version | 7.9.0 Pro |
Hi
My HD is filling up and is now full, but I can’t figure out what is going on. It went from maybe 50% to full and as I have 160 GB it is remarkable. Does anyone have any input about how to handle it?
I’ve found something:
/var/log/syslog.1 → 56G
/var/log/syslog → 51G
What is going on?
Something is filling up your syslog - you need to find out what. Go to Webmin -> System -> System Logs select the log and it will display the last 100 items.
I’ll lay odds they are all postfix/smtpd attempts to login
It seem to be one of my Drupal sites that is freaking out and there is also a terrible lot of messages from MariaDB that started when I upgraded from Debian 11 to Debian 12. I downloaded the syslog to my PC and will try to analyse it there and then I deleted it on the server to free the space.
The new syslog file grow rapidly, 15 MB in just a few minutes, and it is spitting out the 3 below lines. This started when I upgraded from Debian 11 to Debian 12. Any clues about what to do?
2024-01-04T12:58:26.560366+01:00 vps-b68a7783 mariadbd[1160]: 2024-01-04 12:58:26 95 [ERROR] Incorrect definition of table mysql.column_stats: expected column ‘histogram’ at position 10 to have type longblob, found type varbinary(255).
2024-01-04T12:58:26.562733+01:00 vps-b68a7783 mariadbd[1160]: 2024-01-04 12:58:26 95 [ERROR] Incorrect definition of table mysql.column_stats: expected column ‘hist_type’ at position 9 to have type enum(‘SINGLE_PREC_HB’,‘DOUBLE_PREC_HB’,‘JSON_HB’), found type enum(‘SINGLE_PREC_HB’,‘DOUBLE_PREC_HB’).
2024-01-04T12:58:26.562800+01:00 vps-b68a7783 mariadbd[1160]: 2024-01-04 12:58:26 95 [ERROR] Incorrect definition of table mysql.column_stats: expected column ‘histogram’ at position 10 to have type longblob, found type varbinary(255).
As so often nothing to doo with Virtualmin but a problem with a Web App (some bad code or misconfiguration) I am no expert on Drupal.
Those MariaDb errors give a clearer indication to some SQL errors in the code.
Yeah, I should go maybe to a Debian forum, because this happened after upgrading Debian 11 to 12, or maybe to a MariaDB forum. I don’t think the DB errors has anything to do with the Drupal installation, but with MariaDB itself. Chat GPT said:
The error messages you provided don’t explicitly mention the name of the database in which the
mysql.column_statstable is located. However, based on the error messages and the naming convention (mysql.column_stats), it’s likely that this table is part of the MySQL system database named “mysql.”
Thank you !
Is generally thought to be a bad idea! It has the habit of gathering a whole load of garbage from the internet so cannot be relied upon.
You have moved OS which is a big move (a whole version) something that should be done with extreme caution there were breaking changes some of which have been discussed on here.
You may have also moved from a previous version of MariaDB as well.
Drupal (I believe is a CMS written in PHP) PHP is yet another area that has implications for a webserver. You need to check a few things there as well. quite a number of topics on that subject here as well. You should be using the correct PHP executable mode and have the correct php modules installed. If you can give us that info (a screen capture) of the PHP Options page might help.
On top of that which webserver? NGINX or Apache? there can be potential problems there as well especially with .htaccess files. But t the end of the day it is usually down the the code of the web app. Which is why a system upgrade is no light task.
Hmm … I’ve multiple sites on the server, so we can’t be sure the issue comes from this one. Or? The error seem to orginate from the “mysql” database and not from the database connected to the actual Drupal 7 site. The “mysql” database is the database controlling the full MariaDB installation, and something seem to have gone wrong there when upgrading the OS, and probably MariaDB as well. One option is that I make a clean installation of the full OS including Virtualmin, and then I restore the domains by Virtualmin restore.
I gave the errors to Chat GPT and it said:
Based on the error messages you’ve provided, it seems like there are two separate issues with the mysql.column_stats table in your MariaDB database:
Both issues are related to a mismatch between the expected column definitions and the actual column definitions in the mysql.column_stats table.
Here are the solutions for each issue:
Issue 1 (histogram column data type mismatch): The error message indicates that the histogram column at position 10 in the mysql.column_stats table is expected to have a data type of longblob, but it is found to have a data type of varbinary(255).
You can resolve this issue by altering the mysql.column_stats table to change the data type of the histogram column to longblob. Use the following SQL statement:
ALTER TABLE mysql.column_stats
MODIFY COLUMN histogram LONGBLOB;
After making this change, the histogram column will have the correct data type.
Issue 2 (hist_type column enum values mismatch): The error message indicates that the hist_type column at position 9 in the mysql.column_stats table is expected to have an enum data type with three possible values ('SINGLE_PREC_HB', 'DOUBLE_PREC_HB', and 'JSON_HB'), but it is found to have an enum data type with only two possible values ('SINGLE_PREC_HB' and 'DOUBLE_PREC_HB').
To resolve this issue, you need to modify the mysql.column_stats table to include the missing enum value 'JSON_HB' in the hist_type column. Use the following SQL statement:
ALTER TABLE mysql.column_stats
MODIFY COLUMN hist_type ENUM(‘SINGLE_PREC_HB’, ‘DOUBLE_PREC_HB’, ‘JSON_HB’);
After making this change, the hist_type column will have the correct enum values.
Remember to back up your database before making these structural changes to ensure data integrity and to roll back in case of any issues.
What do you think?
I looked in to the columns using PHPMYADMIN and it is exactly as said in the answer from Chat GPT; My first idea was to manually change the columns, but but that is probably a very bad idea, because there is probably a good reason to why they have changed the configuration of the columns and changing them manually will come after me later.
I think I will go for the clean reinstall of the server and then make a restore of the domains. The issues with the Drupal install I’ll take as a separate issue.
Whatever you decide to do.
Do it one step at a time, document each step and test the outcome.
Have you done that PHP Options check? This could be a simple correction to make.
Don’t put all your faith in Chat GPT it might well be pointing you in one possible solution but I doubt its practical use.
But isn’t the PHP option check ment for the database connected to the Drupal installation? The database Errors are not connected to that database, they come from the database named “mysql” that controll the MariaDB itself.
I will go for a vanilla installation, and make domaine restores and I will do as you say and make carefull notes. I now know what to do this WE !
I very much appreciate your input as it helps me thinking and pushes me forward in making decisions.
Thank you.
This is what PHP Options should look like - you may have a different PHP version or you may have different modules but the selected execution mode should be FPM.
and watch out for mod-php ! Its use is a no no.
Yes, but that is for each domain (I use FPM on Apache2 and PHP7.4 up to PHP8.3). My issue is not a database connected to any particular domain, it is the database that controls the Server MariaDB that gives the Errors.
That looks like the database hasn’t been upgraded but the database server has been.
You’d need to read the docs for upgrading between the two versions of MariaDB you had/have (the one on 11 and 12) to figure out what steps you need to take. Some major version changes require a dump/restore, though I think most recent MariaDB versions can upgrade themselves in-place to some degree. Maybe the postinst that was supposed to do something related to that failed. Best to check the MariaDB docs and community for clues, as we don’t have any deep familiarity with the inner workings of MariaDB, we’re just users of it, like all of y’all.
I tried with the solution Chat GPT suggested and it works just great and I base that on the assumption you made that the DB Server was upgraded but not the DB !
But also, the syslog was set to log ALL events on the server, not only syslog events !!! Odd!
Let’s see what happens now !
You’re misunderstanding what syslog does and what its configuration means. (And you don’t really even need syslogd on a systemd system.)
I missunderstand a lot of things, I learn by failing, I’m 100% autodidact. I try to do all by myself so save money in my small company.
Concerning the errors from the DB ‘mysql’ it is because the server can not upgrade the tables due to a password issue.
2024-01-05T14:18:52.407770+01:00 vps-b68a7783 /etc/mysql/debian-start[1242]: Upgrading MySQL tables if necessary.
2024-01-05T14:18:52.444549+01:00 vps-b68a7783 /etc/mysql/debian-start[1245]: Reading datadir from the MariaDB server failed. Got the following error when executing the ‘mysql’ command line client
2024-01-05T14:18:52.444643+01:00 vps-b68a7783 /etc/mysql/debian-start[1245]: ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: NO)
2024-01-05T14:18:52.444693+01:00 vps-b68a7783 /etc/mysql/debian-start[1245]: FATAL ERROR: Upgrade failed
2024-01-05T14:18:52.447142+01:00 vps-b68a7783 /etc/mysql/debian-start[1251]: Checking for insecure root accounts.
This is why I had to do the changements manually using PHPMYADMIN
I don’t understand how to get around this.
You can (temporarily) start Mariadb in safe mode --safe-mode and with --skip-grant-tables so it doesn’t need passwords. I don’t know if you also need to do something else to trigger the updates.
Can someone tell me what is the benefit of having virtualmin pro because YOU CANNOT INSTALL MANY THINGS THAT YOU WANT, jitsi meet, mastodon, ???
