Virtualmin Backup truncates part of MySQL dump without generating error

SYSTEM INFORMATION
OS type and version: Ubuntu 18.04.6 LTS
Webmin version: 1.981
Virtualmin version: 6.17.gpl-3
Related products version: MySQL 5.7.36

I found out that the Virtualmin back-up never showed any warnings/errors because of an error concerning too large packets while exporting the database using mysqldump. (something like “Error 2020: Got packet bigger than ‘max_allowed_packet’ bytes when dumping table”)

The database back-up was included in the back-up files Virtualmin made, but was approx 500 MB (gzipped). No errors appeared in the back-up log.
After increasing the max_allowed_packet in the [mysqldump] section in mysql-config the back-up file increased to 1 GB+ (gzipped again of course) because it could now complete the dump.

Conclusion: some of the mysqldump data missed in the previous back-ups. This didn’t cause any errors even though mysqldump should return some.

Possibly very dangerous because I didn’t know my back-ups were incomplete.

@tfhelp,

Have you checked the miniserv error log to see if Virtualmin maybe logged something in there?

Just checked miniserv.error: nothing I could define as potential relevant (or around that time period)

@tfhelp,

I think MySQL itself has an error log, perhaps that might show something.

Sorry, I’m just offering simple ideas though I am presently doing so from hospital.

No problem, thanks for you help! Hopefully you’ll recover soon!

Yes, I found some error. Didn’t think of it, also because mysqldump returns a specific error directly when using the tool to export the database with too small packet settings, so I thought it was irrelevant in this case.

“2022-01-08T23:02:20.326638Z 4806481 [Note] Aborted connection 4806481 to db: ‘dbname123’ user: ‘root’ host: ‘localhost’ (Got an error writing communication packets)”

@tfhelp,

Alright, it looks like perhaps you may have discovered a key piece of information helpful to the @staff

I do believe despite this being a MySQL error that perhaps it could also be considered a bug in the Virtualmin backup routine, so perhaps when they have a moment one of the Virtualmin @staff might be able to take a look into this.

I’d probably say if anything the bug is that the backup routine isn’t detecting and reporting or halting the backup when it occurs.

Keep us posted. I’ve never had the small packet error using default config when backing up. I’m using CentOS. I don’t know if Ubuntu by default allocate a different MySQL config when dumping data.

How big is max_allowed_packet in your /etc/my.cnf file? In recent installs, Virtualmin bumps this up to 64M to avoid this issue.

I’m not sure, I used the MySQL defaults. Also, I needed to increase it (above 64M) because of some (very) big blobs.

Even though, I think we should focus on the error that didn’t get noticed by the back-up script. I’ve seen file system errors (i.e. not enough disk space) caused by command line actions from the back-up script, so I think the concept is already available in the script.

Can’t the backup script spot errors from the mysqldump-process? So in case there’s some kind of STDERR, it gets returned to the back-up process so that it can spot errors? And/or noticing not-0-return-value endings? I think those 2 tests should notice these kind of errors. There are more errors like this possible and they should be spotted by monitoring this I think.

The backup script currently checks for mysqldump failures, but it looks like the command can also partially fail if the packet size is too low without returning a non-zero exit code. I will add an extra check for this error message in the next Webmin release.

2 Likes

Great! Thank you!
There are more types of feedback possible from ‘mysqldump’. They’re all redirected tot he STDERR.

Only type of warning I know that isn’t relevant is the warning that’s possible beacuse of using password as parameter directly from commandline (it warns the user that it’s not safe to use this type of authentication). But I don’t know whether that’s relevant to Virtualmin. It recommends to use something like an .my.cnf-config-file for the authentication details.

So I think/hope you can conclude something went wrong when there’s some kind of feedback on STDERR. I think that’ll cover most (all?) errors.

Yeah, I know that mysqldump can output all kinds of messages to STDERR, so we can’t just fail the backup entirely if there is any output. The only reliable option is to check for specific errors like this one.