I found out that the Virtualmin back-up never showed any warnings/errors because of an error concerning too large packets while exporting the database using mysqldump. (something like “Error 2020: Got packet bigger than ‘max_allowed_packet’ bytes when dumping table”)
The database back-up was included in the back-up files Virtualmin made, but was approx 500 MB (gzipped). No errors appeared in the back-up log.
After increasing the max_allowed_packet in the [mysqldump] section in mysql-config the back-up file increased to 1 GB+ (gzipped again of course) because it could now complete the dump.
Conclusion: some of the mysqldump data missed in the previous back-ups. This didn’t cause any errors even though mysqldump should return some.
Possibly very dangerous because I didn’t know my back-ups were incomplete.
No problem, thanks for you help! Hopefully you’ll recover soon!
Yes, I found some error. Didn’t think of it, also because mysqldump returns a specific error directly when using the tool to export the database with too small packet settings, so I thought it was irrelevant in this case.
“2022-01-08T23:02:20.326638Z 4806481 [Note] Aborted connection 4806481 to db: ‘dbname123’ user: ‘root’ host: ‘localhost’ (Got an error writing communication packets)”
Alright, it looks like perhaps you may have discovered a key piece of information helpful to the @staff
I do believe despite this being a MySQL error that perhaps it could also be considered a bug in the Virtualmin backup routine, so perhaps when they have a moment one of the Virtualmin @staff might be able to take a look into this.
I’d probably say if anything the bug is that the backup routine isn’t detecting and reporting or halting the backup when it occurs.
Keep us posted. I’ve never had the small packet error using default config when backing up. I’m using CentOS. I don’t know if Ubuntu by default allocate a different MySQL config when dumping data.
I’m not sure, I used the MySQL defaults. Also, I needed to increase it (above 64M) because of some (very) big blobs.
Even though, I think we should focus on the error that didn’t get noticed by the back-up script. I’ve seen file system errors (i.e. not enough disk space) caused by command line actions from the back-up script, so I think the concept is already available in the script.
Can’t the backup script spot errors from the mysqldump-process? So in case there’s some kind of STDERR, it gets returned to the back-up process so that it can spot errors? And/or noticing not-0-return-value endings? I think those 2 tests should notice these kind of errors. There are more errors like this possible and they should be spotted by monitoring this I think.
The backup script currently checks for mysqldump failures, but it looks like the command can also partially fail if the packet size is too low without returning a non-zero exit code. I will add an extra check for this error message in the next Webmin release.
Great! Thank you!
There are more types of feedback possible from ‘mysqldump’. They’re all redirected tot he STDERR.
Only type of warning I know that isn’t relevant is the warning that’s possible beacuse of using password as parameter directly from commandline (it warns the user that it’s not safe to use this type of authentication). But I don’t know whether that’s relevant to Virtualmin. It recommends to use something like an .my.cnf-config-file for the authentication details.
So I think/hope you can conclude something went wrong when there’s some kind of feedback on STDERR. I think that’ll cover most (all?) errors.
Yeah, I know that mysqldump can output all kinds of messages to STDERR, so we can’t just fail the backup entirely if there is any output. The only reliable option is to check for specific errors like this one.
Not that I want to make things difficult, but besides the known message that you shouldn’t use username/password in cli, are there many other STDERR-messages to be expected?
Otherwise, it would be a safe option to make a whitelist of unimported messages instead of a blacklist of errors.
I’m sorry if think of my solution as annoying, it’s just that I hope to realize the best solutions for all of us.
Otherwise: is there some kind of log/location to check whether there are other STDERR messages from mysqldump? I’m interested to know whether there are more errors at the moment possible on my servers.
I see what you’re saying, but it’s tricky because if we treated the backup as a failure if there was any unexpected mysqldump output, it’s possible that a MySQL upgrade would break existing backups.