How to allow emojies to transfer during Virtualmin backup/restore?

Virtualmin
1
SYSTEM INFORMATION
OS type and version CentOS 7.9
Webmin version 2.021
Virtualmin version 7.7
Related packages SUGGESTED - MariaDB 5.5.68

I’m following these instructions to migrate a server:
https://www.virtualmin.com/documentation/system/migrate/

Migration path is:

  1. Generate backup and SCP to another server
  2. Rebuild the server from CentOS 7.9 to AlmaLinux 9.2
  3. SCP the backup back to the server
  4. Restore

After the restoration, nearly all emojies in the database tables are replaced with question marks “?”.

I’m using utf8mb4_unicode_ci collation for all the tables.

How keep emojies during the migration? They are quite widely used in my app.

Any ideas or suggestions appreciated :wink::+1:.

BR; More / Helsinki

3

Any thoughts, anyone? I did some testing:

  1. Set both source and target server default character set to utf8mb4, and default collation to utf8mb4_unicode_ci
  2. Creating a new site and database with Virtualmin. Checked that all database tables and columns were having utf8mb4
  3. Created some records with emojis, like :black_heart::horse::rainbow::cat::sparkles::bulb:
  4. Backed up the site with command line “virtualmin backup-domain”
  5. Deleted the site on Virtualmin UI
  6. Restored the site with “virtualmin restore-domain”

Result: Nearly all of the emojis were translated to ? characters. From those above, only :sparkles: came back normally.

Is there a bug somewhere? Not necessarily in Virtualmin. Is Virtualmin using mysqldump for db backup/restore?

4

Keeping up my monologue. Found out that mysqldump default character set is utf8. When backing up and restoring with it, I get exactly the same result.

With mysqldump, adding parameter --default-character-set=utf8mb4 when backing up resolves the issue and emojis are transferred correctly.

If/when Virtualmin backup-domain is using mysqldump, which I suspect, is there a way to add this parameter for “virtualmin backup-domain” command?. The documentation mentions “–option” parameter, but there are no details if it could be used in this scenario.

Anyone, any thoughts?

5

are you now using mariadb database server in replacement for mysql community server ?
if so you may find that mariadb does not understand the encoding of your database/table/field.
This would be worth checking

6

I would make the next attempt. From the backup I would use (if it is OK) everything except the database. I would make an SQL dump including database create line from the CLI in the source VM, I would go to the destination VM where I would create a database with the same format then I would import the dump. We basically did the Virtualmin part manually.

7

MariaDB seems to support all character set and collation features, that is not the problem, I’m pretty sure.

8

ok no worries I had the issue … the collations were all wrong took hours to manually fix it !
good luck

9

Thanks and yes, that is my plan-B. I just have some ten virtual servers to be handled. Well… mysqldump is pretty easy tool to backup/restore.

10

If you succeed, it means that there is an issue in Virtualmin related to database backup.

11

If you going from mysql to mariaDB I guess (is that a bug though), I don’t think VM backup/restore converts between the 2.

12

No-no, sorry being unclear. I’m going from MariaDB to MariaDB.

1 Like
13

In this case, I doubt that a full SQL dump taken from one side to another will not work.

14

@Jamie any thoughts on why encoding is breaking on backup restore? I assume it’s related to the big version jump between the MariaDB in CentOS 7 and Alma 9, but that’s a guess, and I’m sure there’s some incantation to prevent it.

15

Just to be clear, there’s the same issue when backing up and restoring locally in one and same CentOS 7 server.

1 Like
16

Oh, that’s surprising, and useful information. So, I guess we’re not keeping up with encoding or overriding it somewhere.

17

I thought I addressed this issue already in this patch:

But that would require Webmin 2.102+ to have installed.

18

@Ilia @Joe Thanks for the feedback!

I updated Webmin to the latest, and got 2.101. But as far as I can see, the mentioned collation fix is include starting 2.100. I double checked from the disk and yes, I seem to have it.

However, after testing again (backup and restore in the same system), the problem persists.

Ilia, isn’t the fix you mention relate to collation and not character set? The problem I have is definitely character set related, so collation fixes won’t help me.

The UI “Backup Database” seem to have “Character set for backup” option, which is exactly what I need. I just can’t find the parameter in the command line “virtualmin backup-domain” command.

19

Not 2.100. 2.102. Virtualmin repo does not have it yet. Sorry!

Jamie, just released Webmin 2.103. You can take a small risk and install it manually to see if that works for you.

Also, I would strongly advised against using CentoOS 7 for clean installs in the 4Q of 2023. You should try Rocky 9 or Alma 9 or Debian 12 or Ubuntu 22.04.

20

Thanks a lot, I’ll try it out later.

And yes, Alma 9 is the ultimate target. This is all about migrating my servers from CentOS 7 to fresh-built Alma 9.

21

Unfortunately, no luck:

  1. Updated Webmin 2.103 from the RPM. The dashboard shows “Webmin version 2103”. So, all good.
  2. Prepared a virtual server with small table - db, table and columns are in utf8mb4 character set. And some data with emojis.
  3. Backed up on command line with “virtualmin backup-domain…” command
  4. Deleted the virtual server
  5. Restored back with “virtualmin restore-domain…”

The same result as before: Nearly all emojis are replaced with ? characters.