Most Elegant Way to Sync Virtualman Mailboxes with an IMAP Server?

**Operating system:Most like AlmaLinux or Rocky
**OS version:8.3?

So there’s a pretty good chance I’ll be migrating a client’s server from cPanel to Virtualmin. I’m guessing there’s ~200GB of mail coming along for the ride.

There will be considerable overlap in time between setting up the server and re-pointing DNS, mainly owing to the multiple upgrades and migrations taking place at the same time (CentOS 7 > ???8, cPanel > Virtualmin, Exim > Postfix, and possibly a different MySQL). I’ll need time to assure that everything is working on the new server commissioning it.

Because of the high mail volume, I’d like to enable IMAP sync to keep the mail on the new server in sync with the mail on the old server between installation and commissioning. What’s the best way these days? Fetchmail? doveadm-sync? A console mail client?

There are between 35 and 40 mail accounts, on average, at any given time.

Thanks,

Richard

1 Like

That should do it. Thank you.

Richard

Oh, don’t mention it.

I got it off Google. :innocent:

2 Likes

I haven’t done a leisurely migration in a long time. My last few batches were because the hosting companies went down in flames, and I had to move as quickly as possible. One I had to restore from an Amazon or Backblaze backup because the losing server became unresponsive in the middle of an SCP session.

So since I have the luxury of time with this one, I’d like to make it seamless. The program you Googled to isn’t what I had in mind, but it will work. I can cron it to run, say, once a day until the move, and then do a final run immediately before changing DNS.

Richard

@RJM_Web_Design,

I’ve had the pleasure (or dis-pleasure – depending on how you look at it) to do a large scale server migration, and while there are lots of moving parts involved, if done correctly it can be done with “minimal” downtime and failures.

With regards to email, I was first tasked with this role about 5 years back and ended up writing my own script which essentially uses “rsync” to migrate over the “Maildir” content, maintaining timestamps and open status on each message.

The problem with many migration techniques is that often it’s like downloading the messages and uploading them to the new server which causes them to be seen by the new server as “new messages”.

In a nutshell, I would provision the “new server”, then import JUST the content including but not limited to email.

I would notify clients of these mailboxes in advance that there might be a brief period of mail “appearing” to have disappeared (this is because I’d provision the account, point the DNS to new server – then once DNS propagates start rsyncing the Maildir contents over to the new server)

You might get some flack for the appearance of missing emails, but all said and done most if informed will forgive you knowing that you didn’t disrupt their whole experience during an otherwise smooth and seamless transition.

*** If both new and old server are in a decent data center with good uplinks, rsyncing the data should run pretty smooth and fairly quickly even with over 200 GB of data. ***

Bottom line, good planning, smart executing and communication are the keys!

If you’d like to discuss further and/or setup a personal consult, drop me a line.

Best Regards,
Peter Knowles | TPN Solutions
pknowles@tpnsolutions.com

1 Like

Thanks, Peter.

I actually tried that using the actual losing server, an existing email account on the losing server, and a server that’s almost identical to the proposed gaining server (the only difference being that it’s running CentOS 7.9, which I think is irrelevant in this case). I tried it using rsync, rclone, SCP, tarring and SCP, and tarring and SFTP. I tried it as root and then chown’ing it, and I tried it as the virtual server user.

No joy, no matter how I did it.

What happened was that the account on the gaining server would see the number and size of emails, but nothing else. The headers and content would all be blank. I could “read” the emails using Roundcube, but there would be no content to read; and if I intended to reply, the “To” field would be empty. There were no errors. It just treated it like an empty mail.

I then scrapped and re-created the email account and sent it one email via telnet. That one mail arrived normally. But attempts to manually import the rest yielded the same results as earlier.

So I re-created the account yet again and installed Mutt. That worked. Hence the idea of using an IMAP client to maintain sync after the initial import (which would be accomplished by the Virtualmin migration script).

The site @calport sent me to also worked using a Web-based app that he makes available for free with a 3GB limit (but no limit if you pay for a license). It doesn’t actually maintain synchronization, but it synchronizes changes since the last run. That would also work for my use case.

I should also look around to see if anyone has a plug-in for Roundcube that would work, now that I think about it.

I really don’t know why the mail is unreadable just copying the contents of the folders. It looks like an ownership / permissions problem, but I can’t find any, and there are no overt errors. My guess would be that it has something to do with cPanel. Exim > Postfix shouldn’t make a difference, and both servers use Dovecot.

After a couple of hours, it became a problem I didn’t care about anymore because IMAP sync after the initial import was a workable solution. If I change my mind, I’ll let you know.

Yeah, they’re even in the same DC. That part won’t be a problem. It was more a commentary on some users’ refusal to part with any mail, no matter how old or useless it is. If they only knew how much of their old shit I delete behind their backs, such as Sent messages with image and video attachments that were addressed to my Webmaster account on the same server. :grin:

Richard

Yes, and if you install the FOSS Imapsync on one of your servers then you could use CLI and cron to transfer email very fast and without the issues you experience in this instance when you use rsync et al. Though I must add that I did use the rsync method a few days ago to transfer over 200 GB of email from Virtualmin on Debian 9 to Virtualmin on Debian 10 quite successfully.

1 Like

I can’t think of a good reason why it wouldn’t work other than that it’s cPanel > Virtualmin. If I get really bored, I may try to figure it out.

However, I think I found a solution that’s so stupid simple that it warrants a forehead slap and a hearty DOH!: Just edit the mail_host entry for the users in the Roundcube database to point to mail.domain.tld instead of localhost. I just tried it on one of my own domains and it works fine.

EDIT: Nope. It doesn’t write to the local directories that way. Grrr.

Richard

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.