migration from 3.98.gpl to 4.10.gpl fails

cjfsyntropy · September 17, 2014, 11:09pm

I’m migrating a FreeBSD 8.0 system running webmin 1.620 and virtualmin 3.98.gpl to a fresh Debian Wheezy system running webmin 1.701 and virtualmin 4.10.gpl. There are about 54 domains on the old system.

I installed a fresh Wheezy system. I added my favorite utilities including my own firewall scripts. I added the following to apt/sources.list:

# usermin & webmin
deb http://download.webmin.com/download/repository sarge contrib

# procmail-wrapper virtualmin-base
deb http://software.virtualmin.com/gpl/debian/ virtualmin-wheezy main
deb http://software.virtualmin.com/gpl/debian/ virtualmin-universal main

I ran install.sh

I ran out of disk space in /tmp, so I “mv /tmp/.webmin /var/tmp” and in
Webmin -> Webmin -> Webmin Configuration -> Advanced Options, and changing
“Temporary files directory” to /var/tmp/.webmin

I followed the instructions at http://www.virtualmin.com/documentation/system/migrate

I scp’ied the backups from the FreeBSD system.

I ran virtualmin restore-domain --source /var/migration-backups/virtualmin.tar.gz --all-virtualmin
No apparent errors:

[code]Checking for missing features …
… all features in backup are supported

Checking for errors in backup …
… no errors found

Starting restore…
Extracting backup archive file …
… done

Restoring Virtualmin settings …
Restoring Virtualmin configuration …
… done

Restoring templates and plans ..
.. done

Restoring email templates ..
.. done

Restoring custom fields, links, categories and shells ..
.. done

Restoring custom script installers ..
.. done

Restoring scheduled backups and keys ..
.. done

Restoring FTP directory restrictions ..
.. done

Restoring DKIM settings ..
.. not installed

Restoring greylisting settings ..
.. not installed

Restoring rate limiting settings ..
Restoring mail server configuration ..
.. done

… done

Applying FTP server configuration …
… done

Restore completed successfully.[/code]

I ran virtualmin restore-domain --source /var/migration-backups/ --all-domains --all-features
There is just one error at the end:

[code]Checking for missing features …
… all features in backup are supported

Checking for errors in backup …
… no errors found

Starting restore…
Extracting backup archive files …
… done

Restoring backup for virtual server chinamedicinemall.com …
Restoring virtual server password, quota and other details …
… done

Updating administration password and quotas ..
.. done

Restoring Cron jobs ..
.. done

Extracting TAR file of home directory ..
.. done

Setting ownership of home directory ..
.. done

Restoring Apache virtual host configuration ..
.. done

Checking restored PHP execution mode ..
.. mode Apache mod_php OK for this system

Restoring allowed MySQL hosts ..
.. done

Re-loading MySQL database chinamed ..
.. done

Restoring Webmin ACL files ..
.. done

Re-creating mail and FTP users ..

Error: makemap failed : sh: 1: makemap: not found

Error

makemap failed : sh: 1: makemap: not found[/code]

In Debian makemap is provided by the sendmail-bin package, but that conflicts with postfix which is required for virtualmin. So I do not know how to resolve that error.

Is that error the reason why only 1 of the 45 domains got migrated (according to the “List Virtual Servers” screen in the GUI)?

Is there a listing of all the places virtualmin stores data so I can audit the restore and determine what got migrated and what is missing?

Why is it not seeing the other 55 files like *.{info,dom,tar.gz} in /var/migration-backups/ ? Should I try to specify each domain one at a time? How?

Eric · September 18, 2014, 3:46am

Howdy,

Hmm, it shouldn’t actually need makemap on a Postfix system, and I wouldn’t recommend attempting to install/use Sendmail, that just tends to make things harder

If you look in System Settings -> Virtualmin Config, what is “Mail server to configure” set to on that first screen there?

If your last server was using Sendmail, and you imported the old Virtualmin settings, it may be trying to configure the email users with Sendmail rather than Postfix. You could always change that option to use Postfix.

It sounds like you’re attempting the migration the right way though – just pointing Virtualmin at a directory full of backups. The .dom and .info files are just metadata, the .tar.gz files are your actual backups.

However, by default, if any error occurs during a restore the restore process will stop and not continue on with the other domains.

-Eric

cjfsyntropy · September 23, 2014, 8:13pm

System Settings -> Virtualmin Config -> “Mail server to configure” solved the problem. This is a sendmail to postfix migration too, unfortunately. And that has created another problem. Although virtualmin restore-domain --source /var/migration-backups/ --all-domains --all-features finished without further errors. I now have a convoluted configuration which I’m not sure how to straighten out.

When I go to Postfix Mail Server -> User Mailboxes, mailboxes appear to be in ~USER/Maildir. But postfix delivers mail to /var/mail/USER in mbox format. What kind of postfix configuration do I need for Virtualmin? I assume the migration got it wrong because the old server was sendmail. I assume the migration, therefore, does not touch my local postfix configuration. So I can hack that to make it work without fear that the migration will change it. Is there anything the migration does that I need to be cautious about?

Does virtualmin prefer Maildir or mbox format? Why did the migration configure Dovecot as Maildir? It also put the user’s mail from the old server into ~USER/Maildir. Is Maildir the Virtualmin way to do mail? Will it be easier to migrate postfix to Maildir or Dovecot to mbox?

Finally, this was a test run of the migration. I plan to run the virtualmin restore-domain ... command again for the actual cutover. Is it idempotent? How can I finagle the migration so that mail works after the cutover.

Eric · September 24, 2014, 3:10am

Howdy,

Virtualmin configures the various services on your system to use Maildir by default. It does that during the installation of Virtualmin. While you don’t have to use that, that’s the preferred format.

What format are your current mailboxes in? After performing the migration, are you saying it moved your existing mailboxes over to Maildir?

Virtualmin configures Postfix to use procmail for email delivery… you can verify that email is being delivered to Maildir directories by looking at /etc/procmailrc.

Regarding the restore-domain command – you can run that as often as you want, though note that doing so doesn’t remove the original homedir contents, though it will overwrite files that exist. I believe the “–delete-existing” option should delete the existing homedir contents first.

-Eric

cjfsyntropy · September 24, 2014, 3:24am

Maildir is fine by me. But currently new mails are delivered to /var/mail/MBOX instead of $HOME/Maildir. My /etc/procmailrc looks like this:

LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
DEFAULT=/var/spool/mail/$LOGNAME
ORGMAIL=/var/spool/mail/$LOGNAME
DROPPRIVS=yes

How should it look for a “normal” Virtualmin configuration? How should I use the GUI to choose Maildir in a Virtualmin compatible way?

–delete-existing sounds like exactly what I need: thanks.

Eric · September 24, 2014, 3:52pm

Howdy,

Hmm, what you’re seeing there is a non-default setup, and things shouldn’t appear that way on a freshly installed system.

However, we can reconfigure things so that they use Maildir.

What is the output of this command:

postconf -n

cjfsyntropy · September 24, 2014, 4:35pm

It is a freshly installed system. I added the CA and certs for TLS, but otherwise did not change it from a standard Debian install followed by running Virtualmin’s install.sh and the migration scripts documented above.

alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
allow_percent_hack = no
append_dot_mydomain = no
biff = no
broken_sasl_auth_clients = yes
config_directory = /etc/postfix
home_mailbox = Maildir/
inet_interfaces = all
mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME
mailbox_size_limit = 0
mydestination = polka.omnikoz.com, localhost.omnikoz.com, localhost
myhostname = polka.omnikoz.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
readme_directory = no
recipient_delimiter = +
relayhost =
sender_bcc_maps = hash:/etc/postfix/bcc
smtp_tls_CAfile = /etc/lfrr/ca/ca.crt
smtp_tls_cert_file = /etc/lfrr/ca/thishost.crt
smtp_tls_key_file = /etc/lfrr/ca/thishost.key
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination check_policy_service inet:127.0.0.1:10023
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_tls_CAfile = /etc/lfrr/ca/ca.crt
smtpd_tls_cert_file = /etc/lfrr/ca/thishost.crt
smtpd_tls_key_file = /etc/lfrr/ca/thishost.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
virtual_alias_maps = hash:/etc/postfix/virtual

Eric · September 24, 2014, 5:07pm

Okay, your Postfix config looks good, so it’s just the procmailrc that needs tweaked.

The /etc/procmailrc file on a new setup should look something like this:


LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
:0wi
VIRTUALMIN=|/etc/webmin/virtual-server/lookup-domain.pl $LOGNAME
EXITCODE=$?
:0
* ?/usr/bin/test "$EXITCODE" = "73"
/dev/null
EXITCODE=0
:0
* ?/usr/bin/test "$VIRTUALMIN" != ""
{
INCLUDERC=/etc/webmin/virtual-server/procmail/$VIRTUALMIN
}
ORGMAIL=$HOME/Maildir/
DEFAULT=$HOME/Maildir/
DROPPRIVS=yes
:0
$DEFAULT

Try changing your procmailrc to the above… once you do that, any new emails should be delivered using Maildir.

-Eric

cjfsyntropy · September 25, 2014, 12:27am

Thanks for your help Eric, but it didn’t quite work.

On my system there is no file /etc/webmin/virtual-server/lookup-domain.pl

There is a file /usr/share/webmin/virtual-server/lookup-domain.pl

Is that the correct file? Should I copy it to /etc/webmin/virtual-server/ or just reference the version in /usr/share/webmin/virtual-server/?

Also, on my system there is no directory /etc/webmin/virtual-server/procmail/

Maybe the old server did not have procmail and so the migration failed to create these files? Do I need to generate a site specific procmail file for each of the 54 virtual domains? How would I do that? Is it documented anywhere?

cjfsyntropy · September 25, 2014, 3:54am

When I run virtualmin restore-domain --delete-existing --source /var/migration-backups/ --all-domains --all-features, I get these errors:

Checking for missing features ..
.. all features in backup are supported

Checking for errors in backup ..
.. no errors found

Starting restore..
Extracting backup archive files ..
.. done

Deleting virtual server chinamedicinemall.com ..
    Deleting mail aliases ..
    .. done

    Deleting Webmin login ..
    .. done

    Deleting MySQL database chinamed ..
    .. done

    Deleting MySQL login ..
    .. done

    Deleting virtual website ..
    .. done

    Deleting home directory ..
    .. done

    Deleting administration user ..
    .. done

    Deleting administration group ..
    .. done

    Deleting server details for chinamedicinemall.com ..
    .. done

    Re-loading Webmin ..
    .. done

    Applying web server configuration ..
    .. done

.. done

Re-creating virtual server chinamedicinemall.com ..
    Creating administration group chinamed ..
    .. done

    Creating administration user chinamed ..
    .. done

    Creating aliases for administration user ..
    .. done

    Adding administration user to groups ..
    .. done

    Creating home directory ..
    .. done

    Creating mailbox for administration user ..
    .. done

    Adding new virtual website ..
    .. done

    Performing other Apache configuration ..
    .. done

    Creating MySQL login ..
    .. done

    Creating MySQL database chinamed ..
    .. done

    Creating Webmin user ..
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
    .. done

    Applying web server configuration ..
    .. not running!

    Re-loading Webmin ..
    .. done

    Saving server details ..
    .. done

Restoring backup for virtual server chinamedicinemall.com ..
    Restoring virtual server password, quota and other details ..
    .. done

    Updating administration password and quotas ..
    .. done

    Restoring Cron jobs ..
    .. done

    Extracting TAR file of home directory ..
    .. done

    Setting ownership of home directory ..
    .. done

    Restoring Apache virtual host configuration ..
    .. done

    Checking restored PHP execution mode ..
    .. mode Apache mod_php OK for this system

    Restoring allowed MySQL hosts ..
    .. done

    Re-loading MySQL database chinamed ..
    .. done

    Restoring Webmin ACL files ..
    .. done

    Re-creating mail and FTP users ..
    .. done

    Re-creating mail aliases ..
    .. done

    Restoring mail files ..
    .. done

    Moving additional mail folders ..
    .. done

    Restoring mail and FTP user Cron jobs ..
    .. done

    Updating Webmin user ..
    .. done

[It continues like this for about 11 more virtual domains, then it fails on this restore:]

Deleting virtual server campbellvoiceover.com ..
    Deleting mail aliases ..
    .. done

    Deleting Webmin login ..
    .. done

    Deleting MySQL database campbell ..
    .. done

    Deleting MySQL login ..
    .. done

    Deleting virtual website ..
    .. done

    Removing from email domains list ..
    .. done

    Deleting administration user ..
    .. done

    Deleting administration group ..
    .. done

    Deleting server details for campbellvoiceover.com ..
    .. done

    Applying web server configuration ..
    .. not running!

    Re-loading Webmin ..
    .. done

.. done

Re-creating virtual server campbellvoiceover.com ..
    Creating administration group campbell ..
    .. done

    Creating administration user campbell ..
    .. done

    Creating aliases for administration user ..
    .. done

    Adding administration user to groups ..
    .. done

    Creating home directory ..
    .. done

    Creating mailbox for administration user ..
    .. done

    Adding to email domains list ..
    .. done

    Adding default mail aliases ..
    .. done

    Adding new virtual website ..
    .. done

    Performing other Apache configuration ..
    .. done

    Creating MySQL login ..
    .. done

    Creating MySQL database campbell ..
    .. done

    Creating Webmin user ..
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
    .. done

    Applying web server configuration ..
    .. not running!

    Re-loading Webmin ..
    .. done

    Saving server details ..
    .. done

Restoring backup for virtual server campbellvoiceover.com ..
    Restoring virtual server password, quota and other details ..
    .. done

    Updating administration password and quotas ..
    .. done

    Restoring Cron jobs ..
    .. done

    Extracting TAR file of home directory ..
    .. done

    Setting ownership of home directory ..
    .. done

    Restoring Apache virtual host configuration ..
    .. done

    Checking restored PHP execution mode ..
Error: Failed to copy /etc/php5/apache2/php.ini to /home/campbell/etc/php5/php.ini : cp: cannot create regular file `/home/campbell/etc/php5/php.ini': No such file or directory

Error
-----
Failed to copy /etc/php5/apache2/php.ini to /home/campbell/etc/php5/php.ini : cp: cannot create regular file `/home/campbell/etc/php5/php.ini': No such file or directory

-----

What could have gone wrong? I mv’d /home/campbell out of the way and repeated but it did not work.

I also tried with and without the updated virtualmin.tar.gz* files in /var/migration-backups: I worry about those files changing my postfix mailer again.