Severe problems migrating server

Hello,
I’m trying to migrate a Debian 6 32 Bit VPS to a Debian 6 64 Bit VPS, both on OpenVZ.

I followed the instructions on https://www.virtualmin.com/documentation/system/migrate. However, when I get to the “fun part” (exaclty: virtualmin restore-domain --source /root/backups/ --all-domains --all-features), the restore process fails with an “Out of memory” error message when the first home directory of the first virtual server is being extracted from the archive, even though only half of the real RAM is allocated and the swap isn’t even in use.
On the web-interface of Virtualmin, this action simply stops at the very same point without any error message.

To migrate the server anyway, I tried rsyncing the home directories and backed up the remaining configurations in the old file format using Virtualmin’s webinterface. Nevertheless, the restoration attempt fails, but with the error message “Restore failed : Failed to open /home/[USER]/fcgi-bin/php5.fcgi.webmintmp.5230 : Permission denied” when the Apache configuration is being restored. The access permissions of these files (error occurs with any virtual server) are 755. Strangely, I also cannot delete these files on the new VPS as root.

Thank you in advance for your help!

Regards,
fuggi

Hi,

While this may seem silly, perhaps you could try restoring one domain at a time. Perhaps a bit more time consuming, but may allow for a bit more smooth migration if the machine is complaining about memory or any other resource. It’d also be at the very least a good way of troubleshooting, that is if the issue happens again during a single domain restore, perhaps there’s a bigger issue that needs to be addressed.

-Peter

Hi Peter,
thanks for your quick response! Anyway, I forgot to mention that I already tried to migrate only one of the virtual servers (domains) at a time and encountered the very same errors.

The following is an excerpt of the listing of directory /home/[USER]/fcgi-bin/:
-rwxr-xr-x 1 [USER] [GROUP] 219 Dec 13 06:50 php5.fcgi

Regards,
fuggi

Meanwhile, I also checked the user limits:
root@myserver:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

AFAIK these settings shouldn’t be the reason for the out-of-memory problem.

Anyway, I am more interested in the second way to migrate the servers in order to decrease the downtime during migration. I should mention that I even tried to restore all configurations (without home directory) on Virtualmin’s web interface before rsyncing the home directory, but this failed either with the very same errore message (only the number in the file name behind “webmintmp” might have been different). Up to this point I would guess it is rather a problem with accessing these files in the backup archive. I would suppose they are not included in the archive since I did not include the home directory but the restoration routine tries to access them anyway. (The question may arise how I set up the restoration settings. Well, I chose “Only those selected below …” and selected everything except “Server’s home directory and web pages” in the “Features to restore” section. Additionally, I selected each option under “Virtualmin settings to restore”.)

However, the restoration process results in the same error if I use a full backup archive for restoration (new format including the home directories). Now the confusion is perfect as far as I’m concerned.

Best regards,
fuggi

Howdy,

One of the issues we’ve seen with OpenVZ is that the RAM shown is “free -m” isn’t the only memory restriction.

OpenVZ actually has a few different kinds of RAM, and there’s different ways to run out of each of them.

Can you paste in the contents of /proc/user_beancounters? That will show your current resource limits, and whether you’ve hit any of those limits.

-Eric

Hi Eric,
there you are:
Version: 2.5
uid resource held maxheld barrier limit failcnt
71531: kmemsize 12896184 12929379 96636764 107374182 0
lockedpages 0 0 2059 2059 0
privvmpages 132643 132600 524288 550502 38
shmpages 1082 1082 65536 65536 30
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 57 57 500 500 0
physpages 78735 78689 0 9223372036854775807 0
vmguarpages 0 0 262144 9223372036854775807 0
oomguarpages 78735 78689 9223372036854775807 9223372036854775807 0
numtcpsock 20 20 550 550 0
numflock 8 8 1000 1100 0
numpty 1 1 102 102 0
numsiginfo 0 0 1024 1024 0
tcpsndbuf 350080 350080 5280000 7392000 0
tcprcvbuf 327680 327680 5280000 7392000 0
othersockbuf 252960 252960 3840000 5376000 0
dgramrcvbuf 0 0 3584000 3584000 0
numothersock 139 139 400 400 0
dcachesize 1037595 1045107 14495514 16106127 0
numfile 2731 2733 17600 17600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 38 38 9223372036854775807 9223372036854775807 0

Anyway, do you have any idea regarding the permission-denied problem?

Best regards,
fuggi

Hi Eric,
according to /proc/user_beancounters it looks as if the application actually ran out of memory while trying to restore the backup from the old server. Hence, do you have idea regarding the issue “Restore failed : Failed to open /home/[USER]/fcgi-bin/php5.fcgi.webmintmp.5230 : Permission denied”?

Many thanks for your help in advance!
fuggi

Have you tried that with the new Virtualmin update, 3.76-2?

There were some bugfixes applied in that update which fixed some things like you’re seeing.

-Eric

Hi Eric,
thank you for your follow-up! I will test it as soon as I’ve got another server with more RAM - probably first week of the new year.

Merry Christmas!
fuggi

I know it’s been a while. I’ve been engaged otherwise in the meantime.

Anyway, it looks like this bug has been fixed and it works now. Thank you for your help!

Regards,
fuggi