Hello,
we experience problem with backup via CLI with --shutdown parameter. Sometimes, backup finishes without problem, but sometimes we get this error:
Shutting down immocapserver.immocap.local prior to backup …
server.domain.local: FAILED … shutdown failed : No KVM process found!
After this, VM won’t start.
I dig a little and it seems, most likely problem is with type_kvm_shutdown function in file /usr/share/webmin/server-manager/kvm-type-lib.pl
2423 # Next try shutdown monitor command
2424 if (!$force) {
2425 my ($mok, $merr) = &run_kvm_monitor_command($server,
2426 [ "system_powerdown" ]);
2427 if ($mok) {
2428 # Wait for it to work
2429 my $tries = 0;
2430 while($tries < $wait) {
2431 my ($kout, $kex) = &execute_command_on_server(
2432 $vhost, "kill -0 $proc->{'pid'}");
2433 if ($kex) {
2434 &delete_kvm_cgroup($server, $vhost);
2435 return undef;
2436 }
2437 sleep(1);
2438 $tries++;
2439 }
2440 }
2441 }
2442
2443 $proc = &get_server_kvm_instance($server);
2444 if ($proc) {
2445 # Just kill it
2446 my ($out, $ex, $err) = &execute_command_on_server($vhost,
2447 "kill -9 $proc->{'pid'}");
2448 return $err if ($err);
2449 return $out if ($ex);
2450 &delete_kvm_cgroup($server, $vhost);
2451 return undef;
2452 }
2453 return "No KVM process found!";
2454 }
At first, it tries to shutdown VM properly in cycle (lines 2430-2440). After each try, it waits a second, then increase tries counter and then do next comparison. If VM shuts down on last try, during that 1 second sleep, code will proceed and since VM is no longer running, check on line 2444 will fail.
This is not a good approach. There should be some check at the beginning, if VM is running and then, when needed, another check, if it is still running. If it was running at the beginning and is not now, shutdown should be considered successful. This should never failed.
Currently we can’t rely on our backup, because at random it will fail and leave VM down - which is another thing, if backup with --shutdown parameter fail and VM was running, when backup started, it should always turn it on again. Currently I really don’t recommend automating and scripting those sort of things.