VM Shutdown problem

we experience problem with backup via CLI with --shutdown parameter. Sometimes, backup finishes without problem, but sometimes we get this error:

Shutting down immocapserver.immocap.local prior to backup …
server.domain.local: FAILED … shutdown failed : No KVM process found!

After this, VM won’t start.

I dig a little and it seems, most likely problem is with type_kvm_shutdown function in file /usr/share/webmin/server-manager/kvm-type-lib.pl

2423 # Next try shutdown monitor command
2424 if (!$force) {
2425         my ($mok, $merr) = &run_kvm_monitor_command($server,
2426                                 [ "system_powerdown" ]);
2427         if ($mok) {
2428                 # Wait for it to work
2429                 my $tries = 0;
2430                 while($tries < $wait) {
2431                         my ($kout, $kex) = &execute_command_on_server(
2432                                         $vhost, "kill -0 $proc->{'pid'}");
2433                         if ($kex) {
2434                                 &delete_kvm_cgroup($server, $vhost);
2435                                 return undef;
2436                                 }
2437                         sleep(1);
2438                         $tries++;
2439                         }
2440                 }
2441         }
2443 $proc = &get_server_kvm_instance($server);
2444 if ($proc) {
2445         # Just kill it
2446         my ($out, $ex, $err) = &execute_command_on_server($vhost,
2447                                         "kill -9 $proc->{'pid'}");
2448         return $err if ($err);
2449         return $out if ($ex);
2450         &delete_kvm_cgroup($server, $vhost);
2451         return undef;
2452         }
2453 return "No KVM process found!";
2454 }

At first, it tries to shutdown VM properly in cycle (lines 2430-2440). After each try, it waits a second, then increase tries counter and then do next comparison. If VM shuts down on last try, during that 1 second sleep, code will proceed and since VM is no longer running, check on line 2444 will fail.

This is not a good approach. There should be some check at the beginning, if VM is running and then, when needed, another check, if it is still running. If it was running at the beginning and is not now, shutdown should be considered successful. This should never failed.

Currently we can’t rely on our backup, because at random it will fail and leave VM down - which is another thing, if backup with --shutdown parameter fail and VM was running, when backup started, it should always turn it on again. Currently I really don’t recommend automating and scripting those sort of things.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.