Problem creating new VM via Cloudmin on Xen 4 on CentOS 6 host

I took delivery of a new server today, and decided to try out the new Xen 4 packages from the guys at CentOS. Install went smoothly per instructions at http://wiki.centos.org/HowTos/Xen/Xen4QuickStart. All that was left to do is set up some network bridges which I did manually and I could then add the host to Cloudmin.

Next I tried creating a new Xen VM via Cloudmin. The creation worked OK, but there was one odd glitch. The creation script failed to start up the new VM. The output from the creation process is below.

I logged in to SSH on the host and checked out the logical volume, config file etc and all looked, OK, so I can ‘xm create dave.cfg -c’ and watched the new VM boot up just fine. So I am wondering, why is the Cloudmin creation script failing to boot up the system, when I can boot it fine on the command line?

Interesting now the VM is created I can stop and start it using the System State -> Startup System and System State -> Shutdown System functions in Cloudmin. It was just the automated script that wasn’t working.

Here’s the output from the creation process:

Copying 4.91 GB image file to host system ams2-xen-3-2.localdomain … …
… done, and added to host cache

Creating virtual system with Xen …
… creation started.

Waiting for creation to complete … …
… creation has completed successfully.

Creating swap file of 1024 MB …
… done

Removing missing disks from fstab file …
… done

Mounting new instance’s filesystem …
… mounted on /mnt/xen-dave

Configuring kernel for Xen instance …
… setup to boot Xen system’s kernel with Pv-Grub

Fixing GRUB configuration file …
… done

Setting root password …
… done

Updating configuration files with hostname and IP address …
… done

Updating network configuration files …
… done

Modifying /etc/fstab file …
… done

Allowing SSH logins by root …
… already allowed

Setting SSH server port …
… done

Un-mounting instance’s filesystem …
… done

Adding DNS entry dave.ams2-cloudmin.anu.net. for IP address 83.96.163.8 …
… done

Configuring Xen instance for VNC console access …
… added on dynamic port

Starting up new Xen instance …
… failed to start :

PTY PID: 4525
Using <class ‘grub.GrubConf.GrubConfigFile’> to parse /boot/grub/grub.conf
e(Be)0e[1;24re[me[?7he[?1he=e[He[Je[?1he=
pyGRUB version 0.6
e[0mlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
e[0mx e[0;7mCentOS (2.6.32-279.19.1.el6.x86_64) e[m e[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mxe[72Ce[0mx
e[0mmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
e[70De[0mUse the ^ and v keys to select which entry is highlighted.
e[58DPress enter to boot the selected OS, ‘e’ to edit the
e[52Dcommands before booting, ‘a’ to modify the kernel arguments
e[59Dbefore booting, or ‘c’ for a command line.e[12Ae[26Ce[17Be[68DWill boot selected entry in 10 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 9 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 8 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 7 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 6 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 5 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 4 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 3 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 2 secondse[?1he=
e[Je[17Ae[73Ce[17Be[68DWill boot selected entry in 1 secondse[?1le>e[24;1H
e[?1le>Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-279.19.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Wed Dec 19 07:05:20 UTC 2012
Command line: console=hvc0 xencons=tty0 root=/dev/xvda1 ro crashkernel=no
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000000a0000 (usable)
Xen: 00000000000a0000 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 0000000280000000 (usable)
DMI not present or invalid.
last_pfn = 0x280000 max_arch_pfn = 0x400000000
last_pfn = 0x100000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000100000000
init_memory_mapping: 0000000100000000-0000000280000000
RAMDISK: 02031000 - 04238000
No NUMA configuration found
Faking a node at 0000000000000000-0000000280000000
Bootmem setup node 0 0000000000000000-0000000280000000
NODE_DATA [0000000000008000 - 000000000003bfff]
bootmap [000000000003c000 - 000000000008bfff] pages 50
(9 early reservations) ==> bootmem [0000000000 - 0280000000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [000443b000 - 0004462000] XEN PAGETABLES ==> [000443b000 - 0004462000]
#2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#3 [0001000000 - 0002010024] TEXT DATA BSS ==> [0001000000 - 0002010024]
#4 [0002031000 - 0004238000] RAMDISK ==> [0002031000 - 0004238000]
#5 [0004238000 - 000443b000] XEN START INFO ==> [0004238000 - 000443b000]
#6 [0040000000 - 0280000000] XEN EXTRA ==> [0040000000 - 0280000000]
#7 [0000100000 - 00008db000] PGTABLE ==> [0000100000 - 00008db000]
#8 [0004462000 - 0005068000] PGTABLE ==> [0004462000 - 0005068000]
crashkernel: memory value expected
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00280000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x000000a0
0: 0x00000100 -> 0x00280000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 1 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 280100000 (gap: 280100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 4.2.2-22.el6 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff88002804f000 s94424 r8192 d24360 u126976
pcpu-alloc: s94424 r8192 d24360 u126976 alloc=31*4096
pcpu-alloc: [0] 0
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2583489
Policy zone: Normal
Kernel command line: console=hvc0 xencons=tty0 root=/dev/xvda1 ro crashkernel=no
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture…
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 764004k/10485760k available (5072k kernel code, 388k absent, 9721368k reserved, 7246k data, 1252k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:256
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 83886080 bytes of page_cgroup
please try ‘cgroup_disable=memory’ option if you don’t want memory cgroups
installing Xen timer for CPU 0
Detected 3100.064 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency… 6200.12 BogoMIPS (lpj=3100064)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 2
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
Freeing SMP alternatives: 34k freed
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20987 entries in 83 pages
Performance Events:
no APIC, boot with the “lapic” boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, using software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
Brought up 1 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x280000 max_arch_pfn = 0x400000000
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs…
Freeing initrd memory: 34844k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1372357929.890:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 1560
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring

  • Added public key 822A73AE165296BF
  • User ID: CentOS (Kernel Module GPG key)
    Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
    io scheduler noop registered
    io scheduler anticipatory registered
    io scheduler deadline registered
    io scheduler cfq registered (default)
    pci_hotplug: PCI Hot Plug PCI Core version: 0.5
    pciehp: PCI Express Hot Plug Controller Driver version: 0.4
    acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
    Non-volatile memory driver v1.3
    Linux agpgart interface v0.103
    crash memory driver: version 1.1
    Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
    brd: module loaded
    loop: module loaded
    input: Macintosh mouse button emulation as /devices/virtual/input/input0
    Fixed MDIO Bus: probed
    ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
    ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
    uhci_hcd: USB Universal Host Controller Interface driver
    PNP: No PS/2 controller found. Probing ports directly.
    mice: PS/2 mouse device common for all mice
    rtc_cmos: probe of rtc_cmos failed with error -16
    cpuidle: using governor ladder
    cpuidle: using governor menu
    EFI Variables Facility v0.08 2004-May-17
    usbcore: registered new interface driver hiddev
    usbcore: registered new interface driver usbhid
    usbhid: v2.6:USB HID core driver
    TCP cubic registered
    Initializing XFRM netlink socket
    NET: Registered protocol family 17
    registered taskstats version 1
    XENBUS: Device with no driver: device/vbd/2049
    XENBUS: Device with no driver: device/vbd/2050
    XENBUS: Device with no driver: device/vif/0
    XENBUS: Device with no driver: device/console/0
    drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
    Initalizing network drop monitor service
    Freeing unused kernel memory: 1252k freed
    Write protecting the kernel read-only data: 10240k
    Freeing unused kernel memory: 1052k freed
    Freeing unused kernel memory: 1732k freed
    dracut: dracut-004-284.el6_3.1
    udev: starting version 147
    dracut: Starting plymouth daemon
    xlblk_init: register_blkdev major: 202
    blkfront: xvde1: barriers disabled
    blkfront: xvde2: barriers disabled
    dracut Warning: No root device “block:/dev/xvda1” found

Fetching current status …
… status successfully retrieved (Down)

Re-fetching current status of host system ams2-xen-3-2.localdomain …
… status successfully retrieved (Webmin)

Enabling system at host boot time …
… done

Creating console access user …
… done

Perhaps the device names within the VM are different under Xen 4?

Did you create this VM using an image provided by Cloudmin? Also, did you create it before or after upgrading to Xen 4?

Didn’t think about that. It was from a custom image. I just tried downloading the CentOS 6 x86_64 image provided by Cloudmin, and that works fine.

Any ideas what might be causing this or should I just create new images?

I checked the xen config files for both and can’t see any differences in the disk, vif, kernel or extra settings.

Our own images are based off the Cloudmin ones with some additional packages and configuration, so I’m drawing a blank as to what might be different.

The problem is almost certainly that the device file paths in /etc/fstab in that image are incorrect - I seem to recall that these changes in Xen 4.0. I’d recommend creating a new image, perhaps based on the Cloudmin-provided one.