xen hung at “Checking for hardware changes”

By Michael Altfield, on August 13th, 2009

So, xen is really beginning to piss me off. I turned off all my machines to do a snapshot, and when I tried to bring them back up, they were all in the ‘blocked’ state. Upon further investigation (using virt-manager/xm console), I found that they were hung at the “Checking for hardware changes” item in their boot process. This could be a CentOS/RHEL 5 issue, but I’m putting my money on xen.

domU hangs at "Checking for hardware changes"

To fix it, I did the following:

Mount the DomU image somewhere on my Dom0’s filesystem
‘wipe’ the /etc/init.d/kudzu script in the DomU
unmount
re-start the DomU

Note: This effectively prevents kudzu (the RHEL mechanism to detect hardware changes) from starting on boot. Beware that this is probably a terrible idea; I have no idea what the implications of preforming these actions will have on your system. I *do* know, however, that it got my machine to boot!

[root@dom0 /]# xm list guest
Name                                      ID Mem(MiB) VCPUs State   Time(s)
guest                                       17      511     1 -b----     26.2
[root@dom0 /]# xm destroy guest

[root@dom0 /]# ls /dev/mapper # no loop devices
control
[root@dom0 /]# kpartx -av /var/lib/xen/images/guest.img
add map loop1p1 : 0 6072507 linear /dev/loop1 63
add map loop1p2 : 0 2104515 linear /dev/loop1 6072570
[root@dom0 /]# ls /dev/mapper # note the loop devices
control  loop1p1  loop1p2
[root@dom0 /]# mkdir -p /tmp/mnt/tst
[root@dom0 /]# mount /dev/mapper/loop1p1 /tmp/mnt/tst

[root@dom0 /]# cd /tmp/mnt/tst
[root@dom0 tst]# ls
bin   dev  home  lost+found  misc  net  proc  sbin     srv  tmp  var
boot  etc  lib   media       mnt   opt  root  selinux  sys  usr
[root@dom0 tst]# cd etc/init.d
[root@dom0 init.d]# mv kudzu kudzu.original.20090815
[root@dom0 init.d]# echo "exit 0" &gt; kudzu

[root@dom0 init.d]# cd /
[root@dom0 /]# umount /tmp/mnt/tst
[root@dom0 /]# ls /dev/mapper # note the loop devices
control  loop1p1  loop1p2
[root@dom0 /]# kpartx -d /dev/loop1 # (loopX as seen from the above command)
[root@dom0 /]# ls /dev/mapper # no more loop devices!
control
[root@dom0 /]# xm create guest
Using config file "/etc/xen/guest".
Started domain guest
</code>

I booted successfully! Hope this helps someone…

Michael Altfield

Hi, I’m Michael Altfield. I write articles about opsec, privacy, and devops ➡

About Michael

2009-08-14 at 09:06 · Reply

There are more “proper” (less destructive) ways to do this. From your screen shot above, we see that the problem only occurs when entering run level 3 and it hangs at the change detection (kudzu) line. So, the simplest thing to do is configure kudzu so that it does not start in level 3… which will at least get you past that hang point.

Offline way:
1) Follow your process above to mount your VM
2) Disable the kudzu level 3 launcher by renaming the launch file from capital S to lower case s. This disables the service but leaves all the files and data intact in case you want to turn it back on quickly. Do this by issuing the command: mv /etc/rc3.d/S05kudzu /etc/rc3.d/s05kudzu (again, notice the “s” before “05” has changed case).
3) unmount the disk and power on the VM

Online way:
1) Boot your VM (while watching the console) and hit “e” at the boot loader to go into edit mode
2) Mouse down to the line that begins with “kernel” and hit “e” again to edit that line
3) Add a space and then a 1 to the end of the line, this will boot the system in single user mode (which drops to a shell before your kudzu process hangs at level 3)
4) Hit “b” to boot the system with the new config (this is a one-time setting, it will not be in effect on next boot)
5) Once you’re dropped into a command prompt, disable the kudzu service in run level 3 by issuing the command: chkconfig –level 3 kudzu off
6) Double check your setting with: chkconfig –list | grep kudzu (you should see a “3:off” on the line returned)
7) reboot to start the system normally, the kudzu process will not attempt to start

If you’re using this VM as a server, and the hardware isn’t going to change drastically any time soon, you’ll probably want to leave kudzu disabled anyways.

guttersnipe

2009-08-14 at 13:17 · Reply

@JR

Thanks for your input. I’ll probably do something like that in the future.

As for the “Online way,” I’ve never really seen a boot loader when creating my Xen VMs. It looks like you have to add the line

extra=”1″

…to the xen config file (in this case: /etc/xen/guest) in order to get into single user mode on a DomU. From there, you should be able to pick back up on #5 in your instructions.