Michael Altfield's gravatar

xen hung at “Checking for hardware changes”

So, xen is really beginning to piss me off. I turned off all my machines to do a snapshot, and when I tried to bring them back up, they were all in the ‘blocked’ state. Upon further investigation (using virt-manager/xm console), I found that they were hung at the “Checking for hardware changes” item in their boot process. This could be a CentOS/RHEL 5 issue, but I’m putting my money on xen.

domU hangs at "Checking for hardware changes"

To fix it, I did the following:

  1. Mount the DomU image somewhere on my Dom0’s filesystem
  2. ‘wipe’ the /etc/init.d/kudzu script in the DomU
  3. unmount
  4. re-start the DomU

Note: This effectively prevents kudzu (the RHEL mechanism to detect hardware changes) from starting on boot. Beware that this is probably a terrible idea; I have no idea what the implications of preforming these actions will have on your system. I *do* know, however, that it got my machine to boot!

[root@dom0 /]# xm list guest
Name                                      ID Mem(MiB) VCPUs State   Time(s)
guest                                       17      511     1 -b----     26.2
[root@dom0 /]# xm destroy guest

[root@dom0 /]# ls /dev/mapper # no loop devices
control
[root@dom0 /]# kpartx -av /var/lib/xen/images/guest.img
add map loop1p1 : 0 6072507 linear /dev/loop1 63
add map loop1p2 : 0 2104515 linear /dev/loop1 6072570
[root@dom0 /]# ls /dev/mapper # note the loop devices
control  loop1p1  loop1p2
[root@dom0 /]# mkdir -p /tmp/mnt/tst
[root@dom0 /]# mount /dev/mapper/loop1p1 /tmp/mnt/tst

[root@dom0 /]# cd /tmp/mnt/tst
[root@dom0 tst]# ls
bin   dev  home  lost+found  misc  net  proc  sbin     srv  tmp  var
boot  etc  lib   media       mnt   opt  root  selinux  sys  usr
[root@dom0 tst]# cd etc/init.d
[root@dom0 init.d]# mv kudzu kudzu.original.20090815
[root@dom0 init.d]# echo "exit 0" > kudzu

[root@dom0 init.d]# cd /
[root@dom0 /]# umount /tmp/mnt/tst
[root@dom0 /]# ls /dev/mapper # note the loop devices
control  loop1p1  loop1p2
[root@dom0 /]# kpartx -d /dev/loop1 # (loopX as seen from the above command)
[root@dom0 /]# ls /dev/mapper # no more loop devices!
control
[root@dom0 /]# xm create guest
Using config file "/etc/xen/guest".
Started domain guest
</code>

I booted successfully! Hope this helps someone…

4 comments to xen hung at “Checking for hardware changes”

  • JR

    There are more “proper” (less destructive) ways to do this. From your screen shot above, we see that the problem only occurs when entering run level 3 and it hangs at the change detection (kudzu) line. So, the simplest thing to do is configure kudzu so that it does not start in level 3… which will at least get you past that hang point.

    Offline way:
    1) Follow your process above to mount your VM
    2) Disable the kudzu level 3 launcher by renaming the launch file from capital S to lower case s. This disables the service but leaves all the files and data intact in case you want to turn it back on quickly. Do this by issuing the command: mv /etc/rc3.d/S05kudzu /etc/rc3.d/s05kudzu (again, notice the “s” before “05” has changed case).
    3) unmount the disk and power on the VM

    Online way:
    1) Boot your VM (while watching the console) and hit “e” at the boot loader to go into edit mode
    2) Mouse down to the line that begins with “kernel” and hit “e” again to edit that line
    3) Add a space and then a 1 to the end of the line, this will boot the system in single user mode (which drops to a shell before your kudzu process hangs at level 3)
    4) Hit “b” to boot the system with the new config (this is a one-time setting, it will not be in effect on next boot)
    5) Once you’re dropped into a command prompt, disable the kudzu service in run level 3 by issuing the command: chkconfig –level 3 kudzu off
    6) Double check your setting with: chkconfig –list | grep kudzu (you should see a “3:off” on the line returned)
    7) reboot to start the system normally, the kudzu process will not attempt to start

    If you’re using this VM as a server, and the hardware isn’t going to change drastically any time soon, you’ll probably want to leave kudzu disabled anyways.

  • guttersnipe

    @JR

    Thanks for your input. I’ll probably do something like that in the future.

    As for the “Online way,” I’ve never really seen a boot loader when creating my Xen VMs. It looks like you have to add the line

    extra=”1″

    …to the xen config file (in this case: /etc/xen/guest) in order to get into single user mode on a DomU. From there, you should be able to pick back up on #5 in your instructions.

  • Ed Heron

    An alternate solution would be to press ‘I’, when it says you can, and select ‘N’ when it asks if you want to start kudzu.

    Then, when the VM is booted, run the chkconfig –level 3 kudzu off

    While you’re at it, why are we doing cpu microcode updates on a VM? Should we run chkconfig microcode_ctl off

  • This is a year late, but when I’ve had this happen, its usually that the xenconsoled in the dom0 has died. Restart it and all should be good.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>