I have been setting up a two machines with software raid to use as Xen virtual environments. The two machines have 5 and 4 500GB disks. I decided to run Gentoo as the Dom0 OS because there is lots of good documentation for it and it has always been my favorite GNU/Linux distribution. The install of the OS and xen is easy. I opted for the latest Xen 4.0.1rc3-pre code. Just emerge xen and xen-tools to get all the prerequisites and the unmerge them and build the latest source. Creating the RAID arrays was also easy. I did the following.
for disk in a b c d ; do parted -s /dev/sd$disk rm 1 rm 2 rm 3 rm 4 rm 5 parted -s /dev/sd$disk mklabel gpt parted -s /dev/sd$disk mkpart STUFF 17.4kB 1049kB parted -s /dev/sd$disk set 1 bios_grub on parted -s /dev/sd$disk mkpart BOOT 1049kB 5GB parted -s /dev/sd$disk mkpart SWAP 5GB 21GB parted -s /dev/sd$disk mkpart ROOT 21GB 71GB parted -s /dev/sd$disk mkpart XEN 71GB 500GB for $part in 2 3 4 5 ; do parted -s /dev/sd$disk set $part raid on done done for md in 1 2 3 4 5 ; do echo "Removing array" mdadm --remove --stop /dev/md$md echo "Creating Array" if [ x$md == "x2" ] ; then echo "yes" | mdadm --create /dev/md$md -f --level=1 --raid-devices=4 --metadata=0.90 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md mkfs.ext2 /dev/md$md elif [ x$md == "x5" ] ; then echo "yes" | mdadm --create /dev/md$md -f --level=5 --raid-devices=4 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md mkfs.xfs -f /dev/md$md else echo "yes" | mdadm --create /dev/md$md -f --level=1 --raid-devices=4 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md if [ x$md == "x3" ] ; then mkswap /dev/md$md elif [ x$md == "x4" ] ; then mkfs.ext4 /dev/md$md fi fi done mount /dev/md4 /mnt/gentoo mkdir /mnt/gentoo/boot mkdir /mnt/gentoo/xen mount /dev/md2 /mnt/gentoo/boot swapon /dev/md3 mount /dev/md5 /mnt/gentoo/xen cd /mnt/gentoo tar -xjpf /root/stage3-amd64-20100514.tar.bz2 -C /mnt/gentoo cp -f /root/portage-latest.tar.bz2 /mnt/gentoo cd / mount -t proc proc /mnt/gentoo/proc mount -o bind /dev /mnt/gentoo/dev cp -L /etc/resolv.conf /mnt/gentoo/etc/ chroot /mnt/gentoo /bin/bash env-update && source /etc/profile cd /usr tar -xjpf /portage-latest.tar.bz2 cp /usr/share/zoneinfo/Europe/London /etc/localtime cd /etc echo "127.0.0.1 xen1.sinodun.com xen1 localhost" > hosts hostname xen1 emerge --sync --quiet eselect profile set 7 cat << EOF > /etc/make.conf CFLAGS="-march=native -O2 -pipe" CXXFLAGS="${CFLAGS}" MAKEOPTS="-j5" CHOST="x86_64-pc-linux-gnu" USE="threads -gnome -kde -minimal -qt4 nfs branding dbus hal jpeg lock session startup-notification thunar X hal fglrx lvm qemu-ifup qemu_user_targets_x86_64 qemu_softmmu_targets_x86_64 aio udev libvirtd lxc network phyp policykit caps nfsv4 parted qemu ccache xen nat ioemu hvm mmx sse sse2 X" GENTOO_MIRRORS="ftp://mirror.qubenet.net/mirror/gentoo/ ftp://gentoo.wheel.sk/pub/linux/gentoo/ ftp://ftp.swin.edu.au/gentoo" FEATURES="ccache" CCACHE_DIR="/var/tmp/ccache" CCACHE_SIZE="2G" EOF mkdir /etc/portage/ cat << EOF > /etc/portage/package.keywords dev-python/python-augeas app-emulation/qemulator app-admin/augeas sys-fs/device-mapper net-firewall/shorewall6-lite net-nds/rpcbind net-fs/nfs4-acl-tools app-emulation/qemu-kvm net-analyzer/netcat6 x11-libs/cairo app-emulation/xen app-emulation/xen-tools app-emulation/virt-manager app-emulation/virtinst app-emulation/libvirt sys-kernel/xen-sources net-firewall/ipsec-tools <sys-boot/grub-9999 ** EOF emerge ccache mdadm gentoolkit cat /dev/null > /etc/mdadm.conf for md in 1 2 3 4 5 ; do echo DEVICE /dev/sd[abcd]$md >> /etc/mdadm.conf done for md in 1 2 3 4 5 ; do echo ARRAY /dev/md$md UUID=`mdadm -D /dev/md$md | grep UUID | awk ' { print $3 } '` >> /etc/mdadm.conf done
Then I grabbed a copy of the 2.6.34 kernel and built it with all the Xen features turned on, all the software RAID on and all the drivers for my disk controllers on. I don’t use an initramfs so they all need to be in the kernel for raid auto-detection to work at boot time.
Then the problems started…
I could not get any release of grub2 to mutiboot xen and my kernel. Eventually I found that the fix was to get the latest grub code from Bazaar and to use that with a few work arounds.
- First unmerge any existing grub
- build and install the code from bazaar
- run grub-mkdevicemap. This will give a file called /boot/grub/device.map that looks something like this:
(hd0) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405131 (hd1) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405138 (hd2) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405139 (hd3) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405141 (hd4) /dev/disk/by-id/md-uuid-5e9a37eb:6f0d060b:3cdba84c:474e7631 (hd5) /dev/disk/by-id/md-uuid-66f62ccf:59a6fd08:3cdba84c:474e7631 (hd6) /dev/disk/by-id/md-uuid-69ac7eac:5395e23b:3cdba84c:474e7631 (hd7) /dev/disk/by-id/md-uuid-6fd706d5:94eafa75:3cdba84c:474e7631 (hd8) /dev/disk/by-id/md-uuid-f4168a40:b212cce4:3cdba84c:474e7631
- This needs altering to give md names to the RAID disks. A script like this will do the hard part
for i in 1 2 3 4 5 do echo -n "(md$i) /dev/disk/by-id/md-uuid-" >> /boot/grub/device.map mdadm -D /dev/md$i | grep UUID | awk ' { print $3 } ' >> /boot/grub/device.map done # cut the old lines oiut of the file
- to read like this
(hd0) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405131 (hd1) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405138 (hd2) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405139 (hd3) /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405141 (md1) /dev/disk/by-id/md-uuid-6fd706d5:94eafa75:3cdba84c:474e7631 (md2) /dev/disk/by-id/md-uuid-69ac7eac:5395e23b:3cdba84c:474e7631 (md3) /dev/disk/by-id/md-uuid-f4168a40:b212cce4:3cdba84c:474e7631 (md4) /dev/disk/by-id/md-uuid-66f62ccf:59a6fd08:3cdba84c:474e7631 (md5) /dev/disk/by-id/md-uuid-5e9a37eb:6f0d060b:3cdba84c:474e7631
- Then run grub-mkconfig -o /tmp/grub.cfg. This file will look something like this
Generating grub.cfg ... # # DO NOT EDIT THIS FILE # # It is automatically generated by grub-mkconfig using templates # from /usr/local/etc/grub.d and settings from /usr/local/etc/default/grub # ### BEGIN /usr/local/etc/grub.d/00_header ### if [ -s $prefix/grubenv ]; then load_env fi set default="0" if [ "${prev_saved_entry}" ]; then set saved_entry="${prev_saved_entry}" save_env saved_entry set prev_saved_entry= save_env prev_saved_entry set boot_once=true fi function savedefault { if [ -z "${boot_once}" ]; then saved_entry="${chosen}" save_env saved_entry fi } function load_video { insmod vbe insmod vga insmod video_bochs insmod video_cirrus } insmod raid insmod mdraid insmod part_gpt insmod part_gpt insmod part_gpt insmod part_gpt insmod ext2 set root='(md4)' search --no-floppy --fs-uuid --set 25de7b8d-303c-47fa-8ebb-9b2276f17b5f if loadfont /usr/local/share/grub/unicode.pf2 ; then set gfxmode=640x480 load_video insmod gfxterm fi if terminal_output gfxterm ; then true ; else # For backward compatibility with versions of terminal.mod that don't # understand terminal_output terminal gfxterm fi insmod raid insmod mdraid insmod part_gpt insmod part_gpt insmod part_gpt insmod part_gpt insmod ext2 set root='(md2)' search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce set locale_dir=($root)/grub/locale set lang= insmod gettext set timeout=5 ### END /usr/local/etc/grub.d/00_header ### ### BEGIN /usr/local/etc/grub.d/09_xen ### exec tail -n +3 $0 # This file provides an easy way to add custom menu entries. Simply type the # menu entries you want to add after this comment. Be careful not to change # the 'exec tail' line above. menuentry "vmlinuz-2.6.34-xen" { insmod raid insmod mdraid insmod part_gpt insmod ext2 set root='(md2)' multiboot /xen.gz dummy=dummy module /vmlinuz-2.6.34-xen dummy=dummy root=/dev/md4 max_loop=64 } ### END /usr/local/etc/grub.d/09_xen ### ### BEGIN /usr/local/etc/grub.d/10_linux ### Found linux image: /boot/vmlinuz-2.6.34-xen menuentry 'GNU/Linux, with Linux 2.6.34-xen' --class gnu-linux --class gnu --class os { load_video insmod raid insmod mdraid insmod part_gpt insmod part_gpt insmod part_gpt insmod part_gpt insmod ext2 set root='(md2)' search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce echo 'Loading Linux 2.6.34-xen ...' linux /vmlinuz-2.6.34-xen root=/dev/md4 ro } menuentry 'GNU/Linux, with Linux 2.6.34-xen (recovery mode)' --class gnu-linux --class gnu --class os { load_video insmod raid insmod mdraid insmod part_gpt insmod part_gpt insmod part_gpt insmod part_gpt insmod ext2 set root='(md2)' search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce echo 'Loading Linux 2.6.34-xen ...' linux /vmlinuz-2.6.34-xen root=/dev/md4 ro single } ### END /usr/local/etc/grub.d/10_linux ### ### BEGIN /usr/local/etc/grub.d/20_linux_xen ### ### END /usr/local/etc/grub.d/20_linux_xen ### ### BEGIN /usr/local/etc/grub.d/30_os-prober ### ### END /usr/local/etc/grub.d/30_os-prober ### ### BEGIN /usr/local/etc/grub.d/40_custom ### # This file provides an easy way to add custom menu entries. Simply type the # menu entries you want to add after this comment. Be careful not to change # the 'exec tail' line above. ### END /usr/local/etc/grub.d/40_custom ### ### BEGIN /usr/local/etc/grub.d/41_custom ### if [ -f $prefix/custom.cfg ]; then source $prefix/custom.cfg; fi ### END /usr/local/etc/grub.d/41_custom ### done
- Unfortunately, those two UUIDs are totally wrong. They come from grub-probe (run by grub-mkconfig)
grub-probe -d --target=fs_uuid /dev/md2 9de924a9-a266-4952-90a5-2aec6a9d3cce
- /dev/md2 is my boot disk but according to mdadm the UUID should be:
mdadm -D /dev/md2 /dev/md2: Version : 0.90 Creation Time : Fri Jul 9 11:25:32 2010 Raid Level : raid1 Array Size : 4881664 (4.66 GiB 5.00 GB) Used Dev Size : 4881664 (4.66 GiB 5.00 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Sat Jul 10 13:08:31 2010 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : 69ac7eac:5395e23b:3cdba84c:474e7631 Events : 0.36 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 18 1 active sync /dev/sdb2 2 8 34 2 active sync /dev/sdc2 3 8 50 3 active sync /dev/sdd2
- Fix that by hand in the config don’t forget t change the UUID format to 12345678-1234-1234-1234-123456789abc and copy it to /boot/grub/grub.cfg.
- Finally run grub-install /dev/md1 and you’re done.
- Exit the install
exit umount /mnt/gentoo/proc umount /mnt/gentoo/dev umount /dev/md2 umount /dev/md5 swapoff /dev/md3 umount /dev/md4
Jonathan Bayer