I have been setting up a two machines with software raid to use as Xen virtual environments. The two machines have 5 and 4 500GB disks. I decided to run Gentoo as the Dom0  OS because there is lots of good documentation for it and it has always been my favorite GNU/Linux distribution. The install of the OS and xen is easy. I opted for the latest Xen 4.0.1rc3-pre code. Just emerge xen and xen-tools to get all the prerequisites and the unmerge them and build the latest source. Creating the RAID arrays was also easy. I did the following.

for disk in a b c d ; do
parted -s /dev/sd$disk rm 1 rm 2 rm 3 rm 4 rm 5
parted -s /dev/sd$disk mklabel gpt
parted -s /dev/sd$disk mkpart STUFF 17.4kB 1049kB
parted -s /dev/sd$disk set 1 bios_grub on
parted -s /dev/sd$disk mkpart BOOT 1049kB 5GB
parted -s /dev/sd$disk mkpart SWAP 5GB 21GB
parted -s /dev/sd$disk mkpart ROOT 21GB 71GB
parted -s /dev/sd$disk mkpart XEN 71GB 500GB
for $part in 2 3 4 5 ; do
parted -s /dev/sd$disk set $part raid on
done
done

for md in 1 2 3 4 5 ; do
echo "Removing array"
mdadm --remove --stop /dev/md$md

echo "Creating Array"
if [ x$md == "x2" ] ; then
echo "yes" | mdadm --create /dev/md$md -f --level=1 --raid-devices=4 --metadata=0.90 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md
mkfs.ext2 /dev/md$md
elif [ x$md == "x5" ] ; then
echo "yes" | mdadm --create /dev/md$md -f --level=5 --raid-devices=4 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md
mkfs.xfs -f /dev/md$md
else
echo "yes" | mdadm --create /dev/md$md -f --level=1 --raid-devices=4 /dev/sda$md /dev/sdb$md /dev/sdc$md /dev/sdd$md
if [ x$md == "x3" ] ; then
mkswap /dev/md$md
elif [ x$md == "x4" ] ; then
mkfs.ext4 /dev/md$md
fi
fi
done

mount /dev/md4 /mnt/gentoo
mkdir /mnt/gentoo/boot
mkdir /mnt/gentoo/xen
mount /dev/md2 /mnt/gentoo/boot
swapon /dev/md3
mount /dev/md5 /mnt/gentoo/xen
cd /mnt/gentoo
tar -xjpf /root/stage3-amd64-20100514.tar.bz2 -C /mnt/gentoo
cp -f /root/portage-latest.tar.bz2 /mnt/gentoo
cd /
mount -t proc proc /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev
cp -L /etc/resolv.conf /mnt/gentoo/etc/
chroot /mnt/gentoo /bin/bash
env-update && source /etc/profile
cd /usr
tar -xjpf /portage-latest.tar.bz2

cp /usr/share/zoneinfo/Europe/London /etc/localtime
cd /etc
echo "127.0.0.1 xen1.sinodun.com xen1 localhost" > hosts
hostname xen1
emerge --sync --quiet
eselect profile set 7
cat << EOF > /etc/make.conf
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j5"
CHOST="x86_64-pc-linux-gnu"
USE="threads -gnome -kde -minimal -qt4 nfs branding dbus hal jpeg lock session startup-notification thunar X hal fglrx lvm qemu-ifup qemu_user_targets_x86_64 qemu_softmmu_targets_x86_64 aio udev libvirtd lxc network phyp policykit caps nfsv4 parted qemu ccache xen nat ioemu hvm mmx sse sse2 X"

GENTOO_MIRRORS="ftp://mirror.qubenet.net/mirror/gentoo/ ftp://gentoo.wheel.sk/pub/linux/gentoo/ ftp://ftp.swin.edu.au/gentoo"
FEATURES="ccache"
CCACHE_DIR="/var/tmp/ccache"
CCACHE_SIZE="2G"

EOF

mkdir /etc/portage/
cat << EOF > /etc/portage/package.keywords
dev-python/python-augeas
app-emulation/qemulator
app-admin/augeas
sys-fs/device-mapper
net-firewall/shorewall6-lite
net-nds/rpcbind
net-fs/nfs4-acl-tools
app-emulation/qemu-kvm
net-analyzer/netcat6
x11-libs/cairo
app-emulation/xen
app-emulation/xen-tools
app-emulation/virt-manager
app-emulation/virtinst
app-emulation/libvirt
sys-kernel/xen-sources
net-firewall/ipsec-tools
<sys-boot/grub-9999 **
EOF

emerge ccache mdadm gentoolkit

cat /dev/null  > /etc/mdadm.conf
for md in 1 2 3 4 5 ; do
echo DEVICE /dev/sd[abcd]$md  >> /etc/mdadm.conf
done
for md in 1 2 3 4 5 ; do
echo ARRAY /dev/md$md UUID=`mdadm -D /dev/md$md | grep UUID | awk ' { print $3 } '` >> /etc/mdadm.conf
done

Then I grabbed a copy of the 2.6.34 kernel and built it with all the Xen features turned on, all the software RAID on and all the drivers for my disk controllers on. I don’t use an initramfs so they all need to be in the kernel for raid auto-detection to work at boot time.

Then the problems started…

I could not get any release of grub2 to mutiboot xen and my kernel. Eventually I found that the fix was to get the latest grub code from Bazaar and to use that with a few work arounds.

• First unmerge any existing grub
• build and install the code from bazaar
• run grub-mkdevicemap. This will give a file called /boot/grub/device.map that looks something like this:

(hd0)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405131
(hd1)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405138
(hd2)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405139
(hd3)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405141
(hd4)    /dev/disk/by-id/md-uuid-5e9a37eb:6f0d060b:3cdba84c:474e7631
(hd5)    /dev/disk/by-id/md-uuid-66f62ccf:59a6fd08:3cdba84c:474e7631
(hd6)    /dev/disk/by-id/md-uuid-69ac7eac:5395e23b:3cdba84c:474e7631
(hd7)    /dev/disk/by-id/md-uuid-6fd706d5:94eafa75:3cdba84c:474e7631
(hd8)    /dev/disk/by-id/md-uuid-f4168a40:b212cce4:3cdba84c:474e7631

• This needs altering to  give md names to the RAID disks. A script like this will do the hard part

for i in 1 2 3 4 5
do echo -n "(md$i)    /dev/disk/by-id/md-uuid-"  >> /boot/grub/device.map
mdadm -D /dev/md$i | grep UUID | awk ' { print $3 } ' >> /boot/grub/device.map
done
# cut the old lines oiut of the file

• to read like this

(hd0)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405131
(hd1)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405138
(hd2)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405139
(hd3)    /dev/disk/by-id/ata-SAMSUNG_HD502HJ_S20BJ9CZ405141
(md1)    /dev/disk/by-id/md-uuid-6fd706d5:94eafa75:3cdba84c:474e7631
(md2)    /dev/disk/by-id/md-uuid-69ac7eac:5395e23b:3cdba84c:474e7631
(md3)    /dev/disk/by-id/md-uuid-f4168a40:b212cce4:3cdba84c:474e7631
(md4)    /dev/disk/by-id/md-uuid-66f62ccf:59a6fd08:3cdba84c:474e7631
(md5)    /dev/disk/by-id/md-uuid-5e9a37eb:6f0d060b:3cdba84c:474e7631

• Then run grub-mkconfig -o /tmp/grub.cfg. This file will look something like this

Generating grub.cfg ...
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /usr/local/etc/grub.d and settings from /usr/local/etc/default/grub
#

### BEGIN /usr/local/etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
 load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
 set saved_entry="${prev_saved_entry}"
 save_env saved_entry
 set prev_saved_entry=
 save_env prev_saved_entry
 set boot_once=true
fi

function savedefault {
 if [ -z "${boot_once}" ]; then
 saved_entry="${chosen}"
 save_env saved_entry
 fi
}

function load_video {
 insmod vbe
 insmod vga
 insmod video_bochs
 insmod video_cirrus
}

insmod raid
insmod mdraid
insmod part_gpt
insmod part_gpt
insmod part_gpt
insmod part_gpt
insmod ext2
set root='(md4)'
search --no-floppy --fs-uuid --set 25de7b8d-303c-47fa-8ebb-9b2276f17b5f
if loadfont /usr/local/share/grub/unicode.pf2 ; then
 set gfxmode=640x480
 load_video
 insmod gfxterm
fi
if terminal_output gfxterm ; then true ; else
 # For backward compatibility with versions of terminal.mod that don't
 # understand terminal_output
 terminal gfxterm
fi
insmod raid
insmod mdraid
insmod part_gpt
insmod part_gpt
insmod part_gpt
insmod part_gpt
insmod ext2
set root='(md2)'
search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce
set locale_dir=($root)/grub/locale
set lang=
insmod gettext
set timeout=5
### END /usr/local/etc/grub.d/00_header ###

### BEGIN /usr/local/etc/grub.d/09_xen ###
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry "vmlinuz-2.6.34-xen" {
insmod raid
insmod mdraid
insmod part_gpt
insmod ext2
set root='(md2)'
multiboot /xen.gz dummy=dummy
module /vmlinuz-2.6.34-xen dummy=dummy root=/dev/md4 max_loop=64
}

### END /usr/local/etc/grub.d/09_xen ###

### BEGIN /usr/local/etc/grub.d/10_linux ###
Found linux image: /boot/vmlinuz-2.6.34-xen
menuentry 'GNU/Linux, with Linux 2.6.34-xen' --class gnu-linux --class gnu --class os {
 load_video
 insmod raid
 insmod mdraid
 insmod part_gpt
 insmod part_gpt
 insmod part_gpt
 insmod part_gpt
 insmod ext2
 set root='(md2)'
 search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce
 echo    'Loading Linux 2.6.34-xen ...'
 linux    /vmlinuz-2.6.34-xen root=/dev/md4 ro  
}
menuentry 'GNU/Linux, with Linux 2.6.34-xen (recovery mode)' --class gnu-linux --class gnu --class os {
 load_video
 insmod raid
 insmod mdraid
 insmod part_gpt
 insmod part_gpt
 insmod part_gpt
 insmod part_gpt
 insmod ext2
 set root='(md2)'
 search --no-floppy --fs-uuid --set 9de924a9-a266-4952-90a5-2aec6a9d3cce
 echo    'Loading Linux 2.6.34-xen ...'
 linux    /vmlinuz-2.6.34-xen root=/dev/md4 ro single
}
### END /usr/local/etc/grub.d/10_linux ###

### BEGIN /usr/local/etc/grub.d/20_linux_xen ###
### END /usr/local/etc/grub.d/20_linux_xen ###

### BEGIN /usr/local/etc/grub.d/30_os-prober ###
### END /usr/local/etc/grub.d/30_os-prober ###

### BEGIN /usr/local/etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /usr/local/etc/grub.d/40_custom ###

### BEGIN /usr/local/etc/grub.d/41_custom ###
if [ -f  $prefix/custom.cfg ]; then
 source $prefix/custom.cfg;
fi
### END /usr/local/etc/grub.d/41_custom ###
done

• Unfortunately, those two UUIDs are totally wrong. They come from grub-probe (run by grub-mkconfig)

grub-probe -d --target=fs_uuid /dev/md2
9de924a9-a266-4952-90a5-2aec6a9d3cce

• /dev/md2 is my boot disk but according to mdadm the UUID should be:

mdadm -D /dev/md2
/dev/md2:
 Version : 0.90
 Creation Time : Fri Jul  9 11:25:32 2010
 Raid Level : raid1
 Array Size : 4881664 (4.66 GiB 5.00 GB)
 Used Dev Size : 4881664 (4.66 GiB 5.00 GB)
 Raid Devices : 4
 Total Devices : 4
Preferred Minor : 2
 Persistence : Superblock is persistent

 Update Time : Sat Jul 10 13:08:31 2010
 State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
 Spare Devices : 0

 UUID : 69ac7eac:5395e23b:3cdba84c:474e7631
 Events : 0.36

 Number   Major   Minor   RaidDevice State
 0       8        2        0      active sync   /dev/sda2
 1       8       18        1      active sync   /dev/sdb2
 2       8       34        2      active sync   /dev/sdc2
 3       8       50        3      active sync   /dev/sdd2

• Fix that by hand in the config don’t forget t change the UUID format to 12345678-1234-1234-1234-123456789abc and copy it to /boot/grub/grub.cfg.
• Finally run grub-install /dev/md1 and you’re done.
• Exit the install

exit
umount /mnt/gentoo/proc
umount /mnt/gentoo/dev
umount /dev/md2
umount /dev/md5
swapoff /dev/md3
umount /dev/md4