I said I'd never blog

As probably already pretty obvious here, my main Operating System is NetBSD. Over the past few weeks, I've been trying virtualization, using Xen with NetBSD as the native host. This excellent howto describes the whole procedure better than I could do it.

This wasn't without a few issues though. Here's what was important in my case:

Xen device files

Don't forget to create the device files:

# cd /dev && MAKEDEV xen

The bootloader

For 64-bit hosts, or simply if your / partition is "big" (somewhere between 512MB and 4096MB in my experience), then you must use NetBSD's native bootloader. It requires NetBSD 5.0 (or -current, and you can also get it to work on 4.0 and below).

First, don't forget to copy /usr/mdec/boot to /boot when updating the bootloader (symptom is "unknown command"). Then, copy and modify /boot.cfg from /usr/src/etc/etc.amd64/boot.cfg:

menu=Boot normally:boot /netbsd.gz
menu=Boot Xen:modules enabled;load /netbsd.xen0 bootdev=wd0a ro console=pc;multiboot /xen.gz dom0_mem=1024M
menu=Boot single user:boot /netbsd.gz -s
menu=Disable ACPI:boot /netbsd.gz -2
menu=Disable ACPI and SMP:/boot /netbsd.gz -12
menu=Drop to boot prompt:prompt
default=1
timeout=10

Create the virtual machine

You can use a plain file as the guest, to be able to switch between Xen and qemu at will. I have detailed my procedure below.

Create and partition a 20GB file

I'm not sure if this blocksize is optimal. You may also feel free to set swap space at this stage, I chose to proceed without any:

# dd if=/dev/zero of=ldeb bs=1024*1024 count=20480
# vnconfig -c vnd0 ldeb
# fdisk -u vnd0
[...]
Do you want to change our idea of what BIOS thinks? [n] n

Partition table:
0: <UNUSED>
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>
Bootselector disabled.
No active partition.
Which partition do you want to change?: [none] 0
The data for partition 0 is:
<UNUSED>
sysid: [0..255 default: 169] 131
start: [0..xcyl default: 63, 0cyl, 0MB] 63
size: [0..xcyl default: x, xcyl, xMB] $
bootmenu: []

Partition table:
0: Linux native (sysid 131)
   start 63, size 41942977 (20480 MB, Cyls 0-2610/212/34), Active
       PBR is not bootable: All bytes are identical (0x00)
[...]
Which partition do you want to change?: [none]

We haven't written the MBR back to disk yet.  This is your last chance.
Partition table:
[...]
Should we write new partition table? [n] y

NetBSD also needs a disklabel to identify the different partitions:

# disklabel -Ii vnd0
partition> a
Filesystem type [?] [4.2BSD]: Linux Ext2
Start offset ('x' to start after partition 'x') [0c, 0s, 0M]: 63
Partition size ('$' for all remaining) [xc, ys, zM]: $
partition> W
Label disk [n]? y
Label written
partition> Q

Format and mount the partition

I force a block size of 4KB for compatibility with NetBSD, and 128 bytes per inode for performance:

# mke2fs -b 4096 -I 128 /dev/vnd0a
# mkdir /mnt/ldeb
# mount /dev/vnd0a /mnt/ldeb

Download and extract hackable:1

# wget http://build.hackable1.org/Hackable1-Openmoko-Freerunner-cross-daily.tar.gz
# tar -xzvpf Hackable1-Openmoko-Freerunner-cross-daily.tar.gz -C /mnt/ldeb

Download and install the kernel modules

We're using Xen 3.0.4 now to make sure it is well supported as of today. The kernel itself doesn't need to be on the partition, and will be installed later:

# wget http://bits.xensource.com/oss-xen/release/3.0.4-1/bin.tgz/xen-3.0.4_1-install-x86_64.tgz
# tar xzvf xen-3.0.4_1-install-x86_64.tgz dist/install/lib/modules \

dist/install/boot

# mkdir -p /mnt/ldeb/lib/modules
# mv dist/install/lib/modules/2.6.16.33-xen /mnt/ldeb/lib/modules

Unmount the partition

And take care of the vnd pseudo-device as well:

# umount /mnt/ldeb
# vnconfig -u vnd0

Install the kernel image

Copy the kernel next to the image:

# cp dist/install/boot/vmlinuz-2.6.16.33-xen /home/xen/ldeb.xenU

Configure the virtual machine

This is my configuration file, with the comments stripped off:

# cat > /usr/pkg/etc/xen/ldeb << EOF
kernel = "/home/xen/ldeb.xenU"
memory = 768
name = "ldeb"
cpu = 1
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]
disk = [ 'file:/home/xen/ldeb,0x300,w' ]
root = "hda1"
extra = "root=/dev/hda1 ro"
EOF

Boot the machine

Finally, if all is well:

# xm create -c /usr/pkg/etc/xen/ldeb

You should then be able to login as root. It's recommended to install libc6-xen, some other tweaks may be useful too. I'll be all ears :)

Update:

• the disklabel is better now:
• using "Linux Ext2" instead of "4.2BSD"
• starting at offset 63
• the flag to mke2fs was wrong (it's -I to specify the bytes per inode)
• it's probably vnd0a instead of vnd0e