NexentaCP 2 RC3 on XenServer 5

I recently went through the motions to install Nexenta CP 2 RC3 on XenServer 5, following the directions that I had previously posted for RC1. I’m happy to confirm that the process for RC1 seems to work fine for RC3.

As an aside, a poster to that original article (Bill McGonigle) included some instructions for using lofiadm to alter the miniroot archive from the Nexenta installation DVD image to use with Xen and XenServer (I used a separate OpenSolaris system for that step). Take a look at the comments attached to the previous article for details.

NexentaCP 2 RC1 XenServer Template

I put together a XenServer template for Nexenta. If you’re interested, please give it a try and post your results in the comments:

NexentaCP 2 RC1 Template

Here are the instructions from the README.txt file in that package:

Steps to install this package:

1. Copy the “nexentacp2_rc1.tar.gz” kernel/ramdisk archive to the root directory on the XenServer using scp (or, since you’ll likely be doing this from Windows, you can just use WinSCP):

scp ./nexentacp2_rc1.tar.gz root@xenserver:/

2. At the console of the XenServer (you can use XenCenter or just connect via SSH), decompress the kernel/ramdisk archive (be sure to use the “P” option to maintain the directory structure):

[root@xen /]# tar xvPf ./nexentacp2_rc1.tar.gz

The kernel should now be in /opt/xen/kernels and the ramdisk should be in /opt/xen/ramdisks on your XenServer. There will also be a failsafe ramdisk in the ramdisks folder should you ever need it.

3. Import the XVA into XenCenter. The image size is about 5GB; you’ll need an appropriate amount of space available to execute the import.

Thanks to rootard for posting a mirror of the rather large (~1GB) ZIP file.

UPDATED: Created a slimmer template with a helper script included (see README.txt). Thanks to eXeC001er for helping me to test and validate the original package!

Installing NexentaCP 2 RC1 on XenServer 5

Although Nexenta uses the OpenSolaris kernel, there are a few unique steps that you’ll need to take in order to get Nexenta up and running paravirtualized (PV) on XenServer 5. These steps are a result of the lack of certain files in the default ramdisk on the Nexenta installation CD.

Also, the CDROM device seems to be handled differently in Nexenta than it is in OpenSolaris. As a result, these instructions will require you to fully install Nexenta in HVM mode and then flip the right bits to convert it to PV.

A few notes: I use /opt/kernels on my XenServer to store the kernels and ramdisks I use for my PV systems. You can substitute whatever you like for that directory; technically I think the “correct” place would be somewhere in “/usr/local”. Likewise, the names that I give the ramdisks and kernels are completely subjective; feel free to devise your own scheme.

1. Using the “Other Media” option in XenCenter, install NexentaCP from the installation CD like you’d install a Windows system.
2. Configure the installation appropriately at the presented prompts. Note that due to running in HVM mode, this initial installation will take a significant amount of time. Fear not, things will move much more quickly by the time you’re done.
3. Copy /platform/i86pc/miniroot from the Nexenta installation CD to a system where you can work with it. I used an OpenSolaris system for the next steps.
4. Rename miniroot to miniroot.gz.
5. Decompress the miniroot.gz archive: gunzip miniroot.gz
6. Mount the miniroot archive as a loopback device:
   mount -o loop [/path/to/]miniroot /[mountpoint]
7. Change directories into the /platform directory on the mounted filesystem
8. Copy the entire /platform/i86xpv directory from the HVM Nexenta system via scp over to the mounted filesystem. You might need to do this from the HVM Nexenta system (unless you svcadm enable ssh on the HVM Nexenta system).
9. Move out of the miniroot filesystem on your working machine (the OpenSolaris box for me) and unmount the archive:
   unmount /[mountpoint]
10. Recompress the miniroot archive: gzip miniroot
11. Copy the newly modified miniroot archive over to an appropriate directory on the XenServer:
    scp ./miniroot.gz \
root@xenserver:/opt/kernels/ramdisk_nexenta_install
12. Log in to the Nexenta HVM system and copy the i86xpv kernel over to the XenServer:
    scp /platform/i86xpv/kernel/unix \
root@xenserver:/opt/kernels/kernel_nexenta
13. Log in to the XenServer and configure the Nexenta VM to use the kernel and ramdisk you’ve moved over:
    1. xe vm-param-set uuid=[VM UUID] \
PV-kernel=/opt/kernels/kernel_nexenta
    2. xe vm-param-set uuid=[VM UUID] PV-ramdisk=\
/opt/kernels/ramdisk_nexenta_install
    3. xe vm-param-set uuid=[VM UUID] \
PV-args='/platform/i86xpv/kernel/unix \
-B console=ttya -m milestone=0 -v'
    4. xe vm-param-set uuid=[VM UUID] \
HVM-boot-policy=
14. Reboot the newly paravirtualized Nexenta VM. It will fail to boot – that’s okay.
15. When prompted to log in for maintenance, enter root with a blank password. If you have trouble getting the VM to respond to your keystrokes in XenCenter, try restarting XenCenter; every time I’ve done this, XenCenter has failed at this point and had to be restarted.
16. Import the syspool zpool: zpool import -f syspool
17. Configure the PV-args on the XenServer to specify the bootfs:
    xe vm-param-set uuid=[VM UUID] PV-args=\
'/platform/i86xpv/kernel/unix -B console=ttya,\
zfs-bootfs=syspool/rootfs-nmu-000,\
bootpath="/xpvd/xdf@51712:a"'
18. Reboot your Nexenta system. At this point, it should boot up and let you log in normally.
19. Once booted, plumb your virtualized network interfaces: ifconfig xnf0 plumb
20. Rename /etc/hostname.rtls0 and /etc/hostname6.rtls0 to /etc/hostname.xnf0 and /etc/hostname6.xnf0
21. Reboot.
22. Copy /platform/i86pc/boot_archive from the booted Nexenta system over to the XenServer as something like /opt/kernels/ramdisk_nexenta.
23. Reconfigure the VM parameters to point at the new boot archive:
    xe vm-param-set uuid=[VM UUID] \
PV-ramdisk=/opt/kernels/ramdisk_nexenta
24. Reboot.

Congratulations! You now have a paravirtualized Nexenta core system. Upon booting, the VM screen should look something like:

v3.2.1 chgset ’58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)’
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta…
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: nexenta-test
Reading ZFS config: done.
Mounting ZFS filesystems: (2/2)

NexentaCore 2.0 RC1 (Hardy 8.04/b104+)

nexenta-test console login:

Take particular note: when you update a package that includes a kernel module, be sure to update the boot_archive (bootadm update-archive) and copy that updated archive over to the XenServer as /opt/kernels/ramdisk_nexenta BEFORE rebooting the Nexenta VM. I ran into a problem where the system could not load the console after doing an aptitude safe-upgrade and rebooting without updating the ramdisk.

Also, after installing I had to run an apt-get -f install to finish the installation of libtimedate-perl before doing an aptitude safe-upgrade. That package doesn’t appear to be installed correctly by the installer.

As always, please let me know if you have any comments or suggestions!

NexentaCP 2 RC1 on XenServer: SUCCESS

After my post last night, I did a little more digging; all that was required was to import the syspool from the HVM install (zpool import -f syspool) and reset the PV-args.

It looks like I need to re-configure the interfaces (the rtls0 interface would not plumb, presumably because the PV process changed the name), but I think we’re there. I’ll clean up the process and post comprehensive instructions later tonight.

NexentaCP 2 RC1 on XenServer

Nexenta is an OpenSolaris distribution that combines a GNU userland with an OpenSolaris kernel. Because it’s based on OpenSolaris (kernel 104 for this release), we should be able to get it up and running on XenServer in a manner similar to what I’ve covered in previous blog entries.

Unfortunately, I haven’t yet been fully successful. Nonetheless, I thought it might be useful to post my findings:

1. The boot archive on the installation CD (miniroot) does not include the paravirtualized kernel.
2. The boot process doesn’t seem to recognize the CDROM device as a CDROM.
3. Once booted, all of the disks and virtualized devices seem to be detected successfully (except the CDROM). See the output below for details.
4. Installing via HVM (which does work, but is terribly slow) and attempting to convert the installation to PV doesn’t seem to work – the kernel will not boot the “syspool/root-nmu-000” ZFS environment. This isn’t surprising; I couldn’t get this to work with OpenSolaris either. With OpenSolaris I was able to just install directly into a PV environment, thereby circumventing that difficulty altogether.

I solved the first problem by modifying the miniroot archive (/platform/i86pc/miniroot on the installation CD) using an OpenSolaris box. It’s a gzipped UNIX Fast File System file. To alter it, you’ll need to unzip and mount the file (use “-o loop”) on OpenSolaris and just add the /platform/i86xpv files from a fully installed Nexenta instance to the archive. After adding the files, unmount it, re-gzip it and transfer it over to the XenServer as the VM’s PV-ramdisk.

Similarly, take the /platform/i86xpv/kernel/unix file and move it over to the XenServer as the VM’s PV-kernel.

Now, with the PV-args set thusly:

xe vm-param-set uuid=[VM UUID] PV-args='/platform/i86xpv/kernel/unix -B console=ttya -m milestone=0 -v'

. . . you should be able to boot the PV kernel. What you should see is:

module /platform/i86xpv/kernel/unix: text at [0xf4c00000, 0xf4cdca6f] data at 0xf5000000
module /kernel/genunix: text at [0xf4cdca70, 0xf4ea541f] data at 0xf506c140
v3.2.1 chgset '58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)'
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta...
features: 611e66c6
mem = 1048576K (0x40000000)
Using default device instance data
root nexus = i86xpv
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
ramdisk0 at root
ramdisk0 is /ramdisk
root on /ramdisk:a fstype ufs
/cpus (cpunex0) online
pseudo-device: dld0
dld0 is /pseudo/dld@0
xpvd0 at root
xencons@0, xencons0
xencons0 is /xpvd/xencons@0
cpu0: x86 (chipid 0x0 GenuineIntel F64 family 15 model 6 step 4 clock 2660 MHz)
cpu0: Intel(r) Xeon(tm) CPU 2.66GHz
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: stmf_sbd0
stmf_sbd0 is /pseudo/stmf_sbd@0
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
Hostname: hardy_installcd
iscsi0 at root
iscsi0 is /iscsi
xsvc0 at root: space 0 offset 0
xsvc0 is /xsvc@0,0
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex@1
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap@0
pseudo-device: power0
power0 is /pseudo/power@0
pseudo-device: zfs0
zfs0 is /pseudo/zfs@0
pseudo-device: srn0
srn0 is /pseudo/srn@0
pseudo-device: stmf0
stmf0 is /pseudo/stmf@0
pseudo-device: fct0
fct0 is /pseudo/fct@0
pseudo-device: ucode0
ucode0 is /pseudo/ucode@0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm@0
pseudo-device: fcp0
fcp0 is /pseudo/fcp@0
/xpvd/xnf@0 (xnf0) online
xdf@51712, xdf0
xdf0 is /xpvd/xdf@51712
/xpvd/xdf@51712 (xdf0) online
xdf@51760, xdf1
xdf1 is /xpvd/xdf@51760
/xpvd/xdf@51760 (xdf1) online
xenbus@0, xenbus0
xenbus0 is /xpvd/xenbus@0
domcaps@0, domcaps0
domcaps0 is /xpvd/domcaps@0
balloon@0, balloon0
balloon0 is /xpvd/balloon@0
NOTICE: xdf@51712: failed to read feature-barrier
xdf@51712: 41943040 blocksxdf@51760: 1034668 blocksCD-ROM: discovery failed
ERROR: svc:/system/filesystem/usr:default failed to mount /dev/fd  (see 'svcs -x' for details)
Apr  2 12:42:44 svc.startd[7]: svc:/system/filesystem/usr:default: Method "/lib/svc/method/fs-usr" failed with exit status 95.
Apr  2 12:42:44 svc.startd[7]: system/filesystem/usr:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
Enter user name for system maintenance (control-d to bypass):

From here, you can log in as root (blank password), but you can’t do much. If someone knows how to solve the CDROM detection problem, I think we can move further.

Hopefully Nexenta includes provisions to allow booting the system as a domU in their next release. It seems silly not to, considering that the framework is already pretty much there.