Justin's IT and Security Pages

Installing NexentaCP 2 RC1 on XenServer 5

with 23 comments

Although Nexenta uses the OpenSolaris kernel, there are a few unique steps that you’ll need to take in order to get Nexenta up and running paravirtualized (PV) on XenServer 5. These steps are a result of the lack of certain files in the default ramdisk on the Nexenta installation CD.

Also, the CDROM device seems to be handled differently in Nexenta than it is in OpenSolaris. As a result, these instructions will require you to fully install Nexenta in HVM mode and then flip the right bits to convert it to PV.

A few notes: I use /opt/kernels on my XenServer to store the kernels and ramdisks I use for my PV systems. You can substitute whatever you like for that directory; technically I think the “correct” place would be somewhere in “/usr/local”. Likewise, the names that I give the ramdisks and kernels are completely subjective; feel free to devise your own scheme.

  1. Using the “Other Media” option in XenCenter, install NexentaCP from the installation CD like you’d install a Windows system.
  2. Configure the installation appropriately at the presented prompts. Note that due to running in HVM mode, this initial installation will take a significant amount of time. Fear not, things will move much more quickly by the time you’re done.
  3. Copy /platform/i86pc/miniroot from the Nexenta installation CD to a system where you can work with it. I used an OpenSolaris system for the next steps.
  4. Rename miniroot to miniroot.gz.
  5. Decompress the miniroot.gz archive: gunzip miniroot.gz
  6. Mount the miniroot archive as a loopback device:
    mount -o loop [/path/to/]miniroot /[mountpoint]
  7. Change directories into the /platform directory on the mounted filesystem
  8. Copy the entire /platform/i86xpv directory from the HVM Nexenta system via scp over to the mounted filesystem. You might need to do this from the HVM Nexenta system (unless you svcadm enable ssh on the HVM Nexenta system).
  9. Move out of the miniroot filesystem on your working machine (the OpenSolaris box for me) and unmount the archive:
    unmount /[mountpoint]
  10. Recompress the miniroot archive: gzip miniroot
  11. Copy the newly modified miniroot archive over to an appropriate directory on the XenServer:
    scp ./miniroot.gz \
    root@xenserver:/opt/kernels/ramdisk_nexenta_install
  12. Log in to the Nexenta HVM system and copy the i86xpv kernel over to the XenServer:
    scp /platform/i86xpv/kernel/unix \
    root@xenserver:/opt/kernels/kernel_nexenta
  13. Log in to the XenServer and configure the Nexenta VM to use the kernel and ramdisk you’ve moved over:
    1. xe vm-param-set uuid=[VM UUID] \
      PV-kernel=/opt/kernels/kernel_nexenta
    2. xe vm-param-set uuid=[VM UUID] PV-ramdisk=\
      /opt/kernels/ramdisk_nexenta_install
    3. xe vm-param-set uuid=[VM UUID] \
      PV-args='/platform/i86xpv/kernel/unix \
      -B console=ttya -m milestone=0 -v'
    4. xe vm-param-set uuid=[VM UUID] \
      HVM-boot-policy=
  14. Reboot the newly paravirtualized Nexenta VM. It will fail to boot – that’s okay.
  15. When prompted to log in for maintenance, enter root with a blank password. If you have trouble getting the VM to respond to your keystrokes in XenCenter, try restarting XenCenter; every time I’ve done this, XenCenter has failed at this point and had to be restarted.
  16. Import the syspool zpool: zpool import -f syspool
  17. Configure the PV-args on the XenServer to specify the bootfs:
    xe vm-param-set uuid=[VM UUID] PV-args=\
    '/platform/i86xpv/kernel/unix -B console=ttya,\
    zfs-bootfs=syspool/rootfs-nmu-000,\
    bootpath="/xpvd/xdf@51712:a"'
  18. Reboot your Nexenta system. At this point, it should boot up and let you log in normally.
  19. Once booted, plumb your virtualized network interfaces: ifconfig xnf0 plumb
  20. Rename /etc/hostname.rtls0 and /etc/hostname6.rtls0 to /etc/hostname.xnf0 and /etc/hostname6.xnf0
  21. Reboot.
  22. Copy /platform/i86pc/boot_archive from the booted Nexenta system over to the XenServer as something like /opt/kernels/ramdisk_nexenta.
  23. Reconfigure the VM parameters to point at the new boot archive:
    xe vm-param-set uuid=[VM UUID] \
    PV-ramdisk=/opt/kernels/ramdisk_nexenta
  24. Reboot.

Congratulations! You now have a paravirtualized Nexenta core system. Upon booting, the VM screen should look something like:

v3.2.1 chgset ’58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)’
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta…
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: nexenta-test
Reading ZFS config: done.
Mounting ZFS filesystems: (2/2)

NexentaCore 2.0 RC1 (Hardy 8.04/b104+)

nexenta-test console login:

Take particular note: when you update a package that includes a kernel module, be sure to update the boot_archive (bootadm update-archive) and copy that updated archive over to the XenServer as /opt/kernels/ramdisk_nexenta BEFORE rebooting the Nexenta VM. I ran into a problem where the system could not load the console after doing an aptitude safe-upgrade and rebooting without updating the ramdisk.

Also, after installing I had to run an apt-get -f install to finish the installation of libtimedate-perl before doing an aptitude safe-upgrade. That package doesn’t appear to be installed correctly by the installer.

As always, please let me know if you have any comments or suggestions!

Advertisements

Written by Justin Thomas

April 3, 2009 at 4:28 am

23 Responses

Subscribe to comments with RSS.

  1. Hello!
    I do:
    Enter user name for system maintenance: root
    Enter root password for system maintenance: this blank

    but, not working.
    “Login incorrect or user root not authorized”

    eXeC001er

    April 13, 2009 at 6:00 pm

    • I’ll step through the instructions tonight to see if I can recreate your error.

      justindthomas

      April 13, 2009 at 7:32 pm

    • I do this steps on NCP2 beta 2. Probably problem in it. Today I will try on the version RC1.

      eXeC001er

      April 14, 2009 at 11:44 am

    • Let me know how it goes with the RC1 build. I just completed all of the steps and made a couple of minor modifications to the instructions. At the point where I authenticate in maintenance mode (step 15), my XenCenter software always stops responding to keystrokes until it’s restarted. I added that note to the instructions along with an example of what a successfully booted PV system looks like (to differentiate from an HVM booted system).

      I didn’t run up against the problem that you noted with the root password, though. I’m guessing that you’re right: it’s probably an issue with the beta.

      justindthomas

      April 14, 2009 at 5:28 pm

  2. New probem.
    step 1-17 : ok (reinstall NCP2 beta2 to NCP2 RC1)
    step 18: rebooting NCP and message:
    krtld: Unused kernel arguments: `bootpath=”/xpvd/xdf@51712:a”‘
    and fail to boot .
    Console login service(s) cannot run
    Enter user name for system maintenance (control-d to bypass):
    ….

    eXeC001er

    April 14, 2009 at 8:02 pm

    • Looks like you’re getting the right messages – although your PV-args string is wrong; the error indicates that you’ve set the string to the instructions in step 17, but you’re only at step 15. At step 15, the PV-args string should match what is shown in step 13 – there is no “bootpath” argument at that point.

      Regardless, I would think that you’d be able to log in as “root” with a blank password at that prompt (per step 15).

      justindthomas

      April 14, 2009 at 11:04 pm

    • eXeC001er, give this a try:

      http://www.nexenta.org/~tim/jdthomas/nexentacp2rc1.zip

      Download that file and unzip it on the Windows machine where you’re running XenCenter. Follow the instructions in the README.txt file.

      Once you complete that, you should have a “Nexenta Template” template in your XenCenter that you can use to spawn as many Nexenta servers as you want.

      justindthomas

      April 14, 2009 at 11:07 pm

  3. No, step 1-17 is ok, because i have installed NCP2 RC1.
    Next step is reboot, but boot is fail.
    I will try your templates.

    eXeC001er

    April 15, 2009 at 12:30 pm

    • Could you post the “PV-args” line output from “xe vm-param-list uuid=” into the comments?

      Let me know how the template works; you’ll be the first (that I know of) to give it a try.

      justindthomas

      April 15, 2009 at 3:32 pm

  4. templates not work. 😦
    May be there is a low space on a disk (11 GB free).
    XenCenter gives out a message:
    Importing VM…
    The specified storage repository has insufficient space.
    Check your setting and try again later.
    I will write “PV-args” later.

    eXeC001er

    April 15, 2009 at 5:29 pm

    • Yeah – it’s a 16GB image once it’s all blown up. That’s probably a bit large for Nexenta, but gives some room for growth.

      All right, send over the PV-args, I’m guessing there’s just a typo there; the string should be (all on one line):

      /platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=syspool/rootfs-nmu-000,bootpath="/xpvd/xdf@51712:a"

      justindthomas

      April 15, 2009 at 5:51 pm

    • Good news.
      Has connected other disk. Has begun importation.

      eXeC001er

      April 15, 2009 at 5:51 pm

    • Excellent! Keep me posted.

      justindthomas

      April 15, 2009 at 5:53 pm

    • Feel free to send me any questions in “#nexenta” on irc.freenode.net. I’m “[JT]”.

      justindthomas

      April 15, 2009 at 5:55 pm

  5. I finally got this working under CentOS 5.3 with rc3. Nexenta is using two physical disk partitions and the installed ZFS mirrored them automatically.

    This guide was essential.

    Here are a few quick deltas between what you have here and/or what I’ve used in other situations or wished I’d have known:

    1) use hda,hdb nomenclature for drives under HVM rather than sda,sdb
    3) under nexenta, mount the cdrom image using: mount -Fhsfs -o ro /dev/dsk/c1t0d0p0 /mnt/cdrom/
    6) mount loopback: lofiadm -a miniroot; mount /dev/lofi/1 /mnt/tmp
    11) skip down to 19 & 20 here to get data out. Even under HVM, we now have xnf0, and rtls0 is ‘DLPI link does not exist’. This seems to be newer Xen behavior. I brought it up straight away using ifconfig and route without a reboot.
    13) these are the ‘kernel’, ‘ramdisk’, and ‘extras’ statements in my text config files.
    15) do ‘service xend restart’ when the console crashes
    17) no spaces between the comma-separated elements – aside: it appears 51712 is a xen block device hard-coded number … I was wondering where that came from, but it appears to be consistent, and not machine-specific.

    Besides that, everything else looks good (other than my Solaris being a decade rusty). I didn’t need to get any updates or finish an install with apt, they look good in rc3. I’ll put config files and stuff up on my blog, but after the holiday weekend, which beings – now!

    Bill McGonigle

    May 23, 2009 at 8:23 am

    • Thanks Bill! That’s great information.

      justindthomas

      May 24, 2009 at 4:19 am

  6. Thanks for this tutorial it worked perfectly until step 16. I get the
    xdf@6: 1034668 blocksCD-ROM: discovery failed
    message.
    I have no idea how to continue because zpool import -f syspool did’t work.

    nitro

    May 27, 2009 at 10:35 am

    • If you’re using a distribution newer than RC1, there might be some things that I need to update. I haven’t tried the newer code yet.

      I’m not sure exactly what the issue that you’re seeing is, though. The “CD-ROM: discovery failed” message is normal, in my experience. If you’ve already imported the zpool, (it’s unclear to me whether you actually executed that command, or are unable to reach the prompt to execute the command), then you need to make sure that you’ve modified the PV-args as noted in step 17; the system will not boot unless those are changed.

      justindthomas

      May 27, 2009 at 6:32 pm

    • I reach the prompt and I am able to type in the zpool command but it fails:

      root@hardy_installcd:~# zpool import -f syspool
      cannot import ‘syspool’: no such pool available

      as far as good, where do I get the syspool?

      nitro

      May 28, 2009 at 12:50 pm

  7. @nitro: ‘syspool’ should have been created by the installer the first time through – ‘syspool’ is your zfs pool that the system sets up for itself for the root filesystem. Did it look like the installer completed successfully?

    If so, do you see any indication that it’s not detecting your hard drives successfully on the subsequent boot?

    Bill McGonigle

    May 28, 2009 at 2:34 pm

  8. […] image to use with Xen and XenServer. Take a look at the comments attached to the previous article here for […]

  9. If you’re using more recent versions of Xen (e.g. 3.3), you might have to use xvda, xvdb (etc) instead of hda, hdb in your Xen config for step 16 to work.

    Severn

    July 17, 2009 at 10:41 am

  10. These instructions mostly for NCP 3.01

    if you store the kernels and ramdisks under /opt/kernels under XCP(Xen Cloud Platform) You will get a bad path msg. They must be under /boot/guest

    XenCenter console became much more responsive booting into maint mode after i used the version override trick in XCP to get it to see XCP 1.0 as XenServer 5.6

    Joe

    May 8, 2011 at 11:57 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: