Flower Network Flow Analysis: UPDATE

Things are moving on this item. Here’s a current screenshot of the application in action:

I consider the code for this “alpha” quality at best. If you’re brave and want to give it a try, keep reading:

I still haven’t created a VM image for this yet, but you can download the Java web application (I use Glassfish as the container) and UNIX/Linux (I use Debian) sensor sources from this web site:

https://sourceforge.net/projects/flower-nfa/

All code is available in the Subversion repository.

The current status is that the network map and tabular chart functions work well along with the “constraint” (i.e., specifying traffic patterns to view) mechanism. Both IPv4 and IPv6 traffic can be profiled and queried.

On the client end, Firefox 3 and Safari 4 both work quite well. Safari (really, Webkit) has a minor issue with the title graphic – it doesn’t scale as the window grows and shrinks. That doesn’t impact the functionality at all.

Firefox 3.5 has introduced some pretty significant bugs in SVG rendering that make the scale functionality of the map work poorly; I’ve encountered numerous browser hangs and crashes when I scale the image past about 2.0x and try to move it. It looks to me like maybe they’ve done some work with the text rendering that has introduced this new instability. Hopefully that will be resolved quickly. I’ll try to figure a concise way to file a bug report about it.

Chrome has the same issues as Safari. Chrome also renders the SVG text differently than Safari or Firefox; not a big deal but it does impact readability a little bit.

Opera works, but is “tweaky”. Hard to really describe exactly how.

Internet Explorer does not work at all, of course (no SVG capability). Adobe’s plug-in might work, but I’ve never tried it.

The “flower_config.xml” file in the web application root directory contains the specifications for the networks to profile and some protocol information to normalize non-TCP flows. That must be modified to match your environment.

User name and password for the application are hard-coded as “flower/flower”. I’ll add a real authentication mechanism as things progress towards a beta release.

I use a URL base of “/flower”, but theoretically you should be able to choose whatever you want. (Strike that, the sensor is hard-coded to use “http://. . .:8080/flower’ – I will fix that, though).

Available constraints are: “tcp, udp, esp, ipv4, ipv6, sip , dip , port , protocol ” Additional constraints are easy to add as I find the need.

The analysis engine runs on Debian Linux (and probably most other Linux distributions). It requires libcurl and libpcap. Compilation is a simple: “./autogen.sh;./configure;make” The resulting executable is src/sensor and is executed with the following options:

-b Executes the sensor in the background
-s <address> Analysis server IP address to report to
-d <device> Ethernet device to listen on (i.e., eth0)

If not run as background, you will be presented with a prompt: “FCE>”. Here you can query the running analysis engine. Just type “help” to see the available commands.

Integrating OpenSolaris with Windows 2008 Active Directory

This is a topic that I have struggled with for several months. Until today, I had only been able to convince my OpenSolaris 2009.06 system to share files via SMB as a workgroup; every attempt to join my Windows 2008 Active Directory domain was greeted with a “no domain controllers found” error – even when Kerberos was working flawlessly and LDAP queries were returning everything as they should.

After reading many forums and conversations about what might be causing this error, I finally decided that it must be an idiosyncrasy with OpenSolaris attempting to operate in a multiple-DC environment. I bit the bullet and demoted 2 of my 3 test DCs.

To my chagrin, the OSOL box still refused to join the domain with a “no domain controllers found” error.

I performed a snoop of the traffic and realized that the error message was not entirely truthful. The DC was being found, but the authentication was failing. This differed from my previous attempts which were thwarted when pulling up the “_ldap._tcp.dc._msdcs.domain.name” DNS record and receiving multiple servers in the reply.

I had read on multiple forums that the lmauth level might cause problems with Windows 2008 domains. This command solved that problem:

sharectl set -p lmauth_level=2 smb

I was elated that I could authenticate from my windows machine transparently . . . until I rebooted. Upon reboot, the SMB server service failed. Period. No interesting logs. It just wouldn’t start.

So, for about 30 minutes I gave up. I had almost decided that I’d just run it in workgroup mode and be happy, but I thought I’d do one more Google search. What I discovered was that the SMB server problems were far more common than the other problems I had been running into – in fact, it appeared that snv_114 included some code changes to solve a number of SMB problems. The difficulty was that OpenSolaris was (and is) stuck on snv_111b.

Another alternative exists. Solaris Express Community Edition (SXCE) is currently at snv_115. That distribution does not have some of the niceties of OpenSolaris (package manager, etc.), but appears to be a bit more advanced. I decided to give it a shot.

As it turns out, SXCE rocks. Everything works (almost) right out of the box. I could even install it from the CLI – no messing around with X or VNC just to get the OS installed. It even gives me the option to install a “Core” system, without all of the nonsense I don’t really need. This is what Solaris used to be, and what it should be again.

One problem that came up when I rebooted turned out to be due to unresolvable SID mappings. The symptom was that, again, my domain users were unable to authenticate – log messages in /var/adm/messages reflected the failures. This command solved that issue:


svccfg -s idmap setprop config/unresolvable_sid_mapping = boolean: true
svcadm refresh idmap


I’ll keep an eye on OpenSolaris, but for now I think SXCE will do nicely. I’ll write up a quick blog post on how to get SXCE to play nicely on XenServer in a bit; the process is similar to OpenSolaris, but the CLI install options make things a bit easier.

So to recap: If you want SMB domain authentication with Windows 2008, use SXCE with a single AD domain controller and be sure to execute the sharectl and idmap commands above. Beyond those caveats, all of the instructions peppering OSOL blog posts out there will do fine to help you get things up and running.

Also remember, snoop is your friend.

NexentaCP 2 RC3 on XenServer 5

I recently went through the motions to install Nexenta CP 2 RC3 on XenServer 5, following the directions that I had previously posted for RC1. I’m happy to confirm that the process for RC1 seems to work fine for RC3.

As an aside, a poster to that original article (Bill McGonigle) included some instructions for using lofiadm to alter the miniroot archive from the Nexenta installation DVD image to use with Xen and XenServer (I used a separate OpenSolaris system for that step). Take a look at the comments attached to the previous article for details.

Flower Network Flow Analysis

I’ve been working on building out some critical parts of an application that has been a work in progress for some time. Here’s a screen shot of the latest updates showing the traffic on my messy home network.

More to come on this soon!

NexentaCP 2 RC1 XenServer Template

I put together a XenServer template for Nexenta. If you’re interested, please give it a try and post your results in the comments:

NexentaCP 2 RC1 Template

Here are the instructions from the README.txt file in that package:

Steps to install this package:

1. Copy the “nexentacp2_rc1.tar.gz” kernel/ramdisk archive to the root directory on the XenServer using scp (or, since you’ll likely be doing this from Windows, you can just use WinSCP):

scp ./nexentacp2_rc1.tar.gz root@xenserver:/

2. At the console of the XenServer (you can use XenCenter or just connect via SSH), decompress the kernel/ramdisk archive (be sure to use the “P” option to maintain the directory structure):

[root@xen /]# tar xvPf ./nexentacp2_rc1.tar.gz

The kernel should now be in /opt/xen/kernels and the ramdisk should be in /opt/xen/ramdisks on your XenServer. There will also be a failsafe ramdisk in the ramdisks folder should you ever need it.

3. Import the XVA into XenCenter. The image size is about 5GB; you’ll need an appropriate amount of space available to execute the import.

Thanks to rootard for posting a mirror of the rather large (~1GB) ZIP file.

UPDATED: Created a slimmer template with a helper script included (see README.txt). Thanks to eXeC001er for helping me to test and validate the original package!

OpenSolaris snv_111 on XenServer 5

Just a quick follow-up to my previous post on problems with snv_110 and Xen. snv_111 showed up in IPS recently and I immediately updated one of my OpenSolaris test systems from snv_109 to the new code. The problem with hald/Xen in snv_110 looks to have been solved and everything seems to be working as intended!

Installing NexentaCP 2 RC1 on XenServer 5

Although Nexenta uses the OpenSolaris kernel, there are a few unique steps that you’ll need to take in order to get Nexenta up and running paravirtualized (PV) on XenServer 5. These steps are a result of the lack of certain files in the default ramdisk on the Nexenta installation CD.

Also, the CDROM device seems to be handled differently in Nexenta than it is in OpenSolaris. As a result, these instructions will require you to fully install Nexenta in HVM mode and then flip the right bits to convert it to PV.

A few notes: I use /opt/kernels on my XenServer to store the kernels and ramdisks I use for my PV systems. You can substitute whatever you like for that directory; technically I think the “correct” place would be somewhere in “/usr/local”. Likewise, the names that I give the ramdisks and kernels are completely subjective; feel free to devise your own scheme.

1. Using the “Other Media” option in XenCenter, install NexentaCP from the installation CD like you’d install a Windows system.
2. Configure the installation appropriately at the presented prompts. Note that due to running in HVM mode, this initial installation will take a significant amount of time. Fear not, things will move much more quickly by the time you’re done.
3. Copy /platform/i86pc/miniroot from the Nexenta installation CD to a system where you can work with it. I used an OpenSolaris system for the next steps.
4. Rename miniroot to miniroot.gz.
5. Decompress the miniroot.gz archive: gunzip miniroot.gz
6. Mount the miniroot archive as a loopback device:
   mount -o loop [/path/to/]miniroot /[mountpoint]
7. Change directories into the /platform directory on the mounted filesystem
8. Copy the entire /platform/i86xpv directory from the HVM Nexenta system via scp over to the mounted filesystem. You might need to do this from the HVM Nexenta system (unless you svcadm enable ssh on the HVM Nexenta system).
9. Move out of the miniroot filesystem on your working machine (the OpenSolaris box for me) and unmount the archive:
   unmount /[mountpoint]
10. Recompress the miniroot archive: gzip miniroot
11. Copy the newly modified miniroot archive over to an appropriate directory on the XenServer:
    scp ./miniroot.gz \
root@xenserver:/opt/kernels/ramdisk_nexenta_install
12. Log in to the Nexenta HVM system and copy the i86xpv kernel over to the XenServer:
    scp /platform/i86xpv/kernel/unix \
root@xenserver:/opt/kernels/kernel_nexenta
13. Log in to the XenServer and configure the Nexenta VM to use the kernel and ramdisk you’ve moved over:
    1. xe vm-param-set uuid=[VM UUID] \
PV-kernel=/opt/kernels/kernel_nexenta
    2. xe vm-param-set uuid=[VM UUID] PV-ramdisk=\
/opt/kernels/ramdisk_nexenta_install
    3. xe vm-param-set uuid=[VM UUID] \
PV-args='/platform/i86xpv/kernel/unix \
-B console=ttya -m milestone=0 -v'
    4. xe vm-param-set uuid=[VM UUID] \
HVM-boot-policy=
14. Reboot the newly paravirtualized Nexenta VM. It will fail to boot – that’s okay.
15. When prompted to log in for maintenance, enter root with a blank password. If you have trouble getting the VM to respond to your keystrokes in XenCenter, try restarting XenCenter; every time I’ve done this, XenCenter has failed at this point and had to be restarted.
16. Import the syspool zpool: zpool import -f syspool
17. Configure the PV-args on the XenServer to specify the bootfs:
    xe vm-param-set uuid=[VM UUID] PV-args=\
'/platform/i86xpv/kernel/unix -B console=ttya,\
zfs-bootfs=syspool/rootfs-nmu-000,\
bootpath="/xpvd/xdf@51712:a"'
18. Reboot your Nexenta system. At this point, it should boot up and let you log in normally.
19. Once booted, plumb your virtualized network interfaces: ifconfig xnf0 plumb
20. Rename /etc/hostname.rtls0 and /etc/hostname6.rtls0 to /etc/hostname.xnf0 and /etc/hostname6.xnf0
21. Reboot.
22. Copy /platform/i86pc/boot_archive from the booted Nexenta system over to the XenServer as something like /opt/kernels/ramdisk_nexenta.
23. Reconfigure the VM parameters to point at the new boot archive:
    xe vm-param-set uuid=[VM UUID] \
PV-ramdisk=/opt/kernels/ramdisk_nexenta
24. Reboot.

Congratulations! You now have a paravirtualized Nexenta core system. Upon booting, the VM screen should look something like:

v3.2.1 chgset ‘58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)’
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta…
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: nexenta-test
Reading ZFS config: done.
Mounting ZFS filesystems: (2/2)

NexentaCore 2.0 RC1 (Hardy 8.04/b104+)

nexenta-test console login:

Take particular note: when you update a package that includes a kernel module, be sure to update the boot_archive (bootadm update-archive) and copy that updated archive over to the XenServer as /opt/kernels/ramdisk_nexenta BEFORE rebooting the Nexenta VM. I ran into a problem where the system could not load the console after doing an aptitude safe-upgrade and rebooting without updating the ramdisk.

Also, after installing I had to run an apt-get -f install to finish the installation of libtimedate-perl before doing an aptitude safe-upgrade. That package doesn’t appear to be installed correctly by the installer.

As always, please let me know if you have any comments or suggestions!

NexentaCP 2 RC1 on XenServer: SUCCESS

After my post last night, I did a little more digging; all that was required was to import the syspool from the HVM install (zpool import -f syspool) and reset the PV-args.

It looks like I need to re-configure the interfaces (the rtls0 interface would not plumb, presumably because the PV process changed the name), but I think we’re there. I’ll clean up the process and post comprehensive instructions later tonight.

NexentaCP 2 RC1 on XenServer

Nexenta is an OpenSolaris distribution that combines a GNU userland with an OpenSolaris kernel. Because it’s based on OpenSolaris (kernel 104 for this release), we should be able to get it up and running on XenServer in a manner similar to what I’ve covered in previous blog entries.

Unfortunately, I haven’t yet been fully successful. Nonetheless, I thought it might be useful to post my findings:

1. The boot archive on the installation CD (miniroot) does not include the paravirtualized kernel.
2. The boot process doesn’t seem to recognize the CDROM device as a CDROM.
3. Once booted, all of the disks and virtualized devices seem to be detected successfully (except the CDROM). See the output below for details.
4. Installing via HVM (which does work, but is terribly slow) and attempting to convert the installation to PV doesn’t seem to work – the kernel will not boot the “syspool/root-nmu-000″ ZFS environment. This isn’t surprising; I couldn’t get this to work with OpenSolaris either. With OpenSolaris I was able to just install directly into a PV environment, thereby circumventing that difficulty altogether.

I solved the first problem by modifying the miniroot archive (/platform/i86pc/miniroot on the installation CD) using an OpenSolaris box. It’s a gzipped UNIX Fast File System file. To alter it, you’ll need to unzip and mount the file (use “-o loop”) on OpenSolaris and just add the /platform/i86xpv files from a fully installed Nexenta instance to the archive. After adding the files, unmount it, re-gzip it and transfer it over to the XenServer as the VM’s PV-ramdisk.

Similarly, take the /platform/i86xpv/kernel/unix file and move it over to the XenServer as the VM’s PV-kernel.

Now, with the PV-args set thusly:

xe vm-param-set uuid=[VM UUID] PV-args='/platform/i86xpv/kernel/unix -B console=ttya -m milestone=0 -v'

. . . you should be able to boot the PV kernel. What you should see is:

module /platform/i86xpv/kernel/unix: text at [0xf4c00000, 0xf4cdca6f] data at 0xf5000000
module /kernel/genunix: text at [0xf4cdca70, 0xf4ea541f] data at 0xf506c140
v3.2.1 chgset '58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)'
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta...
features: 611e66c6
mem = 1048576K (0x40000000)
Using default device instance data
root nexus = i86xpv
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
ramdisk0 at root
ramdisk0 is /ramdisk
root on /ramdisk:a fstype ufs
/cpus (cpunex0) online
pseudo-device: dld0
dld0 is /pseudo/dld@0
xpvd0 at root
xencons@0, xencons0
xencons0 is /xpvd/xencons@0
cpu0: x86 (chipid 0x0 GenuineIntel F64 family 15 model 6 step 4 clock 2660 MHz)
cpu0: Intel(r) Xeon(tm) CPU 2.66GHz
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: stmf_sbd0
stmf_sbd0 is /pseudo/stmf_sbd@0
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
Hostname: hardy_installcd
iscsi0 at root
iscsi0 is /iscsi
xsvc0 at root: space 0 offset 0
xsvc0 is /xsvc@0,0
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex@1
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap@0
pseudo-device: power0
power0 is /pseudo/power@0
pseudo-device: zfs0
zfs0 is /pseudo/zfs@0
pseudo-device: srn0
srn0 is /pseudo/srn@0
pseudo-device: stmf0
stmf0 is /pseudo/stmf@0
pseudo-device: fct0
fct0 is /pseudo/fct@0
pseudo-device: ucode0
ucode0 is /pseudo/ucode@0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm@0
pseudo-device: fcp0
fcp0 is /pseudo/fcp@0
/xpvd/xnf@0 (xnf0) online
xdf@51712, xdf0
xdf0 is /xpvd/xdf@51712
/xpvd/xdf@51712 (xdf0) online
xdf@51760, xdf1
xdf1 is /xpvd/xdf@51760
/xpvd/xdf@51760 (xdf1) online
xenbus@0, xenbus0
xenbus0 is /xpvd/xenbus@0
domcaps@0, domcaps0
domcaps0 is /xpvd/domcaps@0
balloon@0, balloon0
balloon0 is /xpvd/balloon@0
NOTICE: xdf@51712: failed to read feature-barrier
xdf@51712: 41943040 blocksxdf@51760: 1034668 blocksCD-ROM: discovery failed
ERROR: svc:/system/filesystem/usr:default failed to mount /dev/fd (see 'svcs -x' for details)
Apr 2 12:42:44 svc.startd[7]: svc:/system/filesystem/usr:default: Method "/lib/svc/method/fs-usr" failed with exit status 95.
Apr 2 12:42:44 svc.startd[7]: system/filesystem/usr:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
Enter user name for system maintenance (control-d to bypass):

From here, you can log in as root (blank password), but you can’t do much. If someone knows how to solve the CDROM detection problem, I think we can move further.

Hopefully Nexenta includes provisions to allow booting the system as a domU in their next release. It seems silly not to, considering that the framework is already pretty much there.

OpenSolaris Kernel Updates

Thus far, most updates have fared well on my OpenSolaris system, following the procedures described in my previous blog entry.

The sole exception is kernel snv_110; there appears to be a problem with hald that impacts Xen systems and is already corrected in snv_111.

For now, snv_109 seems to work with no trouble.