An hour with Solaris Live Upgrade

Last week I had the opportunity to do some work with the Live Upgrade feature of Sun Solaris. I had been vaguely aware of it's capabilities and we had been including a provision for it on our customer server builds, but it was only yesterday that I sat down and tried to do an upgrade to the latest Solaris 10 update 8.

Live Upgrade is a capability where the system administrator can upgrade to a new version of Solaris while the existing operating system in running. The only downtime experienced is a scheduled reboot at the end of the process to initialise the new version. If something goes wrong, the original version of the OS is still available for booting.

The way it works is based around the concept of a "boot environment". The default environment is the operating system you're running at the moment. On our servers, we have been creating a 20GB root (/) filesystem. A second 20GB slice is also created, but not used (nominally mounted as /lu so we remember it's there).

The first step in running the live upgrade is to create a new boot environment. Firstly, the /lu partition was unmounted, and commented out in /etc/vfstab. Once that was done, the new boot environment was setup:

# lucreate -n osupgrade -m /:/dev/md/dsk/d30:ufs

Okay, so here we are creating a new environment called "osupgrade" and saying that the root ("/") filesystem will be installed on the device /dev/md/dsk/d30 and that the filesystem type will be UFS (it can also do ZFS but we didn't have the correct setup on my test system). This bit takes a while, depending on how much you have on your root filesystem.

For those unfamiliar with the /dev/md/ part, this is a Solaris Volume Manager (SVM) metadevice. In reality, "d30" is a mirror that contains two submirrors (probably called d31 and d32). These submirrors are comprised of one of more disk slices. In other words, in the above command, the new boot environment will be installed onto a new mirrored disk.

At the end of the lucreate command, you can actually look at the new, mounted boot environment and see that it's basically a copy of your existing root filesystem. The next step is to upgrade it. To do this, I mounted the install location of our Jumpstart server over NFS and initiated the live upgrade:

# luupgrade -u -n osupgrade -s /mnt/install_sol10_u8_sparc/

This bit takes a while (a bit like installing Solaris...) but basically upgrades the named boot environment using the media specified. At the end of this, all that needs to be done is for the boot environment to be activated:

# luactivate osupgrade

Before initialising the new environment, it's worth noting down your existing, working environment. For me, this was the root filesystem located on /dev/md/dsk/d10. Find out the underlying slices used by d10 (c2t0d0s0 and c2d1d0s0 in my case) Once done, reboot the server and the new boot environment should be loaded.

Now the coolness of this should be immediately apparent! Previously, operating system upgrades would require a backup of the system to tape (always a good idea!), followed by scheduled downtime as the system was upgraded "offline". This also meant a visit to a customer site, typically at a weekend.

Combined with the use of an ILOM interface (for network access to the console), it now becomes perfectly possible to upgrade a Solaris server during the day, while users are on the system. All that it required now is an out-of-hours reboot of the server to initialise the new release.

If there are problems with the upgrade, it's possible to rollback by setting the old boot environment to active. To do this, boot off cdrom or the network (I did the network), by typing the following at the PROM:

ok boot net -s

[wait for the OS to boot)

# mount -Fufs /dev/dsk/c2t0d0s0 /mnt
# /mnt/sbin/luactivate

Exit single user mode and reboot.

Obviously this is only scratching the surface of what Live Upgrade can do. It's possible to merge and split filesystems, detach and build disk mirrors, and much more. The use of Live Upgrade is also greater than the occasional update; it's perfectly possible to use Live Upgrade to apply system patches, with a very easy rollback capability.

Definitely a technology that needs to be investigated more fully...

A first look at Toodledo

My ongoing quest to manage my email, calendar, contacts, tasks and notes from the cloud continues. I've managed to cover most of the above with Google Mail and contacts, Google Calendar and Evernote. Finding a task manager that I like the look of has proven tricky.

I tried to use the brilliantly titled Remember the Milk, but it's not designed to interface with Outlook. Unfortunately I spend a significant amount of my working life in front of Outlook 2007 which has very powerful task management. Using multiple task managers would not make my life easier.

I signed up to Toodledo a while ago, but didn't get around to trying it out. There was an Outlook sync tool, but it was pre-1.0 which didn't inspire confidence. So I left it a while... and went back to it last night.

The sync tool is now post-1.0 so I installed it and took my task list from Outlook 2007 into the cloud. I set Toodledo to map Outlook categories to folders so the organisation structure remains.

There is one downside to the Outlook sync tool: It syncs Tasks, but Outlook 2007 introduced another item called "To-Dos". Think of the "To-Do" list as all types of data that have been marked for action. This includes all tasks, but also all flagged emails, diary events and contacts. Apparently the To-Do list is not stored in the same structure as the Task list, so syncing it is not yet achievable. It's a minor point though and arguably not the fault of the sync tool.

One nice surprise about Toodledo is that it has a Notebook feature. Upon examining this I was surprised to find my Outlook notes had been copied across. They sync changes too, so although I'm primarily using Evernote for taking Notes, it's nice to know that anything I jot down in Outlook is made available as well.

I also spent a whopping £1.79 on the Toodledo iPhone application. This has a very pleasing and easy to use interface. The detail is there, but there's not too much detail to make it unusable. I've not had the chance to play with it too. It works when the iPhone is in airplane mode, so data appears to be cached locally. The aforementioned notebook is not available from the iPhone (that I've seen yet).

There are also integration points for Firefox, Google Gadgets, Twitter, Apple iCal and Dashboard as well as good old email and RSS.

It's early days and I've not used Toodledo enough yet to determine whether I want to commit to basing my task management around it, but initial usage is encouraging. I'll hope to implement some of the popular GTD methodology to it having just received a copy of David Allen's book "Getting Things Done". I just need to get organised enough to read it...

Working with Evernote

When it comes to notebook applications, Microsoft's OneNote is arguably the most powerful product out there. Unfortunately, as I commented in a previous blog entry ("Is OneNote a Cul de Sac"), OneNote - in its current form - is very much a single user, single machine application. Getting data into OneNote is easy, but accessing it remotely over the cloud is not possible.

Perhaps the most popular alternative is Evernote. This provides a cloud-based service that is free when a limited amount of data per month is created (a limit I am nowhere near reaching). Evernote can be accessed through the website, but also provides clients for Windows, Mac and the iPhone. This makes it a very compelling product to use, even if it's not quite a feature-rich as OneNote. If using Windows, it's definitely worth using the 3.5 beta release as the interface is much better (IMHO) than previous releases and is more similar to the Mac version.

One of the things I like about OneNote is the Notebook/Section/Page metaphor. It's possible to have multiple Notebooks (e.g., Work, Personal etc.) and each notebook can have section tabs. Within each section, multiple pages can be created. In comparison, Evernote has the concept of notebooks and pages. In order to work around this, I adopted a naming convention for my notebooks that describes what each notebook is for:

Personal: Holidays
Personal: Home Projects
Work: AIX Notes
Work: Business Planning
Work: Citrix

This effectively works around the lack of sections. (I actually realised after adopting this approach that when using OneNote, I only had a couple of notebooks for Personal and Work anyway and used sections to separate my data)

As for when to use a new notebook, I've adopted the approach that if I was to use a new physical notebook in real life (e.g., I'm attending a course), then I'll create a new notebook for it. This is what I did last week when on a Citrix XenApp course. All my notes were instantly available via the cloud to all my clients. I'm also using new notebooks for different projects.

Evernote isn't perfect. I'd still like the ability to click anywhere and enter text (like OneNote), and wish it handled proper text headings instead of just setting font sizes. But these are pretty minor. A native Linux client would also be nice, but the web interface works well enough for occasional use. One killer feature for me would be rudimentary drawing tools so I could illustrate notes with diagrams. In fact, I'd become a premium user for this feature.

Evernote is definitely worth a look if you're looking for a way to store all your notes together in one place, but access them from anywhere. A cloud win.

Building a ZFS filesystem using RAID60

We are starting to use ZFS as a production filesystem at $WORK. Our disk array of choice is the Sun StorageTek 2540 which provides hardware RAID capabilities. When building a ZFS environment, the decision has to be made on whether to use the hardware RAID and/or the software RAID capabilities of ZFS.

Having watched Ben Rockwood's excellent ZFS tutorial, my understanding of ZFS is much better than before. For our new fileserver, I've created the following:

On the StorageTek 2540, I've created two virtual disks in a RAID6 configuration. Each virtual disk comprises of 5 physical disks (3 for data, 2 for parity) and is assigned to different controllers. On top of each virtual disk, I've created a 100GB volume. This is published as a LUN to the Solaris server and appears as c0d3 and c0d4.

Each LUN is then added to a zpool called "fileserver":

# zpool create fileserver c0d3 c0d4

By default, ZFS treats the above in a variable width stripe, so the hardware and software combined result in a "RAID60" configuration; data is striped across 2 x RAID6 virtual disks for a total width of 10 spindles.

Why RAID6 and not RAID10? Apart from the cost implications, as a fileserver, the majority of operations will be read-only and RAID6 is very good at reads (while being less-good at writes).

Now, when I'm running out of space, I can create a new volume, publish the LUN and add it to the zpool:

# zpool add fileserver c0d5

A quick check of the zpool status shows the disk is added successfully:

# zpool status
pool: fileserver
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
fileserver ONLINE 0 0 0
c0d3 ONLINE 0 0 0
c0d4 ONLINE 0 0 0
errors: No known data errors

Running zpool list reports the newly added space is available:

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
fileserver 199G 69.4G 130G 34% ONLINE -

All told, very simple to do and results in a pretty fast filesystem.

Debugging backup problems

We had some problems with our nightly backup recently. These are tricky to debug as you don't want to be playing around while users are on the system being backed up, so it can become a case of try and fix something during the day, wait until the nightly backup run and try again the next day when it's failed.

The problem for us was that the backup would start, and then fail after about 10 minutes. The software we use is CA ARCserve for Unix and the message logged in the error log was a very unhelpful "unknown scsi error".

The backup server is a Sun V125 running Solaris 10. As noted above, the backup software is ARCserve, version 11.5. The server has a Qlogic fibre channel HBA for SAN connectivity and plugs into a SAN fabric built on Qlogic 5200 (2Gbit) and 5600 (4Gbit) switches. Also plugged into the SAN is an ATTO Fibrebridge. Our tape library, the Sun StorageTek SL48, connects via SCSI to the fibre bridge, which in turn publishes the internal tape drive and the library as LUNs on the SAN.

As you can see, there are a few things that could go wrong. The reason we have this somewhat complicated setup is that when we originally bought the SL48, although it was listed on CA's HCL, it only worked in a fibre attached configuration.

Having checked the obvious; that ARCserve could see the tape library, load, unload and scan the media, that there were sufficient "scratch" tapes available for writing and that the Ingres database that holds all the backup records was consistent and working properly, I turned my attention to the hardware.

The first step was to eliminate ARCserve from the list of suspects. When loaded, ARCserve loads the "cha" device driver that controls the tape library. I rebooted the backup server to ensure that the drivers were definitely not running, and observed that the tape drive could be seen as /dev/rmt/0. Using the SL48 web interface, I loaded a tape and tried to perform a ufsdump of a local filesystem to the tape drive.

This worked for a while and then failed with a write error.

Okay, so it's not ARCserve, and it looks like an error on the tape drive. Perhaps a clean would help. ARCserve is meant to run automatic drive cleans, but perhaps it hadn't. Again, the SL48 web interface provides functionality to do this.

The SL48 complained that the cleaning tape had expired. Unfortunately, I didn't have any spare, but it looked like this might be the issue. I immediately ordered three more tapes, and configured the SL48 to email me when it had warnings as well as errors. This should mean I get notified when the next cleaning tape expires.

A dirty drive was the most likely suspect, but in order to definitely rule out the SAN, I tried directly attaching the SL48 to the V125's internal SCSI port. This hadn't worked with the original ARCserver 11.5, but we had since applied a service pack.

The service pack had updated the device driver list and the SL48 was now detected as a local SCSI drive. I tried the same ufsdump, against the same local filesystem, using the same tape and expecting the same error, but was surprised to see that the backup completed without any problems.

Hmm, perhaps it's the SAN.

Last week, prior to the problems starting, we added a new fibre switch (the 5600) and this required that our existing switches (5200) have a firmware upgrade so they were all at the same level. It's possible that there is something in this latest firmware release that is causing the tape library (or fibre bridge) to choke.

I fired up ARCserve again and kicked off a backup job. It was during the day, but we hadn't had a working backup for several days, so I was content to take the performance hit (not that anyone appeared to notice).

The backup ran for 13 hours and completed successfully.

I've not actually spent more time trying to determine whether the problem is with the switches or the [now redundant] fibre bridge. The backup is now a lot simpler in it's configuration. My belief is that of all the systems we run, the backup should be the simplest, especially when there is a restore to be done!

Subsequent backups have been successful, but in the event of future problems, it's useful to have a template to work from when debugging any issues.

ESX/ESXi networking best practices

VMware Virtual Infrastructure 3 had some fairly well defined best practices for networking. There are three types of network connection that can be setup in an ESX host:

• Service Console
• VMkernel for vMotion
• Virtual Machines network

Although it is possible to have a single physical NIC connecting to a single vSwitch with the above port groups configured, this is not a good approach because the NIC becomes a single point of failure. Adding a second NIC helps and teaming the two physical NICs provides redundancy, but you could still experience performance problems when a bandwidth intensive operation occurs (such as a vMotion) as VM traffic could suffer from a lack of bandwidth.

One solution is to add additional NICs to the vSwitch and hope that the increase in bandwidth is sufficient so that vMotion traffic does not impact on connected users, but there is another potential issue: vMotion traffic is unencrypted, and the content of a vMotion operation is the memory of a virtual machine. The problem here is that if a malicious user is able to eavesdrop on the connection, they might be able to access sensitive data. Using separate VLANs help, but you're still effectively crossing your fingers and hoping everything will be okay.

The safer approach is to separate the Virtual Machine traffic from the vMotion traffic using a separate vSwitch and assigning each vSwitch two physical NICs for redundancy. This ensures that the vMotion traffic is physically isolated from the VMs.

So where to put the Service Console? It is possible to assign it to either the Virtual Machine network vSwitch or the vMotion VMkernel network vSwitch. It's worth pointing out that Virtual Center requires access to the Service Console, so depending on where you run Virtual Center, this might impact on which vSwitch you assign the Service Console. By placing the Service Console on a separate vSwitch to the Virtual Machines network helps to reduce the ability of malicious users of hacking the SC. It's common (especially in environments where servers have 4 physical NICs) to find a configuration where one vSwitch is dedicated to VMs, and the second vSwitch shares the Service Console and vMotion VMkernel port groups.

A four NIC server configuration could look like:

• 2 x NICs for Virtual Machine traffic
• 2 x NICs for vMotion traffic and the Service Console

If six NICs are available, the configuration could look like:

• 2 x NICs for Virtual Machine traffic
• 2 x NICs for vMotion traffic
• 2 x NICs for the Service Console
  

Although the Service Console doesn't require much bandwidth, some sites perform backups from within the SC which can have a significant network overhead.

What about non-Fibre Channel storage?

For users with iSCSI or NFS, things are slightly more complex. The VMware iSCSI initiator lives inside the Service Console but also requires a VMkernel network for actual storage traffic. It would be logical therefore to put these two port groups on the same vSwitch.

Both iSCSI and NFS traffic should be assigned to a separate network to increase security and guarantee that bandwidth is available regardless of whether a vMotion or heavy VM traffic is occuring. Two additional NICs should be allocated, resulting in the following six NIC configuration:

• 2 x NICs for Virtual Machine traffic
• 2 x NICs for vMotion traffic
• 2 x NICs for Service Console and iSCSI/NFS storage

Given an additional two NICs (8 total), the following could be configured:

• 2 x NICs for Virtual Machine traffic
• 2 x NICs for vMotion traffic
• 2 x NICs for Service Console
• 2 x NICs iSCSI/NFS storage

So what's changed in the vSphere 4 world?

For our new deployment, I plan to use ESXi instead of ESX. VMware have stated their intention is to move to the ESXi hypervisor for future releases, and we have no legacy requirement to run the service console in our environment. On first glance, this would appear to remove the need for two of those ports in the above list.

But the question arises, if the Service Console is not present in ESXi, what IP address and interface does the vSphere client connect to? Well, the IP address is that assigned by the sysadmin to the host using the ESXi text mode menu (not sure what the proper terminology is for that). The interface is actually a VMkernel interface.

Replicating the ESX environment configuration in ESXi would therefore look like:

• VMkernel for Administration Network
• VMkernel for vMotion
• Virtual Machines Network

Assuming each of the above has redundant connections, we still need six NICs, although if you are limited to four NICs you could apply the same approach as with ESX 3.x and combine the Vmotion and Administration networks into a single vSwitch.

If you plan on using NFS storage or iSCSI, you will need another VMkernel interface for storage, so add another couple of ports.

One of the new features in vSphere 4 is Fault Tolerance (FT). This features ideally needs a dedicated network between hosts, so that takes the total number of physical ports up to 10:

• 2 x NICs for VMkernel Administration Network
• 2 x NICs for VMkernel Vmotion
• 2 x NICs for Virtual Machine Network
• 2 x NICs for VMkernel NFS/iSCSI
• 2 x NICs for VMkernel FT
  

The above example only accounts for a single vSwitch for Virtual Machine traffic. If there is a reason for a second vSwitch with VM traffic (e.g., you want to segment a DMZ onto a physical network), additional NICs will be needed. Obviously doing this will cause the number of NICs to increase.

Conversely, if your server doesn't support 10 NICs, some sharing of physical NICs / vSwitches will be required.

Our environment only supports 6 NICs per server and we don't use iSCSI. Our NFS usage is limited to ISO datastores so this can be shared with the Administration and vMotion networks, so the approach we'll probably take is:

• 2 x NICs for VMkernel Administration Network, VMkernel vMotion and NFS network
  
• 2 x NICs for Virtual Machine Network
• 2 x NICs for VMkernel FT

This post is based on each NIC having 1Gbit ports. As the 10Gbit NIC becomes more popular, the network design approach might change. Something to think about for a future post...

Adding a SATA disk to ESXi

Having freed up two 500GB SATA disks from my storage server, I wanted to put them in the ESXi server. Although my original intention for this box was to have an essentially disk-less system (USB key boot of the hypervisor only), the reality is that I've not got enough bays in the storage server and don't want to waste two perfectly good disks.

I've also got a little project in the back of my mind that could make use of these disks...

I put the disks in the server and booted ESXi. Using the VI client, I noticed that the disks were recognised, but when I went to add storage, I could select the disk

"Error during the configuration of the host: failed to get disk paritition information"

I booted off the CentOS disk and selected "linux rescue" and destroyed the partition table using fdisk. I wrote these changes and confidently rebooted.

I got the same error.

From the ESXi menu, I viewed the configuration logs and messages and noticed it was reporting the following:

Warning: /dev/sda contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should. Perhaps it was corrupted - possibly by a program that doesn't understand GPT partition tables. Or perhaps you deleted the GPT table, and are now using an msdos partition table. Is this a GPT partition table?

I wasn't aware of what a GPT signature is/was, but it was obviously something that fdisk didn't overwrite. Some googling later suggested the problem could be solved by completely overwriting the start of the disk.

Back into the Linux rescue mode and some dd action (sledgehammer approach perhaps...):

# dd if=/dev/zero of=/dev/sda bs=1M count=1

Rebooted again and this time the disk is selectable and I could add the datastore. Repeated for the second disk and now I've got an additional 1TB of storage for VMs (albeit unmirrored, but that's fine in this non-production environment).

For those unfamiliar with dd, it's a fairly low level command that can copy raw data. The if= specifies the input file, the of= specifies the output file. In the above example, /dev/zero is a special Unix "file" that returns zero when read, and /dev/sda is the disk device I'm writing to. The bs= specifies the size of the block (1M = 1 megabyte) and count= specifies the number of blocks to read. So the above reads 1 block of 1MB size from /dev/zero (effectively 1MB of the "0" character) and writes this out to the disk, starting from the very beginning and overwriting everything there (which includes the partition table).

And this is why an understanding of Unix/Linux can be very useful, even if you don't do Unix stuff in your day job... :-)

SAN/NAS upgrade

All of my important data is stored on my OpenSolaris storage server (an HP ML110 G5). A mirrored pair of 500GB in a ZFS Zpool provided NFS, CIFS and iSCSI sharing. Unfortunately, I ran out of space to the point that ZFS was unable to take snapshots.

I needed to add more storage, but didn't have the drive bays available to do it. So I ordered two 1TB SATA disks with the intention of replacing the two existing disks.

I followed the instructions found at Blog O Matty (a blog I highly recommend). The process was extremely easy:

1. Remove one of the 500GB disks and replace with a 1TB disk.
2. Tell ZFS to "resilver" (aka resync the mirror) the new disk (one command: zpool replace datapool c3d0)
3. Wait a number of hours for the disk to resilver (10 hours when the disks are being used)
   
4. Tell ZFS to clear all error status messages (zpool clear datapool). This puts the pool into an "ONLINE" state for all devices in the pool.
   
5. Remove the second 500GB and replace with the second 1TB disk
6. Tell ZFS to resilver onto the second disk
7. Wait for this second disk to resilver. I did this overnight and it was finished in the morning.
   
8. Tell ZFS to clear the error status on the new disk
9. Check the zpool status (zpool list) and note the new size: Now 928GB
   

The ML110 does not have hot-swap disks, so I needed to power off each time I swapped the disks, but if you have a hot swap capable server, the entire process can be done live with the filesystems mounted. Nice.

With approximately another 500GB free space, I can now experiment with other hypervisors (XenServer and Hyper-V will probably be the first if I can get them working off a USB bootable key drive). I also took the opportunity to add more memory (8GB total) to both the OpenSolaris server and the VM server (the HP ML115). While I was spending money, I also paid out for another 500GB external USB. This means I can now take a backup of the key filesystems (photos, documents etc) and ship them to an off-site facility (aka, my parents house).

Coincidently, T had filled up her C: drive with a huge number of photos and videos. Although the disk is backed up to an external drive, I wanted to move the data to the server. I created a new filesystem in ZFS:

# zfs create datapool/Users/teresa

This filesystem can grow and consume all space in the pool, so I assigned a quota of 30GB:

# zfs set quota=30G datapool/Users/teresa

In order to make this visible to T's Vista PC, I had to share the filesystem over SMB:

# zfs sharesmb=on datapool/Users/teresa

I made sure that T had a Solaris account setup with a password configured so it could authenticate and then mapped the network drive. The UNC path replaces the slashes in the filesystem with underscores: \\opensolaris\datapool_users_teresa

I copied T's documents to the server by changing the locations of the profile shell folders (right-click "Documents", "Pictures", "Videos" etc and select properties, then specify a new location and the contents are moved across - very easy).

It was then I found even more pictures that needed to move across and the 30GB I had allocated to the filesystem was going to be tight in the long term. This was trival to fix:

# zfs set quota=40G datapool/Users/teresa

The change applied instantly and the network drive size increased to 40GB.

It's good to have all our personal data now stored on the ZFS filesystem, with full mirroring, checksumming and backed up.

The now-redundant 500GB disks will be assigned to another blog post...

Upgrading the EeePC 701 to Eeebuntu

I don't tend to use my EeePC 701 4G very much; there's not much point when you have a pretty well setup PC and network. But when it comes to going on holiday, the Eee is a must-pack luggage item.

T and I have just been away for a week in Corfu. Weather: hot. Hotel Wi-Fi: not bad and free to use (guess which is the most important criteria... :-))

It was when using the Eee on holiday that I realised how dated the default Xandros-derived distro is. Some websites even encouraged us to to upgrade to a later release of Firefox. So upon returning, I purchased a 2GB RAM upgrade (from the default 512MB), an 8GB SD card to store my files on and a 4GB USB stick with which I installed Eeebuntu.

I've never been a serious Ubuntu user (or any of its derivatives), being quite happy with OpenSUSE, so installing Eeebuntu has been interesting. Fortunately the website had some decent documentation on building an install USB key (since the Eee doesn't have a CD drive). Once that was set up, it was simply a matter of booting the Eee off the USB stick and following the prompts.

The result is a modern, GNOME-based distro that can take advantage of all the Eee functionality including the Wi-Fi and webcam. It's also a very smart-looking setup with Compiz working out of the box. I took the opportunity to add some extra software that might be useful in the future, including Wireshark and Nessus.

I'm not going to pretend that the Eee is going to be my new, main machine, or that it will be heavily used on a daily basis, but it's a very capable little computer that will be far more useful with the updated OS on it. My initial foray into Eeebuntu has also been very positive. If you're looking to get something better than the default, dated Xandros version, it's worth a look.

Passing the CCNA: My experience

This blog explains why I've been quiet for a few weeks...

I did the CCNA exam back in 2002, but by the time it expired in 2005, I wasn't doing anything specifically with networks so didn't bother re-certifying. A couple of months ago I thought it would be a good cert to pick-up again, so decided to dive in and do some self-study.

The original CCNA was the entry level Cisco exam, but in the last few years this has been replaced by the CCENT. The CCNA is a lot harder than it used to be with many new subjects and a deeper level of understanding required. You can either approach the certification using two tests (ICND1 which gives you the CCENT and ICND2 which results in the CCNA) or by using one combined exam. I opted for the one exam.

I bought the latest version of Todd Lammle's CCNA Study Guide and started studying one chapter per night (there are 14 chapters, but real life meant that it took more than 14 days). I also spent weekends studying as well. The book is generally very good, although I found that because I wasn't replicating the example network used in the book, some sections required me to visualise and absorb what was being shown without any hands on experience. I would highlight the chapter on understanding subnet masks though; probably the best way to learn subnetting I can imagine.

I also purchased the Cisco Press Official Exam Certification Library by Wendell Odom. I planned on using this to get another perspective on the material and started reading bits of this book to clarify areas after I had completed the Lammle book. In comparison with the Lammle book, Odom is a lot more detailed (some might say dry but I enjoyed it). I discovered that, for my learning style, the Odom book helped me more. I found the thorough details showing how something works, step by step, to be very useful.

I managed to borrow some old Cisco kit from work and a laptop from work. This consisted on two 2600 routers, two Catalyst 2900XL switches and a 1700 series router. I didn't have the proper serial cable so couldn't do any WAN activities, but did use the kit to validate my understanding of VLANs, VTP and IOS commands (including the boot process, wiping configs etc).

The final part of my studying involved the CDs provided with the books. The test questions with the Lammle book were pretty straightforward and I was able to get 80%+ in the mock exams without too much trouble.

The Odom book was a completely different matter. These questions are hard. Really hard. I initially found that the time limit in the exam was running out on me, and that I was struggling to get my head around some of the questions at all. Whereas in the original exam, you might be asked a question like:

Given an IP address of 10.2.66.8/18, what is the subnet address, first host, last host and broadcast address?

Now the exam was asking you to look at a network topology diagram with maybe six of these networks and you have to choose a spare subnet range. In other words, you have to do six times the work for single question.

Although I managed to get faster at the questions, and especially enjoyed the simulator questions (where you have to log into simulators of routers and either fix the configuration or use the show commands to identify certain things), I never managed to achieve a pass mark.

I resigned myself to the fact that I wasn't going to pass this exam and instead decided to treat it as an educational experience to work out how difficult it was going to be.

Now the actual exam itself is covered by NDA, so it would be improper of me to comment on the specifics. There were simulations but they weren't too difficult and the majority of the questions were more of the difficulty level found in the Lammle book vs the Odom book.

For those looking at getting the CCNA, I would strongly recommend the two books I used. If you can master the Lammle book, you'll probably do okay. If you can master the Odom book, you'll walk the exam with one arm tied behind your back.

On a personal note, I actually enjoyed the process of learning. I find the networking concepts fascinating, and might even look at doing another cert at some point (we all need a hobby, right?). Not sure T will be too happy about losing me for another month.

At least now I can chillax a bit and enjoy what remains of summer... :-)

Essential Firefox add-ons

Thought I'd better tidy up my Firefox add-ons as I appear to have accumulated a number that either served a specific purpose and are now redundant (Web Developer) or simply aren't needed (FlagFox).

The list of Add-ons I'm currently using on my Linux workstation are:

• Blocksite (essential if you want to block Facebook Beacon)
• Evernote Web Clipper
• Flashblock (enable Flash objects on a per-site basis)
• Gmail Manager (to alert when I get new Gmails)
• TwitterFox (how I generally track Twitter)
• Wikipedia Lookup Extension (highlight word, right click, search in Wikipedia)
• Xmarks (Bookmark sync)

I don't use AdBlock because I'm generally not bothered by the adverts on sites I visit regularly, and I'm not enough of a Firefox hacker to want to bother with Greasemonkey. Perhaps something to manage passwords better would be good though.

Am I missing anything essential?

Another blog worth reading...

My colleague, "JW", has joined the blogosphere and will be documenting some of the adventures he has in the world of Tech. It's off to a good start with a decent write-up of some of the challenges we experienced when we installed VMware and XenServer on a Sun X6240 blade module.

Check it out at Ting Ting Tech.

Samba, Squid and Active Directory authentication

This post is the end of a few weeks of challenging debugging.

At $WORK we are implementing a new proxy server based on Squid. Unlike our old proxy, we want to authenticate each user against Active Directory. In order for this to work, Samba (or more specifically, the Winbind component of Samba) needs to be configured.

Getting Samba setup

Consider that Windows networking in the Active Directory world is built on DNS for name resolution, Kerberos for authentication, LDAP for directory services and the SMB protocol for file, print and RPC.

The first step was to configure the proxy server to use the AD Domain Controllers for DNS resolution. This was done by editing /etc/resolv.conf and configuring /etc/nsswitch.conf.

The second step was to get Kerberos working. I've detailed this in another blog posting. This also needs to point to the AD Domain Controllers for the KDC.

Configuring Samba was also fairly straightforward. There was no need to run smbd (used for file and print serving) or nmbd (the naming service) as this box would not be performing those roles. The winbindd server needs to be running. This is responsible for authenticating against the Active Directory.

The smb.conf can be small:

[global]
workgroup = CSS
realm = CSS.AD.EXAMPLE.COM
server string = Squid Proxy and Samba Server
security = ADS
log file = /var/log/samba/%m.log
max log size = 50
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
dns proxy = No
winbind:ignore domains = MAT LPS LAB MMSC GRP IMCR UPGRADE CENTRAL 4THFLOOR AD CSSDEV NAS
hosts allow = 10.1, 127.

To join the domain, run the net ads join command, using the credentials of a domain administrator. You should be able to confirm whether the trust worked by typing:

# wbinfo -t
checking the trust secret via RPC calls succeeded

If that's okay, try pulling in a list of users, again using wbinfo but this time using the -u flag:

# wbinfo -u

And this is where things started to go wrong for me. Sometimes it would work, but most of the time it would error. This took a lot of investigating and the details can be found here in the Samba mailing list archive. It was this bit that took the time to debug.

Getting Squid working

Having got wbinfo to now return the list of users, it was time to configure Squid to use ntlm_auth (which in turn uses winbindd to perform the authentication request). The /etc/squid/squid.conf needs the following:

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 5

At this point everything should work wonderfully...

Except Squid was unable to authenticate, complaining about the permissions on /var/lib/samba/winbindd_privileged. It appears that Winbind expects this directory to have 750 permissions with root:root ownership, while Squid runs as the "squid" user. According to one post I read about the issue, it may be caused by the way that Red Hat have built Squid. One possible workaround is to use an ACL (yes really, a use for ACLs in Unix!) but it appears my install doesn't have ACL support enabled(!).

So the immediate workaround for me is to create a script that basically does the following:

• Set permissions on /var/lib/samba/winbindd_privileged to 750
• Start Winbind
• Set permissions on /var/lib/samba/winbindd_privileged to 777 (yes, I know...)
• Start Squid

This appears to work okay until I can find a real solution.

So although this is now working, it's taken longer than anticipated. The thing about Samba, and Active Directory integration is that it's so complicated with so many options. The learning curve is steep, but I'm starting to feel I now have a grip on it.

Configuring Kerberos on CentOS 5

Kerberos is a ticket-oriented authentication system that was originally designed for Unix networks, but was also embraced (and extended) by Microsoft in Active Directory. I've been debugging a number of issues involving the Squid proxy server on Linux using Samba to authenticate against Active Directory, and as part of this I had to get familiar with Kerberos.

It's not trivial, so I've documented my workflow here. Hopefully it will useful to others.

The test environment consists of two virtual machines running CentOS 5, imaginatively named centos01 (krbserver) and centos02 (krbclient). For the purpose of this test, centos01 is the Kerberos server and centos02 is the client.

I followed the instructions here and broadly recommend them. These are my additional notes to clarify some parts of the install.

General notes

Make sure that you use the same time source for both client and server. I used NTP to keep the two VMs in sync. The notes do state this but it's worth stressing.

Remember how many IT problems are caused by name resolution errors! Make sure you have both the server and client registered in DNS (or have entries in /etc/hosts). If using /etc/hosts both the standalone hostname and the FQDN should be added:

192.168.192.26 krbserver.local.zone krbserver
192.168.192.108 krbclient.local.zone krbserver

Note the order of the hostname and the FQDN! This is important (see further below).

Configure the server

After installing the packages using YUM, configuring the database and ACL file, adding the first principal user and starting the three services, the server should be ready to go. Confirm this with kinit and klist. Now it's time to configure the client.

Configure the client

Install the packages using YUM and then run the kadmin command and add a new principal for the client machine. It's worth noting that this should be done using the kadmin interactive interface instead of trying to put the "addprinc" parameter on the command line. This is because the -randkey option will be interpreted by kadmin on the command line as "-r andkey" and it will try and authenticate against the "andkey" realm. So for me, the command looked like:

# kadmin -p julian/admin@LOCAL.ZONE
Password for julian/admin@LOCAL.ZONE: ********
kadmin: addprinc -randkey host/krbclient.local.zone

I assume that this is rougly analogous to adding a machine to an Active Directory domain.

Once this entry, export the principal to the workstation's /etc/krb5.keytab file.

In addition to the machine principal, I also created a normal (non-admin) a local user, julian@LOCAL.ZONE. On the client, I log in as my own non-root user ("julian") and type kinit:

$ kinit
Password for julian@LOCAL.ZONE: ********

If this succeeds, you should see the "ticket granting ticket" be assigned:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 11:14:20 07/18/09 11:14:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE

Kerberos 4 ticket cache: /tmp/tkt500 klist: You have no tickets cached

This process shows that communication between the client and server using Kerberos is successful.

Configuring telnet (for testing)

On the server, I then enabled the krb5-telnet service in /etc/xinetd.d and started xinetd. On the client, I then ran:

$ /usr/kerberos/bin/telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
Password:

Problem. It was asking for a password which implied the Kerberos ticket was not being passed correctly. However, when I ran klist, it showed that the ticket for the host was passed correctly:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 10:38:20 07/18/09 10:38:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 10:38:31 07/18/09 10:38:20 host/krbserver.local.zone@LOCAL.ZONE

After running strace against the telnetd process, it appeared that the telnet server was failing when trying to read /etc/krb5.keytab. But all the documentation I had read stated that this should be run on the client and not the server. So, why does the Kerberos server need a keytab file?

Answer: The Kerberos server does not require a keytab file, but the telnet server does! Although they are both running on the same VM, the telnet server is itself a client to the Kerberos server. Simple when you work it out it would have semantically been easier to understand if my telnet server been different from the Kerberos server.

So I ran the kadmin command on the server and created a keytab file using the ktadd command. I restarted the Kerberos services for good measure and cleared my client and server caches using kdestroy, restarted xinetd and tried the telnet:

[julian@krbclient bin]$ ./telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 accepts you as ``julian@LOCAL.ZONE'' ]
Last login: Fri Jul 17 10:48:10 from krbclient
[julian@krbserver ~]$

Result!

Configuring SSH

The instructions state that GSSAPIAuthentication and GSSAPIDelegateCredentials need to be enabled. I did this and restarted the SSH daemon with -ddd (debug) enabled.

The first attempt at running ssh krbserver prompted for a password, but the server debug revealed the following:

debug1: Unspecified GSS failure. Minor code may provide more information
No principal in keytab matches desired name

Okay, so this is weird. Checking the output of klist showed this:

[julian@krbclient ~]$ ssh krbserver
julian@krbserver's password:
Connection closed by 192.168.192.26
[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached


Note that krbserver@ has no realm. This turned out to be because /etc/hosts (on the client) looks like this:

192.168.192.108 krbclient krbclient.local.zone
192.168.192.26 krbserver krbserver.local.zone

Putting the hostname after the FQDN like this:

192.168.192.108 krbclient.local.zone krbclient
192.168.192.26 krbserver.local.zone krbserver

fixes the problem!

[julian@krbclient ~]$ ssh krbserver
Last login: Fri Jul 17 13:54:40 2009 from krbclient.local.zone

Klist now shows:

[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@
07/17/09 13:54:38 07/18/09 13:42:27 host/krbserver.local.zone@LOCAL.ZONE


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Summary

What you see above does not include the time spent trying things out and staring blankly at the screen. Getting Kerberos up and running is not the most trivial process and while there is some decent documentation, there are also a lot of people posting questions and asking for help when it doesn't work properly. Hopefully this will shed some light on it for others.

The future is... iSCSI?

I've recently completed the business planning infrastructure requirements for 2010 at $WORK. As part of this I have specified a new fibre channel switch and some additional fibre attached storage.

Despite this, I'm starting to suspect that the future of SAN connectivity will be iSCSI over copper Ethernet.

Ethernet and IP technologies have basically beaten everything else out there and now dominate computer networks and IP telephony. Why have a different standard for storage networks? Consolidation of the fabric for LAN, SAN and VOIP seems logical to me. Share the components and reduce the total cost.

So why continue to specify fibre channel? We already have a large investment in fibre channel, it's a known quantity and is well supported. Sun Solaris has a very mature FC implementation, and as of VI3, VMware works best on FC (not sure yet if vSphere changes this).

It's also faster (for us). We currently have 2Gbit switches but will be adding 4Gbit switches later this year and early next. Sure, 10Gbit Ethernet is available, but it's still too expensive for us to deploy (especially when adding the cost of switches and NICs).

But fast forward three years and I would expect the following:

• 10Gbit Ethernet switches at a reasonable price with 40Gbit or 100Gbit inter-switch links
  
• 10Gbit NICs with TCP Offload Engine (TOE) as standard and cheap
  
• iSCSI boot as standard on these 10Gbit NICs (some do already, but it's not guaranteed)
  
• Better support in the hypervisor / operating system for iSCSI
  

At the end of the day, managing a single fabric is easier than juggling a bundle of different cable types, protocols, HBAs and drivers.

It's always risky in this business to speculate how things might look in 3 years. If you disagree, please let me know why; it's always good to get alternative views...

Sun StorageTek 2540 and ESX troubleshooting

We experienced a few issues with the StorageTek 2540 array that forms the core of our SAN recently. The symptom was that the array flagged itself as being in a degraded state and that one or more volumes were not assigned to the preferred controller.

The first step was to upgrade the SAN firmware and Common Array Manager (CAM) software to the latest release. Despite this, we observed the problem again. Further digging into the problem found that the failover was happening when we performed a LUN rescan under VMware ESX.

My previous understanding was that there were essentially two types of arrays: active/active and active/passive. In the active/active configuration, both controllers in an array can service I/O requests to a specific volume concurrently. In an active/passive configuration, one [active] controller handles the I/O with the second [passive] controller sitting idle, only servicing I/O if the active controller fails.

I understood the StorageTek 2540 to be an active/passive array; it is only possible to assign a volume to one controller at any time. However, in order to improve the throughput of the array, different volumes can be assigned to different controllers. For example, a volume “VOL1” might be assigned to controller A as its active controller and to controller B for its passive controller, while volume “VOL2” might be assigned to controller B as its active controller and controller A as its passive controller.

It turns out that things are more subtle than this; there is a third type of array configuration: asymmetric.

The asymmetric configuration follows the active/passive model in that only one controller is servicing I/O for a specific volume at any time, but extends this by allowing I/O operations to be received by the second controller. If this happens, the array will automatically failover the volume to the second controller to service the request. This process is called Automatic Volume Transfer (AVT). If the first controller then receives I/O operations, the AVT moves the volume back.

Yes, this could cause some flapping between controllers. It can also cause I/O stalls as the controllers fail across.

Some of the array initiator types (such as Solaris with Traffic Manager (aka MPxIO)) disable AVT, others, including the Linux initiator that we’ve used on our VMware hosts, have AVT enabled.

So the problem we’re having appears to be caused by the array failing over a volume to its second controller. But why is it doing this? The only configuration I had performed on the ESX side was to ensure the multi-pathing option was set to Most Recently Used (MRU); the correct setting for active/passive arrays. What appears to have happened is that when booting, the ESX servers are not mapping to a consistent path. Out of our five ESX servers, three were setting one controller as active, while the other two servers were setting the second controller as active. Presumably, when one of the hosts (that has the wrong active path) performs a scan, the request is sent to the failover controller which invokes AVT and fails over the volume.

How to fix?

Sun have told me that the next version of CAM, due in a few weeks, will include a “VMware” initiator type which will disable AVT. This will negate the need to perform the NVSRAM hack in VMware’s Knowledge Base, but will require a firmware upgrade.

In the meantime, it might be a case of just ensuring that all the ESX hosts are using the same path to connect to each volume. This is all theory as I’m still working this out, but at least it’s all starting to make sense.

Although not specifically VMware or 2540 related, the following links provide some interesting reading around the subject:

Sun discussion forum thread about preferred and owner controllers

Linux kernel mailing list post detailing a bug experienced with multipath and asymmetric arrays

Note-taking on the Cloud

In the "old" days, things were pretty simple; the Palm handled contacts, calendar, tasks and notes well, allowing me to carry everything with me but still sync them with my PC.

Of course, things are more advanced these days: Multiple computers, at home and at work, the iPhone providing near continuous Internet connectivity, web based services and richer software applications. But despite this, I'm still struggling to get perfect syncing across all platforms. Here's where things are for me today:

• Contacts: Google Contacts synced with the iPhone, but no Outlook/Exchange integration.
• Calendar: Google Calendar synced with the iPhone, but primary calendar only.
• Tasks: Still using Outlook/Exchange for this. No sync.
• Notes: Some notes in Outlook, some notes in OneNote, a few notes on the iPhone. Nothing syncs.

The last one is particularly disappointing as note synchronisation shouldn't be difficult. I tried using Google Notebook for a while until Google got bored and dropped it.

It was then I tried Evernote. There is a free version, provided you don't exceed a certain number of notes per month and there are clients for Windows, Mac OS X and the iPhone. There's also a web interface for when I'm in Linux or on a public machine.

My preference in terms of note taking functionality and power is OneNote. The only downside to this application is that notes are basically locked to the client PC, or synced to the corporate SharePoint server at best.

In comparison with OneNote, Evernote has fewer features, and an interface that is less rich. The Mac and Windows versions both have different levels of functionality (the Mac has a nicer set of views IMHO). But in it's favour, any notes I make, on any of my devices, now sync with the cloud.

Evernote is therefore my de facto notes application. For now.

Upgrading to ESXi 4.0

Although VMware vSphere came out a couple of months ago, I haven't had the time to try it out yet. Today I took the opportunity to try and upgrade my ESXi 3.5 server to ESXi 4.0. This won't allow me to test out the new VMware features such as Fault Tolerence, but I'll be able to run the latest vSphere client and see how the interface has changed.

I was a bit reluctant to do this because installing ESXi 3.5 on the USB key drive inside my ML115 G5, although not complicated, consisted of a number of steps and manual copying of files (not to mention having to crawl under the desk and physically install the USB stick).

Well the good news is that ESXi 4.0 is a much easier story. The operating system can be installed from CD straight onto the USB key drive, so no need to pull the side off the case. The install is quick and takes about 5 minutes. Then a reboot and to the DOS-like menu that allows for the networking, password etc to be configured.

A download of the vSphere 4 client and registering the licence key (free for ESXi) and everything is ready to go. I re-added the iSCSI and NFS datastores from my OpenSolaris server and fired up the VMs.

I haven't done anything beyond this yet, but for an upgrade I assumed would be a hassle, ESXi 4.0 has been very straightforward.

Upgrading to OpenSolaris 2009.06

I've been absent from this blog for a while because we've been moving house. The move is now complete and broadband installed in the new house so I'm back online and ready to continue blogging. Today, upgrading OpenSolaris.

My storage server, the ML110 G5 has been running OpenSolaris 2008.11. With the release of OpenSolaris 2009.06, I wanted to upgrade and wondered if I should have left a spare slice of disk so I could perform a live upgrade. With the rest of the network dependent on this server for services (DNS, iSCSI LUNs, file and print serving), a failed upgrade didn't appeal.

It turns out that the the upgrade was very easy and a spare slice was not required:

Step one: Upgrade the Package Manager:

$ pfexec pkg install SUNWipkg

Step two: Run the Package Manager and select "Update All".

Wait while the new package requirements are evaluated, downloaded and installed.

Step three: Reboot

That's it. Checking /etc/release now reports:

OpenSolaris 2009.06 snv_111b X86
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 07 May 2009

Although I'm not an expert in IPS, the OpenSolaris packaging system, my understanding is that the previous version of the operating system is still on the system in the form of a ZFS snapshot, so if I have any problems, I should be able to rollback without too many problems.

Compared with upgrading other operating systems, OpenSolaris has been an absolute dream. Highly recommended.

Oracle to buy Sun

This blog is typically about stuff I'm doing, but having invested 10 years in working with Sun kit, I think this deserves a comment. I'm not particularly qualified to make an informed contribution to the Oracle buyout discussion, but since that doesn't stop most Slashdot commentators, here are my observations:

1) It could have been a lot worse. If IBM had bought Oracle, they would have two directly competing CPU architectures (POWER and SPARC), two directly competing Unix implementations (AIX and Solaris), directly competing storage, Java development tools, application servers, databases etc. The long term outcome would have been that some of these technologies would have been sunset and development consolidated.

2) It could have been much, much worse: HP could have bought Sun. See DEC/Digital and the Compaq merger to see how that would have played out.

3) Sun seems to get a bad rap from the Slashdot crowd, but the number of people who are dismayed that their favourite Sun product is under threat highlights the significance that Sun has in the market. This is especially telling when you see that the object of dismay is one of many different products: Solaris, Java, OpenOffice, Virtualbox, MySQL...

4) Oracle have now migrated from being a software solutions house to a total solutions provider. Although there is talk that Oracle will sell off SPARC to Fujitsu, or port everything to x86, the reality is that SPARC is a very lucrative platform for Sun and will be a revenue generator for Oracle. In fact, Oracle now own a very good portfolio.

5) In the last 12-18 months, Sun has "got" Open Source. I'm not sure Oracle has. Hopefully the Sun culture will impact Oracle (remember how NeXT "absorbed" Apple after being bought?).

So although I'm no expert in the area of business, the future for Sun might be okay after all. Time will tell...

Getting CIFS permissions working in OpenSolaris

I blogged a while back about how I managed to get NFS4 configured on my OpenSolaris server. At the end of that blog, I promised to update on getting CIFS working as well using the OpenSolaris CIFS server.

Finally got around to it tonight. Took a while as every attempt from my Vista machine was reporting an "Access Denied" message when running "net view \\opensolaris".

To cut a long story short, the problem was caused by me setting up a mapping between Unix and Windows using the idmap command. The answer was to remove any mappings and allow the CIFS server to work it out itself. All sorted!

Upgrading to OpenSUSE 11.1

Although it's been out for a number of months (released in December), I've just taken the step of upgrading my workstation to OpenSUSE 11.1. In order to perform a clean upgrade, I decided a reinstall from scratch would be easiest, especially since my services and custom scripts are now running on the OpenSolaris server.

I finally took this opportunity to move all my documents to the server. They now benefit from having regular ZFS snapshots and get backed up to the external USB drive for added security.

Once my home area was clean and having backed up my ~/.mozilla directory, I proceeded to install from the OpenSUSE 11.1 DVD download. I had previously used the 64bit version and hit a few problems with Java and Flash, so this time I opted for a 32bit install (only got 2GB RAM so not a huge hit). All worked as expected, although the default partitioning seems a bit tight - something I've had to rectify this morning using LVM. If installing, it would be worth creating a bigger root filesystem (15GB?) (assuming you're not using multiple partitions for /usr, /var etc.)

The only thing that is bugging me slightly is the beagled indexing daemon causes the CPU fans to spin up when the machine is idle. I need to see if this is because it's doing an initial index build or if this will be permanent, in which case I might turn beagle off.

I was surprised at how easily the Compiz "wobbly windows" was to enable, especially when I remember how difficult and flakey it used to be. I have however turned it off because it's faster without the effects.

All in all, a good distribution upgrade with no major issues.

Copying symbolic links

Sometimes it's useful to copy a directory that contains symlinks. The default cp behaviour is to follow the symlink and copy the file the symlink points to. If you want to preserve the symlinks as symlinks, then the flags the cp command uses depends on the operating system:

Solaris 10 and Linux:

$ cp -rP dir1/* dir2

The -r is recursive (in case dir1 has subdirectories) and the -P preserves the symlink.

AIX 5.2:

$ cp -rh dir1/* dir2

The -r is the same as Solaris, the -h preserves the symlink.

Does XenServer have a future?

Citrix has made their hypervisor based virtualisation product, XenServer, available for free. There has been a free "Express" version out for a while, but this was limited in the number of VMs it could support and came with a limited subset of functionality (similar to ESXi).

The new release makes most of the Advanced functionality available for free, including multi-server management and live migration between hosts. This competes with VMware very well, to the point that a virtualised Citrix (XenApp) solution I am working on will most probably run on the free XenServer and not on the pay-for ESX.

So why do I wonder whether XenServer has a future?

Citrix are looking to get customers hooked on XenServer with the aim of upselling a management suite called "Citrix Essentials for XenServer". The thing that concerns me is they also have a product called "Citrix Essentials for Hyper-V".

On the one hand this makes sense. By providing the same management tools for both XenServer and Hyper-V, Citrix are trying to make the underlying hypervisor a commodity item. The real advantage is in the management layer.

But development of XenServer is not free, and Citrix get no money directly from it. So why bother continuing development in the long term, when they can "superset" on top of Hyper-V, a hypervisor that Citrix don't have to spend any development funds on.

I hope I'm wrong, because XenServer looks interesting, but it wouldn't surprise me to see a future announcement where Citrix drop XenServer and adopt Hyper-V as their favoured hypervisor.

WSUS under Server 2008

I configured a WSUS server under Server 2008 and created a GPO to instruct all the other Windows VMs in my test environment to point to it. Although the GPO was being pushed out, the WSUS console did not detect any computers.

After speaking to some colleagues about this (thanks Rob!), I tried manually running a Windows Update from one of the clients. This produced the following error: 80072EE6

Googling this suggested the problem was an "unknown protocol" and one post suggested checking that the GPO settings for the intranet server included the "http://" prefix before the hostname. I checked and found I didn't have that setting. Adding it in and then forcing a gpupdate worked and the clients have started to appear.

Joining an OpenSolaris CIFS server to an AD domain

These notes are rough, but might prove useful in the future (or to someone else who had the same problem). I wanted to join my newly created OpenSolaris 2008.11 installation to my experimental Windows Server 2008 Active Directory. I followed the tutorial in the Solaris CIFS Administration Guide, but when I attempted the actual join command, it failed with LOGIN_FAILURE.

Here's the answer:

Firstly, make sure you have the IP address setup correctly, name added in /etc/hosts, DNS setup correctly in /etc/resolv.conf and /etc/nsswitch.conf. You make sure you are on the same domain as the Active Directory ("windows.zone" in my case).

Secondly, make sure your system clock is synchronized with the domain controller:

# ntpdate dc01.windows.zone

Then setup Kerberos by editing /etc/krb5/krb5.conf. This is documented in the manual.

Install the SMB Server using the package manager and start it up using:

# svcadm enable -r smb/server

Then try to join the domain:

# smbadm join -u administrator windows.zone

For me, this is where it failed. The mailing lists suggest that the problem might be related to smb signing. On the DC, I opened up the Group Policy Management tool and changed the following:

Computer Configuration\Policies\Administrative Templates\System\Net
Logon\Allow Cryptography Algorithms Compatible with Windows NT 4.0 -> Enabled

I then ran a gpupdate /force.

Finally, I read that the sharectl command on the OpenSolaris server should be run to use NTLMv2 authentication:

# sharectl set -p lmauth_level=2 smb

I re-ran the join command, and it worked properly. OpenSolaris CIFS server now part of the Active Directory domain.

Installing OpenSolaris into VMware ESXi

I've just created a very small VM for installing OpenSolaris onto my ESXi box. The ISO CD booted in live mode to the desktop and I selected the option to install to disk.

The installer loaded, but at the point of "Finding Disks", the installer seemed to hang and no disk was found, even though format was displaying the disk okay.

Upon further investigation, I realised that the amount of RAM I had assigned to the VM (512MB) was causing a problem. By increasing this to 1GB, the disk was found and the installer worked as expected.

My guess is that this is due to Solaris deciding that with only 512MB of RAM, that some swap would be necessary, but it's not able to create it. Only a guess, but at least I now have a workaround.

Getting NFS4 permissions working correctly

Following the installation of the OpenSolaris server, I've had some problems with the NFS mounts. I've created an export for a fileserver (datapool/filestore) and although I have setup identical UID/GID maps between my clients and servers, when I mounted the filesystem on my Linux server, I found that files created were owned by nobody:nobody.

At first I thought this was due to some configuration problem in the OpenSolaris installation, but after trying it with the Mac as well, I realised that the mapping was fine. This then pointed to the OpenSUSE 11.1 installation.

The problem turned out to be the Domain setting in /etc/idmapd.conf. The value in this file was different from the OpenSolaris NFS domain. Changing that and restarting the idmapd process (which I did by rebooting the server as I had a kernel update to do), fixed the problem and I now map correctly.

The next step is getting a Samba and Windows client to authenticate me correctly with the OpenSolaris CIFS server. That might be more difficult, but I'll update here when it's done...

Building a test lab

Thanks to the Technet subscription that work has provided for me, I'm now in a position to build my own test Windows network. The purpose of this is to help me get a grip on Windows Server 2008, some Active Directory, Terminal Services etc., and potentially some other non-MS tech such as Citrix XenDesktop [Express].

I've been thinking through the planning of this test lab and recognise that I need to create a network. I can either create a new, completely virtual network and put my VMs on it, routing this to my physical network using a dual homed VM appliance, or I can create the test network in a different address range as my "live production" network and assign the IP addresses so that they don't overlap.

The latter seems the easiest way of doing it (although I may be proven wrong when it's built!).

So, assume my local network is 192.168.0.0/24 (it's not, but I'm not stupid enough to put my real subnet on the 'net!). I'm going to slice up the subnet as follows:

192.168.0.1 - 192.168.0.99 = static range for production network
192.168.0.100 - 192.168.0.150 = DHCP range for production network
192.168.0.151 - 192.168.0.200 = static range for test lab network
192.168.0.201 - 192.168.0.254 = DHCP range for test lab network

How do I determine whether a plugged in device gets a production or test DHCP address? Ultimately it will depend on which DHCP server responds, but the reality is that it shouldn't really matter. Both servers will allocate an address that is routable to the Internet and will resolve the DNS. For anything that will be permanent, I'll allocate a static IP anyway.

My production network has the DNS suffix of local.zone, and I contemplated creating the Active Directory as a sub-domain (windows.local.zone). I think that it will be easier though if I simply create a new domain (e.g., windows.zone) and manually create a DNS forwarder to local.zone when appropriate. This keeps the production network (primarily non-Windows based Solaris, Linux and Mac OS X with a non-domained Vista) from interfering, or depending on, the test lab.

If either of my readers(!) spots anything obviously wrong here, please let me know!

VMware Certified Professional

After a week of revision, plus a couple of years worth of hands on experience and the VMware Fast Track course, I took the VCP exam this morning. Pass mark is 70 and I managed to get 86 which was fine, especially as I remember the struggle that was the SCSA upgrade.

But the IT world does not stay still, and in these days of economic uncertainty, the market is only going to get more competitive, so with the VCP now under the belt, it's time to turn to the next cert... CCNA refresh? Something Citrix? Red Hat? Microsoft? Hmm.

An OpenSUSE quickie

The command line tool for patch and package management on OpenSUSE is "zypper". I've used zypper to list patch updates using:

# zypper lu

The patches can be added (updated) using:

# zypper up

Because I never got around to reading the man page, I didn't realise that both the above commands have an implicit "-t patch". I also didn't realise that "-t package" applied to the above commands can be used to display and update packages to a later version.

# zypper lu -t package
# zypper up -t package

Currently installing 83 package updates...

Backing up ZFS to an external USB drive

Having a resilient, snapshot managed storage server is all very well, but what happens if your server catches fire? While ZFS is very capable of preventing data loss, and the RAID capabilities compensate for a physical disk failure, the lack of a ufsdump/ufsrestore was a bit troubling.

I'm not claiming to have found the perfect solution, but a bit of playing around today with an external USB disk looks promising. I plugged the USB drive in and OpenSolaris automatically detected it. Running the format command showed it was mapped to c6t0d0.

I partitioned the disk to create a single 500GB(ish) slice 0 and created a traditional UFS filesystem on it. I'm sure I could have used ZFS, but wanted the simplicity of a single filesystem without worrying about pools or other volume manager artifacts.

After creating the filesystem with newfs, I mounted it to /mnt.

# newfs -m5 /dev/rdsk/c6t0d0s0
# mount /dev/dsk/c6t0d0s0 /mnt

I had a test filesystem created (datapool/testfs) and copied a file into it. I then took a snapshot of the filesystem:

# zfs snapshot datapool/testfs@mytest

I backed up the snapshot using the ZFS send syntax:

# zfs send datapool/testfs@mytest > /mnt/testfs.backup

This created a single file (/mnt/testfs.backup) containing the filesystem.

With that completed, I deleted the file I copied across. Now for the restore. This was very easy:

# zfs recv datapool/testfs.recover < /mnt/testfs.backup

A new filesystem was created and mounted in /datapool/testfs.recover, containing the file I wanted to recover which I could then copy back. To test a bit further, I destroyed the original datapool/testfs filesystem and all snapshots. I then did another zfs recv and specified the original filesystem name:

# zfs recv datapool/testfs < /mnt/testfs.backup

And it all came back perfectly!

Obviously this is a simple test and doesn't deal with incrementals etc, but should be sufficient for a keeping a copy of the data on an external disk that can be stored off site. Although I haven't tried, adding encryption to the zfs send pipeline should be very simple to do.

ZFS just gets better and better!