Living on the Cloud

HP Microserver SAN in a box: Benchmarks

2011-10-22T21:27:00.000+01:00

In my previous post, I detailed the build of my new "SAN" using an HP Microserver and the Nexentastor virtual storage appliance. Interested in knowing what the server was capable of, I installed the bonnie++ benchmarking software on the Nexenta VM and ran a number of tests to see how well it would perform as a datastore host for the VMware lab:

Each test was run four times and the average of each result taken. All testing was performed against a four disk RAIDZ parity stripe with an SSD read cache.

For the first test, the ZFS filesystem was configured sync=standard and compression=off. This resulted in the following averages:

Block Writes: 65MB/sec
Rewrites: 39MB/sec
Block Reads: 173MB/sec
Random Seeks: 451.55

For the second test, sync=standard and compression=on:

Block Writes: 148MB/sec
Rewrites: 107MB/sec
Block Reads: 218MB/sec
Random Seeks: 2036.1

As the results show, enabling compression results in a huge performance boost.

Although not recommended in situations where data integrity is important, ZFS supports the option of disabling synchronous writes. The third test was run with sync=disabled and compression=on:

Block Writes: 164MB/sec
Rewrites: 113MB/sec
Block Reads: 216MB/sec
Random Seeks: 2522.76

As expected, disabling synchronous writes improved the write performance of the server and had a corresponding knock-on effect for the rewrites. Block reads, although marginally slower than the second test, were close enough to suggest the difference was environmental.

Although running with synchronous writes disabled resulted in the highest performance, in order to get the best possible data integrity, I opted to run with sync=standard and compression=on.

While running the benchmark with compression=on, I noted in the vSphere client that both the CPU cores in the Microserver ran at nearly 100% (the Nexenta VM has 2 x vCPUs assigned). This suggests that the performance here was limited, not by the disks, but by the rather weak CPU in the Microserver.

HP Microserver: Building a SAN in a box

2011-09-18T14:19:00.000+01:00

Following a recent upgrade to my home lab, my storage now looks like this:

A unified SAN capable of providing both block (iSCSI) and file (NFS/CIFS) data.
Eight disks:

2 x SSD
6 x SATA

Two discrete RAID groups:

a 400GB (usable capacity) two disk mirror
a 1.2TB (usable capacity) four disk parity stripe

Both RAID groups have dedicated 20GB flash read caches.
LUNs can be configured to support compression and/or deduplication
Copy on Write (COW) snapshots for all filesystems and LUNs
Support for replicating filesystems and LUNs to a second SAN

All sounds pretty funky. Must be expensive right?

Actually, the above is all achieved using a very cheap HP Microserver running VMware ESXi and the Nexenta virtual storage appliance. I've assigned the Nexenta VM 4GB RAM, but it would happily use more for it's L1 read cache.

The HP Microserver has 4 x SATA disks (2 x 1TB and 2 x 500GB) with a single 60GB SSD disk.

The Nexenta virtual machine is then assigned VMDK files. The first RAID group is a mirror: one VMDK file on SATA disks 1 and 2. The second RAID group is a RAIDZ parity stripe: one VMDK file on SATA disks 1, 2, 3 and 4. The flash read caches are 20GB VMDK files on the SSD.

The compression, deduplication, snapshot and replication features are provided by the ZFS filesystem.

This is a pictorial representation of the configuration:

And this is what it looks like physically:

Oops. No, that's the NetApp at work. But functionality-wise they are quite similar (obviously the vastly more expensive NetApp is much faster!).

This is the real physical hardware (on the far right, next to the ML110 and ML115):

Pretty small for such a setup. I've currently only got the built-in NIC in the Microserver, but will look at adding another to create a dedicated storage network.

Cisco SG200-26 review

2011-09-17T17:16:00.002+01:00

Until recently I was using a Netgear GS108 switch for my home lab. This eight port, unmanaged switch performed well, but with the addition of a couple of HP Microservers, I ran out of free ports and needed something bigger.

Although not essential to the lab, I wanted a switch with a few more features. I initially looked at the Cisco SG200-18, the HP V1810-24G and a couple of other makes that I hadn't come across before (TP-Link and ZyXEL). The one requirement was that the new switch should be silent. The fans of a Cisco Catalyst switch would dominate the home office and was unacceptable.

I discounted the switches from TP-Link and ZyXEL because I couldn't find any decent reviews of them online. The HP V1810 was then discounted because the price hiked up to over £230. This left the Cisco SG200-18. I then noticed that the SG200-26 was only £3 more expensive at £188 (from Ebuyer), so buying the smaller switch would not have made financial sense. You can't have too many ports, right?

The first thing to say about the Cisco SG200-26 is that it is not an IOS switch. I assume it's the result of the purchase of Linksys. Having said that, the build quality is good, the switch is absolutely silent in operation but doesn't get hot (in contrast, the Netgear was hot to touch). The SG200-26 is a managed, layer 2 switch.

The SG200-26 has 24 standard 10/100/1000 ports, plus another two ports for uplinks. These can be RJ-45 10/100/1000 ports or SFP fibre ports (SFP modules not included). The form factor is standard rack-mount 1U (rack mount kit included) but also has attachable rubber feet for desktop use.

Configuration is through the web interface only (no SSH or serial interface), but does support external logging to a syslog server.

Be sure to upgrade to the latest firmware. This enabled the Cisco Discovery Protocol (CDP) which is very useful in vSphere networking for identifying which physical ports a NIC is plugged into.

In the web interface, ports can be given a description and those of us with OCD can spend a happy evening mapping this information into the switch. The port settings can also be used to state the speed and duplex setting of each port.

The SG200-26 supports up to four Link Aggregation Groups (LAGs) and can load balance based on either MAC address or IP/MAC address. Both static and dynamic (LACP) LAG groups can be configured. Up to eight ports can be assigned to a static LAG and sixteen ports to a dynamic LAG.

Multiple VLANs can be setup and managed as the switch supports 802.1q. Ports can be setup as trunk, general, access or Q-in-Q mode. VLAN pruning can be applied to trunk ports so that only specific VLANs are accessible to particular ports. The interface for this wasn't immediately obvious to me (and setting up the same in IOS initially seemed easier), but once I'd spent some time with it, the VLAN configuration was fairly straightforward. These VLAN options can be applied to either individual ports or a LAG.

In addition to these features, the SG200-26 can also be configured for QoS, there are numerous security features including 802.1X, Smartport macros to configure the port type (e.g, Printer, Desktop, Guest, Server etc.). Jumbo frames can be enabled, although this applies is a global setting that affects all ports (most switches, even expensive Cisco switches, work the same way). A "Green Ethernet" function reduces the power requirements of the switch by calculating the length of cable, and also by turning off unused ports to save energy.

As a lab switch, the SG200-26 is ideal. Personally, I would have liked to see a command line option for configuration as some tasks can be repetitive (e.g., setting up VLANs). Beyond that though, there is little to complain about. The SG200-26 is an excellent entry-level switch, with plenty of ports and a good range of options.

Some useful links:

The Cisco Small Business 200 Series Smart Switch Administration Guide

The Cisco Small Business Online Device Emulators page has a demo of the web interface for the SF300. The 300 series has additional layer 3 functionality, but you can get a good idea what the interface is like on the 200 series.

ISP router ARP cache problems when replacing servers

2011-09-09T13:32:00.000+01:00

I experienced a problem today that took a while to understand so figured it was worth sharing...

Our external mail gateway was due for replacement and a new virtual machine was built, configured and tested alongside the old production server. Happy that everything was functioning as expected, the only remaining task was to disconnect the old server from the network and rename the IP address of the new server from its test IP to that of the old server. This would require no changes to DNS and total downtime would be about a minute.

The change was made and... nothing. No traffic to the new server.

Huh? I tested it from another IP on the public network and it was fine. We tried from another network and... nothing.

I changed the IP back to the test address and the server sprang into life.

After a significant amount of time brainstorming with colleagues as to what was happening, we hit upon the possible problem being an ARP cache issue on the ISP provided router. Unfortunately, we don't have administrative access to this router.

Fortunately, the ISP hadn't locked down the console port of the Cisco router and I was able to connect in and run a "show ip arp" command. Sure enough, it showed the MAC address of the old server. This meant that when packets arrived from the Internet the router was trying to forward them to the old server that was no longer on the network. If I had administrative access to the router, I would have been able to flush the ARP cache and all would have been good. But because this was a "managed" router, I wasn't able to do this. I could see the problem, I knew the solution, but couldn't fix it.

I did some research online to see what the default ARP cache timeout was: typically 4 hours.

I logged a call with the ISP which was not a particularly useful experience. The ISP is a subsidiary of Cable & Wireless, and if you've ever had the misfortune of working with that company you'll understand what I'm talking about! I was told I'd get a call back in 8 hours. Brilliant! Not.

There were a couple of other options: Pulling the Ethernet cable from the router would down the interface which I *think* will cause the ARP cache to flush. I didn't have the luxury of doing this in hours.

The final option was to try and get the new server to send a gratuitous ARP request. This is an ARP request that a server broadcasts about itself. The idea is that other devices on the network will update their ARP caches with the information.

My server however was hidden behind a Cisco ASA firewall.

As I was searching for ways to get this working, the ARP cache timed out (possibly due to the router configuration being lower than the default, although I can't see the config to confirm this) and the new server sprang into life.

At first I wasn't sure whether it was the gratuitous ARP that fixed it, but within the next hour, the ISP called and confirmed they cleared the cache. So fair play to them for getting on with it and sorting the problem.

It's been a learning experience in that even the simplest and quickest network change can have unforeseen side effects!

Slow Windows 2008 install in an ESXi VM

2011-09-03T11:54:00.000+01:00

This is just a quick note on a problem I've just experienced (and found a fix for!).

Having just built a new Windows 2008 VM and mounting the ISO from my NFS ISO datastore, I was surprised to see that the actual install was crawling along very slowly at the "Expanding Windows files" section:

As the above link indicates, the problem was the ESX hosts NIC configuration. Set to "auto-negotiate" (which it should be on gigabit connections), the port had managed to negotiate down to 10Mb half-duplex(!). I changed this to 1000Mbit full-duplex and then back to auto-negotiate (where it stayed at 1000Mbit full-duplex).

And the performance became speedy again!

HP Microserver: Remote Access Card

2011-08-29T14:09:00.000+01:00

Remote access functionality, sometimes called "Lights Out" management, is a standard feature on mid- and high-end servers. It allows a system administrator to remotely access the console of the server as well as performing power on, off and reset operations. Most implementations also allow for remote media management, allowing the administrator to remotely connect CD-ROM or floppy images across the network to the server.

Low end servers, including the original ML110 and ML115 G5 servers, and the newer Microserver do not come with this functionality. However, it can be added as an extra.

I was out of spare slots on my KVM, so when I bought the Microserver, I included the Remote Access Card (RAC) in the purchase. The Microserver has a PCIe 16x and PCIe 1x slot. The RAC fits into the 1x slot, leaving another card free for upgrades.

The easiest way to configure the card is to initially use a keyboard and monitor.

The back of the card has a standard RJ45 Ethernet connector and a VGA port. The monitor needs to be connected to this port and not the onboard VGA port. Once connected, the machine can be powered on.

When prompted, press F10 to enter the ROM setup. From here, select the Advanced page and IPMI Configuration:

Select Set LAN Configuration:

Set the BMC LAN Configuration option to Static and then enter and IP address, subnet mask and default gateway:

While in here, it's also worth tuning the VGA configuration. Since this server isn't running anything graphical, I dropped the VGA RAM allocated down to the minumum. From the Advanced page, select PCI Express Configuration:

Under VGA Memory Size, select 32MB.

Exit the ROM setup and save settings. Reboot the server. If everything has been configured successfully, you can now disconnect the monitor and keyboard.

Once configured, open a browser to the IP port and you should get the login screen:

The default username is admin and the default password is password.

I've had problems sometimes getting past the login. My username/password is accepted, but I'm returned to the login page. To avoid, I always go to the index.html and not the login.html, and I use Firefox's Private Browsing mode. I assume a cookie is getting set incorrectly sometimes and this process seems to work around it.

Once logged in, the RAC presents a menu down the left hand side, with the main content on the right. Most is pretty self-explanatory.

Email settings

Remote power control

SNMP trap configuration

The most interesting are at the bottom and provide access to the virtual media and virtual KVM (Keyboard, Video, Mouse):

Virtual KVM and Media configuration

The Virtual Media is a Java application (loads through Java Webstart) and allows either the local CD/DVD drive, or an ISO image to be connected remotely to the server:

The Virtual KVM is also a Jave Webstart application and provides access to the server console. Special keystrokes such as CTRL-ALT-DEL can be sent using the Macro menu. The following screenshot shows ESXi 5.0 running on the Microserver:

The only problem I had with the Java applications is when I attempted to access them with my Mac. For some reason it had problems opening the file. So I used Windows instead.

So how good is the RAC? While it's probably true to say that I won't be using it all that often, it's a very useful addition to the Microserver, especially if you want to put it somewhere out of the way like the garage or loft.

Unlike the more expensive ILO cards, the RAC does not have an onboard battery, so if the Microserver loses power completely, it's not possible to connect to it. However, if power is connected to the Microserver, you should be able to connect.

HP Microserver: BIOS upgrade

2011-08-27T11:55:00.004+01:00

Despite my attempts to resist, the HP Microserver (with £100 cashback) was too tempting a deal, and I've recently taken ownership of a small server, Remote Access Card (RAC) for ILO functionality and 2 x 4GB memory sticks.

The Microserver comes with 4 internal SATA drive bays. Disks are mounted in the brackets and then slide into the server vertically. There is another drive bay on top for an optional optical (DVD) drive. A USB port on the motherboard can be used for installing a hypervisor like VMware ESXi.

My plan was to put 4 SATA disks into the internal bays and mount the SSD in the "ODD" (Optical Disk Drive) bay which would be used as a cache. The SSD is an OCZ Vertex 2 and is a 2.5" sized drive (as most SSDs are). To fit into a 3.5" bay, an adapter is provided. Another adapter was then required to fit the 3.5" bracket into the 5.25" bay.

The Microserver has six SATA ports. The four internal drives are connected to the Microserver's mainboard via a "MiniSAS" connector. The remaining two ports are configured as the internal optical port and an external eSATA port.

Unfortunately, the ODD SATA port and the external eSATA port are configured in "IDE Emulation" mode instead of the faster AHCI mode. This means that it will be limited to a maximum bus speed of 132MB/sec, significantly less than the 3Gbps that SATA can theoretically handle, and you lose some advanced features such as Native Command Queuing (NCQ). It's obviously not ideal to take your fastest disk and put it on the slowest port!

As I'm running VMware ESXi Hypervisor, this can be seen in the vSphere client. The four SATA disks appear on the SATA controller but the CD drive appears under the separate IDE controller as can be seen here:

Image courtesy of the excellent Techhead Microserver review and used with permission.

A fix appears to exist, courtesy of a Russian hacker, who has patched the Microserver BIOS to enable an option that allows the user to turn off IDE Emulation mode and change the port mode to standard SATA.

I was initially reluctant to install this hack in case it caused problems (and set my BIOS language to Russian; it doesn't!), but there are plenty of people who have used the hack without problems. To install, do the following:

Important: This worked for me. Apply at your own risk. I'm not responsible if this bricks your server! You probably won't be covered under warranty if you have problems.

Download the latest HP Systems ROMPaq Firmware Upgrade. (it doesn't matter what release you download as the modified BIOS will replace the version with its own).

To get started, open the start.htm file in the download and follow the instructions on writing the upgrade to a USB key.

Download the modified BIOS. You can get a copy of it here.

Once the USB keyhas been written, replace the *.ROM file with the modified BIOS, renaming it so that the original filename remains. The provided ROM on my system was called O41040211.ROM and the modified ROM was called O41_AHCI.ROM. I removed the O41040211.ROM and renamed O41_AHCI.ROM to O41040211.ROM.

Insert the USB drive in the Microserver and boot it. The firmware should apply. Once this returns the C:\ prompt, remove the USB drive and reboot.

Enter the BIOS when prompted to press F10.

Select the Chipset menu item and then SouthBridge Configuration (this is new functionality provided by the hack):

Select SB Sata Configuration:

Set SATA IDE Combined Mode to Disabled:

Exit and save the BIOS changes. When ESXi boots, the Storage Adapters should now look like this:

All SATA ports are now running at the optimal AHCI mode allowing for up to six disks to be connected at full speed and with no legacy overhead.

Mac OS X Lion and CUPS printing

2011-08-26T08:21:00.000+01:00

When I originally setup my Mac Mini a few years ago (with Leopard), I had some issues getting the printer setup on my network using CUPS. Having upgraded to Lion a couple of weeks ago, the printing problems returned.

The first problem I had was that my printer, an HP Deskjet 5150, was not on the supported list of Apple printer drivers. HP were similarly useless in not providing a driver.

The answer was found in the open source community. A quick download and install of Ghostscript, Foomatic-RIP and HPIJS make the correct driver available (along with many other printer drivers).

This allowed me to add my printer, but upon trying to print, the print queue window would report that it was "Unable to get printer status". Not helpful.

The remote printer is connected to my Netgear ReadyNAS Duo which runs an embedded Linux distribution and uses CUPS as the print server. Despite trying to dig into the debug options, I was not able to fix the printing error.

The default method of setting up a printer on a Mac is to use Bonjour for auto discovery. This uses the Internet Printing Protocol (IPP) under the hood but was failing. Attempts to setup the IPP queue manually also failed.

The fix that worked for me was to set up the printer as an SMB (Windows) printer. This uses the ReadyNAS's Samba install and printing now works! Not ideal, but does the job.

VMware announce vSphere 5 and why all we're talking about is licensing

2011-07-13T09:26:00.000+01:00

Yesterday, VMware announced the latest version of their flagship product: vSphere 5. This new version further extends VMware's lead over its competitors in the virtualisation space giving users the ability to run more and bigger VMs. Compare the capabilities of vSphere vs XenServer or Hyper-V and the ongoing technical superiority of vSphere is apparent.

The new version offers new features such as the ability to automatically provision ESXi servers, storage enhancements (improvements to VMFS, Storage DRS and Profile-Driven Storage), a rewritten HA component, a Virtual Storage Appliance (which looks interesting for SMBs) and a new ESXi firewall. All of which are useful additions to the product.

Regular Enterprise customers have less to get excited about since the Auto Deploy, Storage DRS and Profile-Driven Storage features join the Distributed vSwitch, Storage- and Network-I/O control and Host Profiles as Plus-only features.

You would imagine that the conversations on Twitter and in blogs would be about these amazing new features. You would be wrong.

In releasing vSphere 5, VMware have made a significant licensing change. Whereas previous versions were licensed per CPU, with a limit on the number of cores per CPU, the new version is licensed per CPU with no restriction on cores. However, the amount of memory that vSphere can use is now licensed which VMware are calling the "vRAM Entitlement". The vRAM entitlement is the amount of RAM used by VMs.

A single CPU licence for vSphere 5 Enterprise comes with a vRAM entitlement of 32GB RAM. In a dual CPU ESXi host, this means the sum of RAM allocated to VMs is 64GB. In real terms, this would be equivalent to 32x2GB RAM VMs or 16x4GB RAM VMs.

The vRAM calculation is based on a pool of all resources, so in a cluster with 8 hosts, each with 2 CPUs and 32GB RAM, the total vRAM licensed is 512GB (8 x 2 x 32).

While the VMware Licensing, Pricing and Packaging PDF tries to make it sound like good news for end users because we will no longer be limited to a number of CPU cores, I wonder how many users are CPU-bound. Looking at the infrastructures that I support, CPU utilisation is never the bottleneck; available RAM is.

The way around this limitation is to purchase additional CPU licences. So, if you have a server with 2 CPUs and 128GB RAM running Enterprise, the 2 CPU licences you have will only support 64GB vRAM, so you'll need to purchase another 2 CPU licences. Ker-ching for VMware!

I have some sympathy for users who have deployed large scale servers such as Cisco UCS blades. With a comparatively "normal" CPU count, but huge memory capacity, the licensing requirement for these environments has just gone through the roof.

One of the advantages of VMware vs its competitors is the number of VMs that can be supported on a single host. With Transparent Page Sharing (TPS) and memory compression, vSphere can typically run more VMs than XenServer or Hyper-V and overcommit allows a 32GB server to run more than 32GBs worth of VMs. Now, the savings made in terms of required hardware is offset by the need to purchase more licences.

For those of us with home labs that we use for testing, the vRAM entitlement may be a significant bottleneck. The HP ML110 G6 is a cheap, single-socket server capable of holding 16GB RAM. The free VMware vSphere Hypervisor (AKA ESXi) will only support 8GB per CPU. Whether a second free licence can be applied is unknown, but if not, many home labs will be limited to 8GB vRAM per host. This means more servers will be required which can cause significant spouse issues.

The VMware perspective is understandable. The ability for servers to run with huge amounts of RAM means that organisations require fewer servers, which means less money to VMware.

So is there any good news in this? The one thing that I can think of is it now makes charge back of VM resources easier to calculate. It appears that instead of managing "traditional" virtual infrastructures, where IT is a cost centre, VMware are shifting to a world where everything is provisioned through a cloud infrastructure and IT is a service. In this new world, the ability to charge back will be a core component.

In summary, some users will be unaffected by this change, some will need to pay a bit more to make full use of their environment, and others will need to pay a lot more. I'm sure that VMware's competitors will have a field day with this.

(All the above is based on the VMware pricing document (linked above). If anyone spots anything incorrect, please comment and I'll fix it. Thanks!).

How Windows Live Mesh broke my ReadyNAS backup

2011-07-10T22:08:00.000+01:00

The following has taken me a while to figure out, but here is the answer and it's hopefully useful to someone else with the same (or similar) problem.

I have a Netgear ReadyNAS appliance which I bought because it there was a very good deal on at the time (buy a unit with a 1TB disk and get another 1TB disk for free). It's sat on my desk and not been doing much as I've had other projects to work on.

I decided to configure it as a backup NAS for some of my other machines, specifically T's HP Pavilion and my Aspire Revo, both running Windows 7. I was interested in backing up documents only, so configured the ReadyNAS to connect to each Windows PC fileshare and pull in the data.

This worked perfectly on T's PC but on mine, the ReadyNAS kept complaining that it could not connect to \\REVO\Users. I checked that T's PC could see the Revo share. It could. It tried it with my Mac and could browse the Revo using Finder. I upgraded the firmware in the ReadyNAS. That made no difference.

I then resorted to running smbtree on the command line to see what was happening. This gave me the following output:

\\REVO
cli_rpc_pipe_open: cli_nt_create failed on pipe \srvsvc to machine REVO. Error was NT_STATUS_ACCESS_DENIED

Okay, this was a clue. I remembered the previous problems I had with Samba (which is also running on the ReadyNAS) and Windows 7. Microsoft had set the default authentication protocol to NTLMv2 which was not supported by older versions of Samba. The workaround was to set the Network Security policy on the Windows 7 box to accept NTLM (v1) instead. I check this on the Revo, but it was setup correctly.

I then tried to run smbclient -L //REVO to list the shares on the Revo. The equivalent command worked fine on T's PC, but I got the following on the Revo:

mac-mini:~$ smbclient -L //REVO
Password:
session setup failed: SUCCESS - 0

Time to turn on debugging (appending -d10 to the above command) and compare the output against the two machines. This showed that there was a difference in the authentication protocols being negotiated.

It was at this point, I remembered a forum post that when I read it seemed irrelevant. The post stated that if Microsoft Live Sign-On Assistant was running, this could cause problems as it introduces another authentication protocol "mechToken". Really? An application can break file sharing?

I run Windows Live Mesh on the Revo, which uses the Windows Live Sign-On Assistant. I uninstalled this. The connection worked! I then tested the ReadyNAS connection and... it worked!

I'm not sure where the "fault" lies here but life would be much easier if these protocols were all properly documented and were designed to gracefully fail if the software sees something unfamiliar.

CrashPlan on Mac OS X

2011-06-09T20:59:00.002+01:00

I've been testing CrashPlan as a method of backing up my files to the cloud. After taking a couple of weeks to get all the data up, it's working pretty well and I'm planning on paying for it on a monthly basis. The one downside is that having CrashPlan loaded appears to slow my old Core 2 Duo Mac Mini down. This seems to be because the Java app consumes several hundred megabytes over time.

CrashPlan comprises of two parts: The front end GUI and a Java-based engine. The first job I did was to customise the backup window so that it only backs up overnight. As my Mac is on all the time, this is not a problem. The reason for this is that I'm asleep and not using the computer, and my ISP, Plusnet, have a monthly bandwidth allowance, but 12am-8am traffic is not counted.

In order to work around the memory leak issue, I've created two root cron jobs to start up the engine just before it's needed, and shut it down again afterwards. To do this requires the UNIX command prompt (open the Terminal app):

First, open a root prompt by running a BASH shell:

$ sudo bash

(enter your password here)

Then edit the crontab by running:

# crontab -e

Enter the following lines:

# Start CrashPlan engine at 5 minutes to midnight in time for overnight run
55 23 * * * /bin/launchctl load /Library/LaunchDaemons/com.crashplan.engine.plist
# Stop CrashPlan engine at 5 minutes past 8am to free memory after overnight run
05 08 * * * /bin/launchctl unload /Library/LaunchDaemons/com.crashplan.engine.plist

Edit the vi editor (hit escape, then type :wq! and hit return).

You can view your crontab by running:

# crontab -l

This gives the benefit of overnight CrashPlan backups, but without having any unnecessary services running in the background during the day.

Passing the CCA: My experience

2011-03-30T20:51:00.001+01:00

I've been quiet on the blogging front recently because I've been studying for my Citrix Certified Administrator for XenApp 5.0 for Windows 2008.

I took the exam yesterday and passed with 85% (the pass mark was 68%, so very pleased with that). As with all certs, the content of the exam is under NDA, but it might be useful to know my revision plan.

I attended the official Citrix course back in 2009, but didn't get around to doing the exam. The course is pretty thorough and details the architecture of XenApp, how to install it and configure the Web Interface, plugins, the Secure Gateway, along with application publishing and streaming, policies and the ever problematic subject of printing.

For revision, I downloaded the Exam Prep Guide and used that as the basis on which to read up from the course notes. I also subscribed to the excellent Citrixxperience site which had a very useful set of study notes as well as a large number of practice questions. Note: This is NOT a brain dump site!

I used the home lab to build a number of VMs based on Technet licences and a 60 day trial licence of XenApp 5.0. To simulate the various components of a XenApp build, I created one VM for Citrix licensing, one for the Web Interface server, two for the XenApp Server Farm and a VM for the streaming profiler. I would have liked to create an additional VM for the Secure Gateway, but this would have required a firewall and I didn't get around to it.

As for the exam itself, the Exam Prep Guide tells you all you need to know: Content, structure, times etc. I found that my experience with the practice questions gave a very similar result to my actual score.

I don't realistically expect to take my Citrix knowledge much deeper. I know enough to make some intelligent decisions when designing a XenApp solution, and it remains perhaps the best way to remotely deploy Windows applications over a web browser or in a thin client environment. Although Windows Server 2008 has improved the Terminal Services capabilities of Windows, Citrix still adds a significant number of useful enhancements with XenApp that are required in larger environments.

Facebook: Protecting yourself from viral links

2011-02-20T19:39:00.004Z

This entry is different from my normal posts. It's a response to the increasing number of viral links cropping up on Facebook. These are more than annoying and could in fact be ways in which unscrupulous people steal your personal data.

Okay, here’s how it works... One of your friends appears to post a comment on their wall urging you to click a link. For example:

Or this:

Or even:

(!)

The first thing to do is think "Why would my friend post this sort of link?". If it seems out of character, think carefully before clicking further.

Note the bit at the bottom. This was posted via “Who Visited You”, "9-9" and "Dad Caught Her Strippin". These are Facebook applications that have written the message. Sometimes these are okay (e.g., posted via iPhone/Android/Blackberry - apps you've installed on a mobile phone or tablet). But in these cases, it should cause alarm bells to ring.

So what happens when you click the link?

The link will try and get you to agree to install an application on your page. It's worth noting that Facebook applications have full access to your profile information, including your list of friends.

Here's the simple rule: Do not allow the application to install!

Applications like “Who Visited You” and others will pull in a list of your friends and write on their walls or update your status, pretending to be you and aiming to trick your friends into clicking the link.

Basically, it’s a computer virus.

Why do the application writers do this? Probably to try and harvest as much marketing information about you as possible, but it could be more insidious. If you’re publishing common “known facts” about you (e.g., your date of birth or what schools you went to), it could be used to steal your identity. Think of some of those security questions you get prompted for when you forget your email or online shopping account password. Are those answers in your profile?

Clearing up after it's happened...

If you’ve been caught out by one of these scamming apps, click the Account button at the top right of the Facebook window and select Privacy Settings:

Under “Apps and websites”, click the “Edit your settings” link:

Under “Apps you use”, click the link for “Remove unwanted or spammy apps”.

Delete (click the X on the right hand side next to the app) for all the apps you don’t want to have access to your personal information (the apps in the screenshot below are all valid apps, but you should look out for the dodgy ones).

In addition to the really obvious dodgy apps (such as those illustrated above), consider if you really want "How Blonde Are You?", "Which 80's song describes your life?", "Are you a potato?" (seriously?) or... "FarmVille" to have access to your personal data. Because when you sign up for one of these questionnaires or games, that's what you're doing.

When all the dodgy apps are removed, change your Facebook password!

Keeping safe on Facebook

Finally, if you don't believe me, at least watch this short YouTube video from anti-virus company Sophos that shows how these applications try and trick Facebook users into giving away personal data:

Your identity is important. Look after it.

Safe browsing!

NexentaStor Community Edition: Troubleshooting the slow web interface

2011-01-22T10:49:00.000Z

Although my experience with the NexentaStor Community Edition VSA has been largely positive, I found the web interface to be slow at times. I thought I'd do a bit of troubleshooting to see what was wrong...

The first step to troubleshooting is to get to a proper Unix prompt (remember that NexentaStor is built on the Solaris codebase). I opened an SSH session to the VSA and logged in as "admin". By default the admin shell is a bit special, and for real troubleshooting, we needed the root account. To get this, run the "su" command:

admin@nexenta01:~$ su
Password:
root@nexenta01:/export/home/admin#

Note that I ran "su" and not "su -".

VMware ESX admins may be familiar with "esxtop", and Linux admins with "top". The Solaris equivalent is "prstat":

   PID USERNAME SIZE   RSS STATE PRI NICE      TIME CPU PROCESS/NLWP
   877 root       36M   33M sleep   59    0   0:00:35 0.7% python2.5/15
7312 root     4320K 3468K cpu0    59    0   0:00:00 0.4% prstat/1
   855 root       58M   26M sleep   59    0   0:00:33 0.3% nms/1
   196 root     7812K 4716K sleep   59    0   0:00:00 0.2% nscd/32
   576 root     6952K 4676K sleep   59    0   0:00:05 0.2% vmtoolsd/1
3716 root       17M   15M sleep   44    5   0:00:04 0.1% volume-check/1
   596 root       39M 9092K sleep   59    0   0:00:00 0.1% nmdtrace/1
7213 admin    7744K 5272K sleep   59    0   0:00:00 0.1% sshd/1
   953 root       58M   54M sleep   59    0   0:00:13 0.1% nms/1
3560 root       16M   15M sleep   44    5   0:00:01 0.0% disk-check/1
3564 root       18M   17M sleep   59    0   0:00:02 0.0% hosts-check/1
   434 root     3472K 2104K sleep   59    0   0:00:03 0.0% dbus-daemon/1
   324 root        0K    0K sleep   99 -20   0:00:02 0.0% zpool-testpool/136
     5 root        0K    0K sleep   99 -20   0:00:01 0.0% zpool-syspool/136
   509 root       18M   10M sleep   59    0   0:00:01 0.0% fmd/21
1273 root       58M   54M sleep   59    0   0:00:05 0.0% nms/1
   392 root       13M 8400K sleep   59    0   0:00:00 0.0% smbd/18
   515 www-data   17M 6716K sleep   59    0   0:00:00 0.0% apache2/28
   234 root     2604K 1584K sleep 100    -   0:00:00 0.0% xntpd/1
7231 root     4628K 2568K sleep   59    0   0:00:00 0.0% bash/1
   519 www-data   17M 6564K sleep   59    0   0:00:00 0.0% apache2/28
Total: 91 processes, 732 lwps, load averages: 0.55, 0.53, 0.55

When troubleshooting, I noticed that the process using the most CPU was "nms". This is a custom command provided by Nexenta. Curious to what this command was doing, I ran the truss command against the process id:

root@nexenta01:/export/home/admin# truss -f -p 855
855:    pollsys(0x08047AA0, 1, 0x08047B58, 0x00000000) (sleeping...)
855:    pollsys(0x08047AA0, 1, 0x08047B58, 0x00000000)    = 1
855:    read(4, " l01\00118\0\0\08D\0\0\0".., 2048)    = 176
855:    read(4, 0x0AB8DB78, 2048)            Err#11 EAGAIN
855:    stat64("/tmp/.nza", 0x0813E078)            = 0
855:    stat64("/tmp/.nza", 0x0813E078)            = 0
855:    stat64("/tmp/.nza/.appliance", 0x0813E078)    = 0
855:    open64("/tmp/.nza/.appliance", O_RDWR)        = 9
855:    fstat64(9, 0x0813DFE8)                = 0
855:    fcntl(9, F_SETFD, 0x00000001)            = 0
855:    llseek(9, 0, SEEK_CUR)                = 0
855:    fcntl(9, F_SETLKW64, 0x08047410)        = 0
855:    llseek(9, 0, SEEK_CUR)                = 0

The truss command traces system calls and although the output appear quite scary, you can learn a lot about what a process is doing without needing to know what the system calls are doing. Useful calls to look for are:

open() - opens a file for reading/writing. The number returned (on the right after the = sign) is the file descriptor.
close() - closes a file.
read() and write() - the first number after the "(" is the file that is being read from or written to. Cross reference it with the open() call.
stat() and stat64() - tests to see if a file exists. Don't worry if you get errors returned here as it might be the process looking for a file that may existing in multiple places (e.g., when scanning the PATH for an executable).

The -f option in truss means that child processes will be "followed". So if the process you are tracing forks another process, you will get the data on the child process as well. The -p tells truss to trace the numbered process (obtained from ps or prstat output).

That was a really quick intro to truss for the purposes of explaining how I debugged the problem. Truss is capable of a lot more than I've just described. See the man page ("man truss") for more details.

Back to the performance problem...

The truss output showed me that the nms process was scanning through a lot of ZFS snapshots. There seemed to be a lot of these snapshots. I obtained a list of snapshots on the system:

# zfs list -t snapshot

...and got hundreds back! Something was creating a large number of snapshots. On closer inspection, it appeared I was getting a new snapshot of some filesystems every 7 minutes:

filestore/Shared@snap-daily-1-2011-01-21-2122     1.23M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2129     1.22M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2136     1.39M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2143     1.22M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2150     1.12M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2157     1.06M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2201     1.04M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2208     1.04M      - 50.5G -
filestore/Shared@snap-daily-1-2011-01-21-2215         0      - 50.5G -
filestore/Shared@snap-daily-1-latest                  0      - 50.5G -

It appeared that the filesystems with the large number of snapshots were also the filesystems that I had set to replicate to my second VSA using the auto-tier service. As a test, I listed all the auto-tier services and disabled the suspects:

root@nexenta01:/# svcs -a | grep auto-tier
online         20:33:40 svc:/system/filesystem/zfs/auto-tier:filestore-Software-000
online         20:33:41 svc:/system/filesystem/zfs/auto-tier:filestore-ISOs-000
online         20:33:42 svc:/system/filesystem/zfs/auto-tier:filestore-Shared-000
online         21:33:24 svc:/system/filesystem/zfs/auto-tier:filestore-Home-000

root@nexenta01:/# svcadm disable svc:/system/filesystem/zfs/auto-tier:filestore-Home-000
root@nexenta01:/# svcadm disable svc:/system/filesystem/zfs/auto-tier:filestore-Shared-000

The snapshots stopped.

To determine if the number of snapshots was the problem (I'd seen similar problems before), I destroyed all the snapshots for that filesystem:

root@nexenta01:/# for snapshot in $(zfs list -t snapshot | grep Home | grep snap-daily-1-2011-01 | awk '{ print $1 }'); do zfs destroy $snapshot; echo "Destroyed $snapshot"; done

The web interface performance was fast.

Okay, so that was the problem, but why was it happening? At first, I couldn't work it out and deleted and recreated the auto-tier jobs. Everything then worked fine... for a couple of days. Then the number of snapshots increased again.

This time I was able to identify a change in the configuration. I had had to reboot the second VSA because it had run out of memory (I assigned too little). This appears to have caused the link between the two to have been broken and the auto-tier jobs were running out of control.

Knowing that a new snapshot would fire every 7 minutes, I waited and ran the "ptree" command (shows the list of processes in a tree view showing the parent/child relationships) until I spotted the auto-tier job:

2260 sh -c /lib/svc/method/zfs-auto-tier svc:/system/filesystem/zfs/auto-tier:filest
    2266 /usr/bin/perl /lib/svc/method/zfs-auto-tier svc:/system/filesystem/zfs/auto-tie
      2315 rsync -e ssh --delete --exclude-from=/var/lib/nza/rsync_excl.txt --inplace --ig
        2316 ssh nexenta02.local.zone rsync --server -lHogDtpre.isf --delete --ignore-errors

The problem here was the zfs-auto-tier service (process 2260). Although the full command is truncated, I compared it with the output from svcs (see above) and guessed it to be:

sh -c /lib/svc/method/zfs-auto-tier svc:/system/filesystem/zfs/auto-tier:filestore-Home-000

To examine the properties of this service, I ran:

root@nexenta01:/# svccfg -s svc:/system/filesystem/zfs/auto-tier:filestore-Shared-000 listprop
zfs                                application
zfs/action                         astring
zfs/day                            astring 1
zfs/depth                          astring 1
zfs/dircontent                     astring 0
zfs/direction                      astring 1
zfs/exclude                        astring
zfs/from-fs                        astring /volumes/filestore/Shared
zfs/from-host                      astring localhost
zfs/from-snapshot                  astring
zfs/fs-name                        astring filestore/Shared
zfs/keep_days                      astring 7
zfs/method                         astring tier
zfs/minute                         astring 0
zfs/options                        astring "--delete --exclude-from=/var/lib/nza/rsync_excl.txt --inplace --ignore-errors -HlptgoD"
zfs/proto                          astring rsync+ssh
zfs/rate_limit                     astring 0
zfs/to-fs                          astring /volumes/backup
zfs/to-host                        astring nexenta02.local.zone
zfs/trace_level                    astring 1
zfs/type                           astring daily
zfs/retry-timestamp                astring 1295492474
zfs/period                         astring 1
zfs/hour                           astring 2
zfs/last_replic_time               astring 12
zfs/time_started                   astring 21:43:18,Jan21
zfs/retry                          astring 1
startd                             framework
startd/duration                    astring transient
general                            framework
general/enabled                    boolean true
start                              method
start/exec                         astring "/lib/svc/method/zfs-auto-tier start"
start/timeout_seconds              count    0
start/type                         astring method
stop                               method
stop/exec                          astring "/lib/svc/method/zfs-auto-tier stop"
stop/timeout_seconds               count    0
stop/type                          astring method
refresh                            method
refresh/exec                       astring "/lib/svc/method/zfs-auto-tier refresh"
refresh/timeout_seconds            count    0
refresh/type                       astring method
restarter                          framework    NONPERSISTENT
restarter/auxiliary_state          astring none
restarter/logfile                  astring /var/svc/log/system-filesystem-zfs-auto-tier:filestore-Shared-000.log
restarter/start_pid                count    867
restarter/start_method_timestamp   time     1295642022.247594000
restarter/start_method_waitstatus integer 0
restarter/transient_contract       count
restarter/next_state               astring none
restarter/state                    astring online
restarter/state_timestamp          time     1295642022.255217000

The property that stood out was zfs/retry-timestamp and I guessed the value was a timestamp counting in seconds since the epoch. Converting the value turned it into a human-readable date:

Thu Jan 20 2011 03:01:14 GMT+0000 (BST)

This date was in the past, so was the script running because of this?

I editing the value:

svccfg -s svc:/system/filesystem/zfs/auto-tier:filestore-Home-000 setprop zfs/retry-timestamp=0

And waited...

No new snapshot was created!

I assume this a bug. The temporary failure of the second device should not cause the primary VSA to run amok! Fortunately, this fix appears to have worked and the auto-tier service is now working correctly. The web interface is also performing as expected!

NexentaStor Community Edition: Compression and Deduplication Benchmarks

2011-01-18T21:06:00.000Z

If you read my previous post on benchmarking the NexentaStor VSA and want even more benchmarking information, this post is for you!

ZFS filesystems can be configured to support compression, and in the later releases, deduplication. While both these features are useful in maximising the use of disk space, what is the impact on performance running with these options configured?

The base configuration of the appliance is the same as test 4 from the previous configuration: A mirrored pair of SATA disks, with a SSD L2ARC. The filesystem is configured to use the ZFS Intent Log (ZIL) for synchronous operations (the default) but a separate log is not configured.

The performance data for the base configuration is:

Sequential Block Reads: 59934K/sec (58.5MB/sec)
Sequential Block Writes: 28793K/sec (28.1MB/sec)
Rewrite: 17127K/sec (16.7MB/sec)
Random Seeks: 517.45/sec

The benchmark will use bonnie++ running on a Solaris 11 Express VM connecting to the NexentaStor appliance over an internal vSwitch. The bonnie++ command line is:

#  /usr/local/sbin/bonnie++ -uroot -x 4 -f -s 4096 -d /mnt

See the previous blog post for an explanation of these options.

Test 1: Set compression=on

Enabling compression for a specific filesystem is very simple:

# zfs set compression=on testpool/testing

The results of the test were:

Sequential Block Reads: 52830K/sec (51.5MB/sec)
Sequential Block Writes: 38811K/sec (37.9MB/sec)
Rewrite: 18659K/sec (18.2MB/sec)
Random Seeks: 1188.95/sec

Reads were lower with compression enabled, but writes and rewrites were faster. Random seeks are much faster, but I cannot explain that, although suspect that if the bonnie++ data is highly compressible, this may cause "odd" results such as this.

Test 2: Set deduplication=on

For this test, compression was turned off and de-duplication turned on:

# zfs set compression=off testpool/testing

# zfs set dedup=on testpool/testing

The results of the test were:

Sequential Block Reads: 45806K/sec (44.7MB/sec)
Sequential Block Writes: 27550K/sec (26.9MB/sec)
Rewrite: 15179K/sec (14.8MB/sec)
Random Seeks: 464/sec

This shows that there is a performance penalty for enabling data deduplication. There is also a RAM overhead as the operating system needs to store the dedupe table in memory (not measured as part of this test).

Test 3: Set compression=on, deduplication=on

For this test, both compression and de-duplication were turned on:

# zfs set compression=on testpool/testing

# zfs  set dedup=on testpool/testing

The results of the test were:

Sequential Block Reads: 53844K/sec (52.5MB/sec)
Sequential Block Writes: 34315K/sec (33.5MB/sec)
Rewrite: 17654K/sec (17.2MB/sec)
Random Seeks: 1331/sec

These results suggest that if deduplication is required (to save space), then the additional overhead of compression improves both the read and write performance. As with compression turned on, random seeks are improved significantly.

Conclusion

In conclusion, for maximum read performance, do not turn on compression or deduplication. For maximum write and rewrite performance, turn on compression. If deduplication is required, consider turning on compression as well as this improves dedupe performance. There is a CPU and memory overhead using these features, but as with most things, it's a case of balancing the cost vs the benefit.

NexentaStor Community Edition: Benchmarking

2011-01-14T22:37:00.000Z

In a previous post I discussed how I implemented a Virtual Storage Appliance (VSA) on my VMware home server running the NexentaStor Community Edition operating system. While getting everything working was fairly straightforward, knowing how well it was running required some benchmarking.

I've used the bonnie++ benchmark program before and generally like the way it works. Although I suspect most of this testing could be done through the web interface (setting up the disks etc.), I found it easier and quicker to use the command line and the Solaris commands.

For the test, I create a new VMDK (20GB) on my primary SATA drive and published it to the VSA. I then created a new pool and added the disk:

# zpool create testpool c1t5d0

I then created a filesystem in the pool:

# zfs create testpool/testing

For this testing, I did not enable compression or deduplication (perhaps a topic for another day...).

I ran the bonnie++ benchmark with the following command line:

#  /usr/local/sbin/bonnie++ -uroot -s 8192 -d /testpool/testing

The size (8192) tells bonnie++ to create test data that is 8GB in size. This is twice the RAM allocated to the VM, so prevents the results being skewed by using data cached in memory. I then ran each test 4 times and averaged the results. No other significant activity was taking place while the tests were running. To provide a consistent environment, I used CPU and memory reservations for the VM. I opted to focus on the sequential block reads, sequential block writes (ZFS buffers random writes and writes them sequentially), rewrite and random seek performance. A good guide to understanding bonnie++ output in a ZFS contact can be found here.

Test 1: One SATA based disk

This is the basic starting point: One disk in the pool:

Sequential Block Reads: 75492K/sec (73.5MB/sec)
Sequential Block Writes: 61966K/sec (60.5MB/sec)
Rewrite: 26873K/sec (26.2MB/sec)
Random Seeks: 278.8/sec

Test 2: Mirrored SATA disks

I added a second VMDK to the NexentaStor VM locating it on the second SATA disk. The new disk was then added to the test pool as a mirror. Once the resilvering was complete, the test was re-run:

Sequential Block Reads: 76141K/sec (74.3MB/sec)
Sequential Block Writes: 52331K/sec (51.1MB/sec)
Rewrite: 31525K/sec (30.7MB/sec)
Random Seeks: 292.6/sec

So we can see that in a mirrored configuration, block reads are marginally faster, block writes are slower, rewrites of existing blocks is faster and the number of random seeks has increased. I was a bit surprised that the block read was not much higher given that running a "zpool iostat" on the test pool shows that the read load is balanced across both disks. The slower writes are no surprise as the kernel has to write the same data to two separate devices.

Test 3: Mirrored SATA disks with SSD L2ARC

I added another VMDK to the NexentaStor VM locating it on the SSD datastore. The new disk was added to the test pool as a cache device, implementing a L2ARC:

Sequential Block Reads: 116837K/sec (114MB/sec)
Sequential Block Writes: 65454K/sec (63.9MB/sec)
Rewrite: 37598K/sec (36.7MB/sec)
Random Seeks: 440/sec

The L2ARC has improved the read performance significantly, and surprisingly the write performance is faster too (not sure why this is because the L2ARC is a read-only cache). Rewrite is a bit faster and random seeks is much higher (to be expected with an SSD).

So at this point, we have a pretty good idea what the performance of the NexentaStor appliance is writing to local disk. The next test is to see what the performance is like over NFS...

Test 4: NFS test from Solaris 11 Express VM

The Solaris 11 Express VM is running on the same host and is connected to the VSA by the same vSwitch.

The mount operation was performed by running:

# mount -F nfs nexenta01:/testpool/testing /mnt

The bonnie++ command was:

# /usr/local/sbin/bonnie++ -uroot -x 4 -f -s 4096 -d /mnt

The size of the testing dataset was reduced from the 8192MB on the VSA because the Solaris 11 Express VM only has 2GB of RAM and 4096MB is enough to ensure the VM isn't caching the data in its RAM. It's less than the RAM in the VSA, but at this point we're interested in the performance over the network to clients, not the speed of the appliance itself.

The default behaviour for Solaris NFS is to perform synchronous writes (see my last blog post for a quick primer on NFS/ZFS interactions). Using the zilstat script, I was able to confirm that the ZIL was written to during the benchmark run, proving that the write operations were indeed synchronous. As expected, performance was much worse:

Sequential Block Reads: 59934K/sec (58.5MB/sec)
Sequential Block Writes: 28793K/sec (28.1MB/sec)
Rewrite: 17127K/sec (16.7MB/sec)
Random Seeks: 517.45/sec

Of course, the network stack will be an overhead, but it's worth seeing if we can improve on these times...

Test 5: NFS test from Solaris 11 Express VM, sync=disabled on NexentaStor NFS server

The ZIL is used in ZFS to log synchronous writes to a secure place before writing the data to the pool. The NFS client will wait until the server has confirmed the write to the ZIL before continuing processing. Very good for secure data, but does slow things down. The ZFS sync=disabled option bypasses the ZIL and buffers the request in the server's RAM until it is commited to disk. In real world terms, it's more unreliable, but it's about the same as other non-ZFS based NFS servers such as Linux.

The command to disable synchronous writes (on a per-filesystem basis), is:

# zfs set sync=disabled testpool/testing

The tests were then re-run:

Sequential Block Reads: 69438K/sec (67.8MB/sec)
Sequential Block Writes: 49177K/sec (48MB/sec)
Rewrite: 20737K/sec (20.2MB/sec)
Random Seeks: 520.9/sec

As the test ran, I monitored the ZIL utilisation using zilstat and confirmed that the ZIL was not being used. The results show a significant improvement in writes of approximately 20MB/sec and a smaller improvement in rewrites.

Test 6: NFS test from Solaris 11 Express VM, sync=standard, separate slog on NexentaStor NFS server

Disabling the ZIL improved NFS performance, but what would happen if the ZIL was placed on a separate SSD disk? To do this, I created a new disk from the SSD datastore and attached it to the pool:

# zpool add testpool log c1t4d0

I changed the sync property back to standard (re-enabling synchronous writes) and ran the tests, using zilstat to confirm that the ZIL was being written to:

Sequential Block Reads: 55142K/sec (53.8MB/sec)
Sequential Block Writes: 23931K/sec (23.3MB/sec)
Rewrite: 16518.5K/sec (16.1MB/sec)
Random Seeks: 628.1/sec

Well this was unexpected! The separate ZIL has produced worse results than using the pool SATA disks writing the data twice! The surprising drop in performance may be due to the type of SSD I'm using (OCZ Vertex 2). This is a Multi Level Cell (MLC) type device which is optimised for read operations (most consumer SSDs are MLC). For high performance writes, Single Level Cell (SLC) SSDs are recommended, but they are far more expensive.

Conclusion

To wrap this up then, there are two options to consider when running NFS on ZFS:

Enable the ZIL, experience slower performance but know the data is secure
Disable the ZIL, experience faster performance but understand the risks

The "best" option depends on the environment the VSA is serving. Fortunately the ZIL can be turned on or off on a per-filesystem basis. This means that non-critical test lab VMs can sit on a filesystem with no ZIL for maximum performance, while critical data (e.g., family photos/videos and the copy of your tax return) can be configured with end-to-end consistency.

If you are running a VMware home lab and are looking for a decent virtual storage appliance, NexentaStor CE is definitely worth a look, and as you can see, has plenty of features!

Understanding NFS and ZFS interactions

2011-01-13T23:08:00.001Z

This post was originally going to document the benchmarking of my NexentaStor VSA. Although most of this work has been done (and will be posted soon - promise!), the results were somewhat confusing and required me to dive into the guts of how Solaris (on which NexentaStor is based) handles filesystem operations and NFS. This might be useful if you are trying to debug some performance issues:

ZFS is designed so that all writes are transactional. A write is either successful and the data is written to disk, or it fails and no data gets written. This means that the data on-disk is always consistent.

From an application's perspective, there are two types of write operations to a POSIX compliant filesystem: asynchronous and synchronous.

An asynchronous write passes the data to the filesystem and then continues, effectively assuming that the data is safe. In comparison, a synchronous write passes the data to the filesystem and then waits until the filesystem acknowledges that the data is safe.

Asynchronous Writes

Synchronous Writes

The difference can be seen in what happens if the filesystem or server has a problem (e.g., power fail). In an asynchronous write, the data may only be buffered in RAM and is therefore lost, although the application believes it is safe. With synchronous writes, the data will be definitely be safe because the filesystem only reports back to the application when it is definitely written.

So while the ZFS on-disk data is consistent, it's possible, using asynchronous writes, to lose data in transit. For synchronous writes, ZFS uses a feature called the ZFS Intent Log (ZIL).

When a synchronous write request is made to the filesystem, the data is first written to the ZIL. This ensures the data is safe on disk and the application is free to continue. ZFS will then flush the contents of ZIL to the filesystem at a specified interval (roughly every 5 seconds).

By default, the ZIL is allocated disk blocks from the pool. This leads to a situation where data is written to disk twice, once to the ZIL and secondly to the actual filesystem. This double writing can slow things down.

One option to speed things up is to use a Separate Log (SLOG in ZFS terminology) on which to locate the ZIL. This is typically a flash/SSD drive. Synchronous writes can be logged quickly to the SLOG and then written to slower disk later, speeding up the response to the application. When using a SLOG, the recommendation is to use a mirrored ZIL to ensure that the data is truly safe before being committed to disk.

Synchronous Writes with separate ZIL

How does this impact on NFS?

It is common for an NFS client to write synchronously to an NFS server in order to get an acknowledgement that the data is safely committed to disk. This means that an NFS server with a ZFS filesystem will be performing ZIL writes and come with the associated overheads. This also means the NFS client will be blocked until it receives an acknowledgement back from the server. This can result in "poor performance" when running ZFS and NFS together.

NFS with synchronous writes

There are a couple of options that can speed things up:

The NFS client may opt to mount the filesystem asynchronously. This means that the data will sit in the server's RAM buffers until a transaction group commit and is therefore vulnerable to power failure. NFS clients mounting asynchronously may therefore lose data.

Another option is to disable the ZIL on the NFS server. This effectively makes all synchronous writes asynchronous from the ZFS perspective. Again, data in transit may be lost. Recent versions of ZFS refer to this as sync=standard (default, synchronous writes are written to the ZIL), sync=always (paranoid, everything is written synchronously) or sync=disabled (makes all writes asynchronous).

NFS with async or ZIL disables

Whether NFS async or disabling the ZIL is a risk worth taking depends on the nature of your data. VMware vSphere appears (based on my reading which I'm assuming to be true) to use synchronous writes when using NFS datastores, which can impact on performance. Applying some of the tuning detailed above may help improve performance in a virtual environment.

NexentaStor Community Edition - first impressions

2011-01-05T20:58:00.001Z

During the Christmas break I took the opportunity to upgrade my HP ML110 G5 from the sadly future-less OpenSolaris to another platform. I opted to turn it into a VMware ESXi 4.1 install to run alongside my existing HP ML115 G5 lab server.

The ML110 G5 was fitted with 2 x 1TB SATA drives and a 60GB SSD drive. All three were presented as datastores to ESXi.

For file and block level storage, I opted to use NexentaStor Community Edition. This operating system is derived from the OpenSolaris code base and builds on many Solaris technologies, including ZFS. The enterprise version is pay-for, but the free Community Edition supports datasets up to 18TB, which is easily enough for a home lab environment.

I installed NexentaStor CE on a fairly small volume and created a larger (400GB) VMDK which I then added to the ZFS pool. I assigned 4GB of RAM to the VM, the majority of which will be used as the ARC cache (see below for details).

A (brief) ZFS Primer

In ZFS, physical disks are grouped together in pools. Writes to a pool are striped across all disks in the pool by default, but disks within the pool can be mirrored to each other, or configured in parity RAID comprising of one, two or three parity disks (called RAIDZ, RAIDZ2 and RAIDZ3 respectively) to provide additional resilience.

ZFS filesystems are created from space in the pool and can have many properties applied including size reservations, quotas, compression and deduplication. Filesystems can be shared over NFS, CIFS, or both concurrently.

In addition to ZFS filesystems, Zpools can also contain Zvols. These are basically ZFS filesystems without the filesystem formatted. Zvols provide many of the same properties as a ZFS filesystem including compression and deduplication. Zvols can be shared over iSCSI and formatted by the initiator to hold a server's native filesystem (such as VMFS, NTFS, Ext3, HFS+ etc.).

NexentaStor CE VSA data integrity

With a single 400GB VMDK created and assigned to the VM, I create a new zpool (called Datasets by Nexenta and configured through the web interface - command line mojo not required) and started creating new ZFS filesystems (called Shares, one to hold software installers, another for ISO images, a third for documents etc.).

Obviously a single disk is no good if there is a problem with the underlying drive, so I created a second 400GB VMDK on the other physical disk and presented it to the appliance (all disk rescanning is done without a reboot necessary). The second 400GB was then added to the zpool as a mirror. The process of copying data from the original disk to the mirror is called resilvering and can take some time.

This mirroring is within the VSA and will not help if the primary disk fails as the VM configuration files and boot VMDK are not mirrored. So why mirror the data?

ZFS stores a checksum for the data it writes and when configured in a mirror or RAID-Z, the filesystem is able to reconstruct the data in the event of disk write errors using the redundant data. See here for more information on the end-to-end checksumming and data integrity.

This means that while the VSA will not survive the primary disk physically dying, any corruptions that occur as a disk starts to die will be caught and corrected. A scheduled housekeeping job called a scrub runs weekly to ensure the checksums and data are correct.

NexentaStor CE VSA performance tuning

SATA disks are slow and SSD is fast. Unfortunately SSD is much more expensive than SATA. While one option is to put performance critical data on the SSDs and less important VMs on SATA, the alternative is to use flash disk as cache.

ZFS utilises an in-memory cache called the "Adaptive Replacement Cache" (ARC). This is very fast (being in RAM) and speeds up disk reads, but is limited to the physical memory in the machine (approximately 3GB in a 4GB VM). However, ZFS can support two additional caches: The L2ARC (Level 2 Adaptive Replacement Cache) and ZIL (ZFS Intent Log). The L2ARC is designed to speed up reads, while the ZIL speeds up metadata writes. The best practice for creating a ZIL is to use mirrored flash drives on devices separate from the L2ARC, but as I only had one SSD, I opted to create a single L2ARC.

The L2ARC was created as a 20GB VMDK disk on the SSD datastore and added to the VM. The new volume was then added to the zpool as a cache device. While 20GB is not huge in terms of disk, it represents a significant amount of cache memory.

The performance advantages of the cache are not immediately obvious given that it takes time for the cache to populate. However, once data has been read, future reads will be taken from SSD instead of SATA. I've not had the chance to do meaningful benchmarks yet, but plan to do so soon.

NexentaStor CE VSA snapshots and replication

On top of the data resilience provided by the checksum, ZFS supports copy-on-write snapshots. These can be automatically scheduled on a per-filesystem basis to provide a point in time snapshot. This can be configured so document data is snapshotted daily (or hourly), while more static data such as the ISO store taken weekly or monthly.

The final step was to add even more resilience to the configuration. For this, I created a second NexentaStor CE VM on my HP ML115 G5 lab machine. This VM is smaller with only 1GB RAM. I created a 400GB disk but did not bother with mirroring. Using the NexentaStor web interface, I paired the machines and configured some scheduled jobs to replicate specific filesystems from the primary VSA to this secondary VSA (using snapshot copies over SSH). Nexenta refers to this as a "tiering service". This means that in the event the original server dies, the important data will still be available.

Overkill? Perhaps, but part of this work was to see what could be done with ZFS and the result is a very powerful storage setup.

There are a couple of concerns. One surrounds the longterm viability of ZFS given the Oracle takeover. Although NetApp have settled with Oracle, I don't know if the agreement covers other users of ZFS. Secondly, there will be a performance overhead by running the NexentaStor CE as a VSA on top of the ESXi storage subsystem. While it might be possible to squeeze a bit more performance by running NexentaStor CE directly on the bare-metal, ESXi allows me to run a few other VMs alongside the VSA. The trade-off is worth it in my mind.

In summary, NexentaStor Community Edition is a very powerful piece of software (and this post only scratches the surface - no mention of its AD integration, iSCSI functionality etc.) that gives some high-end functionality *for free* and is certainly worth considering for your home lab.

Home network update

2010-12-22T18:49:00.000Z

My home network has been faithfully served by an installation of OpenSolaris on my HP ML110 G5. Unfortunately Oracle's actions towards the open source community have sadly killed what was an excellent project. While there is a small hope that the Illumos folks might get a fork organised, remaining on OpenSolaris was not practical.

To replace OpenSolaris, I would need an environment that provided all the features I was currently running. This meant I needed a CIFS and NFS server, iSCSI target server, internal DNS server, CUPS print server, private IMAP server (for my old pre-Gmail mail archive) and Windows 7 virtual machine courtesy of VirtualBox. Yeah, OpenSolaris was a *very* capable platform.

The solution I opted for was to turn the ML110 G5 into another VMware ESXi server, running a number of virtual machines the provide the above services. I would also take this opportunity to fix a couple of niggling problems with the way it was setup.

This change coincided with a number of new purchases for the home network:

Acer Revo Aspire
Netgear ReadyNAS Duo
OCZ Vertex 2 SSD

The Acer Revo was bought because I was fed up with using Windows 7 over RDP on the Mac. Little issues like the backslash key not working with a UK keyboard (sounds trivial, but you try using Windows seriously without entering backslash) meant I wanted something I could connect to directly via my KVM. The Revo won't win any awards for high performance, is a capable enough machine (with 2GB RAM) and can run a number of apps (including the vSphere client and Office 2010) without any problems.

I bought the Netgear ReadyNAS Duo after looking at some of the alternatives. I was originally tempted by the Iomega ix2-200d, but was put off by the fact the filesystem is proprietary and requires you to send the unit back if there is a problem. In contrast, the ReadyNAS Duo uses Ext3, but the real dealmaker was an offer to get a second 1TB disk *for free* (via mail-in coupon). My initial playing with this unit has been positive and it's nice and quiet, but I've not spent a huge amount of time with it yet.

The OCZ Vertex 2 SSD (60GB) was purchased because I wanted to experiment with the NexentaStor [Community Edition] virtual storage appliance. Built on top of the open-sourced Solaris codebase, Nexenta have built a storage solution around ZFS. Although the SSD is pretty small for a disk, it can be added as an "L2ARC" (Level 2 Adaptive Read Cache) to boost performance. This will require a block post of it's own to detail.

Finally, although I am not pleased with the way that Oracle have gutted Sun, the Solaris operating system remains excellent and I will be using it in my work for the foreseeable future. The preview release of Solaris 11 Express demanded a look.

I'll be blogging about some of these developments in future posts, coming soon...

Building a vLab Part 6: Configuring Lab Networking

2010-11-20T14:10:00.000Z

In the last post in this series, we configured our two vESXi servers to connect to an OpenFiler storage appliance. This was done by creating a dedicated vSwitch connecting to a Storage LAN. We need to now finish the network configuration to add resilience to our VM network, and add another vSwitch for vMotion.

At the moment, our vESXi network configuration looks like this:

Although we could separate our Management Network from our VM Network (and in the real world, there are some good arguments for doing this), in this vLab, we will use vSwitch0 for both of these functions. vSwitch0 also has a single vmnic which would be unacceptable in a real-world environment.

To setup vSwitch0 for VM traffic, edit the Properties and click Add. Select the Connection Type of "Virtual Machine" and complete the wizard with the default options. Back at the vSwitch0 Properties window, select the Network Adapters tab and add the unclaimed adapter that is attached to the same LAN subnet. When this is setup, close the Properties window.

Next, to setup a private vMotion network, create a new vSwitch by clicking "Add Networking" and specifying the Connection Type of VMkernel. Assign both remaining unassigned vmnic adapters to the new vSwitch. Label the network (e.g., "vMotion LAN") noting that this is case sensitive! Tick the box for "Use this port group for vMotion". Enter an IP address on a new subnet (e.g., 192.168.30.0/24). Don't worry about the VMkernel Default Gateway as the vMotion network does not need to be routable. The resulting configuration should look like this:

Repeat the network configuration on the second vESXi host.

The networking is setup and the vLab should be ready for some VMs!

Building a vLab Part 5: Configuring The Lab Storage

2010-11-14T16:49:00.000Z

At this point in the vLab build, we have done the following:

Installed ESXi on a physical host
Created VMs for Active Directory and vCenter Server
Created a Vyatta VM to act as a router
Created an OpenFiler VM to act as a SAN/NAS storage array
Created two virtual (nested) ESXi server

At this point in the build, we need to connect everything together.

In order to allow the vESXi hosts to access the "SAN", they need to be connected to the correct LAN. To do this, first note down the MAC addresses of the interfaces you have assigned to the Storage LAN. These details can be obtained by using the vSphere Client connected to the pESXi server and editing the settings of the vESXi VM:

With the MAC addresses noted, switch to the vSphere Client connected to the vCenter Server, select one of the vESXi hosts, Configuration, Network Adapters. Identify the vmnics that correspond to the Storage LAN adapters.

Staying on the Configuration screen, select Networking and Add Networking.

Specify the Connection Type as VMkernel, Create a virtual switch and select the two vmnics that are connected to the Storage LAN. Give the network a suitable label (e.g., "Storage LAN"), then assign an IP address on the storage LAN subnet (e.g., 192.168.20.101).

The end result should appear similar to the following:

Repeat this for the second vESXi host.

At this point, the hosts are ready to connect to the storage. The next step is to configure our OpenFiler "SAN/NAS appliance" and share out some storage.

Setup SAN storage

Log into the OpenFiler web interface (https://192.168.20.1:446) as the openfiler user (default password is "password"). In the original build, I added two 100GB hard disks in addition to the 8GB install disk. We will create one as an NFS share and the second as an iSCSI target.

OpenFiler is built on Linux, so an understanding of Linux LVM is useful. A very basic summary of Linux LVM:

Physical disks are encapsulated and referred to as "Physical Volumes" (PVs)
One of more PVs are combined together into a Volume Group (VG)
A VG is carved up into Logical Volumes (LVs)
A filesystem is created on an LV

Click the Volumes button
Click the link to "Create new physical volumes"
Select the first non-OS disk (/dev/sdb on my configuration)
Create a partition with the following properties:

Mode: Primary
Partition Type: Physical Volume

Accept the start and end cylinders and click Create
Click Volume Groups
Under the "Create a new volume group" section, enter a name (e.g., "vmware"), put a tick next to the newly created physical volume and click "Add Volume Group".

First, let's create an NFS datastore. To do this, click Add Volume.
Under "Create a volume in ", enter a name (e.g., "ds01"), a description (e.g., "NFS datastore"), select the size (e.g., 50GB) and choose a filesystem type (I used ext3). Then click Create. This creates a new Logical Volume in the "vmware" volume group that is 50GB in size and formats an ext3 filesystem onto it. The create operation may take a couple of minutes.

When this is complete, create an iSCSI datastore by clicking Add Volume again.
Enter a new name (e.g., "ds02"), a description (e.g., "iSCSI datastore"), assign all remaining space in the volume group and select the partition type as iSCSI. Then click Create. The result should look similar to this:

The OpenFiler appliance will now have the two datastores configured, but they are not published yet. Click the Services button and enable the "NFSv3 server" and the "iSCSI target server".

Click the System button and scroll down to the section titled "Network Access Control". In order for a host to see the OpenFiler storage, it needs to match an ACL entry. The most secure way to do this is to enter the IP address of each VM host. The easiest way is to specify the entire storage LAN subnet (192.168.20.0/24):

Map the iSCSI volume to the ACL by clicking the Volumes button, then select "iSCSI Targets". The system will present a new iSCSI target name. Click Add. To assign the iSCSI volume to the target, click the LUN Mapping button and click Map.

Click the Network ACL button and change the host access configuration to Allow and Update.

Share the NFS volume by clicking the Shares button. Click the NFS Datastore and create a sub-folder (e.g., "VMs"):

Click on the new sub-folder and select "Make Share". Scroll to the bottom of the new window and change the NFS setting to RW. Click the Edit button and set the UID/GID mapping to no_root_squash:

With these options set, click Update.

Finally, change Share Access Control Mode to "Public guest access" and click Update.

Switch back to the vSphere Client connected to the vCenter Server, select the first VM host, select Configure, Storage Adapters. Select the iSCSI Software Adapter (probably vmhba33) and select Properties. Click the Configure button and put a tick next Enabled to turn on iSCSI. Click OK and then select the Dynamic Discovery tab. Click Add and enter the IP address of the OpenFiler server. With iSCSI enabled and configured, click Rescan All, scanning for new storage devices and new VMFS volumes. If all is successful, you should see the new iSCSI volume appear.

Click Storage and Add Storage. Select Disk/LUN and the iSCSI disk should appear. Select it, click Next and create a new partition. Enter a datastore name (e.g., OpenFiler iSCSI) and choose a maximum file size (doesn't matter which since the disk is only small). Finish clicking through the wizard and the new datastore should appear in the list.

Click Add Storage again. Select Network File System and click next. Enter the IP address of the OpenFiler server (192.168.20.1) with a folder path in the format of /mnt/volumegroup/logicalvolume/sharename (e.g., /mnt/vmware/ds01/VMs). Give the Datastore Name something suitable (e.g., OpenFiler NFS).

The datastore view in vSphere client should now look similar to the following:

Repeat the adding of storage on the second vESXi host. After scanning for the iSCSI storage, the datastore should appear automatically. NFS storage will still need to be entered manually.

The next step will be to finish configuring out networks and setting up vMotion...

Building a vLab Part 4: Virtual ESXi install

2010-11-05T18:48:00.000Z

In the last post in this mini-series, we created a VM for vCenter Server and set it up in such a way that the install could be rapidly rebuilt in the future. The next step is to create our virtual ESXi instances (called vESXi in this blog).

When sizing the VMs, it's worth thinking of the vESXi hosts as if they were physical. For example, most physical ESXi servers will have multiple network cards, so we'll put several in our vESXi servers.

Create the virtual machine as you normally would, specifying the following parameters:

Guest OS: Other (64-bit)
CPU: 2 vCPU
Memory: 4096MB

Use the datastore available to the pESXi server and not the OpenFiler VM. Accept the default virtual disk size of 8GB and tick "Allocate and commit space on demand (Thin Provisioning)".

Edit the settings of the VM and add more network adapters (remember, we need this VM to appear similar to a physical host). For the vLab, use a total of 6 network adapters in pairs for the LAN, Storage LAN and vMotion LAN respectively.

The other networking step required is to edit each vSwitch, select the port group and edit the security settings:

Promiscuous Mode: Accept
MAC Address Changes: Accept
Forged Transmits: Accept

Do this for all the vSwitches.

While this is enough to get ESXi installed, you won't be able to power on any nested VMs without making the following change. When the vESXi VM is created (but not powered on), right click on the VM and select "Edit Settings". On the Options tab, select Advanced, General, Configuration Parameters. Add a new row and give it the name monitor_control.restrict_backdoor with a value of true:

Now insert/connect the ESXi install CD/ISO and power on the vESXi VM. To be honest, this next bit should be simple (especially as you've already installed the physical ESXi server).

Once installed into the VM, assign a static IP address.

Now open a second vSphere client instance and connect to the vCenter Server. Within the vCenter Server, create a new Datacenter and add a new host, pointing to the IP address of the newly created vESXi VM. When added, the new server should look similar to this:

There's not much point in having a lab with only a single server, so repeat the process detailed in this post and create a second VM. In the next post, we'll configure it all up...

Building a vLab Part 3: vCenter Server

2010-10-29T21:13:00.002+01:00

Previously on "Building a vLab": Part 1: The Design and Part 2: Infrastructure Build.

For a production environment, many people run vCenter on a server that connects to a SQL Server database on another server (possibly as part of a cluster). However, as part of this vLab, we're going for the default install of a single VM using a local SQL Server Express database.

The vCenter VM has 1 vCPU, 4GB RAM, 40GB hard disk, 1 vNIC connecting to the vLab LAN with an IP address of 192.168.10.2/24. This specification is smaller than that recommended by VMware, but it's enough to get started with. The vCenter server is running Windows Server 2008 R2 as vCenter 4.1 requires a 64bit version of Windows.

Once built, the vCenter Server is assigned the default gateway of the Vyatta VM (192.168.10.33) and the DNS server of the domain controller. The vCenter Server is named "vcenter" and then joined to the vLab domain.

As I do not have permanent VMware vSphere licences in my home lab, I wanted to create an environment where rebuilding from scratch would be a fairly painless experience.

I first created a new 64bit Windows Server 2008 R2 virtual machine. After it was assigned a static IP address and given the correct hostname, I created a small command script called install-vcenter.cmd based on the VMware "Performing a Command-Line Installation of vCenter Server" and copied it to the administrator's desktop.

Having got the base Windows install done, I then exported the VM as an OVF template for future use.

The next step was to mount the vCenter ISO and run the install-vcenter.cmd script. This performed a silent default installation of vCenter including the installation of the .NET runtime and SQL Server Express install. There are many customisable options that can be passed to the setup but these work well enough for my needs:

set EXE=D:\vpx\VMware-vcserver.exe
start /wait %EXE% /q /s /w /L1033 /v" /qr DB_SERVER_TYPE=Bundled FORMAT_DB=1 /L*v \"%TEMP%\vmvcsvr.txt\""

This means that when the vCenter licence expires, I can wipe it out, re-import the template VM, rejoin the domain and run the install-vcenter.cmd script to rebuild a new vCenter installation. It won't keep all my previous settings, and won't configure all the VMs, but it's a start.

UPDATE (22 Jan 2011): If the lab isn't used for a couple of months, the Active Directory trust relationship will fail and the installation will fail. To fix, export the VM when built, but before it is on the domain. I then wrote a short cmd file to automatically join the domain with the following command:

netdom join %computername% /Domain:VSPHERE /UserD:Administrator /PasswordD:mypassword /REBoot:20

In the next part of this series, we'll build our virtual ESXi servers.

Building a vLab Part 2: Infrastructure Build

2010-10-24T20:22:00.004+01:00

The journey begins! In order to build the vLab as detailed in part one, I'll be using my HP ML115 G5. This is a quad core, single CPU tower server in which I've installed 8GB RAM. It's also got 2 x 500GB SATA drives of which I'll be using one for the vLab environment (the other will be used for other projects). The ML115 G5 has an internal USB socket and ESXi can easily be installed on it, reserving the disk space for the VMs.

There is little point in recreating the same installation instructions over and over again when there is a perfectly good reference point. In this case, I'm using the excellent "Installing VMware ESXi 4.0 on a USB Memory Stick The Official Way" post from TechHead (the install for 4.1 is pretty much the same).

As the only physical VMware server, I'll be referring to it as the pESXi box.

At the end of the install process, I have an empty VM host connected to my physical network on which to build the VMs that represent the "physical" items in my virtual infrastructure: The router and the SAN/NAS and the Active Domain controller that we'll need when vCenter Server is installed.

Building the LAN

VMware best practice is to use multiple networks for different types of traffic. The vLab will require four different VLANs (virtual machine network traffic, vMotion traffic, IP storage and access to the physical network). In order to enable this, the pESXi server needs four switches. All four of these switches contain "Virtual Machine" port groups. The switch containing the management interface to the pESXi box also has a VMkernel port.

vSwitch0: Connects to the physical (non-vLab) network
vSwitch1: The vLab LAN for management access and connecting VMs
vSwitch2: The vLab storage network for iSCSI and/or NFS traffic
vSwitch3: The vLab vMotion network

On the pESXi box, the networks look as follows:

You will notice that vSwitches1-3 are not connected to any physical adapter yet. We will use the Vyatta router to accomplish this.

For reference, I'll be using the following subnets:

192.168.192.0/24 - main network connected to the physical network
192.168.10.0/24 - vLab Virtual Machine network
192.168.20.0/24 - vLab storage network
192.168.30.0/24 - vLab vMotion network

Add the routes to the network on your management PC. Alternatively, it might be more useful to add these on your main router so that traffic to these networks route correctly.

The router is necessary because I want to give my vLab a completely separate IP range to the rest of my kit. Therefore, in order for my non-lab kit to communicate with the vLab, I need a layer 3 router. Vyatta have a free "Core" edition that can be installed. For this, I created a new VM with the following sizings:

1 vCPU
256MB RAM
8GB Hard Disk (thin provisioned)
3 x Network Interfaces

1 connecting to the default VM network (i.e., the physical LAN)
1 connecting to the "vSphere Lab in a Box LAN"
1 connecting to the "vSphere Lab in a Box Storage LAN"

For information on installing Vyatta, see this guide. My Vyatta configuration (as displayed using the show -all command) is:

interfaces {
     ethernet eth0 {
         address 192.168.192.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:5b
         speed auto
     }
     ethernet eth1 {
         address 192.168.10.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:65
         speed auto
     }
     ethernet eth2 {
         address 192.168.20.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:6f
         speed auto
     }
     loopback lo {
     }
}
system {
     gateway-address 192.168.192.1
     host-name vyatta
     login {
         user root {
             authentication {
                 encrypted-password $1$VBYqK71jAsu3bsoAznh22mx0pqp31nU/
             }
             level admin
         }
         user vyatta {
             authentication {
                 encrypted-password $1$FdjsdebjGneXOIVw9exHrXRAcaN.
             }
             level admin
         }
     }
     ntp-server 69.59.150.135
     package {
         auto-sync 1
         repository community {
             components main
             distribution stable
             password ""
             url http://packages.vyatta.com/vyatta
             username ""
         }
     }
     time-zone GMT
}

I mapped the Vyatta ethernet adapters (eth0, eth1 and eth2) to the correct network by comparing the MAC address with those listed against each adapter in the vSphere client. The default route (pointing to my Internet router) will make things like downloading software from within the vLab easier.

Building the SAN

VMware vSphere shines when the hosts have access to shared storage. The vLab ESXi servers will connect to an IP based (iSCSI) SAN. There are multiple ways to achieve this and one of the most common ways for home lab users to get shared storage is to use the Linux-based OpenFiler distribution.

Again, in an attempt to avoid reinventing the wheel, I'll point to the excellent Techhead post on configuring OpenFiler.

The specifics for the vLab is that the OpenFiler VM is connected to a the vLab storage LAN and not the vLab VM LAN. The IP address for the OpenFiler VM is 192.168.20.1. In addition to the install disk of 8GB, I've also created a 100GB thin provisioned disk on which to install VMs. The OpenFiler storage will be used for the VMs that I'll install on the virtual ESXi servers. The Active Directory Domain Controller, Vyatta VM and vCenter Server will also install directly onto the 500GB SATA datastore.

Installing the Active Directory Domain Controller

VMware vCenter Server requires Active Directory, so we'll need a domain controller for the vLab. Best practice requires at least two domain controllers for resilience, but we'll make do with just the one (this is a VM lab not a Windows lab). I sized the DC VM to be very small: 1 vCPU with 256MB RAM, 8GB hard disk and 1 vNIC connecting to the vLab LAN with an IP address of 192.168.10.1/24. Although I prefer Windows Server 2008, the DC will run Windows Server 2003 because of it's lower footprint.

The setup was a standard Windows Server 2003 install, followed by running dcpromo. I called the host "labdc", the domain "vsphere.lab" and we're off.

Okay, so we have everything ready apart from our virtual ESXi hosts and the vCenter server. We'll continue the journey in part 3.

Building a vLab Part 1: The Design

2010-10-22T11:46:00.002+01:00

Like many in the VMware community, having a home lab on which to try things out is something I've been working on for some time. As I prepared to update my VCP3 to the VCP4, I thought it would be good to build myself a new "vLab" to test - and break - things without worrying about production systems and complaining users.

This mini series is partly inspired by a posting on TechHead's blog. I think there was originally going to be a series there, but for whatever reason (I'm guessing time was the issue), it didn't really develop.

In order to be able to run through the VCP4 syllabus, I needed the following:

2 x ESX servers
1 x VMware vCenter server
1 x Active Directory Domain Controller
1 x SAN/NAS shared storage array
1 x router/firewall that isolates the vLab from the rest of my home network

This is what I'm aiming for:

Obviously this is going to take up a fair amount of space, will be noisy and hot. So the approach I'm going to aim for is to run the entire vLab on a single HP ML115 G5 with 1 x quad core CPU and 8GB RAM. I'll run VMware ESXi as the base hypervisor and then install the following VMs:

1 x Windows Server 2008 R2 VM to run vCenter Server (4.1 requires a 64bit OS)
1 x Windows Server 2003 VM to run as an Active Directory Domain Controller
1 x OpenFiler VM to run as an iSCSI and NFS server
1 x Vyatta VM to run as a router/firewall
2 x VMware vSphere Hypervisor (ESXi) VMs to run the lab VMs

For the Windows Server 2003/2008 licences I'll use my Technet subscription and the OpenFiler and Vyatta installs are free. For the vCenter Server and enterprise features of vSphere, I'll have to use the evaluation licences.

With the software components downloaded and ready to do, it's time to do the build...http://livingonthecloud.blogspot.com/2010/10/building-vlab-part-2-infrastructure.html

Sun X4100 M2 firmware upgrade

2010-10-08T10:43:00.000+01:00

This is a very short note that others might run into...

I was trying to upgrade the firmware on a Sun X4100 M2 server to the latest release and the System BIOS upgrade was failing. I was picking the firmware image up from a network drive which may have been the problem, as copying the image to my C: drive and then installing the upgrade worked fine.

Not sure why this should be the case, but the upgrade has now worked.

Configure Solaris 10 for mail relaying

2010-10-07T15:14:00.001+01:00

We have a number of devices on our network that can send email alerts. It makes sense to have a central server that can act as a mail relay. We have a Solaris 10 server "sol10" that comes bundled with sendmail, but this is not configured to act as a mail relay.

In order to make the Solaris server relay messages to another host involves editing the /etc/mail/sendmail.cf file and setting the value:

# "Smart" relay host (may be null)
DSmailserver.my.domain

Obviously replace "mailserver.my.domain" with the FQDN of your real mail server that you want to send email through. Restart sendmail by running:

svcadm restart /network/smtp

This setting will allow mail that originates on "sol10" to be sent out, but does not help when you want other devices on your network to use sol10 as it's relay. The answer was surprisingly easy:

Create a new file /etc/mail/relay-domains. In this file, put the networks you want sol10 to accept email from. For example, if you have devices on the 10.0.0.0/8, 172.16.0.0/16 and 192.168.20.0/24 networks and want to use sol10 as the relay, enter the following lines in /etc/mail/relay-domains:

10

172.16

192.168.20

Once done, restart sendmail again (same command as above), configure your clients to use the Solaris server as their SMTP server and check the output in /var/log/syslog while you send a test message.

Upgrading to ARCserve r15

2010-10-04T17:24:00.001+01:00

We've been running ARCserve 11.5 on UNIX for a number of years, but CA have effectively stopped development on it. The Windows version was not suitable for our environment because it was (at the time) unable to perform incremental and differential backups of UNIX/Linux filesystems.

The latest ARCserve release (r15) now supports incremental/differential backups of UNIX/Linux filesystems, so we made the jump.

As most of our Windows and Linux servers are now VMs running in our VMware cluster, we have opted to perform block level VM backups. To do this, we installed the ARCserve Backup Agent for Virtual Machines on our vCenter server.

ARCserve uses the VMware Virtual Disk Development Toolkit (VDDK) to provide integration with the VMware Data Protection APIs. I installed this on our vCenter server and configured the ARCserve server to backup all the VMs using the vCenter server as a proxy (note: this is not a VCB proxy as it does not copy the files, rather the VDDK provides a direct way of accessing the underlying VMDKs).

The problem we had was that the VM backups failed with the following error:

VMDKInit() : Initialization of VMDKIoLib failed

To cut a long story short, the problem was that the vCenter server is 64bit (required by vSphere 4.1). The fix, provided by CA support, was to extract the vddk64.zip file in C:\Program Files (x86)\VMware\VMware Virtual Disk Development Kit\bin.

The problem here appears to be that the VMware installer for the VDDK does not create the 64bit files when installing on a 64bit version of Windows. By adding these and restarting the ARCserve processes, the backup worked successfully.

Now to test the overnight backup of all our VMs...

Edit: The backup appears to be working fine!

Edit 2: Forgot to mention that the CA support representative also modified the system PATH variable to include the 64bit VDDK driver: C:\Program Files (x86)\VMware\VMware Virtual Disk Development Kit\bin\vddk64

Running a Windows 7 client

2010-09-04T10:44:00.000+01:00

Although I never made a conscious decision to switch to the Mac, I've noticed that more of my home computing time is spent in Mac OS X. I've actually given up on using Linux as my main desktop OS, primarily because my Shuttle PC is so loud in comparison with the practically silent Mac Mini (and the Mac provides a decent UNIX environment in a few clicks if necessary).

But when I'm doing home lab stuff, I need access to a Windows PC. I tried to run Windows 7 (thanks to Technet) in a virtual machine on the Mac, but the memory overhead was too great as I've only got 2GB of RAM.

The solution I've decided on is to run Windows 7 under VirtualBox on my OpenSolaris server. As the server is always on, it means I've got quick and ready access to Windows 7 whenever I need it. The server has 8GB RAM, so running a 2GB VM is perfectly usable.

For connectivity to the Windows 7 VM, I'm using the Microsoft RDP client for Mac OS X. This has a couple of nice features: A full screen mode activated by Command-2 (Command being the Windows key on my keyboard) and the ability to switch the windowed view between full size pixels and a scaled fit-to-window option with Command-1.

The Windows 7 VM is used for running the vSphere client, PowerCLI etc., but also has the typical essential apps (Firefox, Evernote, Office 2010) installed as well. Performance isn't as great as a physical Windows 7 server (no Aero Glass for example) and I'm not using it for multimedia apps, but it doesn't require additional space or electricity and it doesn't generate extra heat in the office.

iPad first impressions

2010-08-30T15:03:00.000+01:00

When Apple announced the iPad, my reaction was one of "Looks nice, but it's just a big iPod Touch". I couldn't see where an iPad would fit into my workflow and thought I'd give it a miss.

Two days ago I was out shopping and wandered into Currys. There were some iPads on display and I spent a few minutes playing around with one. As with most things created by Apple, the user interface was beautifully designed and this really impressed me. I had been considering an upgrade to my EeePC to a device with a better display, but after using the iPad, the netbooks on display looked decidedly underwhelming. Although the iPad was more expensive, the difference in capabilities were significant.

The next day, T and I returned to Currys to have another look at the iPad. Despite my demoing the browsing and email capabilities, it was the version of Labyrinth for the iPad that won T over(!). We walked out of the shop with a 16GB Wi-Fi model.

I opted not to get the 3G version as a) it was £100 more expensive and b) the data plans were not cheap. The vast majority of places I'll be using it will have Wi-Fi, but if I really need 3G, I'll probably look to get a Mi-Fi which will enable me to create a local wireless gateway for up to 5 devices.

Here are the first impressions:

The iPad is a large screen iPod Touch. True, but it's amazing how much more you can do with a large screen. Apps look fantastic on the big screen.
The built in apps are beautifully designed. One of my complaints about the iPhone (coming from a Palm PDA background) is the calendar application is very lacking. The iPad version is much improved, finally adding a week view.
Evernote for the iPad is a killer app. It really is brilliant.
The on-screen keyboard is very usable and I can type pretty quickly on it.
Battery life so far is very good (although I've not done much audio/video playback yet).
It's quite heavy. Holding it in one hand for a long time will be uncomfortable.
The Apple foldback protective case is nice and you can create a decent angled stand for typing.

I have synced my Google mail, calendar and contacts to the iPad, installed the Evernote client, installed the Toodledo client, MobileRSS, Twitteriffic, Box.net and many more applications. Within a hour or so, I had access to all my cloud data.

I've also added Wikipanion (a really nice Wikipedia application), BBC News, YouVersion's Bible application, updated versions of RDP, telnet and VNC clients, Connect (for Google Docs reading and soon, editing), Whiteboard HD (for diagrams and doodles) and GoodReader (for PDF reading). Not all of these are free, but pay-for apps come in under £5.

So first impressions are extremely positive. I spend a lot of time at home sat in front of my computer, but with the iPad I can get the same experience for many of these tasks from anywhere with wi-fi. The iPad interface is typical Apple: extremely well designed and very consistent. As a device, the iPad is occupying that vague space between the smartphone and the laptop, but despite being a first generation product, it is polished and is a very welcome addition to my kitbag.

Customising gVim

2010-07-14T14:06:00.001+01:00

While other text editors may be available, I prefer to use vi for my editing needs when running on a Unix or Linux box. I get my vi fix on Windows by running the excellent gVim. When I upgraded to a new work laptop running 64bit Windows 7, installing gVim was one of my first tasks.

There are a couple of things that need to be done to make gVim work correctly. The first is a registry change to add an "Edit with VIM" context menu item in Explorer. The following is the contents of a gvim.reg file I ran to add this functionality:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\Edit with Vim]

[HKEY_CLASSES_ROOT\*\shell\Edit with Vim\command]
@="C:\\Program Files (x86)\\Vim\\vim72\\gvim.exe \"%1\""

The second thing to do is add some customisation. gVim can use a _vimrc file (the underscore is necessary at the start) and uses the HOME variable to locate it. I setup a HOME variable that was pointing to "%USERPROFILE%" (c:\users\jr) and created a text file in %HOME% with my desired settings:

colorscheme slate
set guifont=Lucida\ Console
set columns=132
set lines=50
set nobackup
set number

This doesn't do much apart from set a nice colour scheme, font, window size, prevents backup files being created whenever a file is edited, and turns on line numbers. There are hundreds of options that power users can add to customise gVim, but it's a good start.

With the final edition of adding the gVim icon to my task bar, I now have a comfortable working environment.

Passing the VCP for vSphere 4

2010-06-23T22:32:00.000+01:00

Tonight I took and passed the VCP410 exam, upgrading my VCP for VI3 to the latest release. The scoring is between 100 and 500, with 300 being the pass mark. I got 338 which wasn't great; I actually found the exam pretty tough (being at 6pm on a very warm day probably didn't help either!).

As with my CCNA post, I thought it might be useful to share some of the resources I used to study.

I used both Scott Lowe's Mastering VMware vSphere 4 and Mike Laverick's VMware vSphere 4 Implementation. Both were very good at explaining the underlying technologies, but both had sections that were out of date. Always compare with the official VMware docs!

I also used the following sites:

Simon Long's blog and (practice exam)
vReference
B3RG

The essential VMware exam blueprint and online documentation is a must read.

I also set up a home lab using virtualised ESX servers on my ML115 using the 60 day trial licences. Getting hands on is essential and having an environment where breaking things is not a problem makes revision much easier.

In addition to this, I work with VMware nearly every day (not every component and feature, but I get regular, hands on experience).

The NDA prevents me from talking about questions in the exam, but I will say this: I thought the mock exams were much easier than the real thing. I was getting > 90% in the mocks, so was slightly disappointed to get such as low pass score. Having said that, I'm extremely relieved that I don't have to go back to more revision!

Onto the next thing now...

Exporting and importing SharePoint sites

2010-05-21T10:24:00.002+01:00

A number of our users have SharePoint (WSS 3.0) sites hosted in another office and wanted to move the contents down to our local WSS 3.0 install. This was not as straightforward as you might imagine. We hit a number of gotchas and had to provide workarounds that are documented here so that others can benefit from our experiences.

Running out of disk space on the C: drive

When running an export using stsadm, we kept filling up the C: drive despite exporting to a separate drive. The reason for this is that SharePoint writes temporary files to the location defined by the %TMP% variable. This defaults to the C: drive!

To fix, open a command prompt and type:

set TMP=E:\Tmp

(replace E:\Tmp with the drive and folder you want to use for your temporary storage). Then run the stsadm export and it should work!

Commands to export and import a named site

The command we used to export the site was:

stsadm -o export -url http://old-sharepointserver/hostedsites/development -filename e:\development.cab -includeusersecurity -versions 4 -overwrite

The above command will export the site called "development" referenced at http://old-sharepointserver/hostedsites/development to a file called development.cab. The security information will be included in the export as will all versions of documents.

To import, the following command was used on the new server:

stsadm -o import -url http://new-sharepointserver/development -filename development.cab -includeusersecurity

Note that we are importing the site "development" into the top level and not as a subsite beneath hostedsites. If the name of the site is omitted, the top level site is overwritten!

The gotchas

When running the import, we received the following message:

"The file cannot be imported because its parent web <site path> does not exist"

This error is not helpful and for us the problem was permission related. We had used users (albeit domain admin accounts) to export and import the data that were different from the site collection administrators. To fix we had to do the following:

Make sure the site collection administrator is the same on both the source and destination servers.

When running the export and import, make sure you are running the stsadm commands as the site collection administrator. This ensures the permissions are aligned and the import should work.

stsadm Import error: The 'ASPXPageIndexMode' attribute is not declared

Not sure what the cause of this error is, but we found a fix online:

To get round it I edited C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\XML\DeploymentManifest.xsd on the destination server:

under section



I added the following.

<xs:attribute name="ASPXPageIndexMode" type="xs:string" use="optional"></xs:attribute>
<xs:attribute name="NoCrawl" type="xs:boolean" use="optional"></xs:attribute>
<xs:attribute name="CacheAllSchema" type="xs:boolean" use="optional"> </xs:attribute>
<xs:attribute name="AllowAutomaticASPXPageIndexing" type="xs:boolean" use="optional"></xs:attribute>

With these gotchas overcome, we were able to successfully import the new site.

IBM pSeries (AIX) to Sun StorageTek 2540 - Part 2

2010-05-05T16:31:00.005+01:00

The saga continues...

At the end of my previous port, the SAN LUN was being successfully seen by AIX as a single device using the Cambex driver.

With the multipathing fixed, it was time to build some WPARs. Everything went smoothly until we rebooted. At which point, hdisk10 was visible but I could no longer see the logical volumes on the disk. Furthermore, I couldn't activate the volume group I'd created "wparvg", getting the message:

bash-3.2# varyonvg wparvg
0516-013 varyonvg: The volume group cannot be varied on because there are no good copies of the descriptor area.

To cut a long story short (that primarily consists of me rebooting, removing the device in smit and running cfgmgr is various combinations), the Cambex install (/usr/lpp/cbxdpf) includes some useful commands. Running the dpfinfo command showed that hdisk10 was configured in the following way:

=== /usr/lpp/cbxdpf/dpfutil listall ===
# Device Active Standby
hdisk10 cbx1 (fscsi0 0x040200,1) cbx0 (fscsi0 0x030200,1)

This means that it's using path cbx1 as its active path, with cbx0 as the failover path. Some exploring with the dpfutil command showed it supports the following options:

dpfutil []
Commands may be abbreviated:
HELP - Display this message
LISTALL - List devices and path configuration
ACTIVATE [cbxN] - Manually switch virtual disk to path [cbxN]
VARYOFFLINE [cbxN] - Mark path [cbxN] unavailable
VARYONLINE [cbxN] - Mark path [cbxN] available
MARKFORDELETE [cbxN] - Force path off even if open (may crash)
LIST_HBAS - List HBAs with DPF paths
HBA_SET_WWN [cbxN] [no|yes] - Set WWN preferred path
TARGET_SET_WWN [cbxN] [yes|no] - Set target preferred path

I tried to manually switch over the paths:

bash-3.2# ./dpfutil activate cbx0
bash-3.2# ./dpfutil listall
# Device Active Standby
hdisk10 cbx0 (fscsi0 0x030200,1) cbx1 (fscsi0 0x040200,1)

With this done, I then tried the varyonvg again:

bash-3.2# varyonvg wparvg
bash-3.2# lsvg wparvg
VOLUME GROUP: wparvg VG IDENTIFIER: 00048ada0000d3000000012865034072
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 999 (255744 megabytes)
MAX LVs: 256 FREE PPs: 979 (250624 megabytes)
LVs: 2 USED PPs: 20 (5120 megabytes)
OPEN LVs: 0 QUORUM: 2 (Enabled)
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 1024 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
bash-3.2#

Result!

Not sure what this says about the failover capabilities of the driver... It appears that when the VG is active, manually failing over the paths works okay and the VG remains active.

Fortunately this isn't a mission critical production box (it's a development compile box for porting our code from Solaris to AIX).

IBM pSeries (AIX) to Sun StorageTek 2540

2010-04-28T14:53:00.007+01:00

We have a IBM pSeries 505 running AIX 6.1 that we use for product compilation and testing. The 505 is a 1U, entry-level POWER server with the capacity for two internal disks. In order to provision additional disk space so we can run Workload Partitions (wpars), we've added a single port Fibre Channel HBA.

The Common Array Manager (CAM) software that Sun provides to manage the 25x0 series of arrays (that form the heart of our SAN) allows the administrator to define an initiator which has a "host type" (i.e., what OS the host is running). Among the list of supported host types are the following for AIX:

AIX
AIX (with Veritas DMP)
AIX (Discretionary Access Control)

The "right" option depends on the software running on the server. As I don't have Veritas DMP, and don't know what Discretionary Access Control is, I opted for AIX. Note, this doesn't appear to be documented in any of Sun's manuals...!

So with a volume setup on the array and mapped to the AIX server, it was time to see what the AIX server discovered.

In theory, booting the AIX server should find the new disks, but if you don't want to reboot, run the cfgmgr command. This appears to scan for new devices and the output can be checked by running:

bash-3.2# lsdev -Cc disk

hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive

hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive

hdisk2 Available 01-08-01 Other FC SCSI Disk Drive

hdisk3 Available 01-08-01 Other FC SCSI Disk Drive

hdisk4 Available 01-08-01 Other FC SCSI Disk Drive

hdisk5 Available 01-08-01 Other FC SCSI Disk Drive

hdisk6 Available 01-08-01 Other FC SCSI Disk Drive

hdisk7 Available 01-08-01 Other FC SCSI Disk Drive

hdisk8 Available 01-08-01 Other FC SCSI Disk Drive

hdisk9 Available 01-08-01 Other FC SCSI Disk Drive

hdisk10 Available 01-08-01 Other FC SCSI Disk Drive

hdisk11 Available 01-08-01 Other FC SCSI Disk Drive

The first two disks are internal SCSI. The disks labelled hdisk2 to hdisk9 are the management LUNs for the two 2540 arrays we have and can be ignored. The final two disks, hdisk10 and hdisk11, are two views of the LUN published from the array.

The reason there are two LUNs is because the 2540 has two controllers. Although the array is asymmetric active/passive, the server can see the LUN through both controllers at the same time.

To work around this, we need some multipath I/O software. Hunting around Sun's website found the "Dynamic Path Failover (DPF) Drivers for AIX Operating System 63 General Availability" of which there was a download for AIX 6.1. The download requires a software licence (more on that below...).

With the file downloaded to /tmp, the package could be installed by running smit and selecting the download directory (/tmp) and installing the driver. The software installs into /usr/lpp/cbxfc.

Now, a little diversion into the licensing of this driver. You need to register for a licence with Sun, providing your serial number, contract number and site id which proves you own a 2540 array. After that, the licence is free. Quite why you'd want the driver if you didn't own the array is beyond my powers of comprehension! A 30 day licence is provided in /usr/lpp/cbxfc and can be activated by copying the file "license.30day" to "license". It might be interesting to see what happens if you registered the licence a long way in the future and then changing the date back to today... (purely academic interest only, but might be useful if Sun take their time sending the licence through!).

Having installed the driver, it's another trip into smit, devices and unconfiguring the previously discovered SAN disks, hdisk2 through to hdisk11. When removing, specify the option to remove from the device database as well. After doing this, lscfg should not show the SAN disks.

With the SAN disks no longer visible to the AIX server, re-run cfgmgr and then run lscfg again. If everything is working correctly, the disks should reappear, but be labelled differently:

bash-3.2# lsdev -Cc disk
hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk3 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk4 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk5 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk6 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk7 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk8 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk9 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk10 Available 01-08-01-01 StorageTek FlexLine with DPF V4.31P

Ignoring the management LUNs (hdisk2 to hdisk9), note how there is only one SAN LUN visible, hdisk10. Running /usr/lpp/cbxdpf/dputil utility shows the multipathed disk:

# Device Active Standby
hdisk10 cbx0 (fscsi0 0x010200,1) cbx1 (fscsi0 0x030200,1)

The new disk also appears as a physical volume:

bash-3.2# lspv
hdisk0 00048ada53748080 rootvg active
hdisk1 none None
hdisk10 00048ada40cd9d27 None

The disk can now be added to a volume group and used by the AIX system.

Special thanks to @cgibbo on Twitter who spotted my cry for help when I was struggling to get the multipathing working and got in touch. Thanks for the pointers Chris!
.

OpenSolaris: Very slow boot times

2010-01-16T21:34:00.003Z

Today I lost power to my servers due to a power outage. The UPS wasn't up to coping and almost instantly died (I was running five computers on the one, small UPS...).

Booting the OpenSolaris server back up reminded me of the painfully slow boot times that can occur. We're talking *hours* to get the server up.

The reason for this is due to the number of ZFS snapshots on a system. Here's the experiment:

I booted off the OpenSolaris 2009.06 CD and ran the format command. This displayed the disks on the system. I then imported the zpools into the running installation:

# zpool import rpool -f
# zpool import datapool -f

The -f is required because the system thinks the zpools have been assigned to another server (useful if the zpool is on a SAN LUN). The first command was relatively quick, the second was much, much slower.

Running prstat revealed that devfsadm was consuming an entire CPU. The purpose of devfsadm is to dynamically add and remove devices on the system. It was stating each of the snapshots in the datapool and creating entries in /dev. After running for a few hours, it had created over 4000(!) devices in /dev/zvol/dsk/datapool and /dev/zvol/rdsk/datapool.

The number of snapshots is thanks to the automatic snapshot service which takes frequent snapshots of the filesystems in the pool. This list is not automatically cleared down, so can grow huge. Not a problem usually because the uptime of OpenSolaris is fantastic, but is a real pain when you need the server to boot.

So, in order to keep your OpenSolaris boot times down, keep an eye on the number of snapshots on your system.

Book Review: OpenSolaris Bible

2010-01-10T14:48:00.002Z

The relationship between OpenSolaris and Solaris is similar to that between Fedora and Red Hat Enterprise Linux. OpenSolaris is Sun's "in development" operating system that introduces many new features that will eventually become available in a Solaris 10 update or in a future Solaris 11 release.

So it would seem sensible for Solaris system administrators to have some familiarity with OpenSolaris and while it's possible to transfer a lot of existing Solaris knowledge across, having a comprehensive book alongside can be very useful.

Enter, the OpenSolaris Bible by Solter, Jelinek and Miner; a book I received just before Christmas and have been reading through recently.

The book covers the release of OpenSolaris as of 2008 which suggests the book was based around release 2008.05 or 2008.11 (OpenSolaris releases have a YYYY.MM version number). Since that date, there has been a 2009.06 release and 2010.02 is anticipated next month. However, do not let this put you off considering this book. OpenSolaris development is fast paced, but there is an awful lot of stuff in this book to absorb that still remains relevant in newer releases.

The book is broken into six parts:

Introduction to OpenSolaris
Using OpenSolaris
OpenSolaris File Systems, Networking and Security
OpenSolaris Reliability, Availability and Serviceability
OpenSolaris Virtualization
Deploying and Developing on OpenSolaris

The first part is a typical introduction and covers the history of Solaris, Open Source, as well as instructions on installing OpenSolaris and a basic "crash course" on using the GNOME desktop and the Unix shell. Experienced administrators will be able to skim this section.

Part two covers using the desktop in more detail, printing and software management using the Image Packaging System (IPS). This is an essential read as IPS is a new feature in OpenSolaris and printing can sometimes be a bit tricky.

Part three provides a very comprehensive introduction to Solaris disks, pseudo filesystems such as devfs, tmpfs, lofs and swap, UFS, Solaris Volume Manager, iSCSI, quotas, backups and restores, mounting and unmounting as well as a full chapter on ZFS, before moving onto network configuration including IPMP, link aggregation, virtual LAN interfaces, network services (DNS, DHCP, FTP, NTP, Mail, HTTP etc.), routing and the IP Filter firewall. Part three of the book then finishes with a chapter on network file systems and directory services (NFS, CIFS, NIS and LDAP) and security (PAM, RBAC, SSH, auditing and Kerberos). There is a lot of good content here.

Part four details the Fault Management architecture in OpenSolaris, the Service Management Framework (SMF) introduced in Solaris 10 as well as monitoring with conventional tools and Dtrace, ending with a chapter on high-availability clustering.

Part five covers resource management (projects, tasks, caps and pools) along with a number of Sun virtualisation technologies (Zones, xVM, LDOMs and VirtualBox). The xVM section is only relevant to x64 installs and the LDOM section requires Sun UltraSPARC T-series processors, while Zones can be used one either architecture and is certainly worth a read.

The final part consists of a chapter on deploying a web stack (Apache, PHP, MySQL, Tomcat and Glassfish) and a chapter on software development (Java, C/C++, etc.). I have no strong interest in these subjects at the moment, so haven't read this section.

While I have not read the whole book yet (Having ignored most of the coverage of GNOME desktop applications as if you are familiar with Linux, there's not a lot of new stuff to learn), there are plenty of sections that have made the book worthwhile. Whether this book is suitable for you or not, depends on where you're starting from:

If you are a Windows administrator looking to develop some Solaris experience, the OpenSolaris Bible is well worth a read. The first two parts provide a gentle introduction to the Unix operating system to get you started, and subsequent chapters dive pretty deep into the capabilities of OpenSolaris.

If you are experienced with Linux but have minimal Solaris experience, the OpenSolaris Bible is highly recommended! FMA, SMF, Zones, ZFS, UFS/SVM, Clustering, Dtrace and IPS are not found in Linux, so the OpenSolaris Bible provides a single point of reference for a whole lot of new learning.

Even experienced Solaris administrators will find things to like in this book. The IPS is certainly a new feature that I assume will impact us when Solaris 11 is released, and while ZFS, Zones, FMA, SMF etc are already present in Solaris 10, the book provides a very good overview of these technologies that can otherwise only be found by attending a course or reading the online documentation.

It's probably fair to say that if you read through the whole book, put into practice the features described, and you understand them, you'll have a wider understanding than many existing Solaris system administrators.

The OpenSolaris Bible can be bought at Amazon.

Highly recommended. 9/10.

Hands on Solaris IP Multipathing

2010-01-08T11:45:00.000Z

Today I had a first look at IP multipathing on Sun Solaris. What is IP multipathing? It's a feature that can be used to provide additional network resilience to a server with multiple physical network interfaces. By ensuring that at least two interfaces are on the same subnet, IP Multi Pathing (IPMP) provides continuous uptime if one of the links goes down, migrating the IP address transparently over to the other interface. IPMP operates at the IP layer (layer 3) and does not do link aggregation (although Solaris does support this, I haven't tried that yet).

To test, I booted a Solaris 10 VM on VMware ESXi. I had assigned two NICs to the VM and ensured they were connected and plumbed in.

The first step is to create a "group" for the IPMP by assigning the interfaces e1000g0 and e1000g1 together. I unimaginatively called the group "mygroup".


bash-3.00# ifconfig e1000g0 group mygroup
bash-3.00# ifconfig e1000g1 group mygroup

I then assigned an IP address to each interface and brought the interface up:


bash-3.00# ifconfig e1000g0 192.168.192.105 up
bash-3.00# ifconfig e1000g1 192.168.192.106 up

The output of ifconfig -a looks similar to normal with the addition of the groupname flag:


bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

With IPMP setup, I setup a continuous ping from another machine and then edited the VM in the vSphere client, disconnecting the second NIC from the network. Immediately, the following was reported in /var/adm/messages:


Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 215189 daemon.error] The link has gone down on e1000g1
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 594170 daemon.error] NIC failure detected on e1000g1 of group mygroup
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g1 to NIC e1000g0

No packet loss so far. What does ifconfig -a now show?


bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g0:1: flags=1000843 mtu 1500 index 2
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
e1000g1: flags=19000802 mtu 0 index 3
      inet 0.0.0.0 netmask 0
      groupname mygroup
      ether 0:c:29:f9:5d:ee

While e1000g1 was still present, it now had a status of FAILED and was no longer UP. The in.mpathd daemon had initiated a new virtual interface, e1000g0:1, on the other interface and assigned the failed IP address.

With the test complete, I then reattached the NIC in the vSphere client and noted the following in /var/adm/messages:


Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 820239 daemon.error] The link has come up on e1000g1
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 299542 daemon.error] NIC repair detected on e1000g1 of group mygroup
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g1

This looked good, and a final check of ifconfig -a showed:


bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

This was so easy I don't know why I didn't do this years ago...

Applying GTD principles to Outlook 2007

2010-01-07T20:23:00.000Z

I have now read a large chunk of David Allen's "Getting Things Done" book and have started to apply some (not all!) of the principles to my home and work processes. This started with me tidying my desk before Christmas leave, and filing away reference documents in folders. It's amazing how positive a tidy desk can make you feel.

I also attacked the 13000+ items in my inbox and split them into Archive sub-folders, one for each year (Archive - 2009, Archive - 2008 etc.). This meant that when I started back in January, my inbox had zero items in it!

The heavy snow has resulted in me working from home this week and the productivity benefit has been huge! I have successfully completed almost all of the 69 items I have marked in Outlook "For Follow Up". This had become a catch-all view of emails I wanted to review, or that required action. While it was better than marking the email unread (which some of my colleagues do), it wasn't ideal and could quickly become unwieldy.

So my new plan is to attempt an "Inbox Zero" approach to email. If an email requires an action that takes less than five minutes, I'll do it immediately, otherwise I'll assign it to a category and archive it.

I've also created some additional categories, @Deferred, @Someday/Maybe and @Waiting. GTD followers will recognise these. I've also created a new "Archive - 2010" folder to put emails that are dealt with but I need to keep. All read and actioned emails go here.

Manually setting these categories and dragging completed emails to the archive folder works, but I wanted to have toolbar buttons. Unfortunately my VBA skills are non-existent, but a quick Google found these two sites:

http://blogs.msdn.com/swiss_dpe_team/archive/2007/12/11/office-2007-outlook2007-macros-vba-how-to-work-better-with-categories.aspx

http://verychewy.com/archive/2006/04/12/outlook-macro-to-move-an-email-to-folder.aspx

The former site provides a macro for assigning categories to selected items, and the latter provides a macro for moving selected messages to a designated folder. The first of those two links also provides details on creating a self-signed cert to enable macros in Outlook. Full credit for this code goes to the above sites.

Once I'd modified the macros for my own needs, I created a new toolbar which gives me one click actions to the categories and email archiving:

The source code to the macros is very simple:


Sub Waiting()
Call updateCategoryMain("@Waiting")
End Sub

Sub Someday()
Call updateCategoryMain("@Someday/Maybe")
End Sub

Sub Deferred()
Call updateCategoryMain("@Deferred")
End Sub

Function updateCategoryMain(cat As String)
  Dim myOlExp As Outlook.Explorer
  Dim myOlSel As Outlook.Selection
  Set myOlExp = Application.ActiveExplorer
  Set myOlSel = myOlExp.Selection
  Dim i As Integer
  For i = 1 To myOlSel.Count
      Call updateCategory(myOlSel(i), cat)
  Next i
End Function

Function updateCategory(mi As Object, cat As String)
  Dim pos As Integer
  pos = InStr(1, mi.categories, cat, vbTextCompare)
  If pos > 0 Then
      a = Left(mi.categories, pos - 1)
      b = Right(mi.categories, Len(mi.categories) - pos - Len(cat) + 1)
      res = a & b
      mi.categories = res
  Else
      mi.categories = mi.categories + "," + cat
  End If
  mi.Save
End Function

Sub Archive()
  On Error Resume Next
  Dim objFolder As Outlook.MAPIFolder
  Dim objInbox As Outlook.MAPIFolder
  Dim objNS As Outlook.NameSpace
  Dim objItem As Outlook.MailItem
  Set objNS = Application.GetNamespace("MAPI")
  Set objInbox = objNS.GetDefaultFolder(olFolderInbox)
  Set objFolder = objNS.Folders.Item("Mailbox - R, J (XXX)").Folders.Item("Archive - 2010")

  'Assume this is a mail folder
  If objFolder Is Nothing Then
      MsgBox "This folder doesn't exist!", vbOKOnly + vbExclamation, "INVALID FOLDER"
  End If

  If Application.ActiveExplorer.Selection.Count = 0 Then
      'Require that this procedure be called only when a message is selected
      Exit Sub
  End If

  For Each objItem In Application.ActiveExplorer.Selection
      If objFolder.DefaultItemType = olMailItem Then
          If objItem.Class = olMail Then
              objItem.Move objFolder
          End If
      End If
  Next

  Set objItem = Nothing
  Set objFolder = Nothing
  Set objInbox = Nothing
  Set objNS = Nothing
End Sub

Okay, so there's a fair amount of hard coding, but I didn't want to spend more time than absolutely necessary to get this working!

The final improvement I've done is to assign colours to my incoming emails. By selecting Tools, Organize and then selecting "Using Colors", it's possible to configure Outlook so that mails that are only to me are in blue, mails from my line manager(s) are in red and mails from T are in green. Everything else is in black, but it provides a nice visual clue.

No promises that Inbox Zero and the GTD approach will work for me, but so far I've managed to stay on top of incoming requests and finding things in my mailbox is refreshingly speedy.

An hour with Solaris Live Upgrade

2009-12-03T18:12:00.003Z

Last week I had the opportunity to do some work with the Live Upgrade feature of Sun Solaris. I had been vaguely aware of it's capabilities and we had been including a provision for it on our customer server builds, but it was only yesterday that I sat down and tried to do an upgrade to the latest Solaris 10 update 8.

Live Upgrade is a capability where the system administrator can upgrade to a new version of Solaris while the existing operating system in running. The only downtime experienced is a scheduled reboot at the end of the process to initialise the new version. If something goes wrong, the original version of the OS is still available for booting.

The way it works is based around the concept of a "boot environment". The default environment is the operating system you're running at the moment. On our servers, we have been creating a 20GB root (/) filesystem. A second 20GB slice is also created, but not used (nominally mounted as /lu so we remember it's there).

The first step in running the live upgrade is to create a new boot environment. Firstly, the /lu partition was unmounted, and commented out in /etc/vfstab. Once that was done, the new boot environment was setup:

# lucreate -n osupgrade -m /:/dev/md/dsk/d30:ufs

Okay, so here we are creating a new environment called "osupgrade" and saying that the root ("/") filesystem will be installed on the device /dev/md/dsk/d30 and that the filesystem type will be UFS (it can also do ZFS but we didn't have the correct setup on my test system). This bit takes a while, depending on how much you have on your root filesystem.

For those unfamiliar with the /dev/md/ part, this is a Solaris Volume Manager (SVM) metadevice. In reality, "d30" is a mirror that contains two submirrors (probably called d31 and d32). These submirrors are comprised of one of more disk slices. In other words, in the above command, the new boot environment will be installed onto a new mirrored disk.

At the end of the lucreate command, you can actually look at the new, mounted boot environment and see that it's basically a copy of your existing root filesystem. The next step is to upgrade it. To do this, I mounted the install location of our Jumpstart server over NFS and initiated the live upgrade:

# luupgrade -u -n osupgrade -s /mnt/install_sol10_u8_sparc/

This bit takes a while (a bit like installing Solaris...) but basically upgrades the named boot environment using the media specified. At the end of this, all that needs to be done is for the boot environment to be activated:

# luactivate osupgrade

Before initialising the new environment, it's worth noting down your existing, working environment. For me, this was the root filesystem located on /dev/md/dsk/d10. Find out the underlying slices used by d10 (c2t0d0s0 and c2d1d0s0 in my case) Once done, reboot the server and the new boot environment should be loaded.

Now the coolness of this should be immediately apparent! Previously, operating system upgrades would require a backup of the system to tape (always a good idea!), followed by scheduled downtime as the system was upgraded "offline". This also meant a visit to a customer site, typically at a weekend.

Combined with the use of an ILOM interface (for network access to the console), it now becomes perfectly possible to upgrade a Solaris server during the day, while users are on the system. All that it required now is an out-of-hours reboot of the server to initialise the new release.

If there are problems with the upgrade, it's possible to rollback by setting the old boot environment to active. To do this, boot off cdrom or the network (I did the network), by typing the following at the PROM:

ok boot net -s

[wait for the OS to boot)

# mount -Fufs /dev/dsk/c2t0d0s0 /mnt
# /mnt/sbin/luactivate

Exit single user mode and reboot.

Obviously this is only scratching the surface of what Live Upgrade can do. It's possible to merge and split filesystems, detach and build disk mirrors, and much more. The use of Live Upgrade is also greater than the occasional update; it's perfectly possible to use Live Upgrade to apply system patches, with a very easy rollback capability.

Definitely a technology that needs to be investigated more fully...

A first look at Toodledo

2009-11-11T16:41:00.005Z

My ongoing quest to manage my email, calendar, contacts, tasks and notes from the cloud continues. I've managed to cover most of the above with Google Mail and contacts, Google Calendar and Evernote. Finding a task manager that I like the look of has proven tricky.

I tried to use the brilliantly titled Remember the Milk, but it's not designed to interface with Outlook. Unfortunately I spend a significant amount of my working life in front of Outlook 2007 which has very powerful task management. Using multiple task managers would not make my life easier.

I signed up to Toodledo a while ago, but didn't get around to trying it out. There was an Outlook sync tool, but it was pre-1.0 which didn't inspire confidence. So I left it a while... and went back to it last night.

The sync tool is now post-1.0 so I installed it and took my task list from Outlook 2007 into the cloud. I set Toodledo to map Outlook categories to folders so the organisation structure remains.

There is one downside to the Outlook sync tool: It syncs Tasks, but Outlook 2007 introduced another item called "To-Dos". Think of the "To-Do" list as all types of data that have been marked for action. This includes all tasks, but also all flagged emails, diary events and contacts. Apparently the To-Do list is not stored in the same structure as the Task list, so syncing it is not yet achievable. It's a minor point though and arguably not the fault of the sync tool.

One nice surprise about Toodledo is that it has a Notebook feature. Upon examining this I was surprised to find my Outlook notes had been copied across. They sync changes too, so although I'm primarily using Evernote for taking Notes, it's nice to know that anything I jot down in Outlook is made available as well.

I also spent a whopping £1.79 on the Toodledo iPhone application. This has a very pleasing and easy to use interface. The detail is there, but there's not too much detail to make it unusable. I've not had the chance to play with it too. It works when the iPhone is in airplane mode, so data appears to be cached locally. The aforementioned notebook is not available from the iPhone (that I've seen yet).

There are also integration points for Firefox, Google Gadgets, Twitter, Apple iCal and Dashboard as well as good old email and RSS.

It's early days and I've not used Toodledo enough yet to determine whether I want to commit to basing my task management around it, but initial usage is encouraging. I'll hope to implement some of the popular GTD methodology to it having just received a copy of David Allen's book "Getting Things Done". I just need to get organised enough to read it...

Working with Evernote

2009-11-07T17:27:00.000Z

When it comes to notebook applications, Microsoft's OneNote is arguably the most powerful product out there. Unfortunately, as I commented in a previous blog entry ("Is OneNote a Cul de Sac"), OneNote - in its current form - is very much a single user, single machine application. Getting data into OneNote is easy, but accessing it remotely over the cloud is not possible.

Perhaps the most popular alternative is Evernote. This provides a cloud-based service that is free when a limited amount of data per month is created (a limit I am nowhere near reaching). Evernote can be accessed through the website, but also provides clients for Windows, Mac and the iPhone. This makes it a very compelling product to use, even if it's not quite a feature-rich as OneNote. If using Windows, it's definitely worth using the 3.5 beta release as the interface is much better (IMHO) than previous releases and is more similar to the Mac version.

One of the things I like about OneNote is the Notebook/Section/Page metaphor. It's possible to have multiple Notebooks (e.g., Work, Personal etc.) and each notebook can have section tabs. Within each section, multiple pages can be created. In comparison, Evernote has the concept of notebooks and pages. In order to work around this, I adopted a naming convention for my notebooks that describes what each notebook is for:

Personal: Holidays
Personal: Home Projects
Work: AIX Notes
Work: Business Planning
Work: Citrix

This effectively works around the lack of sections. (I actually realised after adopting this approach that when using OneNote, I only had a couple of notebooks for Personal and Work anyway and used sections to separate my data)

As for when to use a new notebook, I've adopted the approach that if I was to use a new physical notebook in real life (e.g., I'm attending a course), then I'll create a new notebook for it. This is what I did last week when on a Citrix XenApp course. All my notes were instantly available via the cloud to all my clients. I'm also using new notebooks for different projects.

Evernote isn't perfect. I'd still like the ability to click anywhere and enter text (like OneNote), and wish it handled proper text headings instead of just setting font sizes. But these are pretty minor. A native Linux client would also be nice, but the web interface works well enough for occasional use. One killer feature for me would be rudimentary drawing tools so I could illustrate notes with diagrams. In fact, I'd become a premium user for this feature.

Evernote is definitely worth a look if you're looking for a way to store all your notes together in one place, but access them from anywhere. A cloud win.

Building a ZFS filesystem using RAID60

2009-10-20T12:56:00.005+01:00

We are starting to use ZFS as a production filesystem at $WORK. Our disk array of choice is the Sun StorageTek 2540 which provides hardware RAID capabilities. When building a ZFS environment, the decision has to be made on whether to use the hardware RAID and/or the software RAID capabilities of ZFS.

Having watched Ben Rockwood's excellent ZFS tutorial, my understanding of ZFS is much better than before. For our new fileserver, I've created the following:

On the StorageTek 2540, I've created two virtual disks in a RAID6 configuration. Each virtual disk comprises of 5 physical disks (3 for data, 2 for parity) and is assigned to different controllers. On top of each virtual disk, I've created a 100GB volume. This is published as a LUN to the Solaris server and appears as c0d3 and c0d4.

Each LUN is then added to a zpool called "fileserver":

# zpool create fileserver c0d3 c0d4

By default, ZFS treats the above in a variable width stripe, so the hardware and software combined result in a "RAID60" configuration; data is striped across 2 x RAID6 virtual disks for a total width of 10 spindles.

Why RAID6 and not RAID10? Apart from the cost implications, as a fileserver, the majority of operations will be read-only and RAID6 is very good at reads (while being less-good at writes).

Now, when I'm running out of space, I can create a new volume, publish the LUN and add it to the zpool:

# zpool add fileserver c0d5

A quick check of the zpool status shows the disk is added successfully:

# zpool status
pool: fileserver
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
fileserver ONLINE 0 0 0
c0d3 ONLINE 0 0 0
c0d4 ONLINE 0 0 0
errors: No known data errors

Running zpool list reports the newly added space is available:

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
fileserver 199G 69.4G 130G 34% ONLINE -

All told, very simple to do and results in a pretty fast filesystem.

Debugging backup problems

2009-10-10T17:23:00.003+01:00

We had some problems with our nightly backup recently. These are tricky to debug as you don't want to be playing around while users are on the system being backed up, so it can become a case of try and fix something during the day, wait until the nightly backup run and try again the next day when it's failed.

The problem for us was that the backup would start, and then fail after about 10 minutes. The software we use is CA ARCserve for Unix and the message logged in the error log was a very unhelpful "unknown scsi error".

The backup server is a Sun V125 running Solaris 10. As noted above, the backup software is ARCserve, version 11.5. The server has a Qlogic fibre channel HBA for SAN connectivity and plugs into a SAN fabric built on Qlogic 5200 (2Gbit) and 5600 (4Gbit) switches. Also plugged into the SAN is an ATTO Fibrebridge. Our tape library, the Sun StorageTek SL48, connects via SCSI to the fibre bridge, which in turn publishes the internal tape drive and the library as LUNs on the SAN.

As you can see, there are a few things that could go wrong. The reason we have this somewhat complicated setup is that when we originally bought the SL48, although it was listed on CA's HCL, it only worked in a fibre attached configuration.

Having checked the obvious; that ARCserve could see the tape library, load, unload and scan the media, that there were sufficient "scratch" tapes available for writing and that the Ingres database that holds all the backup records was consistent and working properly, I turned my attention to the hardware.

The first step was to eliminate ARCserve from the list of suspects. When loaded, ARCserve loads the "cha" device driver that controls the tape library. I rebooted the backup server to ensure that the drivers were definitely not running, and observed that the tape drive could be seen as /dev/rmt/0. Using the SL48 web interface, I loaded a tape and tried to perform a ufsdump of a local filesystem to the tape drive.

This worked for a while and then failed with a write error.

Okay, so it's not ARCserve, and it looks like an error on the tape drive. Perhaps a clean would help. ARCserve is meant to run automatic drive cleans, but perhaps it hadn't. Again, the SL48 web interface provides functionality to do this.

The SL48 complained that the cleaning tape had expired. Unfortunately, I didn't have any spare, but it looked like this might be the issue. I immediately ordered three more tapes, and configured the SL48 to email me when it had warnings as well as errors. This should mean I get notified when the next cleaning tape expires.

A dirty drive was the most likely suspect, but in order to definitely rule out the SAN, I tried directly attaching the SL48 to the V125's internal SCSI port. This hadn't worked with the original ARCserver 11.5, but we had since applied a service pack.

The service pack had updated the device driver list and the SL48 was now detected as a local SCSI drive. I tried the same ufsdump, against the same local filesystem, using the same tape and expecting the same error, but was surprised to see that the backup completed without any problems.

Hmm, perhaps it's the SAN.

Last week, prior to the problems starting, we added a new fibre switch (the 5600) and this required that our existing switches (5200) have a firmware upgrade so they were all at the same level. It's possible that there is something in this latest firmware release that is causing the tape library (or fibre bridge) to choke.

I fired up ARCserve again and kicked off a backup job. It was during the day, but we hadn't had a working backup for several days, so I was content to take the performance hit (not that anyone appeared to notice).

The backup ran for 13 hours and completed successfully.

I've not actually spent more time trying to determine whether the problem is with the switches or the [now redundant] fibre bridge. The backup is now a lot simpler in it's configuration. My belief is that of all the systems we run, the backup should be the simplest, especially when there is a restore to be done!

Subsequent backups have been successful, but in the event of future problems, it's useful to have a template to work from when debugging any issues.

ESX/ESXi networking best practices

2009-09-26T13:48:00.002+01:00

VMware Virtual Infrastructure 3 had some fairly well defined best practices for networking. There are three types of network connection that can be setup in an ESX host:

Service Console
VMkernel for vMotion
Virtual Machines network

Although it is possible to have a single physical NIC connecting to a single vSwitch with the above port groups configured, this is not a good approach because the NIC becomes a single point of failure. Adding a second NIC helps and teaming the two physical NICs provides redundancy, but you could still experience performance problems when a bandwidth intensive operation occurs (such as a vMotion) as VM traffic could suffer from a lack of bandwidth.

One solution is to add additional NICs to the vSwitch and hope that the increase in bandwidth is sufficient so that vMotion traffic does not impact on connected users, but there is another potential issue: vMotion traffic is unencrypted, and the content of a vMotion operation is the memory of a virtual machine. The problem here is that if a malicious user is able to eavesdrop on the connection, they might be able to access sensitive data. Using separate VLANs help, but you're still effectively crossing your fingers and hoping everything will be okay.

The safer approach is to separate the Virtual Machine traffic from the vMotion traffic using a separate vSwitch and assigning each vSwitch two physical NICs for redundancy. This ensures that the vMotion traffic is physically isolated from the VMs.

So where to put the Service Console? It is possible to assign it to either the Virtual Machine network vSwitch or the vMotion VMkernel network vSwitch. It's worth pointing out that Virtual Center requires access to the Service Console, so depending on where you run Virtual Center, this might impact on which vSwitch you assign the Service Console. By placing the Service Console on a separate vSwitch to the Virtual Machines network helps to reduce the ability of malicious users of hacking the SC. It's common (especially in environments where servers have 4 physical NICs) to find a configuration where one vSwitch is dedicated to VMs, and the second vSwitch shares the Service Console and vMotion VMkernel port groups.

A four NIC server configuration could look like:

2 x NICs for Virtual Machine traffic
2 x NICs for vMotion traffic and the Service Console

If six NICs are available, the configuration could look like:

2 x NICs for Virtual Machine traffic
2 x NICs for vMotion traffic
2 x NICs for the Service Console

Although the Service Console doesn't require much bandwidth, some sites perform backups from within the SC which can have a significant network overhead.

What about non-Fibre Channel storage?

For users with iSCSI or NFS, things are slightly more complex. The VMware iSCSI initiator lives inside the Service Console but also requires a VMkernel network for actual storage traffic. It would be logical therefore to put these two port groups on the same vSwitch.

Both iSCSI and NFS traffic should be assigned to a separate network to increase security and guarantee that bandwidth is available regardless of whether a vMotion or heavy VM traffic is occuring. Two additional NICs should be allocated, resulting in the following six NIC configuration:

2 x NICs for Virtual Machine traffic
2 x NICs for vMotion traffic
2 x NICs for Service Console and iSCSI/NFS storage

Given an additional two NICs (8 total), the following could be configured:

2 x NICs for Virtual Machine traffic
2 x NICs for vMotion traffic
2 x NICs for Service Console
2 x NICs iSCSI/NFS storage

So what's changed in the vSphere 4 world?

For our new deployment, I plan to use ESXi instead of ESX. VMware have stated their intention is to move to the ESXi hypervisor for future releases, and we have no legacy requirement to run the service console in our environment. On first glance, this would appear to remove the need for two of those ports in the above list.

But the question arises, if the Service Console is not present in ESXi, what IP address and interface does the vSphere client connect to? Well, the IP address is that assigned by the sysadmin to the host using the ESXi text mode menu (not sure what the proper terminology is for that). The interface is actually a VMkernel interface.

Replicating the ESX environment configuration in ESXi would therefore look like:

VMkernel for Administration Network
VMkernel for vMotion
Virtual Machines Network

Assuming each of the above has redundant connections, we still need six NICs, although if you are limited to four NICs you could apply the same approach as with ESX 3.x and combine the Vmotion and Administration networks into a single vSwitch.

If you plan on using NFS storage or iSCSI, you will need another VMkernel interface for storage, so add another couple of ports.

One of the new features in vSphere 4 is Fault Tolerance (FT). This features ideally needs a dedicated network between hosts, so that takes the total number of physical ports up to 10:

2 x NICs for VMkernel Administration Network
2 x NICs for VMkernel Vmotion
2 x NICs for Virtual Machine Network
2 x NICs for VMkernel NFS/iSCSI
2 x NICs for VMkernel FT

The above example only accounts for a single vSwitch for Virtual Machine traffic. If there is a reason for a second vSwitch with VM traffic (e.g., you want to segment a DMZ onto a physical network), additional NICs will be needed. Obviously doing this will cause the number of NICs to increase.

Conversely, if your server doesn't support 10 NICs, some sharing of physical NICs / vSwitches will be required.

Our environment only supports 6 NICs per server and we don't use iSCSI. Our NFS usage is limited to ISO datastores so this can be shared with the Administration and vMotion networks, so the approach we'll probably take is:

2 x NICs for VMkernel Administration Network, VMkernel vMotion and NFS network
2 x NICs for Virtual Machine Network
2 x NICs for VMkernel FT

This post is based on each NIC having 1Gbit ports. As the 10Gbit NIC becomes more popular, the network design approach might change. Something to think about for a future post...

Adding a SATA disk to ESXi

2009-09-19T21:50:00.000+01:00

Having freed up two 500GB SATA disks from my storage server, I wanted to put them in the ESXi server. Although my original intention for this box was to have an essentially disk-less system (USB key boot of the hypervisor only), the reality is that I've not got enough bays in the storage server and don't want to waste two perfectly good disks.

I've also got a little project in the back of my mind that could make use of these disks...

I put the disks in the server and booted ESXi. Using the VI client, I noticed that the disks were recognised, but when I went to add storage, I could select the disk

"Error during the configuration of the host: failed to get disk paritition information"

I booted off the CentOS disk and selected "linux rescue" and destroyed the partition table using fdisk. I wrote these changes and confidently rebooted.

I got the same error.

From the ESXi menu, I viewed the configuration logs and messages and noticed it was reporting the following:

Warning: /dev/sda contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should. Perhaps it was corrupted - possibly by a program that doesn't understand GPT partition tables. Or perhaps you deleted the GPT table, and are now using an msdos partition table. Is this a GPT partition table?

I wasn't aware of what a GPT signature is/was, but it was obviously something that fdisk didn't overwrite. Some googling later suggested the problem could be solved by completely overwriting the start of the disk.

Back into the Linux rescue mode and some dd action (sledgehammer approach perhaps...):

# dd if=/dev/zero of=/dev/sda bs=1M count=1

Rebooted again and this time the disk is selectable and I could add the datastore. Repeated for the second disk and now I've got an additional 1TB of storage for VMs (albeit unmirrored, but that's fine in this non-production environment).

For those unfamiliar with dd, it's a fairly low level command that can copy raw data. The if= specifies the input file, the of= specifies the output file. In the above example, /dev/zero is a special Unix "file" that returns zero when read, and /dev/sda is the disk device I'm writing to. The bs= specifies the size of the block (1M = 1 megabyte) and count= specifies the number of blocks to read. So the above reads 1 block of 1MB size from /dev/zero (effectively 1MB of the "0" character) and writes this out to the disk, starting from the very beginning and overwriting everything there (which includes the partition table).

And this is why an understanding of Unix/Linux can be very useful, even if you don't do Unix stuff in your day job... :-)

SAN/NAS upgrade

2009-09-13T14:52:00.000+01:00

All of my important data is stored on my OpenSolaris storage server (an HP ML110 G5). A mirrored pair of 500GB in a ZFS Zpool provided NFS, CIFS and iSCSI sharing. Unfortunately, I ran out of space to the point that ZFS was unable to take snapshots.

I needed to add more storage, but didn't have the drive bays available to do it. So I ordered two 1TB SATA disks with the intention of replacing the two existing disks.

I followed the instructions found at Blog O Matty (a blog I highly recommend). The process was extremely easy:

Remove one of the 500GB disks and replace with a 1TB disk.
Tell ZFS to "resilver" (aka resync the mirror) the new disk (one command: zpool replace datapool c3d0)
Wait a number of hours for the disk to resilver (10 hours when the disks are being used)
Tell ZFS to clear all error status messages (zpool clear datapool). This puts the pool into an "ONLINE" state for all devices in the pool.
Remove the second 500GB and replace with the second 1TB disk
Tell ZFS to resilver onto the second disk
Wait for this second disk to resilver. I did this overnight and it was finished in the morning.
Tell ZFS to clear the error status on the new disk
Check the zpool status (zpool list) and note the new size: Now 928GB

The ML110 does not have hot-swap disks, so I needed to power off each time I swapped the disks, but if you have a hot swap capable server, the entire process can be done live with the filesystems mounted. Nice.

With approximately another 500GB free space, I can now experiment with other hypervisors (XenServer and Hyper-V will probably be the first if I can get them working off a USB bootable key drive). I also took the opportunity to add more memory (8GB total) to both the OpenSolaris server and the VM server (the HP ML115). While I was spending money, I also paid out for another 500GB external USB. This means I can now take a backup of the key filesystems (photos, documents etc) and ship them to an off-site facility (aka, my parents house).

Coincidently, T had filled up her C: drive with a huge number of photos and videos. Although the disk is backed up to an external drive, I wanted to move the data to the server. I created a new filesystem in ZFS:

# zfs create datapool/Users/teresa

This filesystem can grow and consume all space in the pool, so I assigned a quota of 30GB:

# zfs set quota=30G datapool/Users/teresa

In order to make this visible to T's Vista PC, I had to share the filesystem over SMB:

# zfs sharesmb=on datapool/Users/teresa

I made sure that T had a Solaris account setup with a password configured so it could authenticate and then mapped the network drive. The UNC path replaces the slashes in the filesystem with underscores: \\opensolaris\datapool_users_teresa

I copied T's documents to the server by changing the locations of the profile shell folders (right-click "Documents", "Pictures", "Videos" etc and select properties, then specify a new location and the contents are moved across - very easy).

It was then I found even more pictures that needed to move across and the 30GB I had allocated to the filesystem was going to be tight in the long term. This was trival to fix:

# zfs set quota=40G datapool/Users/teresa

The change applied instantly and the network drive size increased to 40GB.

It's good to have all our personal data now stored on the ZFS filesystem, with full mirroring, checksumming and backed up.

The now-redundant 500GB disks will be assigned to another blog post...

Upgrading the EeePC 701 to Eeebuntu

2009-09-11T18:07:00.000+01:00

I don't tend to use my EeePC 701 4G very much; there's not much point when you have a pretty well setup PC and network. But when it comes to going on holiday, the Eee is a must-pack luggage item.

T and I have just been away for a week in Corfu. Weather: hot. Hotel Wi-Fi: not bad and free to use (guess which is the most important criteria... :-))

It was when using the Eee on holiday that I realised how dated the default Xandros-derived distro is. Some websites even encouraged us to to upgrade to a later release of Firefox. So upon returning, I purchased a 2GB RAM upgrade (from the default 512MB), an 8GB SD card to store my files on and a 4GB USB stick with which I installed Eeebuntu.

I've never been a serious Ubuntu user (or any of its derivatives), being quite happy with OpenSUSE, so installing Eeebuntu has been interesting. Fortunately the website had some decent documentation on building an install USB key (since the Eee doesn't have a CD drive). Once that was set up, it was simply a matter of booting the Eee off the USB stick and following the prompts.

The result is a modern, GNOME-based distro that can take advantage of all the Eee functionality including the Wi-Fi and webcam. It's also a very smart-looking setup with Compiz working out of the box. I took the opportunity to add some extra software that might be useful in the future, including Wireshark and Nessus.

I'm not going to pretend that the Eee is going to be my new, main machine, or that it will be heavily used on a daily basis, but it's a very capable little computer that will be far more useful with the updated OS on it. My initial foray into Eeebuntu has also been very positive. If you're looking to get something better than the default, dated Xandros version, it's worth a look.

Passing the CCNA: My experience

2009-08-27T13:46:00.004+01:00

This blog explains why I've been quiet for a few weeks...

I did the CCNA exam back in 2002, but by the time it expired in 2005, I wasn't doing anything specifically with networks so didn't bother re-certifying. A couple of months ago I thought it would be a good cert to pick-up again, so decided to dive in and do some self-study.

The original CCNA was the entry level Cisco exam, but in the last few years this has been replaced by the CCENT. The CCNA is a lot harder than it used to be with many new subjects and a deeper level of understanding required. You can either approach the certification using two tests (ICND1 which gives you the CCENT and ICND2 which results in the CCNA) or by using one combined exam. I opted for the one exam.

I bought the latest version of Todd Lammle's CCNA Study Guide and started studying one chapter per night (there are 14 chapters, but real life meant that it took more than 14 days). I also spent weekends studying as well. The book is generally very good, although I found that because I wasn't replicating the example network used in the book, some sections required me to visualise and absorb what was being shown without any hands on experience. I would highlight the chapter on understanding subnet masks though; probably the best way to learn subnetting I can imagine.

I also purchased the Cisco Press Official Exam Certification Library by Wendell Odom. I planned on using this to get another perspective on the material and started reading bits of this book to clarify areas after I had completed the Lammle book. In comparison with the Lammle book, Odom is a lot more detailed (some might say dry but I enjoyed it). I discovered that, for my learning style, the Odom book helped me more. I found the thorough details showing how something works, step by step, to be very useful.

I managed to borrow some old Cisco kit from work and a laptop from work. This consisted on two 2600 routers, two Catalyst 2900XL switches and a 1700 series router. I didn't have the proper serial cable so couldn't do any WAN activities, but did use the kit to validate my understanding of VLANs, VTP and IOS commands (including the boot process, wiping configs etc).

The final part of my studying involved the CDs provided with the books. The test questions with the Lammle book were pretty straightforward and I was able to get 80%+ in the mock exams without too much trouble.

The Odom book was a completely different matter. These questions are hard. Really hard. I initially found that the time limit in the exam was running out on me, and that I was struggling to get my head around some of the questions at all. Whereas in the original exam, you might be asked a question like:

Given an IP address of 10.2.66.8/18, what is the subnet address, first host, last host and broadcast address?

Now the exam was asking you to look at a network topology diagram with maybe six of these networks and you have to choose a spare subnet range. In other words, you have to do six times the work for single question.

Although I managed to get faster at the questions, and especially enjoyed the simulator questions (where you have to log into simulators of routers and either fix the configuration or use the show commands to identify certain things), I never managed to achieve a pass mark.

I resigned myself to the fact that I wasn't going to pass this exam and instead decided to treat it as an educational experience to work out how difficult it was going to be.

Now the actual exam itself is covered by NDA, so it would be improper of me to comment on the specifics. There were simulations but they weren't too difficult and the majority of the questions were more of the difficulty level found in the Lammle book vs the Odom book.

For those looking at getting the CCNA, I would strongly recommend the two books I used. If you can master the Lammle book, you'll probably do okay. If you can master the Odom book, you'll walk the exam with one arm tied behind your back.

On a personal note, I actually enjoyed the process of learning. I find the networking concepts fascinating, and might even look at doing another cert at some point (we all need a hobby, right?). Not sure T will be too happy about losing me for another month.

At least now I can chillax a bit and enjoy what remains of summer... :-)

Essential Firefox add-ons

2009-08-09T21:42:00.002+01:00

Thought I'd better tidy up my Firefox add-ons as I appear to have accumulated a number that either served a specific purpose and are now redundant (Web Developer) or simply aren't needed (FlagFox).

The list of Add-ons I'm currently using on my Linux workstation are:

Blocksite (essential if you want to block Facebook Beacon)
Evernote Web Clipper
Flashblock (enable Flash objects on a per-site basis)
Gmail Manager (to alert when I get new Gmails)
TwitterFox (how I generally track Twitter)
Wikipedia Lookup Extension (highlight word, right click, search in Wikipedia)
Xmarks (Bookmark sync)

I don't use AdBlock because I'm generally not bothered by the adverts on sites I visit regularly, and I'm not enough of a Firefox hacker to want to bother with Greasemonkey. Perhaps something to manage passwords better would be good though.

Am I missing anything essential?

Another blog worth reading...

2009-08-06T22:35:00.003+01:00

My colleague, "JW", has joined the blogosphere and will be documenting some of the adventures he has in the world of Tech. It's off to a good start with a decent write-up of some of the challenges we experienced when we installed VMware and XenServer on a Sun X6240 blade module.

Check it out at Ting Ting Tech.

Samba, Squid and Active Directory authentication

2009-07-30T09:29:00.003+01:00

This post is the end of a few weeks of challenging debugging.

At $WORK we are implementing a new proxy server based on Squid. Unlike our old proxy, we want to authenticate each user against Active Directory. In order for this to work, Samba (or more specifically, the Winbind component of Samba) needs to be configured.

Getting Samba setup

Consider that Windows networking in the Active Directory world is built on DNS for name resolution, Kerberos for authentication, LDAP for directory services and the SMB protocol for file, print and RPC.

The first step was to configure the proxy server to use the AD Domain Controllers for DNS resolution. This was done by editing /etc/resolv.conf and configuring /etc/nsswitch.conf.

The second step was to get Kerberos working. I've detailed this in another blog posting. This also needs to point to the AD Domain Controllers for the KDC.

Configuring Samba was also fairly straightforward. There was no need to run smbd (used for file and print serving) or nmbd (the naming service) as this box would not be performing those roles. The winbindd server needs to be running. This is responsible for authenticating against the Active Directory.

The smb.conf can be small:

[global]
workgroup = CSS
realm = CSS.AD.EXAMPLE.COM
server string = Squid Proxy and Samba Server
security = ADS
log file = /var/log/samba/%m.log
max log size = 50
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
dns proxy = No
winbind:ignore domains = MAT LPS LAB MMSC GRP IMCR UPGRADE CENTRAL 4THFLOOR AD CSSDEV NAS
hosts allow = 10.1, 127.

To join the domain, run the net ads join command, using the credentials of a domain administrator. You should be able to confirm whether the trust worked by typing:

# wbinfo -t
checking the trust secret via RPC calls succeeded

If that's okay, try pulling in a list of users, again using wbinfo but this time using the -u flag:

# wbinfo -u

And this is where things started to go wrong for me. Sometimes it would work, but most of the time it would error. This took a lot of investigating and the details can be found here in the Samba mailing list archive. It was this bit that took the time to debug.

Getting Squid working

Having got wbinfo to now return the list of users, it was time to configure Squid to use ntlm_auth (which in turn uses winbindd to perform the authentication request). The /etc/squid/squid.conf needs the following:

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 5

At this point everything should work wonderfully...

Except Squid was unable to authenticate, complaining about the permissions on /var/lib/samba/winbindd_privileged. It appears that Winbind expects this directory to have 750 permissions with root:root ownership, while Squid runs as the "squid" user. According to one post I read about the issue, it may be caused by the way that Red Hat have built Squid. One possible workaround is to use an ACL (yes really, a use for ACLs in Unix!) but it appears my install doesn't have ACL support enabled(!).

So the immediate workaround for me is to create a script that basically does the following:

Set permissions on /var/lib/samba/winbindd_privileged to 750
Start Winbind
Set permissions on /var/lib/samba/winbindd_privileged to 777 (yes, I know...)
Start Squid

This appears to work okay until I can find a real solution.

So although this is now working, it's taken longer than anticipated. The thing about Samba, and Active Directory integration is that it's so complicated with so many options. The learning curve is steep, but I'm starting to feel I now have a grip on it.

Configuring Kerberos on CentOS 5

2009-07-17T10:49:00.007+01:00

Kerberos is a ticket-oriented authentication system that was originally designed for Unix networks, but was also embraced (and extended) by Microsoft in Active Directory. I've been debugging a number of issues involving the Squid proxy server on Linux using Samba to authenticate against Active Directory, and as part of this I had to get familiar with Kerberos.

It's not trivial, so I've documented my workflow here. Hopefully it will useful to others.

The test environment consists of two virtual machines running CentOS 5, imaginatively named centos01 (krbserver) and centos02 (krbclient). For the purpose of this test, centos01 is the Kerberos server and centos02 is the client.

I followed the instructions here and broadly recommend them. These are my additional notes to clarify some parts of the install.

General notes

Make sure that you use the same time source for both client and server. I used NTP to keep the two VMs in sync. The notes do state this but it's worth stressing.

Remember how many IT problems are caused by name resolution errors! Make sure you have both the server and client registered in DNS (or have entries in /etc/hosts). If using /etc/hosts both the standalone hostname and the FQDN should be added:

192.168.192.26 krbserver.local.zone krbserver
192.168.192.108 krbclient.local.zone krbserver

Note the order of the hostname and the FQDN! This is important (see further below).

Configure the server

After installing the packages using YUM, configuring the database and ACL file, adding the first principal user and starting the three services, the server should be ready to go. Confirm this with kinit and klist. Now it's time to configure the client.

Configure the client

Install the packages using YUM and then run the kadmin command and add a new principal for the client machine. It's worth noting that this should be done using the kadmin interactive interface instead of trying to put the "addprinc" parameter on the command line. This is because the -randkey option will be interpreted by kadmin on the command line as "-r andkey" and it will try and authenticate against the "andkey" realm. So for me, the command looked like:

# kadmin -p julian/admin@LOCAL.ZONE
Password for julian/admin@LOCAL.ZONE: ********
kadmin: addprinc -randkey host/krbclient.local.zone

I assume that this is rougly analogous to adding a machine to an Active Directory domain.

Once this entry, export the principal to the workstation's /etc/krb5.keytab file.

In addition to the machine principal, I also created a normal (non-admin) a local user, julian@LOCAL.ZONE. On the client, I log in as my own non-root user ("julian") and type kinit:

$ kinit
Password for julian@LOCAL.ZONE: ********

If this succeeds, you should see the "ticket granting ticket" be assigned:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 11:14:20 07/18/09 11:14:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE

Kerberos 4 ticket cache: /tmp/tkt500 klist: You have no tickets cached

This process shows that communication between the client and server using Kerberos is successful.

Configuring telnet (for testing)

On the server, I then enabled the krb5-telnet service in /etc/xinetd.d and started xinetd. On the client, I then ran:

$ /usr/kerberos/bin/telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
Password:

Problem. It was asking for a password which implied the Kerberos ticket was not being passed correctly. However, when I ran klist, it showed that the ticket for the host was passed correctly:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 10:38:20 07/18/09 10:38:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 10:38:31 07/18/09 10:38:20 host/krbserver.local.zone@LOCAL.ZONE

After running strace against the telnetd process, it appeared that the telnet server was failing when trying to read /etc/krb5.keytab. But all the documentation I had read stated that this should be run on the client and not the server. So, why does the Kerberos server need a keytab file?

Answer: The Kerberos server does not require a keytab file, but the telnet server does! Although they are both running on the same VM, the telnet server is itself a client to the Kerberos server. Simple when you work it out it would have semantically been easier to understand if my telnet server been different from the Kerberos server.

So I ran the kadmin command on the server and created a keytab file using the ktadd command. I restarted the Kerberos services for good measure and cleared my client and server caches using kdestroy, restarted xinetd and tried the telnet:

[julian@krbclient bin]$ ./telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 accepts you as ``julian@LOCAL.ZONE'' ]
Last login: Fri Jul 17 10:48:10 from krbclient
[julian@krbserver ~]$

Result!

Configuring SSH

The instructions state that GSSAPIAuthentication and GSSAPIDelegateCredentials need to be enabled. I did this and restarted the SSH daemon with -ddd (debug) enabled.

The first attempt at running ssh krbserver prompted for a password, but the server debug revealed the following:

debug1: Unspecified GSS failure. Minor code may provide more information
No principal in keytab matches desired name

Okay, so this is weird. Checking the output of klist showed this:

[julian@krbclient ~]$ ssh krbserver
julian@krbserver's password:
Connection closed by 192.168.192.26
[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Note that krbserver@ has no realm. This turned out to be because /etc/hosts (on the client) looks like this:

192.168.192.108 krbclient krbclient.local.zone
192.168.192.26 krbserver krbserver.local.zone

Putting the hostname after the FQDN like this:

192.168.192.108 krbclient.local.zone krbclient
192.168.192.26 krbserver.local.zone krbserver

fixes the problem!

[julian@krbclient ~]$ ssh krbserver
Last login: Fri Jul 17 13:54:40 2009 from krbclient.local.zone

Klist now shows:

[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@
07/17/09 13:54:38 07/18/09 13:42:27 host/krbserver.local.zone@LOCAL.ZONE

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Summary

What you see above does not include the time spent trying things out and staring blankly at the screen. Getting Kerberos up and running is not the most trivial process and while there is some decent documentation, there are also a lot of people posting questions and asking for help when it doesn't work properly. Hopefully this will shed some light on it for others.

The future is... iSCSI?

2009-07-12T12:18:00.000+01:00

I've recently completed the business planning infrastructure requirements for 2010 at $WORK. As part of this I have specified a new fibre channel switch and some additional fibre attached storage.

Despite this, I'm starting to suspect that the future of SAN connectivity will be iSCSI over copper Ethernet.

Ethernet and IP technologies have basically beaten everything else out there and now dominate computer networks and IP telephony. Why have a different standard for storage networks? Consolidation of the fabric for LAN, SAN and VOIP seems logical to me. Share the components and reduce the total cost.

So why continue to specify fibre channel? We already have a large investment in fibre channel, it's a known quantity and is well supported. Sun Solaris has a very mature FC implementation, and as of VI3, VMware works best on FC (not sure yet if vSphere changes this).

It's also faster (for us). We currently have 2Gbit switches but will be adding 4Gbit switches later this year and early next. Sure, 10Gbit Ethernet is available, but it's still too expensive for us to deploy (especially when adding the cost of switches and NICs).

But fast forward three years and I would expect the following:

10Gbit Ethernet switches at a reasonable price with 40Gbit or 100Gbit inter-switch links
10Gbit NICs with TCP Offload Engine (TOE) as standard and cheap
iSCSI boot as standard on these 10Gbit NICs (some do already, but it's not guaranteed)
Better support in the hypervisor / operating system for iSCSI

At the end of the day, managing a single fabric is easier than juggling a bundle of different cable types, protocols, HBAs and drivers.

It's always risky in this business to speculate how things might look in 3 years. If you disagree, please let me know why; it's always good to get alternative views...

Sun StorageTek 2540 and ESX troubleshooting

2009-07-10T11:22:00.001+01:00

We experienced a few issues with the StorageTek 2540 array that forms the core of our SAN recently. The symptom was that the array flagged itself as being in a degraded state and that one or more volumes were not assigned to the preferred controller.

The first step was to upgrade the SAN firmware and Common Array Manager (CAM) software to the latest release. Despite this, we observed the problem again. Further digging into the problem found that the failover was happening when we performed a LUN rescan under VMware ESX.

My previous understanding was that there were essentially two types of arrays: active/active and active/passive. In the active/active configuration, both controllers in an array can service I/O requests to a specific volume concurrently. In an active/passive configuration, one [active] controller handles the I/O with the second [passive] controller sitting idle, only servicing I/O if the active controller fails.

I understood the StorageTek 2540 to be an active/passive array; it is only possible to assign a volume to one controller at any time. However, in order to improve the throughput of the array, different volumes can be assigned to different controllers. For example, a volume “VOL1” might be assigned to controller A as its active controller and to controller B for its passive controller, while volume “VOL2” might be assigned to controller B as its active controller and controller A as its passive controller.

It turns out that things are more subtle than this; there is a third type of array configuration: asymmetric.

The asymmetric configuration follows the active/passive model in that only one controller is servicing I/O for a specific volume at any time, but extends this by allowing I/O operations to be received by the second controller. If this happens, the array will automatically failover the volume to the second controller to service the request. This process is called Automatic Volume Transfer (AVT). If the first controller then receives I/O operations, the AVT moves the volume back.

Yes, this could cause some flapping between controllers. It can also cause I/O stalls as the controllers fail across.

Some of the array initiator types (such as Solaris with Traffic Manager (aka MPxIO)) disable AVT, others, including the Linux initiator that we’ve used on our VMware hosts, have AVT enabled.

So the problem we’re having appears to be caused by the array failing over a volume to its second controller. But why is it doing this? The only configuration I had performed on the ESX side was to ensure the multi-pathing option was set to Most Recently Used (MRU); the correct setting for active/passive arrays. What appears to have happened is that when booting, the ESX servers are not mapping to a consistent path. Out of our five ESX servers, three were setting one controller as active, while the other two servers were setting the second controller as active. Presumably, when one of the hosts (that has the wrong active path) performs a scan, the request is sent to the failover controller which invokes AVT and fails over the volume.

How to fix?

Sun have told me that the next version of CAM, due in a few weeks, will include a “VMware” initiator type which will disable AVT. This will negate the need to perform the NVSRAM hack in VMware’s Knowledge Base, but will require a firmware upgrade.

In the meantime, it might be a case of just ensuring that all the ESX hosts are using the same path to connect to each volume. This is all theory as I’m still working this out, but at least it’s all starting to make sense.

Although not specifically VMware or 2540 related, the following links provide some interesting reading around the subject:

Sun discussion forum thread about preferred and owner controllers

Linux kernel mailing list post detailing a bug experienced with multipath and asymmetric arrays

Note-taking on the Cloud

2009-07-04T22:35:00.003+01:00

In the "old" days, things were pretty simple; the Palm handled contacts, calendar, tasks and notes well, allowing me to carry everything with me but still sync them with my PC.

Of course, things are more advanced these days: Multiple computers, at home and at work, the iPhone providing near continuous Internet connectivity, web based services and richer software applications. But despite this, I'm still struggling to get perfect syncing across all platforms. Here's where things are for me today:

Contacts: Google Contacts synced with the iPhone, but no Outlook/Exchange integration.
Calendar: Google Calendar synced with the iPhone, but primary calendar only.
Tasks: Still using Outlook/Exchange for this. No sync.
Notes: Some notes in Outlook, some notes in OneNote, a few notes on the iPhone. Nothing syncs.

The last one is particularly disappointing as note synchronisation shouldn't be difficult. I tried using Google Notebook for a while until Google got bored and dropped it.

It was then I tried Evernote. There is a free version, provided you don't exceed a certain number of notes per month and there are clients for Windows, Mac OS X and the iPhone. There's also a web interface for when I'm in Linux or on a public machine.

My preference in terms of note taking functionality and power is OneNote. The only downside to this application is that notes are basically locked to the client PC, or synced to the corporate SharePoint server at best.

In comparison with OneNote, Evernote has fewer features, and an interface that is less rich. The Mac and Windows versions both have different levels of functionality (the Mac has a nicer set of views IMHO). But in it's favour, any notes I make, on any of my devices, now sync with the cloud.

Evernote is therefore my de facto notes application. For now.

Upgrading to ESXi 4.0

2009-06-29T16:42:00.003+01:00

Although VMware vSphere came out a couple of months ago, I haven't had the time to try it out yet. Today I took the opportunity to try and upgrade my ESXi 3.5 server to ESXi 4.0. This won't allow me to test out the new VMware features such as Fault Tolerence, but I'll be able to run the latest vSphere client and see how the interface has changed.

I was a bit reluctant to do this because installing ESXi 3.5 on the USB key drive inside my ML115 G5, although not complicated, consisted of a number of steps and manual copying of files (not to mention having to crawl under the desk and physically install the USB stick).

Well the good news is that ESXi 4.0 is a much easier story. The operating system can be installed from CD straight onto the USB key drive, so no need to pull the side off the case. The install is quick and takes about 5 minutes. Then a reboot and to the DOS-like menu that allows for the networking, password etc to be configured.

A download of the vSphere 4 client and registering the licence key (free for ESXi) and everything is ready to go. I re-added the iSCSI and NFS datastores from my OpenSolaris server and fired up the VMs.

I haven't done anything beyond this yet, but for an upgrade I assumed would be a hassle, ESXi 4.0 has been very straightforward.

Upgrading to OpenSolaris 2009.06

2009-06-20T13:23:00.003+01:00

I've been absent from this blog for a while because we've been moving house. The move is now complete and broadband installed in the new house so I'm back online and ready to continue blogging. Today, upgrading OpenSolaris.

My storage server, the ML110 G5 has been running OpenSolaris 2008.11. With the release of OpenSolaris 2009.06, I wanted to upgrade and wondered if I should have left a spare slice of disk so I could perform a live upgrade. With the rest of the network dependent on this server for services (DNS, iSCSI LUNs, file and print serving), a failed upgrade didn't appeal.

It turns out that the the upgrade was very easy and a spare slice was not required:

Step one: Upgrade the Package Manager:

$ pfexec pkg install SUNWipkg

Step two: Run the Package Manager and select "Update All".

Wait while the new package requirements are evaluated, downloaded and installed.

Step three: Reboot

That's it. Checking /etc/release now reports:

OpenSolaris 2009.06 snv_111b X86
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 07 May 2009

Although I'm not an expert in IPS, the OpenSolaris packaging system, my understanding is that the previous version of the operating system is still on the system in the form of a ZFS snapshot, so if I have any problems, I should be able to rollback without too many problems.

Compared with upgrading other operating systems, OpenSolaris has been an absolute dream. Highly recommended.

Oracle to buy Sun

2009-04-21T18:24:00.002+01:00

This blog is typically about stuff I'm doing, but having invested 10 years in working with Sun kit, I think this deserves a comment. I'm not particularly qualified to make an informed contribution to the Oracle buyout discussion, but since that doesn't stop most Slashdot commentators, here are my observations:

1) It could have been a lot worse. If IBM had bought Oracle, they would have two directly competing CPU architectures (POWER and SPARC), two directly competing Unix implementations (AIX and Solaris), directly competing storage, Java development tools, application servers, databases etc. The long term outcome would have been that some of these technologies would have been sunset and development consolidated.

2) It could have been much, much worse: HP could have bought Sun. See DEC/Digital and the Compaq merger to see how that would have played out.

3) Sun seems to get a bad rap from the Slashdot crowd, but the number of people who are dismayed that their favourite Sun product is under threat highlights the significance that Sun has in the market. This is especially telling when you see that the object of dismay is one of many different products: Solaris, Java, OpenOffice, Virtualbox, MySQL...

4) Oracle have now migrated from being a software solutions house to a total solutions provider. Although there is talk that Oracle will sell off SPARC to Fujitsu, or port everything to x86, the reality is that SPARC is a very lucrative platform for Sun and will be a revenue generator for Oracle. In fact, Oracle now own a very good portfolio.

5) In the last 12-18 months, Sun has "got" Open Source. I'm not sure Oracle has. Hopefully the Sun culture will impact Oracle (remember how NeXT "absorbed" Apple after being bought?).

So although I'm no expert in the area of business, the future for Sun might be okay after all. Time will tell...

Getting CIFS permissions working in OpenSolaris

2009-04-11T21:37:00.002+01:00

I blogged a while back about how I managed to get NFS4 configured on my OpenSolaris server. At the end of that blog, I promised to update on getting CIFS working as well using the OpenSolaris CIFS server.

Finally got around to it tonight. Took a while as every attempt from my Vista machine was reporting an "Access Denied" message when running "net view \\opensolaris".

To cut a long story short, the problem was caused by me setting up a mapping between Unix and Windows using the idmap command. The answer was to remove any mappings and allow the CIFS server to work it out itself. All sorted!

Upgrading to OpenSUSE 11.1

2009-04-11T10:30:00.002+01:00

Although it's been out for a number of months (released in December), I've just taken the step of upgrading my workstation to OpenSUSE 11.1. In order to perform a clean upgrade, I decided a reinstall from scratch would be easiest, especially since my services and custom scripts are now running on the OpenSolaris server.

I finally took this opportunity to move all my documents to the server. They now benefit from having regular ZFS snapshots and get backed up to the external USB drive for added security.

Once my home area was clean and having backed up my ~/.mozilla directory, I proceeded to install from the OpenSUSE 11.1 DVD download. I had previously used the 64bit version and hit a few problems with Java and Flash, so this time I opted for a 32bit install (only got 2GB RAM so not a huge hit). All worked as expected, although the default partitioning seems a bit tight - something I've had to rectify this morning using LVM. If installing, it would be worth creating a bigger root filesystem (15GB?) (assuming you're not using multiple partitions for /usr, /var etc.)

The only thing that is bugging me slightly is the beagled indexing daemon causes the CPU fans to spin up when the machine is idle. I need to see if this is because it's doing an initial index build or if this will be permanent, in which case I might turn beagle off.

I was surprised at how easily the Compiz "wobbly windows" was to enable, especially when I remember how difficult and flakey it used to be. I have however turned it off because it's faster without the effects.

All in all, a good distribution upgrade with no major issues.

Copying symbolic links

2009-03-11T10:39:00.003Z

Sometimes it's useful to copy a directory that contains symlinks. The default cp behaviour is to follow the symlink and copy the file the symlink points to. If you want to preserve the symlinks as symlinks, then the flags the cp command uses depends on the operating system:

Solaris 10 and Linux:

$ cp -rP dir1/* dir2

The -r is recursive (in case dir1 has subdirectories) and the -P preserves the symlink.

AIX 5.2:

$ cp -rh dir1/* dir2

The -r is the same as Solaris, the -h preserves the symlink.

Does XenServer have a future?

2009-03-04T12:44:00.002Z

Citrix has made their hypervisor based virtualisation product, XenServer, available for free. There has been a free "Express" version out for a while, but this was limited in the number of VMs it could support and came with a limited subset of functionality (similar to ESXi).

The new release makes most of the Advanced functionality available for free, including multi-server management and live migration between hosts. This competes with VMware very well, to the point that a virtualised Citrix (XenApp) solution I am working on will most probably run on the free XenServer and not on the pay-for ESX.

So why do I wonder whether XenServer has a future?

Citrix are looking to get customers hooked on XenServer with the aim of upselling a management suite called "Citrix Essentials for XenServer". The thing that concerns me is they also have a product called "Citrix Essentials for Hyper-V".

On the one hand this makes sense. By providing the same management tools for both XenServer and Hyper-V, Citrix are trying to make the underlying hypervisor a commodity item. The real advantage is in the management layer.

But development of XenServer is not free, and Citrix get no money directly from it. So why bother continuing development in the long term, when they can "superset" on top of Hyper-V, a hypervisor that Citrix don't have to spend any development funds on.

I hope I'm wrong, because XenServer looks interesting, but it wouldn't surprise me to see a future announcement where Citrix drop XenServer and adopt Hyper-V as their favoured hypervisor.

WSUS under Server 2008

2009-02-27T08:40:00.003Z

I configured a WSUS server under Server 2008 and created a GPO to instruct all the other Windows VMs in my test environment to point to it. Although the GPO was being pushed out, the WSUS console did not detect any computers.

After speaking to some colleagues about this (thanks Rob!), I tried manually running a Windows Update from one of the clients. This produced the following error: 80072EE6

Googling this suggested the problem was an "unknown protocol" and one post suggested checking that the GPO settings for the intranet server included the "http://" prefix before the hostname. I checked and found I didn't have that setting. Adding it in and then forcing a gpupdate worked and the clients have started to appear.

Joining an OpenSolaris CIFS server to an AD domain

2009-02-08T21:18:00.002Z

These notes are rough, but might prove useful in the future (or to someone else who had the same problem). I wanted to join my newly created OpenSolaris 2008.11 installation to my experimental Windows Server 2008 Active Directory. I followed the tutorial in the Solaris CIFS Administration Guide, but when I attempted the actual join command, it failed with LOGIN_FAILURE.

Here's the answer:

Firstly, make sure you have the IP address setup correctly, name added in /etc/hosts, DNS setup correctly in /etc/resolv.conf and /etc/nsswitch.conf. You make sure you are on the same domain as the Active Directory ("windows.zone" in my case).

Secondly, make sure your system clock is synchronized with the domain controller:

# ntpdate dc01.windows.zone

Then setup Kerberos by editing /etc/krb5/krb5.conf. This is documented in the manual.

Install the SMB Server using the package manager and start it up using:

# svcadm enable -r smb/server

Then try to join the domain:

# smbadm join -u administrator windows.zone

For me, this is where it failed. The mailing lists suggest that the problem might be related to smb signing. On the DC, I opened up the Group Policy Management tool and changed the following:

Computer Configuration\Policies\Administrative Templates\System\Net
Logon\Allow Cryptography Algorithms Compatible with Windows NT 4.0 -> Enabled

I then ran a gpupdate /force.

Finally, I read that the sharectl command on the OpenSolaris server should be run to use NTLMv2 authentication:

# sharectl set -p lmauth_level=2 smb

I re-ran the join command, and it worked properly. OpenSolaris CIFS server now part of the Active Directory domain.

Installing OpenSolaris into VMware ESXi

2009-02-08T19:06:00.003Z

I've just created a very small VM for installing OpenSolaris onto my ESXi box. The ISO CD booted in live mode to the desktop and I selected the option to install to disk.

The installer loaded, but at the point of "Finding Disks", the installer seemed to hang and no disk was found, even though format was displaying the disk okay.

Upon further investigation, I realised that the amount of RAM I had assigned to the VM (512MB) was causing a problem. By increasing this to 1GB, the disk was found and the installer worked as expected.

My guess is that this is due to Solaris deciding that with only 512MB of RAM, that some swap would be necessary, but it's not able to create it. Only a guess, but at least I now have a workaround.

Getting NFS4 permissions working correctly

2009-01-31T18:13:00.002Z

Following the installation of the OpenSolaris server, I've had some problems with the NFS mounts. I've created an export for a fileserver (datapool/filestore) and although I have setup identical UID/GID maps between my clients and servers, when I mounted the filesystem on my Linux server, I found that files created were owned by nobody:nobody.

At first I thought this was due to some configuration problem in the OpenSolaris installation, but after trying it with the Mac as well, I realised that the mapping was fine. This then pointed to the OpenSUSE 11.1 installation.

The problem turned out to be the Domain setting in /etc/idmapd.conf. The value in this file was different from the OpenSolaris NFS domain. Changing that and restarting the idmapd process (which I did by rebooting the server as I had a kernel update to do), fixed the problem and I now map correctly.

The next step is getting a Samba and Windows client to authenticate me correctly with the OpenSolaris CIFS server. That might be more difficult, but I'll update here when it's done...

Building a test lab

2009-01-15T12:24:00.003Z

Thanks to the Technet subscription that work has provided for me, I'm now in a position to build my own test Windows network. The purpose of this is to help me get a grip on Windows Server 2008, some Active Directory, Terminal Services etc., and potentially some other non-MS tech such as Citrix XenDesktop [Express].

I've been thinking through the planning of this test lab and recognise that I need to create a network. I can either create a new, completely virtual network and put my VMs on it, routing this to my physical network using a dual homed VM appliance, or I can create the test network in a different address range as my "live production" network and assign the IP addresses so that they don't overlap.

The latter seems the easiest way of doing it (although I may be proven wrong when it's built!).

So, assume my local network is 192.168.0.0/24 (it's not, but I'm not stupid enough to put my real subnet on the 'net!). I'm going to slice up the subnet as follows:

192.168.0.1 - 192.168.0.99 = static range for production network
192.168.0.100 - 192.168.0.150 = DHCP range for production network
192.168.0.151 - 192.168.0.200 = static range for test lab network
192.168.0.201 - 192.168.0.254 = DHCP range for test lab network

How do I determine whether a plugged in device gets a production or test DHCP address? Ultimately it will depend on which DHCP server responds, but the reality is that it shouldn't really matter. Both servers will allocate an address that is routable to the Internet and will resolve the DNS. For anything that will be permanent, I'll allocate a static IP anyway.

My production network has the DNS suffix of local.zone, and I contemplated creating the Active Directory as a sub-domain (windows.local.zone). I think that it will be easier though if I simply create a new domain (e.g., windows.zone) and manually create a DNS forwarder to local.zone when appropriate. This keeps the production network (primarily non-Windows based Solaris, Linux and Mac OS X with a non-domained Vista) from interfering, or depending on, the test lab.

If either of my readers(!) spots anything obviously wrong here, please let me know!

VMware Certified Professional

2009-01-09T17:12:00.004Z

After a week of revision, plus a couple of years worth of hands on experience and the VMware Fast Track course, I took the VCP exam this morning. Pass mark is 70 and I managed to get 86 which was fine, especially as I remember the struggle that was the SCSA upgrade.

But the IT world does not stay still, and in these days of economic uncertainty, the market is only going to get more competitive, so with the VCP now under the belt, it's time to turn to the next cert... CCNA refresh? Something Citrix? Red Hat? Microsoft? Hmm.

An OpenSUSE quickie

2009-01-08T17:58:00.003Z

The command line tool for patch and package management on OpenSUSE is "zypper". I've used zypper to list patch updates using:

# zypper lu

The patches can be added (updated) using:

# zypper up

Because I never got around to reading the man page, I didn't realise that both the above commands have an implicit "-t patch". I also didn't realise that "-t package" applied to the above commands can be used to display and update packages to a later version.

# zypper lu -t package
# zypper up -t package

Currently installing 83 package updates...

Backing up ZFS to an external USB drive

2009-01-02T18:00:00.004Z

Having a resilient, snapshot managed storage server is all very well, but what happens if your server catches fire? While ZFS is very capable of preventing data loss, and the RAID capabilities compensate for a physical disk failure, the lack of a ufsdump/ufsrestore was a bit troubling.

I'm not claiming to have found the perfect solution, but a bit of playing around today with an external USB disk looks promising. I plugged the USB drive in and OpenSolaris automatically detected it. Running the format command showed it was mapped to c6t0d0.

I partitioned the disk to create a single 500GB(ish) slice 0 and created a traditional UFS filesystem on it. I'm sure I could have used ZFS, but wanted the simplicity of a single filesystem without worrying about pools or other volume manager artifacts.

After creating the filesystem with newfs, I mounted it to /mnt.

# newfs -m5 /dev/rdsk/c6t0d0s0
# mount /dev/dsk/c6t0d0s0 /mnt

I had a test filesystem created (datapool/testfs) and copied a file into it. I then took a snapshot of the filesystem:

# zfs snapshot datapool/testfs@mytest

I backed up the snapshot using the ZFS send syntax:

# zfs send datapool/testfs@mytest > /mnt/testfs.backup

This created a single file (/mnt/testfs.backup) containing the filesystem.

With that completed, I deleted the file I copied across. Now for the restore. This was very easy:

# zfs recv datapool/testfs.recover < /mnt/testfs.backup

A new filesystem was created and mounted in /datapool/testfs.recover, containing the file I wanted to recover which I could then copy back. To test a bit further, I destroyed the original datapool/testfs filesystem and all snapshots. I then did another zfs recv and specified the original filesystem name:

# zfs recv datapool/testfs < /mnt/testfs.backup

And it all came back perfectly!

Obviously this is a simple test and doesn't deal with incrementals etc, but should be sufficient for a keeping a copy of the data on an external disk that can be stored off site. Although I haven't tried, adding encryption to the zfs send pipeline should be very simple to do.

ZFS just gets better and better!

Building a SAN with OpenSolaris and ZFS

2008-12-30T13:26:00.003Z

The one line summary: OpenSolaris and ZFS is very, very cool.

Having created a ZFS pool "datapool" consisting of two 500GB disks in a RAID1 mirror, I then created a few filesystems, including datapool/filestore which was to be the new fileserver. The whole process took two commands:

# zpool create datapool mirror c3d1 c4d1
# zfs create datapool/filestore

To make the new filesystem available over NFS to my Linux machine took another command:

# zfs set sharenfs=rw,anon=0 datapool/filestore

Okay, but what about the Windows box that doesn't support NFS natively? For this, I had to install the SMB server package (along with the kernel extension) and reboot the server. But after that it was simply a case of one more command:

# zfs set sharesmb=on datapool/filestore

The Mac has a very nifty backup tool called Time Machine. To use it you must directly attach a second hard disk, or tweak the configuration to allow for "unsupported" devices to work. I wanted to add in another hard disk to allow me to run Time Machine. Again, OpenSolaris/ZFS to the rescue.

I installed the iSCSI target server software (a couple of ticks in the package manager) and then ran the following commands:

# zfs create -s -V 100GB datapool/mac_backup
# zfs set shareiscsi=on datapool/mac_backup
# iscsitadm list target

The first command creates a "zvol" - a block device that is built from the zpool but does not have ZFS on it. The -s creates a sparse volume so that the space is not-preallocated. The second command makes the volume available over iSCSI, while the third command lists the available iSCSI targets so you can easily get the iSCSI address.

I then downloaded and installed the free globalSAN software for the Mac which provides an iSCSI initiator. Five minutes later (including a reboot of the Mac because it's another kernel extension), I had a new block device ready for partitioning in the Disk Utility. I created a Mac HFS journaled filesystem which Time Machine is able to use.

So one server can now provide simulateous NFS, CIFS/SMB and iSCSI to all my servers. It really is a small SAN at home!

OpenSolaris, Courier IMAP and FAM

2008-12-24T20:40:00.002Z

I started the final part of migrating to the new OpenSolaris server today: Moving the internal IMAP server.

This server exists to collect email that is sent to my old address, T's old Hotmail emails (slurped down using a Thunderbird webmail extension and imported into IMAP), as well as act as an archive of all my old email dating back a number of years.

The new mail server will be a Solaris zone called "mailserver" on the OpenSolaris host. I built the zone and installed Fetchmail (to collect mail from my ISP using POP3), Procmail (to filter the email into the correct mailboxes) and Courier IMAP (to serve the email internally over IMAP).

Once I got Courier IMAP installed, and the config file copied across, I hit a problem:

Dec 24 19:19:32 mailserver imapd: [ID 702911 mail.error] Error: I/O error
Dec 24 19:19:32 mailserver imapd: [ID 702911 mail.error] Check for proper operation and configuration
Dec 24 19:19:32 mailserver imapd: [ID 702911 mail.error] of the File Access Monitor daemon (famd).

Hmm, so I need to install FAM. I did the install, but this still didn't work. Turns out that the FAM package doesn't make the necessary additions to /etc/rpc (FAM is controlled by rpcbind) and /etc/inetd.conf.

For reference, this is what needs to go in /etc/rpc:

sgi_fam 391002

And this is what needs to go in /etc/inetd.conf:

sgi_fam/1-2 stream rpc/tcp wait root /opt/csw/bin/fam fam

To register the inetd config, the inetconv command needs to be run at which point, starting Courier IMAP works and the above error has gone.

Job done!

New kit, new project, new stuff to learn

2008-12-23T23:10:00.003Z

Although I bought the ML110 G5 a few months ago, I have only been using it as a "play" machine. Earlier this month, Ebuyer got the ML115 G5 back in stock, so I took the opportunity to snap up one of those, along with 2 x 500GB disks, a 4GB RAM upgrade, 8 port Gigabit switch and a 4GB USB key drive.

You see, this is the plan:

The ML110 G5 will be the "production" server, running OpenSolaris 2008.11and playing the role of a storage server along with some core infrastructure services. I took the 250GB disk out of the ML115 and put it in the ML110 and mirrored the existing 250GB disk. I also added the 2 500GB disks in a mirrored pool. All of this is using ZFS for resilience and only takes a handful of commands to setup.

The OpenSolaris server is now running my print server software (CUPS) and works with the Linux machines, the Mac and even T's Vista PC. I'm running a small BIND DNS server for keeping track of the internal machines, and have a number of NFS shares setup for file serving, an ISO store and (in the near future), a VM datastore. The beauty of ZFS is that in addition to serving filesystems using NFS (and CIFS using the new kernel based service), other filesystems can use the ZFS technology thanks to zvols (essentially block devices in a zpool that can be shared using iSCSI and FC).

The 4GB USB key drive has VMware ESXi installed on it and the ML115 boots this. The USB port is internal, on the motherboard, so everything is nicely integrated. The actual VMs will be on the storage server - initially using NFS, but potentially using iSCSI in the next OpenSolaris release as well. Sounds like a SAN at home? Yep! :-) (this is why I've upgraded the switch to gigabit).

One of the things I'm finding with OpenSolaris is that although I've been using Solaris for years, there are a lot of changes (that presumably will make it into Solaris 11). The biggest one I've hit so far is the Image Packaging System (IPS). This appears to use a network repository for installing software (so no local .pkg files) and creates some new differences with Zones (they are now branded as "ipkg" and I think I've lost the ability to create sparse root zones in this release).

Still a lot to do and play with, but OpenSolaris certainly seems very feature-rich, and nothing else currently provides stuff like ZFS.

I might even have a look at getting xVM installed as well, but that might take a bit longer...

Vista PDF printing problem

2008-12-02T22:13:00.002Z

T has a problem printing PDF files from her Vista PC. The print server is a Solaris VM running CUPS and all other documents seem to print without any issues. It appears that selecting Print from the Adobe Acrobat Reader application doesn't send anything to the print queue.

The reason I'm blogging about this is the result of T noticing that she never gets a mention and therefore "Why don't you blog about it and see if anyone knows the answer".

What is this? A Helpdesk? :-)

iMovie 08 - the good and the bad

2008-11-25T21:02:00.003Z

I'm working on a 5 week project for church where we are recording a 25-35 minute teaching sermon per week for the purpose of creating a DVD resource. We have three cameras that are mixed by a Datavideo SE-500 video mixer. This outputs to a DVD-RAM recorder with a separate hard disk recorder for backup.

As we shoot and mix the whole thing live, I've been editing minor glitches in iMovie 08. This is the workflow:

1) Insert DVD-RAM disc in the Mac Mini and copy the VR_MOVIE.VRO file onto the disk.
2) Copy this across to my Linux workstation and re-encode it into DV using ffmpeg-kino.
3) Import the resulting (large) DV file into iMovie.

This works very well and I've been able to edit out the odd glitch very quickly. A minor error (where the speaker then repeats the offending line) can be tidied up in about 2 minutes. It's been taking me no more than 15 minutes to edit the source video. This is the good.

The bad occurs when I want to insert graphic shots from PowerPoint over the video (without interrupting the audio). To do this, import the graphics into iPhoto and then in iMovie, drag and drop the graphic over the video and set the duration. iMovie will then create a smooth cross fade into the graphic and back while the audio continues.

Very smooth.

Until you want to put more than about three graphics into the movie in this way. At this point, the live preview stops working (so you can't skim over the video and see the transition render). It gets worse when I export the "completed" video file; the inserted graphics are rendered in and fade correctly, but the order of the graphics is messed up.

I've tried to split the source video into smaller clips but this has not helped. I've logged it as a bug with Apple, but iMovie is too buggy to use for anything beyond basic clip editing in its current state. Shame really as when it works, it works very well. Apple - please fix!

Google Calendar Sync not synching... the fix!

2008-10-05T21:38:00.003+01:00

A while back I noticed that Google Calendar Sync was not correctly synchronising all my calendar entries. While some were okay, others refused to appear. Having spent a couple of hours last night working on it, I have found what I believe to the be the solution.

The problem was either me being too clever, or Google Calendar Sync not being clever enough.

Back when I started using Outlook as my main calendar, I wanted to prevent my work colleagues from viewing my personal appointments. The easiest way to do this was by setting all appointments to be private by default, and then change them if it was a business item. This involved editing the appointment form in the Outlook designer and saving it as a new form (that I called "Private Appointment".

Unfortunately, Google Calendar Sync only tries to synchronise calendar items that are Meetings or Appointments. My custom form was not being synchronised. I've now changed the default form back, but will need to find a way to convert all my private appointments into "normal" appointments.

Once I knew what the problem was, I was able to Google and find that numerous people have had similar problems when they've installed an Outlook add-on such as WebEx or integration with Live Maps.

Now I know this, hopefully I'll be able to get my calendar working properly again and join it up with the rest of my cloud activities.

Moving VMs from local disk to shared storage

2008-09-30T09:39:00.003+01:00

We've had VMware Infrastructure 3 for over a year, but our usage of it has been "organic" (read: unplanned and unstructured). Yesterday I started making moves to rectify that.

One of the things we didn't get right at the start was setting up the SAN to store all VMs. This resulted in a number of VMs crammed onto internal storage. Some VMs were loaded on SAN LUNs, but we didn't have the LUN mappings setup to allow multiple ESX servers to see them.

As I said, it was an organic install.

So yesterday I created a new 250GB LUN on the SAN and mapped it to our "infrastructure" ESX servers (three HP DL360 G4 servers - not super fast, but sufficient for domain controllers etc.). For the first time, multiple ESX servers could see the same storage.

My first action was to create a small Linux server on the shared storage and test Vmotion. This worked fine, so I started copying the VMs from internal storage to the shared LUN. To do this, I shutdown the VM, browsed to the VM folder in the datastore browser, highlighted it and selected "Move to..." from the context sensitive menu. I then selected the new SAN LUN and started the move.

Once moved, the VMX file needs to have the execute bit set (using chmod from the ESX service console), and I then removed the VM from the Virtual Center inventory. I then re-added the VM by right-clicking the VMX file in the datastore browser and selecting "Add to inventory".

Upon starting the VM, a question needs to be answered (the icon for the VM changes to a speech bubble - right click and select Answer Question). Because the VM has been moved, elect to "Keep" the VM id. The VM will then fire up properly.

Or it did in most cases. One VM gave an error that it was "Unable to lock file". A quick Google suggested a LCK file, but I didn't have one of those. To resolve this problem, I edited the VM settings and removed the hard disk (from the VM, NOT the disk file!). I then re-added the same disk, and the VM started without problems.

Migration to Google Mail

2008-09-27T22:20:00.003+01:00

Although I've had a Google Mail account for a few years, I've really only used it as a test account, or when I've not had access to my personal email. Instead, my main (church) email address has been accessed by Thunderbird on my local machine. Obviously this is not ideal from a cloud computing perspective.

So tonight I took the step of migrating all my email from Thunderbird to Google. I did this using the IMAP interface to Gmail and dragging and dropping messages up to the cloud.

In order to ensure that all new messages go to Google, I configured Gmail to download my email using POP. I've also got the webmaster account on a Google Apps domain for a local charity, which I've now changed to forward to Gmail.

The net result was an inbox with > 1700 messages and taking 325MB (out of a quota of 7178MB).

The next step was to use the Gmail filter capabilities to add tags for my emails based on destination email address. Once this was done, I spent a couple of minutes archiving all email older than 1 month.

Now I have a tidy inbox, accessible from everywhere I can use a browser, collecting email from three different addresses and automatically tagging incoming messages.

VMware ESXi on an HP ML110 G5

2008-09-06T15:49:00.003+01:00

I recently bought an HP ML110 G5 from Ebuyer because it was going very cheap (£220 for a dual core Xeon, 1GB RAM and 250GB hard disk). This box will be my new server, possibly for trying things out with, but maybe as a replacement for my Shuttle that currently runs VMware Server on OpenSUSE.

Unfortunately, the first attempt at installing ESXi failed with the message that no storage devices could be found. After some searching, I found that by changing the disk settings in the BIOS from "Auto" to "SATA" fixed the problem and the disk was found (full details here.

The installer again failed with the following error: "Unable to write image to the selected disk. This maybe caused by bad sectors on the device or another hardware problem."

One comment I read suggested the amount of memory could be a problem. As this is to be a VM server, I ordered another 4GB RAM (CT810056 4GB kit (2GBx2), 240-pin DIMM Upgrade for a HP - Compaq ProLiant ML110 G5 System) from Crucial and fitted it this morning.

ESXi installed without any problems. Now to install some VMs...