OpenSolaris: Very slow boot times

Today I lost power to my servers due to a power outage. The UPS wasn't up to coping and almost instantly died (I was running five computers on the one, small UPS...).

Booting the OpenSolaris server back up reminded me of the painfully slow boot times that can occur. We're talking *hours* to get the server up.

The reason for this is due to the number of ZFS snapshots on a system. Here's the experiment:

I booted off the OpenSolaris 2009.06 CD and ran the format command. This displayed the disks on the system. I then imported the zpools into the running installation:

# zpool import rpool -f
# zpool import datapool -f

The -f is required because the system thinks the zpools have been assigned to another server (useful if the zpool is on a SAN LUN). The first command was relatively quick, the second was much, much slower.

Running prstat revealed that devfsadm was consuming an entire CPU. The purpose of devfsadm is to dynamically add and remove devices on the system. It was stating each of the snapshots in the datapool and creating entries in /dev. After running for a few hours, it had created over 4000(!) devices in /dev/zvol/dsk/datapool and /dev/zvol/rdsk/datapool.

The number of snapshots is thanks to the automatic snapshot service which takes frequent snapshots of the filesystems in the pool. This list is not automatically cleared down, so can grow huge. Not a problem usually because the uptime of OpenSolaris is fantastic, but is a real pain when you need the server to boot.

So, in order to keep your OpenSolaris boot times down, keep an eye on the number of snapshots on your system.

Book Review: OpenSolaris Bible

The relationship between OpenSolaris and Solaris is similar to that between Fedora and Red Hat Enterprise Linux. OpenSolaris is Sun's "in development" operating system that introduces many new features that will eventually become available in a Solaris 10 update or in a future Solaris 11 release.

So it would seem sensible for Solaris system administrators to have some familiarity with OpenSolaris and while it's possible to transfer a lot of existing Solaris knowledge across, having a comprehensive book alongside can be very useful.

Enter, the OpenSolaris Bible by Solter, Jelinek and Miner; a book I received just before Christmas and have been reading through recently.

The book covers the release of OpenSolaris as of 2008 which suggests the book was based around release 2008.05 or 2008.11 (OpenSolaris releases have a YYYY.MM version number). Since that date, there has been a 2009.06 release and 2010.02 is anticipated next month. However, do not let this put you off considering this book. OpenSolaris development is fast paced, but there is an awful lot of stuff in this book to absorb that still remains relevant in newer releases.

The book is broken into six parts:

• Introduction to OpenSolaris
• Using OpenSolaris
• OpenSolaris File Systems, Networking and Security
• OpenSolaris Reliability, Availability and Serviceability
• OpenSolaris Virtualization
• Deploying and Developing on OpenSolaris

The first part is a typical introduction and covers the history of Solaris, Open Source, as well as instructions on installing OpenSolaris and a basic "crash course" on using the GNOME desktop and the Unix shell. Experienced administrators will be able to skim this section.

Part two covers using the desktop in more detail, printing and software management using the Image Packaging System (IPS). This is an essential read as IPS is a new feature in OpenSolaris and printing can sometimes be a bit tricky.

Part three provides a very comprehensive introduction to Solaris disks, pseudo filesystems such as devfs, tmpfs, lofs and swap, UFS, Solaris Volume Manager, iSCSI, quotas, backups and restores, mounting and unmounting as well as a full chapter on ZFS, before moving onto network configuration including IPMP, link aggregation, virtual LAN interfaces, network services (DNS, DHCP, FTP, NTP, Mail, HTTP etc.), routing and the IP Filter firewall. Part three of the book then finishes with a chapter on network file systems and directory services (NFS, CIFS, NIS and LDAP) and security (PAM, RBAC, SSH, auditing and Kerberos). There is a lot of good content here.

Part four details the Fault Management architecture in OpenSolaris, the Service Management Framework (SMF) introduced in Solaris 10 as well as monitoring with conventional tools and Dtrace, ending with a chapter on high-availability clustering.

Part five covers resource management (projects, tasks, caps and pools) along with a number of Sun virtualisation technologies (Zones, xVM, LDOMs and VirtualBox). The xVM section is only relevant to x64 installs and the LDOM section requires Sun UltraSPARC T-series processors, while Zones can be used one either architecture and is certainly worth a read.

The final part consists of a chapter on deploying a web stack (Apache, PHP, MySQL, Tomcat and Glassfish) and a chapter on software development (Java, C/C++, etc.). I have no strong interest in these subjects at the moment, so haven't read this section.

While I have not read the whole book yet (Having ignored most of the coverage of GNOME desktop applications as if you are familiar with Linux, there's not a lot of new stuff to learn), there are plenty of sections that have made the book worthwhile. Whether this book is suitable for you or not, depends on where you're starting from:

If you are a Windows administrator looking to develop some Solaris experience, the OpenSolaris Bible is well worth a read. The first two parts provide a gentle introduction to the Unix operating system to get you started, and subsequent chapters dive pretty deep into the capabilities of OpenSolaris.

If you are experienced with Linux but have minimal Solaris experience, the OpenSolaris Bible is highly recommended! FMA, SMF, Zones, ZFS, UFS/SVM, Clustering, Dtrace and IPS are not found in Linux, so the OpenSolaris Bible provides a single point of reference for a whole lot of new learning.

Even experienced Solaris administrators will find things to like in this book. The IPS is certainly a new feature that I assume will impact us when Solaris 11 is released, and while ZFS, Zones, FMA, SMF etc are already present in Solaris 10, the book provides a very good overview of these technologies that can otherwise only be found by attending a course or reading the online documentation.

It's probably fair to say that if you read through the whole book, put into practice the features described, and you understand them, you'll have a wider understanding than many existing Solaris system administrators.

The OpenSolaris Bible can be bought at Amazon.

Highly recommended. 9/10.

Hands on Solaris IP Multipathing

Today I had a first look at IP multipathing on Sun Solaris. What is IP multipathing? It's a feature that can be used to provide additional network resilience to a server with multiple physical network interfaces. By ensuring that at least two interfaces are on the same subnet, IP Multi Pathing (IPMP) provides continuous uptime if one of the links goes down, migrating the IP address transparently over to the other interface. IPMP operates at the IP layer (layer 3) and does not do link aggregation (although Solaris does support this, I haven't tried that yet).

To test, I booted a Solaris 10 VM on VMware ESXi. I had assigned two NICs to the VM and ensured they were connected and plumbed in.

The first step is to create a "group" for the IPMP by assigning the interfaces e1000g0 and e1000g1 together. I unimaginatively called the group "mygroup".

bash-3.00# ifconfig e1000g0 group mygroup
bash-3.00# ifconfig e1000g1 group mygroup

I then assigned an IP address to each interface and brought the interface up:

bash-3.00# ifconfig e1000g0 192.168.192.105 up
bash-3.00# ifconfig e1000g1 192.168.192.106 up

The output of ifconfig -a looks similar to normal with the addition of the groupname flag:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

With IPMP setup, I setup a continuous ping from another machine and then edited the VM in the vSphere client, disconnecting the second NIC from the network. Immediately, the following was reported in /var/adm/messages:

Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 215189 daemon.error] The link has gone down on e1000g1
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 594170 daemon.error] NIC failure detected on e1000g1 of group mygroup
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g1 to NIC e1000g0

No packet loss so far. What does ifconfig -a now show?

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g0:1: flags=1000843 mtu 1500 index 2
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
e1000g1: flags=19000802 mtu 0 index 3
      inet 0.0.0.0 netmask 0
      groupname mygroup
      ether 0:c:29:f9:5d:ee

While e1000g1 was still present, it now had a status of FAILED and was no longer UP. The in.mpathd daemon had initiated a new virtual interface, e1000g0:1, on the other interface and assigned the failed IP address.

With the test complete, I then reattached the NIC in the vSphere client and noted the following in /var/adm/messages:

Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 820239 daemon.error] The link has come up on e1000g1
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 299542 daemon.error] NIC repair detected on e1000g1 of group mygroup
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g1

This looked good, and a final check of ifconfig -a showed:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

This was so easy I don't know why I didn't do this years ago...

Applying GTD principles to Outlook 2007

I have now read a large chunk of David Allen's "Getting Things Done" book and have started to apply some (not all!) of the principles to my home and work processes. This started with me tidying my desk before Christmas leave, and filing away reference documents in folders. It's amazing how positive a tidy desk can make you feel.

I also attacked the 13000+ items in my inbox and split them into Archive sub-folders, one for each year (Archive - 2009, Archive - 2008 etc.). This meant that when I started back in January, my inbox had zero items in it!

The heavy snow has resulted in me working from home this week and the productivity benefit has been huge! I have successfully completed almost all of the 69 items I have marked in Outlook "For Follow Up". This had become a catch-all view of emails I wanted to review, or that required action. While it was better than marking the email unread (which some of my colleagues do), it wasn't ideal and could quickly become unwieldy.

So my new plan is to attempt an "Inbox Zero" approach to email. If an email requires an action that takes less than five minutes, I'll do it immediately, otherwise I'll assign it to a category and archive it.

I've also created some additional categories, @Deferred, @Someday/Maybe and @Waiting. GTD followers will recognise these. I've also created a new "Archive - 2010" folder to put emails that are dealt with but I need to keep. All read and actioned emails go here.

Manually setting these categories and dragging completed emails to the archive folder works, but I wanted to have toolbar buttons. Unfortunately my VBA skills are non-existent, but a quick Google found these two sites:

http://blogs.msdn.com/swiss_dpe_team/archive/2007/12/11/office-2007-outlook2007-macros-vba-how-to-work-better-with-categories.aspx

http://verychewy.com/archive/2006/04/12/outlook-macro-to-move-an-email-to-folder.aspx

The former site provides a macro for assigning categories to selected items, and the latter provides a macro for moving selected messages to a designated folder. The first of those two links also provides details on creating a self-signed cert to enable macros in Outlook. Full credit for this code goes to the above sites.

Once I'd modified the macros for my own needs, I created a new toolbar which gives me one click actions to the categories and email archiving:


The source code to the macros is very simple:

Sub Waiting()
Call updateCategoryMain("@Waiting")
End Sub

Sub Someday()
Call updateCategoryMain("@Someday/Maybe")
End Sub

Sub Deferred()
Call updateCategoryMain("@Deferred")
End Sub

Function updateCategoryMain(cat As String)
  Dim myOlExp As Outlook.Explorer
  Dim myOlSel As Outlook.Selection
  Set myOlExp = Application.ActiveExplorer
  Set myOlSel = myOlExp.Selection
  Dim i As Integer
  For i = 1 To myOlSel.Count
      Call updateCategory(myOlSel(i), cat)
  Next i
End Function

Function updateCategory(mi As Object, cat As String)
  Dim pos As Integer
  pos = InStr(1, mi.categories, cat, vbTextCompare)
  If pos > 0 Then
      a = Left(mi.categories, pos - 1)
      b = Right(mi.categories, Len(mi.categories) - pos - Len(cat) + 1)
      res = a & b
      mi.categories = res
  Else
      mi.categories = mi.categories + "," + cat
  End If
  mi.Save
End Function

Sub Archive()
  On Error Resume Next
  Dim objFolder As Outlook.MAPIFolder
  Dim objInbox As Outlook.MAPIFolder
  Dim objNS As Outlook.NameSpace
  Dim objItem As Outlook.MailItem
  Set objNS = Application.GetNamespace("MAPI")
  Set objInbox = objNS.GetDefaultFolder(olFolderInbox)
  Set objFolder = objNS.Folders.Item("Mailbox - R, J (XXX)").Folders.Item("Archive - 2010")

  'Assume this is a mail folder
  If objFolder Is Nothing Then
      MsgBox "This folder doesn't exist!", vbOKOnly + vbExclamation, "INVALID FOLDER"
  End If

  If Application.ActiveExplorer.Selection.Count = 0 Then
      'Require that this procedure be called only when a message is selected
      Exit Sub
  End If

  For Each objItem In Application.ActiveExplorer.Selection
      If objFolder.DefaultItemType = olMailItem Then
          If objItem.Class = olMail Then
              objItem.Move objFolder
          End If
      End If
  Next

  Set objItem = Nothing
  Set objFolder = Nothing
  Set objInbox = Nothing
  Set objNS = Nothing
End Sub

Okay, so there's a fair amount of hard coding, but I didn't want to spend more time than absolutely necessary to get this working!

The final improvement I've done is to assign colours to my incoming emails. By selecting Tools, Organize and then selecting "Using Colors", it's possible to configure Outlook so that mails that are only to me are in blue, mails from my line manager(s) are in red and mails from T are in green. Everything else is in black, but it provides a nice visual clue.

No promises that Inbox Zero and the GTD approach will work for me, but so far I've managed to stay on top of incoming requests and finding things in my mailbox is refreshingly speedy.