Home Lab: Storage Upgrade

I've been quiet over the last few months due to a very busy work project. With that now out of the way (for now!), I can update on a few of the things I've been doing with the home lab...

As detailed in a previous post, I've been running my NAS/SAN on Nexenta. This was originally running as a VSA under ESXi. While this configuration worked very well for me, I did notice that on the Microserver, the Nexenta VM was being CPU and RAM constrained. When my old ML110 G5 was replaced by the G7 servers, I decided to reprovision the Nexenta VM onto dedicated hardware.

The ML110 G5 has six SATA ports and I configured them as follows:

1. 250GB SATA
2. 60GB SSD
3. 1TB SATA
4. 1TB SATA
5. 1TB SATA
6. 1TB SATA

The Nexenta operating system is installed on the 250GB disk. While this is not mirrored, there is nothing particularly important on this disk and if it dies then it can be replaced, Nexenta re-installed and the important zpool reimported.

The 60GB SSD is configured as a L2ARC (read) cache device.

The four 1TB hard drives are configured in a RAID10 configuration. This gives better performance than my original configuration which used RAID-Z (aka RAID5) and is used for both NFS and iSCSI shares. It's where the "important stuff" is stored.

I added a Lights Out board for the ML110 G5 which I found on Ebay. All my other servers have lights out and it means I don't need to have a monitor attached.

In addition to the capacity and improved performance due to the faster RAID, the other advantage of going physical is that I can dedicate the entire 8GB of RAM in the server to Nexenta (up from 4GB on the VSA). Nexenta also benefits from having two Xeon cores dedicated to it, a significant increase over the cores in the Microserver.

I've not had the chance to benchmark the new build, but it "feels" faster than running under a VM and it also frees up CPU and RAM on my Microservers for more VMs.

How a disposable email address shows who is sharing your details...

I bought my iPhone 3G from an O2 store on the day of its release. A couple of years later I upgraded it to an iPhone 4, again from the same O2 store.

Over the last couple of weeks, I've had a number of missed calls to my mobile. No voicemail, just a missed call from a number I didn't recognise. I entered the number into Google and it appears to be associated with the Carphone Warehouse. The search results revealed a lot of people complaining that Carphone Warehouse were cold calling their mobiles to try and sell an upgrade.

Annoying, but since I'd missed the calls, no big deal.

Then, today, I received an email from Carphone Warehouse telling me I was eligible for an upgrade. Now, I'm not a customer of the Carphone Warehouse, so how did they know I was eligible?

The answer is in the email...

I use Yahoo mail for non-important stuff because it has disposable addresses. Basically you create a new address such as "mymail-" and you can then append a word to it depending on who you are dealing with. In this instance, the email was addressed to "mymail-o2@yahoo.co.uk". This address (and I've modified the first bit as it's not really "mymail-") was used when I signed up with O2.

I don't use it for anything else.

So if Carphone Warehouse are sending me emails to this address, it means that either a) I've given Carphone Warehouse my details using my O2 address (I haven't), or b) that O2 have passed/sold my details to a third party.

To be fair, it's probably hidden in the small print, but it's not cool. Who here likes receiving cold calls and junk email?

No, didn't think so.


So I complained on Twitter:

And got a reply, which started a conversation:

I'm not sure why me being a PAYG or Pay Monthly customer matters, or why recently upgrading would make a difference. I'm going to give the O2 Twitter operator the benefit of the doubt here. It's possible he/she doesn't know that the data is passed on.

But the reality is that somehow Carphone Warehouse have an address that should only exist in O2's customer database.

The last message I received today from O2 was:

That I will. And update this post.

Of course, if the data hasn't been intentionally passed to a third party, then there is always the possibility that O2 have experienced a data breach... (I don't honestly expect this to be the case).

Perhaps someone from O2 would like to clarify how Carphone Warehouse know the private address that only O2 and I know, unless the data is passed on?

Remember, O2, actions that result in your customers getting cold called and spammed is BAD.

And Carphone Warehouse? No. Not a chance.

*** UPDATE 19th December 2012 ***

Carphone Warehouse continue to spam me:

O2: Please make this stop. I have sent a STOP message and forwarded the text to the O2 spam reporting number.

VMworld Europe 2012: Wrap up and the next steps...

Having blogged about VMworld Europe 2012 as it happened, I felt it was important to summarise the week and highlight the key points that will shape the next 12 months:

1. Get the basics right

A solid infrastructure foundation is essential. As vSphere deployments grow, both in terms of number of VMs and complexity of applications provisioned, the need to ensure best practices for storage, networks and vSphere configurations is a pre-requisite of any new projects. This can be accomplished through the following of vendor white papers and policy management tools such as Host Profiles.

However, a pragmatic approach needs to be taken to balance any performance gains against operational complexity. For example, while it may be possible to tweak the round robin path selection parameter, or implement jumbo frames on storage switches, the additional complexity can introduce additional management problems later on. So, while tuning the number of IOPS per path may result in a performance improvement for a small number of VMs, the benefit may be negligible when dealing with a large number of hosts, datastores and VMs. Similarly, jumbo frames may provide a minor throughput improvement, but if a new switch is added later, the system administration team must remember to apply the same settings to the new switch or else experience frame fragmentation.

In other words, keep it simple.

2. Learn to automate

VMware vCenter Orchestrator is moving from being a peripheral product to a core part of a complex cloud infrastructure. Therefore, I'd consider it a must-have skill to acquire in the coming year. The knock on requirement is that vSphere admins will need to have a basic understanding of programming languages and development methodologies.

What about other scripting approaches such as PowerCLI? In many ways, Orchestrator and PowerCLI are complimentary. PowerCLI scripts can be used to make repetitive individual tasks easier to perform, while Orchestrator is more about enabling workflow (of larger tasks). It's probable that PowerCLI (or other PowerShell) scripts will be called as part of an Orchestrator workflow.

Bottom line: Learn both.

3. Virtualise more

VMware's "Software Defined Data Center" (SDDC) strategy extends the existing virtual infrastructure and introduces the automation and orchestration of edge devices. If implemented well, this means that provisioning new services should be faster and less error prone. The opportunity to upgrade Enterprise Plus to vCloud Suite Standard enables the rollout of vCNS and vCloud Director, thereby forming the first step in moving to this new "agile" data centre, and also provides the foundations to build a private cloud on top of vSphere.

In order to ensure that a private cloud implementation is "done right", the VMware vCloud Architecture Toolkit will be followed. This will dictate some changes to the existing infrastructure, further extending the use of a dedicated management cluster and separate production clusters.

There is one challenge to consider: The added complexity that comes with the additional functionality makes the learning curve significantly steeper. Today, it is possible for the IT generalist to work with VMware in addition to their regular day job (which may involve supporting everything else in the IT estate). At first glance, the SDDC makes this a lot more difficult. There are more virtual devices to manage and more places to configure more settings.

The responsibility is therefore on the VMware senior administrators/architects to design using clear principles, document and create the orchestration workflows to ensure that complex tasks are well managed so that the IT staff responsible for operations can do their job without needing to fully understand the complexity of the environment.  

IT architects: Be prepared to invest time learning how all this fits together.

4. Make it easier for users

The truth is that today, end users have to jump through too many hoops to get their VMs provisioned and the cost model is still unclear.

Although VMware have demonstrated vCloud Automation Manager, capable of handling the provision request/approval process, by limiting it to customers running vCloud Suite Enterprise, the rest of us are left with no ability to provide a request/approval mechanism for VM creation. It is likely that something will need to be written in-house, perhaps using something like WaveMaker Studio front-ending a series of Orchestrator workflows.

There are other ways to make life easier for our users. As admins, we currently use vSphere templates to make VM deployments simple, but this should be extended to create a full vCloud Director application catalogue for our end users. For example, if our end users want a Red Hat Enterprise Linux server with Apache Tomcat already installed, we should be in a position to make a catalogue item available for this purpose.

Using the vFabric suite to deploy these applications is probably overkill, but there are alternatives. It would be a useful exercise to do some work with Puppet to look into the provisioning of applications.

5. Monitor and plan

Finally, as things get more abstracted and complex, there is a real need to manage this infrastructure and proactively plan for future growth. VCOPS Foundation will be the starting point for this, but it's possible (probable?) that a fuller featured edition may be required over time.

There is going to be a real need to perform capacity planning to ensure that existing infrastructure resources are utilised properly, and that additional hardware is available when required.

Rationale

The above needs to be done in order to allow IT to keep up with the business demands of our users.  We need to stop being a stumbling block and instead come up with ways to deliver services faster.

The amount of servers and applications we now manage is greater than ever before and there is no sign that this will change in the near future (probably the opposite, and demand will continue to increase!).

We can't keep doing things the same way (it doesn't scale and it's too slow). We need to be smarter and more proactive about managing our infrastructure and providing the applications and environments our users need.

VMworld Europe 2012: Day Three

The final day of VMworld began and I again opted to spend the majority of my time in the hands on labs.

Back to the Hands on Lab

The first lab of the day was based around the new features and capabilities built into the vCloud Networking and Security (vCNS) product, previously known as vShield Edge and App. This product is a virtual appliance based firewall with some advanced capabilities such as IPsec VPN and load balancing. Previous versions of vShield sat at the border of a vSwitch port group and provided only two interfaces (inside and outside). The new version can support multiple networks with the example given being a traditional three legged design for external, DMZ and internal networks.

Another useful feature of vCNS is the ability to put two appliances in an active/passive configuration. As part of the lab, the active instance was powered off and the failover kicked in, losing only three ping packets before the peer picked up the load. As previous versions of vShield Edge represented a single point of failure, this addition is welcome for mission critical or highly available environments.

The second lab was an epic example of tying a number of products together to illustrate the automation and provisioning of applications. Understatedly titled "Deliver Your IT Services in the Cloud", the lab utilised VMware Service Manager to request a new vApp as a consumer, Zimbra for the administrator to receive the request and then action it in VMware Service Manager, which then kicked off a workflow in Orchestrator that provisioned a VM in vCloud Director and then installed the web application using vFabric Application Manager. This advanced level of automation and integration across the various products provided perhaps the best illustration of where VMware are going with its Software Defined Datacenter strategy. In contrast, it's interesting to see how little focus there is on the "traditional" VMware strengths of virtualising compute, network and storage. A hint to competing hypervisor vendors: the world has moved on and it's now about building and managing automated application stacks built on top of public/private/hybrid clouds.

The third lab was a focus on the new features of the Distributed vSwitch. As new Enterprise Plus users, the Distributed vSwitch will be a new addition to our infrastructure and the lab provided some useful troubleshooting tips.

The final lab covered the new features in the latest release of VCOPS. The analytics engine and presentation of data in VCOPS is amazing (no change there) and it was interesting to see how this is becoming the management platform for VMware.

vCenter Operations Manager

By which point my brain was struggling to assimilate any new information! There was no big bang event to close the conference, just a steady stream of people finishing up and heading back to their hotels.

There is still much to reflect on, and my deliberate strategy of tackling the Hands on Labs at the expense of attending the sessions means there will be significant ongoing catching up online over the coming weeks and months.

The value of attending a conference such as VMworld is that it gives an opportunity to deeply dive into the technology we use everyday, familiarise ourselves with the developments that will be with us soon, and allows us the space to think, plan, learn, discuss and ultimately equips us to do our jobs. For those of us who are passionate about the work we do, it's an amazing experience and I'm personally very grateful to my company for making it possible to attend.

VMworld Europe 2012: Day Two

Day two started with another keynote, this time focused on "end user computing". VMware demonstrated a number of new features to View, Horizon and application delivery to iOS and Android devices.

None of this really interests me, so, meh.

My first session of the day was around the vCloud Architecture Toolkit, which I only stayed in for the first ten minutes. The pace of the presentation, along the presenter providing a definition of cloud (really? It's 2012!), led to me bailing and going the Hands On Lab instead.

Hands on Lab

At the lab, I took several sessions throughout the day. Two sessions were based around the vFabric Application Director and vFabric Data Director. These two products are designed to facilitate the provisioning of applications to a cloud environment. In the first lab, you had to provision a database (MySQL), application (Tomcat) and web (Apache HTTP) server, connect them dynamically, and then scale out the middle tier. The resulting blueprint is then built as a vApp which is available through vCloud Director. Clever stuff, but quite a steep learning curve (following the guides were easy, but starting from scratch would be a formidable challenge!).

Of interest was the list of options of components that can be used in building an application blueprint. In addition to the open source stacks were objects for SQL Server and IIS, although the lab didn't touch on these components.

The vFabric Data Director lab was similarly interesting, providing a single web front end into the management and deployment of Oracle and Postgres databases. The database instances could be cloned, resources hot-added and SQL run through the interface. Of interest to me was that the underlying mechanism to manage these instances is to manipulate and clone VMs. In other words, the object that is being managed is a VM. The application doesn't appear to be doing anything to enable multi tenancy within a single database installation (or if it does, this wasn't covered in the lab). It was very impressive to be able to take a Postgres 9.0 installation, rapidly copy the VM using linked clones, and then apply an upgrade to 9.1!

For more information (and to see what the above looks like):

A short video introduction to vFabric Application Director
A short video introduction to vFabric Data Director

The final lab of the day was on the new vCenter Orchestrator, a product most of us VMware admins have had for years, but few of us have used. This lab went through the creation of workflows that are then run against objects in vCenter. The lab then extended this with the use of plugins, showing how Orchestrator can be used to manipulate Active Directory. This is something that deserves a greater look and as a result, I bought the VMware Press book, Automating vSphere with VMware vCenter Orchestrator.

While in the lab, I noticed a plasma with a display that looked as if it was showing a Splunk dashboard. On wandering over to the screen, I discovered that this was labelled "VMware Strata". On enquiring with the lab staff, no one appeared to know what it was, but they thought it may be a new product based on a VMware acquisition that would eventually make it into that all-powerful monitoring and reporting tool, vCenter Operations Manager.

Mysterious VMware Application

Following the labs was the "Hall Crawl", providing another opportunity to meet the vendors. This included a very useful discussion with HP on the use of 3PAR storage and the unique features it provides. The thin provisioning and zero reclaim are very impressive ways to optimise and manage storage usage, although there is currently no deduplication or compression as found in NetApp filers.

The day was finished with the VMworld Party, a huge event with a cool 80s retro computer games theme, many activities and games, live acts and a constant supply of food and drink that ran late into the evening.

The VMworld Party

VMworld Europe 2012: Day One

Arrived yesterday afternoon in sunny, hot and humid Barcelona in preparation for VMworld Europe 2012. Having registered, taken a lab session (on the vCloud Suite), settled into the hotel and eaten an evening meal in a small, local cafe (where the owner knew no English and we knew no Spanish, which made the whole affair very interesting), I made sure I had an early night because this conference is going to be packed.

And so it is!

And so it begins

The Keynote

The keynote kicked off at 9am. Not a huge amount of new stuff announced after the US show, but there is a new version of vCenter Operations Manager (aka VCOPS) which now includes application awareness so you can drill into a VM and get the state of the underlying application (such as the RAM utilisation of an individual Oracle instance). While this is very impressive, unfortunately, VCOPS remains reassuringly expensive (but see below for some good news).

A new plugin was shown that allows the vSphere client to manage third party hypervisors. This, along with both DynamicOps and Nicira supporting multiple hypervisors, is a new approach by VMware and probably reflects the reality in many datacenters.

The vCloud Automation Center was revealed, which is based on VMware's acquisition of DynamicOps. This showed the ability for end users to request new VMs from a catalogue and to have management over their VMs (power on, off, restart etc.) through a web portal. This looks like an essential component in the provisioning process, fulfilling the self-service aspect of the cloud while facilitating an approval process, but according to the VMware website, it's priced at $400 per managed VM(!), which makes it too expensive for all but the largest customers.

The new vFabric Application Director was also shown, with the introduction of the idea of "application blueprints" which take the concept of VM templates to the next level, allowing a drag and drop approach to building an application stack. The Cloud Application Marketplace (aka App Store) was shown where third parties can publish their own blueprint components, allowing for even more sophisticated application stacks. As an example, the screenshot showed some Riverbed components, so you could imagine integrating WAN acceleration into your one-click-to-deploy vApp. The list price for this is $6250, which, like the above, is going to be difficult for many of us to justify, no matter how cool.

A couple of screens were shown of the IT Business Management Suite, which would make a CIO very happy. Lots of cost related metrics and pretty dashboards, but not something I'll personally have any need for.

The keynote ended with a demo of a social networking project. In the demo, hosts, clusters and VMs are social network entities that can be "followed" and "liked" in the same way our Facebook friends can be "followed" and "liked". If a host loses a datastore, it posts a message to the news stream. Other hosts that have the same problem "like" the original message which can be used for analytics. If dozens of hosts all report that they have lost access to a datastore, then there's probably a serious issue. All sounds a bit gimmicky, but it did look pretty decent and got the biggest, most spontaneous applause of the keynote(!).

The main things I took away from the keynote are:

VMware offer so much more than just hypervisor-based virtualisation. Where other hypervisor vendors sometimes compare vMotion/Live Migration capabilities, play "My VM is bigger than your VM" (VMware are guilty of this too), and fight to see who offers the most features for free, the VMware strategy is now about the datacentre and everything in it, automating and orchestrating according to policy.

In VMware language, this policy based automation is referred to as the "Software Defined Datacenter" (SDDC) and is a logical continuation of the ability to virtualise resources. We're now comfortable virtualising compute, storage and network connectivity, but VMware are looking to extend this to network edge devices (load balancers, WAN compression, firewalls etc.). While virtual appliances already exist for some of these, the big difference is that the SDDC is built on the idea that these devices can be scripted, orchestrated and therefore automated by policy. This further extends (through the vFabric suite) into the application layer.

Stop and think about this for a bit, because this is actually a big deal. What happens when your entire datacentre is software defined? You get to throw the word "agile" into the mix...

Want to know more about the SDDC? Check this short video: http://www.youtube.com/watch?v=Jf0KdjpxgCI

A negative comment regarding the keynote announcements, it is frustrating that customers who opted for the top end solution in the VI3.5 days with the Enterprise edition were left in the cold with 4.0's introduction of Enterprise Plus and subsequently upgraded, are now migrated to vCloud Suite Standard which still leaves many useful features out, only available in vCloud Suite Advanced and Ednterprise. I understand that each edition adds many new features, but it seems like a ploy by VMware to constantly generate new revenue by making customers pay out for ever bigger software solutions.

Post keynote

Following the keynote, I had a quick wander around the Solutions Exchange and got into a conversation with a Cisco or NetApp employee about the new ExpressPod which appears to be a low cost FlexPod built around the C-series rack mount server and Nexus 3000 switches (with NetApp storage). This may be worth investigating further...

The Solution Exchange

Also in the Solution Exchange, I had a conversation with Veeam about support for vCloud Director and end user, self-service restores on backed up VMs. Nothing yet announced, but Backup and Replication release 7 should be interesting next year. There was also the suggestion that tape support was coming, but on later reflection, this may have limitations for those of us running a purely virtual Veeam deployment.

The Solution Exchange was also the venue for an unexpected encounter with Brent Spiner who was signing photos and chatting to people!

It's Data!

And, to the probable dismay of my wife, I now own a red fedora, courtesy of Red Hat (and she thought last year's bandana was bad...).

The rest of the day was filled with sessions, including an excellent multi-vendor presentation on storage best practices by Chad Sakac (EMC) and Vaughn Stewart (NetApp). Storage is a topic of interest to me and I've read up a fair amount on it, but was pleased to find myself noting down many "to-do" items for the infrastructure I manage. Very useful!

The second session attended was on future storage developments, including vVols, Virtual SAN and Virtual Flash. Interesting technology and it's useful to know this stuff is coming down the road.

While sitting in the Hang Space reading up on the new developments, I came across a reference on VMware's website that a new, entry level version of VCOPS called vCenter Operations Manager Foundation, will be FREE to all vSphere customers. While this removes about 90% of the product's feature set, it will be a good starting point for infrastructure management and alerting.

Part of the Hang Space

The early evening was spent in the Solutions Exchange for the Welcome Reception, basically an opportunity to engage with vendors while eating canapés and drinking free wine/beer/juice/water.
And that, for me, was enough for the day! There are parties running late into the evening, but tomorrow promises to be an equally busy day, so time to get some head down time.

Other thoughts...

On a less IT related note, this venue is very impressive (but not noticeably better or worse than Copenhagen). The conference centre is huge, requiring the use of those automated walkways normally found in airports to get around. There is a constant availability of free food and drink on offer and plenty of space to sit, relax and reflect on all the new developments. Credit to the event organisers who really understand their audience and do everything to keep attendees happy and focused!

As with last year's conference, it's interesting to note the use of smartphones and tablets, alongside laptops. It seems that most people have converged down onto two devices, not one. Of all laptops being used by conference attendees, I'd guess that easily 50% are MacBooks, implying that either VMware admins prefer to spend time doing VMware stuff and not installing Windows patches, or they are easily seduced by shiny hardware...

Passing the CCNA Security exam

The CCNA certification is valid for 3 years and mine was due to expire at the end of August 2012. I could either retake the same exam and recertify, or take another CCNA "concentration" exam that would give me a new certification and renew the original certification at the same time. I opted to tackle the CCNA Security exam, IINS 640-553.

I'd originally bought the Cisco Press "Authorized Self-Study Guide", Implementing Cisco IOS Network Security by Catherine Paquet back in 2010, but the material is a bit dry and I didn't have the motivation to get into it very far. By booking the exam, I suddenly acquired the motivation required.

As things happen, the 640-553 exam is due to be retired in September 2012, to be replaced by 640-554. The main difference in the new exam appears to be an additional focus on the Cisco ASA platform, as well as de-emphasising the Cisco Secure Device Manager (SDM). This means that any advice I give here will be redundant soon, and also I'm bound by the NDA, so can't obviously comment on what is in the exam.

What I can do though is give some general thoughts on the revision process:

The Good

The Implementing Cisco IOS Network Security book is very thorough. It covers a lot of detail and assumes little prior knowledge of security. Some of it is dry, especially the first chapter which weighs in at about 100 pages and gives an introduction to the world of security. Once that's passed, the content gets better and even the chapter on cryptography was interesting(!).

I also bought the Cisco CCNA Security Lab manual for the CCNA Security course. This gave some very good exercises to run through which were very useful in grounding the theory in the practical.

All of this was made possible using the amazing GNS3 router simulation software. I installed this on a meaty Windows Server VM and was able to run the 3 routers and 2 XP images (in Virtualbox, under ESXi) without any problems. The ability to save configurations and easily re-import them was a great time saver. GNS3 doesn't do everything (specifically switches, due to the custom silicon in them), but it made the whole process of learning the syllabus a lot easier.

There is some very good material at the Cisco Learning Network including free study chapters, training videos and discussions. Highly recommended.

The Bad

Cisco sell the book for self-study, but make it very difficult to practice because IOS images are not available without having the correct support contract. If you work for a large company with either old routers sat on a shelf or a contract with the ability to download the image then you'll be okay. Otherwise I guess you'll be searching the Internet for a dodgy copy of an old image. Seriously Cisco, how about making them freely available? You can do the study material with a 2600 series router and how old is that?

The same is true of the IPS signatures. A valid contract is required just to learn how the IPS works and again, this could mean a trip to the darker parts of the Internet to find them.

The Cisco Press book covers the Cisco Access Control Server software but it's not in the syllabus or lab manual. It can be used to learn about AAA and specifically authentication and authorization with RADIUS and TACACS+. Unfortunately Cisco don't have a trial version to help self-studying students.

The Ugly

Getting the Cisco Security Device Manager (SDM) working requires jumping through a number of hoops. To cut a long story short, you need an old version of Java (1.4 worked for me) and Windows XP. I'm guessing the latter requirement is due to Internet Explorer 6 as I couldn't get it working on Server 2008 R2 no matter what settings I tried.

Conclusion

Having worked through the labs a number of times and then setting things up "blind" (without referring to any notes), I felt fairly confident as I went into the exam. I passed with a good mark well above the passing level, so I'm naturally very pleased with this. It's a good subject to read up on since security requirements impact on so much of what we design these days. The CCNA Security should demonstrate I now have a solid grounding in the subject, even if I'm still a long way from being an expert.