So much power in so little space.

Over the years I have tweaked and built up many different ESXi whitebox “servers” for lab use. I wanted to share some experiences with you in a blog post and recommend some new hardware that will help you get a nice little home lab setup for ESXi 5.

I’ve bought vanilla Intel motherboards and have had a lot of success with those. The Intel network adapters seem to have the greatest compatibility overall. My most recent builds have been based on the Shuttle XPC platform. They work right out of the box and this post will focus primarily on those.

If you’re looking to build a hardware based VMware home lab, you will have the most flexibility overall. When you nest your lab inside VMware Workstation, there are some features that you will not be able to use like Fault Tolerance and EVC. If you hack around a bit you can get them to work though. I’ve just always went with a hardware lab so we will stick with that.

The ESXi 5 Host:

It’s recommended to have two hosts minimum so you can do all of the cool stuff. Without the second one, it will still work fine but what fun is it without HA/DRS and vMotion?!

Shuttle XPC SH67H3 - We will use this as our base system. It’s a compact system, supports new generation processors and up to 32GB memory. It is recommended to update to the newest BIOS on the Shuttle website. This will allow us to use the PCI Express slot for a NIC and not lock us into using it for a video card. There were early reports that old revisions of the BIOS would only allow you to use it for video card expansions. We want to have the option to use a dual NIC to bring the host up to a total of 3 NICs. The integrated adapter works fine out of the box with no modifications to the ESXi 5 install media. These machines have been running flawlessly for almost a year now in my lab.

Intel PRO/1000 Dual Port Server Adapter - This adapter works in the spare slot that the Shuttle has. Again, as of this writing it works flawlessly out of the box with the vanilla ESXi 5 installer. No modifications needed. Just be sure to flash the BIOS to the newest revision.

Your Choice: Intel Core i7 2600 Sandy Bridge Processor or Intel Core i5 2500 Processor - I went with the i7 processor but this is entirely your choice. If you’re on a tighter budget the i5 processor would save you some cash. I would recommend that, if you are on a budget, go with the i5 and buy two hosts vs. one host with an i7. You are almost always out of memory before CPU anyway.

Your Choice: 8GB Corsair DDR3 (2 x 4GB) 1333 MHz or 16GB Corsair DDR3 (2 x 8GB) 1333 MHz - I’ve had a ton of success with Corsair memory. This is one of those things where you probably will want to shop around but this memory has been priced well and just works every time you buy it. Remember that the Shuttle system above can do a maximum of 32GB DDR3 across 4 total slots. So if it’s in your budget order up 2x of the 16GB kits and you’re set. I went with 16GB in each box.

Kingston Digital 8 GB USB 2.0 Flash Drive - I’ve used 4GB USB sticks before with no issue, but went with this 8GB stick on the last build. I am sure you have plenty of these things laying around. Any of them should do. I try to avoid the really cheap vendor ones given out at all the VMUGs/conferences etc since I want it to run relatively stable and not die within a weeks use.

This guy might be proud of his lab, but it's time to upgrade.

The Storage Setup/Options:

Now we meet a crossroads. The stuff above gets you your basic host but we all know that a standalone host can only go so far. We have a couple options when it comes to storage. If you’re totally new to the homebrew ESXi lab then you will need to decide what path to take.

• The cheapest option: Find an old computer (you know you have one) and install openfiler or FreeNAS on it and share it out. This is the cheapest option, but may not perform great depending on the box/disk/network etc.
• No old computer to re-purpose? Then pick up a Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s- I’ve used these before but you can buy any simple and cheap SATA drive and install it inside of the Shuttle standalone host. Try to stay away from the green drives if you can. Especially if you’re putting them in a NAS. Then you could build a massive virtual machine on the host and share out the storage over iSCSI/NFS to the host(s). You can use openfiler or FreeNAS if you’d like inside of the virtual machine. Remember that in this case your VMs (disk files specifically) that your build will live inside of this new storage VM itself. Think of the movie Inception. Kind of like a dream within a dream.
• Buy a new NAS. I own a device called a Thecus N5200 Pro. I cannot recommend it. Thecus tries to say this is a great device but they never update the firmware. Avoid them if you can. I would recommend and have heard great reviews of Synology. A Synology DS411 NAS. It’s not the cheapest solution but the performance is great and Synology has impressed as of late. Throw in 4 of the Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s disks and you’ve got a winner! Kyle Ruddy has a recent post and review on his Synology DS411.

So you can see with storage in a VMware home lab you have a lot of options. I would recommend starting small and growing it later.

The Networking Setup:

In order to be as real world as possible in this home lab you’ll want to find a way to setup some VLANs and keep the traffic that your homelab generates off of your “Production” network.

I would recommend using a router based on dd-wrt and setup seperate VLANs for your storage, vMotion, etc. I use dd-wrt and have had success with it and pair it with a 24 port managed Trendnet switch. It works but I’ve also recently started playing with an HP 1810G-8 switch as well which is exactly what I need for my hosts.

Host Details:

Leave your home lab components in the comments below! Lets get a good thread going with hardware that’s supported on ESXi 5.

Duncan over at Yellow Bricks announced that he’s releasing the vSphere 4.1 HA/DRS kindle version for free for two days only. It explains good details of VMware HA (Remember this book focuses on the vSphere 4 implementation of HA) and DRS.

If you’re looking for a great book on vSphere 5 HA which has an entirely new architecture, check out the other book here. The new book is a great deal as well at $9.99 for the kindle version.

Download Here

Thought I would pass along a great slide deck about troubleshooting vSphere storage.

• vCenter Performace Charts & esxtop
• Disk Alignment
• Hard Disks/Storage Arrays
• SCSI Reservations
• APD (All Paths Down) Issues
• Multipathing considerations
• Communication

VMware has released version 1 of the vSphere 5 hardening guide.

The format of this guide has changed from previous versions. The guide is being released as a Excel spreadsheet only.  The guideline metadata from earlier guides has been greatly expanded and standardized.

For additional information, please see the Intro tab of the spreadsheet.

Download Here.

VMware made the announcement this morning and I am happy to say that I have been selected as a 2012 VMware vExpert.  I look forward to being included in the group this year!

The list below is a list of the recipients who’ve been selected globally. It’s my first time on the list and what a group to be included in!

A bit from the VMware Blogs announcement:

We are very happy to announce that the judging for the vExpert 2012 title is now complete and we have the results available.

vExpert 2012 list – rev 1.0

If your name is on the list, you can begin celebrating.

The on-boarding process for the new vExperts will start during the week of April 16th. You will receive an email notification from Alex or John with the explanation of the next steps.

Congratulations to all the vExperts! Thank you for your ongoing contributions to help your fellow VMware users be successful with virtualization technology!

VMware Education has released three hours of free training videos on their site.

Here are some of my favorite ones (all of the links are below):

vSphere Storage Profiles:

vSphere Distributed Switches:

VMware Thick/Thin Provisioning

vSphere VMFS 5

I had the pleasure of speaking at the March Indy VMware User Group meeting last night. I spoke for about 30 minutes on P2Vs. Here is an outline of the slides I presented and also a link to the presentation itself. [P2V_Best_Practices]. If you did not make it, I also gave a live demo of VMware Fault Tolerance as well. They were recording the sessions so I might link to that as well later.

I must give credit to Kendrick Coleman as well. I point out his tool here on several occasions. I had mentioned that I had some scripts to do some of these tasks before I noticed this tool, now you’re the lucky one who doesn’t have to reinvent the wheel!

If you have any questions, comments or things to add to this list, leave them in the comments section below!

Let’s not get physical.
A P2V converts existing physical infrastructure to virtual.
VMware provides a free tool to do this.
Third party tools are available to help you as well.

We will focus on “VMware Converter” which is the free utility provided by VMware.

VMware used to provide a converter plugin for vCenter. In vSphere 5 the vCenter client is not available. Standalone client is the only client.

VMware also no longer provides an offline “cold boot” option to P2V anymore. Your choice is this Standalone Client. Learn to love it!

What can you convert?
Symantec Backup Exec System Recovery (formerly LiveState Recovery)
Symantec Backup Exec
Norton Ghost
Acronis True Image
StorageCraft
Parallels Desktop
Microsoft Virtual PC, Virtual Server and Hyper-V
Live powered on physical systems.

Pre-P2V.
Pull up ‘cmd’ and do an ipconfig /all > c:\ipconfig.txt
After the P2V you will need to re-enter this information because even though the data inside the VM remains the same, you’ll have new hardware (NICs) presented to the VM.
Cache your credentials or have local admin credentials ready.
It’s always nice to not have to dig for this data!

P2V Process: Split up partitions.
Split the partitions into separate .vmdks.
Keeps your disk layout cleaner so that way when your users decide they don’t need that E: drive, your life is easier.
You do this under “Data to copy” section.

P2V Process: Speed them up.
By default, VMware Converter uses SSL to transmit data. If you’re P2V’ing on a local LAN, you might want to turn this off as it significantly speeds the process up. Talk to your security team before doing this.

There is an xml file located in:
C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\converter-client.xml
Look in the NFC section:

<nfc>
<useSsl>false</useSsl> --Change to false.
</nfc>

Post-P2V
A few steps I recommend after completing the P2V:
Remove vendor software.
HP, Dell, IBM, etc all have software that no longer will apply. Remove it!

There are scripts you can use to get most of the software removed.
(See Kendrick Coleman’s VM_Advanced.iso)
Includes an HP and Dell Cleanup utility.
NOTE: Although the script removes a large portion, you’ll still want to verify it as the vendors add new software and the cleanup utilities are not updated.

Remove Windows services that are no longer needed.

Remove non-present devices in Device Manager.
Start>Run and type “cmd” (without quotes).
Set devmgr_show_nonpresent_devices=1
devmgmt.msc

I know what you’re thinking. That’s a lot of right clicking.
There are scripts for this. (See VM_Advanced.iso) and the script below:

REM http://www.vmdude.fr/tips-tricks/suppression-des-peripheriques-caches/
REM http://support.microsoft.com/kb/311272
REM Back up the list of every device in a temporary file
DEVCON Find * | FIND /I /V "matching device (s)" > "%temp%\DevconFind.txt"

REM Remove temporary file
DEL "%Temp%\DevconFind.txt"

Cleanup “Local Area Connections”
Often, because there are old hidden devices, Windows says your new devices are “Local Area Connection 3, 4, etc. Be sure to clean them up.
There is an automated tool for this on VM_Advanced.iso

Be sure that your HALs are updated.
If you went from a multi-processor environment to a single core, downgrade it.
If you went from a single core to a multi processor VM, upgrade the HAL.

Be sure that your disks are aligned.
See the Uberalign tool on VM_Advanced.iso.
A correctly aligned disk can yield a large increase in performance over a non-aligned disk.

Links

VMware Converter
http://www.vmware.com/products/converter/

Kendrick Coleman‘s VM_Advanced.iso
http://kendrickcoleman.com/images/documents/VM_Advanced.iso

Over on VMware blogs, they have released some real jewels lately. This post looks at the protocol purely from a vSphere perspective. It doesn’t cover performance because they state another team inside VMware does that, but this is a great post, so check it out. If you’re looking for performance documents and comparisons the links at the bottom are what you want.

http://blogs.vmware.com/vsphere/2012/02/storage-protocol-comparison-a-vsphere-perspective.html

Here are some VMware whitepapers which cover storage performance comparison:

•  vSphere 5.0 achieves 1 million iops (via FC)
•  http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf
•  http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

The Indy VMUG announced the date/location of the March meeting. I will be building a presentation and speaking about P2V Best Practices. See you there?

The next Indy VMUG meeting will be held at Harrison College on Thursday March 22nd 2012 from 5:30 to 9:00PM.

Location:
Harrison College
6300 Technology Center Drive
Indianapolis, IN 46278 <view map>

Agenda:
5:30 p.m. Sign-in and Networking
6:30 p.m. VKernel Presentation: Top vCenter Performance Metrics that Matter
7:30 p.m. Break and Networking
7:45 p.m. VMware Product Discussion and Q&A – Cameron Smith, VMware
8:15 p.m. P2V Best Practices – Ryan Birk, VCI (@ryanbirk | ryanbirk.com)
9:00 p.m. Discussion and Wrap-up

<Register Now>

Event sponsored by:

With vSphere 5, the VMware HA infrastructure has been completely changed.

In vSphere 5.0 the new HA agent is “FDM” (Fault Domain Manager) and replaces the old AAM agent (from EMC Legato).

Other notable changes:
The old Primary / Secondary node concept has been replaced by a new and more simple Master/Slave node concept
A new Datastore Heartbeat function as been added (HA Architecture Series – Datastore Heartbeating)
No dependency on DNS
Syslog functionality

See this slideshare for more information: