The Perfect VMware vSphere 5 Home Lab
So much power in so little space.
Over the years I have tweaked and built up many different ESXi whitebox “servers” for lab use. I wanted to share some experiences with you in a blog post and recommend some new hardware that will help you get a nice little home lab setup for ESXi 5.
I’ve bought vanilla Intel motherboards and have had a lot of success with those. The Intel network adapters seem to have the greatest compatibility overall. My most recent builds have been based on the Shuttle XPC platform. They work right out of the box and this post will focus primarily on those.
If you’re looking to build a hardware based VMware home lab, you will have the most flexibility overall. When you nest your lab inside VMware Workstation, there are some features that you will not be able to use like Fault Tolerance and EVC. If you hack around a bit you can get them to work though. I’ve just always went with a hardware lab so we will stick with that.
The ESXi 5 Host:
It’s recommended to have two hosts minimum so you can do all of the cool stuff. Without the second one, it will still work fine but what fun is it without HA/DRS and vMotion?!
Shuttle XPC SH67H3 - We will use this as our base system. It’s a compact system, supports new generation processors and up to 32GB memory. It is recommended to update to the newest BIOS on the Shuttle website. This will allow us to use the PCI Express slot for a NIC and not lock us into using it for a video card. There were early reports that old revisions of the BIOS would only allow you to use it for video card expansions. We want to have the option to use a dual NIC to bring the host up to a total of 3 NICs. The integrated adapter works fine out of the box with no modifications to the ESXi 5 install media. These machines have been running flawlessly for almost a year now in my lab.
Intel PRO/1000 Dual Port Server Adapter - This adapter works in the spare slot that the Shuttle has. Again, as of this writing it works flawlessly out of the box with the vanilla ESXi 5 installer. No modifications needed. Just be sure to flash the BIOS to the newest revision.
Your Choice: Intel Core i7 2600 Sandy Bridge Processor or Intel Core i5 2500 Processor - I went with the i7 processor but this is entirely your choice. If you’re on a tighter budget the i5 processor would save you some cash. I would recommend that, if you are on a budget, go with the i5 and buy two hosts vs. one host with an i7. You are almost always out of memory before CPU anyway. Be sure to buy the non K processors. (The ones linked are the ones you want in a virtualization lab.) K processors do not support Intel TXT, Intel VT-d and vPro. (Yes, Sandy Bridge works with Fault Tolerance.)
Your Choice: 8GB Corsair DDR3 (2 x 4GB) 1333 MHz or 16GB Corsair DDR3 (2 x 8GB) 1333 MHz - I’ve had a ton of success with Corsair memory. This is one of those things where you probably will want to shop around but this memory has been priced well and just works every time you buy it. Remember that the Shuttle system above can do a maximum of 32GB DDR3 across 4 total slots. So if it’s in your budget order up 2x of the 16GB kits and you’re set. I went with 16GB in each box.
Kingston Digital 8 GB USB 2.0 Flash Drive - I’ve used 4GB USB sticks before with no issue, but went with this 8GB stick on the last build. I am sure you have plenty of these things laying around. Any of them should do. I try to avoid the really cheap vendor ones given out at all the VMUGs/conferences etc since I want it to run relatively stable and not die within a weeks use.
This guy might be proud of his lab, but it’s time to upgrade.
The Storage Setup/Options:
Now we meet a crossroads. The stuff above gets you your basic host but we all know that a standalone host can only go so far. We have a couple options when it comes to storage. If you’re totally new to the homebrew ESXi lab then you will need to decide what path to take.
- The cheapest option: Find an old computer (you know you have one) and install openfiler or FreeNAS on it and share it out. This is the cheapest option, but may not perform great depending on the box/disk/network etc.
- No old computer to re-purpose? Then pick up a Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s- I’ve used these before but you can buy any simple and cheap SATA drive and install it inside of the Shuttle standalone host. Try to stay away from the green drives if you can. Especially if you’re putting them in a NAS. Then you could build a massive virtual machine on the host and share out the storage over iSCSI/NFS to the host(s). You can use openfiler or FreeNAS if you’d like inside of the virtual machine. Remember that in this case your VMs (disk files specifically) that you build will live inside of this new storage VM itself. Think of the movie Inception. Kind of like a dream within a dream.
- Buy a new NAS. I own a device called a Thecus N5200 Pro. I cannot recommend it. Thecus tries to say this is a great device but they never update the firmware. Avoid them if you can. I would recommend and have heard great reviews of Synology. A Synology DS411 NAS. It’s not the cheapest solution but the performance is great and Synology has impressed as of late. Throw in 4 of the Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s disks and you’ve got a winner! Kyle Ruddy has a recent post and review on his Synology DS411.
So you can see with storage in a VMware home lab you have a lot of options. I would recommend starting small and growing it later.
The Networking Setup:
In order to be as real world as possible in this home lab you’ll want to find a way to setup some VLANs and keep the traffic that your homelab generates off of your “Production” network.
I would recommend using a router based on dd-wrt and setup seperate VLANs for your storage, vMotion, etc. I use dd-wrt and have had success with it and pair it with a 24 port managed Trendnet switch. It works but I’ve also recently started playing with an HP 1810G-8 switch as well which is exactly what I need for my hosts.
Host Details:
Leave your home lab components in the comments below! Lets get a good thread going with hardware that’s supported on ESXi 5.
Hi Ryan,
Nice article. I run a pair of HP N36L’s and a QNAP 439 Pro II myself. It’s a modest setup, but lacks the grunt of your shuttle configuration.
One thing I thought your readers may also wish to check out if they are building home labs for VMware is the AutoLab. Alastair Cooke created this automated build system particularly for home / lab use. You can find it at labguides.com
Cheers,
Nick Marshall
Nick,
How ironic is it that last night I was on the AutoLab site and checking it out!
I am going to play with it soon and start mentioning it in class to students I think. Maybe do a review and post on here. It looks like a great idea. I’ve found that a lot of people want to start out with nested ESXi and have kind of a hybrid lab.
I’ve heard good things on the QNAPs as well. I bought by Thecus several years ago and wish I had went with something else.
Ryan
2600 is actually better in the case of virtualization as it supports vt-d where the 2600k doesn’t. The best option for you is not the best option for everyone.
K processors do not support Intel TXT, Intel VT-d and vPro.
Hi John, thanks for pointing that out! I am actually using the 2600 processor. From your comment, I just realized I put the wrong one up there. I’ve updated it to not throw anybody off.
I also wonder why you are not installing 32GB of RAM since RAM has become cheaper and you state that you will run out of RAM before CPU? I know we are not all made out of money, but these things could really kick butt with some more RAM chips in them.
Good question. I started building them up last summer right when these boxes came out. At the time I priced them out I saved a few bucks going with 16GB of memory knowing that I’d eventually upgrade to 32. I’ve had pretty good luck at 16 though for what I’m doing. You know, the birthday is coming up soon and the wife just asked me what I wanted. Hehehe
Assuming you had access to a copy of VMware Workstation, would one of these be enough if you went with the ‘inception’ approach? At least enough to give you practice with HA and DRS?
Rich,
It works well in Workstation 8. It will actually auto detect the ESXi media. You just want to make sure you go under the CPU section after you build the VM, give it two CPUs (install will fail as two cores is required minimum) and click the option that allows you to virtualize VT-d.
You might want also want to check out Nicks comment above. Autolab. http://www.labguides.com/autolab/ You can download some pre-built VMs ready to go inside a nested environment for use inside VMware Workstation.
I have a few hosts nested in my lab in both Workstation 8 and a standalone ESXi 5 host and they work well. I might try to do a new post that will go through setting up all the workstation details soon.
Ryan,
The pre-built VMs are shells that need other pieces downloaded. I am sure that Rich will be able to read it, but it isn’t quite as easy as just deploying the OVF/OVA as other work is involved.
Another good blog to read if you’re looking into a VMware Workstation 8 lab: http://boerlowie.wordpress.com/2011/11/30/building-the-ultimate-vsphere-lab-part-1-the-story/
Ryan, these seem pretty awesome! Was thinking about the “baby dragon” system I have heard a lot about to get some remote console connectivity so I could make my lab sit in my basement, but dang these are so much better on the wallet. I do love finding perfection.
Great site and write up.
The shuttle systems are superb, really good products and well built.
Considering the pricing scheme, if you had to create your lab again, would you consider the HP ML110 G7′s over your spec (even though they are limited at 16gb) ?
They are £230 after cashback inc vat, so you would be saving about £230 per host (roughly) [based on shuttle chassis and CPU]
Additionally, what is the noise like in your lab? Do the shuttle’s fans spin up extensively on certain work loads?
Many thanks,
G.
Gabi,
If I were to do it again I would probably stick with the Shuttles. I looked up the specs on those ML110 G7′s and from what it looks the Shuttles performance would be better (possibility of an i7, 32GB RAM, etc). Plus the small form factor is really nice!! If you could save significant money on the HP’s and are building your first lab, I would say go for those and buy a nicer NAS device!
The configuration I use in my lab makes the hosts themselves almost entirely silent. There is only a single case fan in the rear of the unit. The processor is cooled via a heat spreader that is connected to the rear fan. I thought it would be interesting at first when I built them but it’s worked out really well and the units have stayed cool.
I also have no spinning disk in my hosts since it’s all in my NAS so that makes them even quieter!
Ryan
@Ryan Birk
Hi Ryan,
Thank you for taking your time to reply.
The small form factor is certainly an added benefit, they are great and also the ability of the increase in RAM would be superb.
The price increase is certainly a lot more, but I guess it’s not wasted money.
I am also looking at using the SG300 Cisco Small Business L3 switch.
The prices for the CPU’s are quite a bit. Are you finding you can test all the functionality?
Having a NAS is a great way to test the enterprise features too, currently have an NVX, but looking to change to something else.
Excellent stuff,
Thanks,
Gabi.
Hi Ryan.
I have decided to give your lab design a go, also I have decided to blog the build and setup of my system.
http://innocentbystanders.wordpress.com/
Hoping it will help others in their endeavours.
Ryan – I was a New Horizons student back in the middle of July and taking that ESX course inspired me to build my own rig. I’m in the process of doing that now and due to cost I went with AMD vs Intel. My rig will not be nearly as robust as the one you have outlined, but still 6 cores, 16GB of RAM and I’ll throw about 5TB of disk at it.
I’m curious what you are using (or if you know of anyone using) for AV at the hypervisor level?
Thanks.
@Nick Marshall
Thanks for the heads up on Autolab. It looks pretty cool. Didn’t know it existed and I will definitely be checking it out.
I bought this exact same set up (with the exception of the nic) and I have been fighting with this thing for days. It came with the latest bios on the website (from May 2012). No matter what keyboard I plug in, the BIOS menu is unusable as hitting one key translates to about 40 on screen. Gave up on installing from USB because of that, installed a DVD drive, and ESXi 5.0 gets to that “Welcome to the VMWare ESXi 5.0 Installation” screen and basically stops there and won’t allow me to continue. I was hoping to use my Synology for the back end storage, but I think I still have some fighting to do. Oh well. I’ll figure something out.
Just wanted to post an update that I managed to catch a quick flash of an error in red text before it freezes on that “Welcome” install screen. I actually had to video tape and play it back to see it. Although fuzzy, it looks like it says todays date, my cpu, then something about logs being stored on persistent storage and consulting documentation to configure a scratch partition. Any one ever experience anything like this?
Hack, what version of the motherboard do you have by chance? I’m looking into this as well. I see Shuttle released a new version a short time ago. Mine is a v1 motherboard. I’d like to compare and see if the issues might be regarding that new motherboard version. See http://cinlortech.wordpress.com/2012/04/20/ivy-bridge-support-on-the-shuttle-sh67h3/
Ryan, if Hack’s H3 has the BIOS from May 2012 (v.202) then it is the V2 motherboard that supports Ivy Bridge.
Forgive my ignorance, but why is it better to nest storage in a big VM and share it back to the host instead of using the local storage directly?
I have set up a Shuttle SH67V7 with twin 1TB drives with ESXi 5 booting from a USB stick using the drives directly for virtual machines. They are not part of a RAID setup, In intend to use a separate NAS box for back-up of the important VM’s and data.
Again, why would it be better to assign both disks to a VM and share it back to the host? Would it be to support RAID, or are there other reasons?
Thank you in advance for explaining it …
John, it allows you to simulate a real SAN environment. If you use just local disk, you don’t get that feel/options. You can get block level disk and do what you want with it. It’s more useful if you’ve got more than 1 host however.
Have You tried VMware SiteSurvey Report ? Or is it possible You to try it?
My main problem is, that it reports, that my i7 2600 is not supported by FT!
CPU type Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz is not supported by FT.
I have not tried SiteSurvey before but the i7-2600 I use allows me to use FT. I use it on a weekly basis.
hey ryan,
will one shuttle with esxi work, say versus two shuttle’s? just ordered a synology nas and disks.
great info here
Yes, most definitely. You can put together a lab with one host, just remember you won’t be able to do any clustering features.
when money allows, i’ll add a second shuttle. i’m clearer on advantages of having 2 or more, i.e., clustering. thx
this i7-2600 3.4GHz supports Vt-d, yes? i believe so.
have you had luck passing graphics card to a VM? not sure what GPU would work well for this
Thank you for responding to my question. I see why it adds value when having more than one host. I am currently happy with one host with local discs plus a QNAP NAS.
I am investigating a new setup including vCenter to be able to use the web client for the newest features as it looks like vmware do not plan to maintain the vsphere client.
Is there a good setup including vCenter on one host. or should I invest in a second host?
the synology arrived. it now has one huge vol (regular files). for home labs do users make more use of mulitple iSCSI LUNS/Targets (Block Level) versus (regular files)?
i want to store movies/songs on part of this box and also have some VMs running off it. guess you cant have regular files and block level
You should be able to split it into different volumes so you can do that. You’ll probably need to delete the existing volume that it has and start new. Synology might also be able to re-size the existing volume as well. It should work great for that! Most people will use a mix of both iSCSI and NFS in their labs to do testing. You might want to consider using one of each!
Hey Ryan,
Just an FYI for you and your readers, Shuttle has their 1st NAS server coming out the end of this month. It’s an 8 TB, only 2 bays and diskless, but looks like a pretty nice system for $229.99. I have a write up on it on my blog cinlortech/wordpress.com titled Shuttle-rolls-out-its-first-low-energy-nas-server-OMNINAS-KD20.
Hey Ryan, may be a dumb question but with running the two ESX hosts in your lab what are you using as the VCenter, DNS, etc. A third system?
Not a dumb question at all. I have a VM running on one of the ESXi hosts that runs vCenter and another VM is a small little domain controller/DNS.
@Ryan Birk
Thanks! Purchasing all of the PC’s now. I was thinking I would need another system that would have to manage(VCenter) these since they are ESX hosts
No, just use the vSphere client to hit them directly and install the VMs, then add the hosts to vCenter. Works good! You could run them on another physical box but what fun is that?
My SH67H3 has an I7-3770, 32 GB Corsair RAM and a 256 GB SSD. After a week this drive wasn’t seen by ESXi so I moved my VMs to a 1TB iSCSI LUN hosted by my Synology DS410j NAS. I probably going to repurpose that SSD in my next work laptop. My next purchase is that dual NIC you have.
Other thoughts, I’ve experienced some general weirdness with the vSphere client, and am leaning (and learning) more of the CLI.
Thanks for your input on this build.
All, I wanted to post quickly. I got a v2 motherboard (one that supports Ivy Bridge) and it works just fine with ESXi 5. In fact, I installed it with the 5.1 media to an 8GB usb stick and 32GB of memory. Thanks for the recommendation on these machines Ryan. They’re great! I ALMOST want to run them in production.
Ryan, with an Ivy Bridge CPU, is there any benefit to moving up to the ‘built-for IB’ chipset (Comparing the SH67H3 with the SZ77R5, so the chipset comparison is H67 vs Z77; from what I’ve been able to dig up it means PCI-E 3.0, native USB 3.0, and native DDR3 speed moves from 1066/1333 to 1333/1600) for a $110 step-up per box. Also an (unneeded in my mind) 200W boost to the PSU (300W->500W) for this role.
Not sure if any of this will really benefit an ESXi stack, aside from the faster RAM. (Been an AMD guy for awhile, and haven’t followed the Intel chipset/CPU featureset)
Hi,
I am about to purchase a Shuttle XH61V Ivy Bridge, i will out an i7+16GB of mem and use it as a ESX 5.1 Lab, can anyone tell me if this setup will run ESXi out of the box.
Would very much appreciate your input!
Kind regards
Charles
Hey Charles, I’ve not tried it on that platform yet. The Ethernet ports are Realtek 8111E’s so those should probably work and it looks like it will support VT-d. Let us know how it all works out for you. I’m curious!
Hi,
First of all, thanks a lot for sharing this information !
I read from Jared’s post that SSD is not recognized. Is it possible to use an SSD with ESXi 5.1 ? I suppose it’s much better from a performance point of view ? I will have less drive space but this could be solved by using linked clones.
Best regards,
Wim
Yes, you can use SSD with 5.1. In fact ESXi 5 and up will recognize the disk and tag it as an SSD. I do exactly that when I am playing with View in my home lab actually. Linked Clones on SSD!
Ryan,
Pretty new to this and have a question. I have the same setup with two Shuttle PC’s running ESXi5.1 and a DS1512+. With three NIC’s in each Shuttle, how should I configure them for management, vMotion and or FT. I have vCenter also and was wondering the best way to setup my lab. Do I dedicate one for management, one for vMotion and the other for FT or combine some?
Thanks,
Scott
Scott, do you have the ability to use some sort of VLAN on your lab network? What I would do is setup VLANs and run them all on the same cables so you’ve got a bit more flexibility. @Scott Fella
I do have the ability to setup vlans, but how should I break it apart? Thats what I’m confused on. Should I have management only on one nic, vmotion on another and FT on the 3rd nic?
@Ryan Birk
FYI. Running the following configuration now :
Shuttle SH67H3, Intel Core i7 3770, 2 x Seagate ST1000DM003 Barracuda 1TB, 4 x Kingston 8 GB DIMM DDR3 1600MHz CL11
The support list for CPU, memory,… on the Shuttle site is also usefull : http://us.shuttle.com/barebone/Models/SH67H3.html
Best regards,
Wim
That support list is great. I love these little boxes! Thanks, Wim.
Has anyone had experience with the Shuttle SX79R5? This shuttle is based off the X79 chipset. I would think this would be a serious power house for a ESXi lab. Here is a link to the shuttle.
http://global.shuttle.com/main/productsSpec?productId=1607
What do you guys think? Is there any reason I should not go this route if I want to create a home lab. I’m thinking of buying two of them. Thoughts.
Has anyone got the SX79R5 to work with VMWare ESXi or Even ESX?
Hey Ryan, wicked post, just built my 32gb, i7 3770S shuttle. Was just wondering, have any of you guys been able to get esxi provide temperature information? From what I understand it’s not supported by the hardware and that’s (one reason) why HP/Dell, etc provide custom builds…it’s kind of annoying having to reset and jump into BIOS.
@Nav For these boxes, I have not been able to pull temperature. Maybe we need to talk Shuttle into providing us some CIM providers.
Haha.
@Mr Rex I have not, if you go that route, I’d love to have you do a guest post on the blog and send us some feedback. Either you or @mrrob2g . I don’t see a reason they wouldn’t work. Keep me posted!
@Scott Fella I enable all the VLANs across all the cables and then setup each port group to override the vswitch. So with 3 NICs in total, I use management on vmnic0 and then vmnic1/2 as standby. For vMotion I do vmnic1 primary and then vmnic0/2 standby. Setup one big vSwitch and then edit each port group itself to override the switch. Hope that makes more sense.
Excellent guide Ryan. Some interesting comments too. Think I will embark on a project like this myself.
Ryan, this is great post! I’m interested to know how you enabled FT as it didn’t pass site survey and i7-2600 isn’t on the official FT hcl, thanks.
@Ryan Birk
Haha, I came to the same conclusion. I’m still looking for other solutions workarounds (other than resetting and entering bios). Would be nice to, at least, access bios whilst esxi is running :/
Excellent Guide, Mr Birk !! What do You have for power consumption on this Shuttle boxes? I think it would be nice to know how mutch “juice” the need running WM
@Mr Rex
Mr Rex i had the same idea as you and have a Shuttle SX79R5 built it up as ESXi home virtual. Installed esxi 5.1 no problems added to vcenter, everything going well….so you think )) transferring large amounts of data eg migrating vm’s crashes and shuts down the management network to fix reboot and start again and it shut it down again.
So thought i would put Xen on it and wow same issue…
begrudgingly i put hyper vomit 2012 on it and is currently running, only problem is there arent any supported drivers for 2012…sounds like vista all over again….
If you have them already would be interested on your experiences but if you havent got them yet then i would advise against it.
Happy days
Great write up, I noticed that several talked about using the new SZ77R5 or SX79R5. My main issue is I don’t believe either has AMT (vPro) Ip-KVM abilities? I would love to use one of these system but require remote power and remote KVM access. Any thought on this?
Hello Ryan,
I am planing to build my first home lab.
But I have a few questions.
I want to use the following parts:
-Shuttle XPC Barebone SH67H7
-Intel Core i7-3770, CPU “Ivy Bridge”
-2x Corsair DIMM 16 GB DDR3-1333 Kit, CMX16GX3M2A1333C9, XMS
-Intel PRO/1000 PT Dual Port Server Adapter or Intel PRO/1000 PT Quad Port Server Adapter
-Sandisk Cruzer Fit 8 GB, USB-Stick
-Four-drive 2.5″ SATA cage Icy Dock MB994SP-4SB-1
-Intel 520series 2,5″ SSD 240 GB
-HTS721010A9E630 HDD 2,5″ SATA3 1000GB/7200RPM HGST
On the Sandisk Cruzer Fit Usb-Stick i want to install the Esxi 5.1.
The SSD is planed to used as datastore for iso-images and 2 or 3 VM’s.
The harddisk is used as second datastore and all the other VM’s.
At the moment I don’t know if I should buy a dual port or quad port Intel
NIC. What do you suggest? How many cards can I install in the shuttle?
1 is onboard + 2 NIC’s in addition (PCIex) right?
In another VM-Forum I have read that someone installed 3 NIC’s in the Shuttle SH67H3. Makes a total of 4.
- NIC: Realtek RTL 8111E (Onboard)
- NIC: Intel Pro 1000 PT Dual Port (PCI-e 1x)
- NIC: Intel Pro 1000 PT Dual Port (PCI-e 16x)
- NIC: Intel Pro 1000 PT Single Port (PCI-e 1x)
How did he install the 3 NIC’s?
Specification SH67H3 from shuttle website
Expansion Slots
1x PCI-Express x16 v2.0 slot (PEG, for graphics cards only)
1x PCI-Express x1 v2.0 slot
1x Mini-PCI-Express x1v2.0 slot (for the optional WLAN module)
Supports Dual-slot (double-width) graphics cards -
in this case the second PCI-Express slot will be occupied.
A 6 pin power connector is available for the graphics card.
So I can only use the 1x PCI-Express x1 v2.0 slot.
Right?
I’m a bit confused… How did he install the 3 Intel cards?
What do you think about my configuration? Is it ok?
Hope to hear from you soon to get some answers to my questions.
Thank you.
best regards
Thomas
Looking for a little help I have a shuttle setup for the last year running with no problems until the other day I had a power outage. I boot from a USB now I get fatal error 33 inconsistent data. I recreated a new USB with the same iso but now after passing all the way to the first yellow screens I lose the keyboard. So I get stuck reboot try again no good. Any advice??
@Hack
Hack, were you ever able to get a fix for the keyboard issue you had, looks like i am having the exact same issue. If you dont mind sharing with me…
Thanks in advance
What BIOS version are you running?
I’ll have to double check once i get home but if i am not mistaken it is version 1.15 build date 11/24/2011.
Hi Ryan
I just installed an esxi5 on top of my sh67h3
I also configure a raid1 volume with my both sata internal disk and internal raid controller
But when I boot my esxi5 I still see my both sata disk
Normally I should see only my raid1 volume0 ?
Many thanks for your help
I’ve not tried anything as RAID1 on these boxes. I’m not sure that ESXi would have a driver for the RAID controller actually.