The Perfect VMware vSphere 5 Home Lab

So much power in so little space.

Update: 10/2/2013: For those installing ESXi 5.5, be sure to read my post about building an iso that includes the Realtek drivers. That post will help you if you’re building this particular lab on 5.5 instead of 5.1. 

Over the years I have tweaked and built up many different ESXi whitebox “servers” for lab use. I wanted to share some experiences with you in a blog post and recommend some new hardware that will help you get a nice little home lab setup for ESXi 5.

I’ve bought vanilla Intel motherboards and have had a lot of success with those. The Intel network adapters seem to have the greatest compatibility overall. My most recent builds have been based on the Shuttle XPC platform. They work right out of the box and this post will focus primarily on those.

If you’re looking to build a hardware based VMware home lab, you will have the most flexibility overall. When you nest your lab inside VMware Workstation, there are some features that you will not be able to use like Fault Tolerance and EVC. If you hack around a bit you can get them to work though. I’ve just always went with a hardware lab so we will stick with that.

The ESXi 5.x Host:

It’s recommended to have two hosts minimum so you can do all of the cool stuff. Without the second one, it will still work fine but what fun is it without HA/DRS and vMotion?!

Shuttle XPC SH67H3 – We will use this as our base system. It’s a compact system, supports new generation processors and up to 32GB memory. It is recommended to update to the newest BIOS on the Shuttle website. This will allow us to use the PCI Express slot for a NIC and not lock us into using it for a video card. There were early reports that old revisions of the BIOS would only allow you to use it for video card expansions. We want to have the option to use a dual NIC to bring the host up to a total of 3 NICs. The integrated adapter works fine out of the box with no modifications to the ESXi 5 install media. These machines have been running flawlessly for almost a year now in my lab.

Intel PRO/1000 Dual Port Server Adapter – This adapter works in the spare slot that the Shuttle has. Again, as of this writing it works flawlessly out of the box with the vanilla ESXi 5 installer. No modifications needed. Just be sure to flash the BIOS to the newest revision. (update 9/1/2013: All systems Shuttle has shipped within the last year don’t require or need the BIOS update!)

Your Choice: Intel Core i7 2600 Sandy Bridge Processor or Intel Core i5 2500 Processor – I went with the i7 processor but this is entirely your choice. If you’re on a tighter budget the i5 processor would save you some cash. I would recommend that, if you are on a budget, go with the i5 and buy two hosts vs. one host with an i7. You are almost always out of memory before CPU anyway. Be sure to buy the non K processors. (The ones linked are the ones you want in a virtualization lab.) K processors do not support Intel TXT, Intel VT-d and vPro and yes, Sandy Bridge works with Fault Tolerance. The new v2 units also work with Ivy Bridge, so you might want to consider that as well!

Your Choice: 8GB Corsair DDR3 (2 x 4GB) 1333 MHz or 16GB Corsair DDR3 (2 x 8GB) 1333 MHz – I’ve had a ton of success with Corsair memory. This is one of those things where you probably will want to shop around but this memory has been priced well and just works every time you buy it. Remember that the Shuttle system above can do a maximum of 32GB DDR3 across 4 total slots. So if it’s in your budget order up 2x of the 16GB kits and you’re set. I went with 16GB in each box but I’ve tested it with 32 in each and it works great.

Kingston Digital 8 GB USB 2.0 Flash Drive – I’ve used 4GB USB sticks before with no issue, but went with this 8GB stick on the last build. I am sure you have plenty of these things laying around. Any of them should do. I try to avoid the really cheap vendor ones given out at all the VMUGs/conferences etc since I want it to run relatively stable and not die within a weeks use.

This guy might be proud of his lab, but it’s time to upgrade.

The Storage Setup/Options:

Now we meet a crossroads. The stuff above gets you your basic host but we all know that a standalone host can only go so far. We have a couple options when it comes to storage. If you’re totally new to the homebrew ESXi lab then you will need to decide what path to take.

  • The cheapest option: Find an old computer (you know you have one) and install openfiler or FreeNAS on it and share it out. This is the cheapest option, but may not perform great depending on the box/disk/network etc.
  • No old computer to re-purpose? Then pick up a Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s– I’ve used these before but you can buy any simple and cheap SATA drive and install it inside of the Shuttle standalone host. Try to stay away from the green drives if you can. Especially if you’re putting them in a NAS. Then you could build a massive virtual machine on the host and share out the storage over iSCSI/NFS to the host(s). You can use openfiler or FreeNAS if you’d like inside of the virtual machine. Remember that in this case your VMs (disk files specifically) that you build will live inside of this new storage VM itself. Think of the movie Inception. Kind of like a dream within a dream.
  • Buy a new NAS. I own a device called a Thecus N5200 Pro. I cannot recommend it. Thecus tries to say this is a great device but they never update the firmware. Avoid them if you can. I would recommend and have heard great reviews of Synology. A Synology DS411 NAS. It’s not the cheapest solution but the performance is great and Synology has impressed as of late. Throw in 4 of the Seagate Barracuda 7200 1 TB 7200RPM SATA 6 Gb/s disks and you’ve got a winner! Kyle Ruddy has a recent post and review on his Synology DS411.

So you can see with storage in a VMware home lab you have a lot of options. I would recommend starting small and growing it later.

The Networking Setup:

In order to be as real world as possible in this home lab you’ll want to find a way to setup some VLANs and keep the traffic that your homelab generates off of your “Production” network.

I would recommend using a router based on dd-wrt and setup seperate VLANs for your storage, vMotion, etc. I use dd-wrt and have had success with it and pair it with a 24 port managed Trendnet switch. It works but I’ve also recently started playing with an HP 1810G-8 switch as well which is exactly what I need for my hosts.

Yes, you can run ESXi 5.5! Be sure to read my post about adding the Realtek drivers in!

SH67H3-5.5

SH67H3-5.5-nics-viewSH67H3-5.5-dcui-view

This entry was posted in Home Lab, vSphere. Bookmark the permalink.

120 Responses to The Perfect VMware vSphere 5 Home Lab

  1. Hi Ryan,

    Nice article. I run a pair of HP N36L’s and a QNAP 439 Pro II myself. It’s a modest setup, but lacks the grunt of your shuttle configuration.

    One thing I thought your readers may also wish to check out if they are building home labs for VMware is the AutoLab. Alastair Cooke created this automated build system particularly for home / lab use. You can find it at labguides.com

    Cheers,
    Nick Marshall

  2. Ryan Birk says:

    Nick,

    How ironic is it that last night I was on the AutoLab site and checking it out!

    I am going to play with it soon and start mentioning it in class to students I think. Maybe do a review and post on here. It looks like a great idea. I’ve found that a lot of people want to start out with nested ESXi and have kind of a hybrid lab.

    I’ve heard good things on the QNAPs as well. I bought by Thecus several years ago and wish I had went with something else.

    Ryan

  3. John Roberts says:

    2600 is actually better in the case of virtualization as it supports vt-d where the 2600k doesn’t. The best option for you is not the best option for everyone.

    K processors do not support Intel TXT, Intel VT-d and vPro.

  4. Ryan Birk says:

    Hi John, thanks for pointing that out! I am actually using the 2600 processor. From your comment, I just realized I put the wrong one up there. I’ve updated it to not throw anybody off. 🙂

  5. John Roberts says:

    I also wonder why you are not installing 32GB of RAM since RAM has become cheaper and you state that you will run out of RAM before CPU? I know we are not all made out of money, but these things could really kick butt with some more RAM chips in them.

  6. Ryan Birk says:

    Good question. I started building them up last summer right when these boxes came out. At the time I priced them out I saved a few bucks going with 16GB of memory knowing that I’d eventually upgrade to 32. I’ve had pretty good luck at 16 though for what I’m doing. You know, the birthday is coming up soon and the wife just asked me what I wanted. Hehehe 🙂

  7. Rich says:

    Assuming you had access to a copy of VMware Workstation, would one of these be enough if you went with the ‘inception’ approach? At least enough to give you practice with HA and DRS?

  8. Ryan Birk says:

    Rich,

    It works well in Workstation 8. It will actually auto detect the ESXi media. You just want to make sure you go under the CPU section after you build the VM, give it two CPUs (install will fail as two cores is required minimum) and click the option that allows you to virtualize VT-d.

    You might want also want to check out Nicks comment above. Autolab. http://www.labguides.com/autolab/ You can download some pre-built VMs ready to go inside a nested environment for use inside VMware Workstation.

    I have a few hosts nested in my lab in both Workstation 8 and a standalone ESXi 5 host and they work well. I might try to do a new post that will go through setting up all the workstation details soon.

  9. Pingback: Ryan Birk – Virtual Insanity » VCP-5 Exam Thoughts

  10. Ryan Birk says:

    Another good blog to read if you’re looking into a VMware Workstation 8 lab: http://boerlowie.wordpress.com/2011/11/30/building-the-ultimate-vsphere-lab-part-1-the-story/

  11. Steven Smith says:

    Ryan, these seem pretty awesome! Was thinking about the “baby dragon” system I have heard a lot about to get some remote console connectivity so I could make my lab sit in my basement, but dang these are so much better on the wallet. I do love finding perfection.

  12. Gabi says:

    Great site and write up.

    The shuttle systems are superb, really good products and well built.

    Considering the pricing scheme, if you had to create your lab again, would you consider the HP ML110 G7’s over your spec (even though they are limited at 16gb) ?

    They are £230 after cashback inc vat, so you would be saving about £230 per host (roughly) [based on shuttle chassis and CPU]

    Additionally, what is the noise like in your lab? Do the shuttle’s fans spin up extensively on certain work loads?

    Many thanks,

    G.

  13. Ryan Birk says:

    Gabi,

    If I were to do it again I would probably stick with the Shuttles. I looked up the specs on those ML110 G7’s and from what it looks the Shuttles performance would be better (possibility of an i7, 32GB RAM, etc). Plus the small form factor is really nice!! If you could save significant money on the HP’s and are building your first lab, I would say go for those and buy a nicer NAS device!

    The configuration I use in my lab makes the hosts themselves almost entirely silent. There is only a single case fan in the rear of the unit. The processor is cooled via a heat spreader that is connected to the rear fan. I thought it would be interesting at first when I built them but it’s worked out really well and the units have stayed cool.

    I also have no spinning disk in my hosts since it’s all in my NAS so that makes them even quieter!

    Ryan

  14. Gabi says:

    @Ryan Birk

    Hi Ryan,

    Thank you for taking your time to reply.

    The small form factor is certainly an added benefit, they are great and also the ability of the increase in RAM would be superb.

    The price increase is certainly a lot more, but I guess it’s not wasted money.

    I am also looking at using the SG300 Cisco Small Business L3 switch.

    The prices for the CPU’s are quite a bit. Are you finding you can test all the functionality?

    Having a NAS is a great way to test the enterprise features too, currently have an NVX, but looking to change to something else.

    Excellent stuff,

    Thanks,

    Gabi.

  15. Ozy says:

    Hi Ryan.

    I have decided to give your lab design a go, also I have decided to blog the build and setup of my system.

    http://innocentbystanders.wordpress.com/

    Hoping it will help others in their endeavours.

  16. Chris McQ says:

    Ryan – I was a New Horizons student back in the middle of July and taking that ESX course inspired me to build my own rig. I’m in the process of doing that now and due to cost I went with AMD vs Intel. My rig will not be nearly as robust as the one you have outlined, but still 6 cores, 16GB of RAM and I’ll throw about 5TB of disk at it.

    I’m curious what you are using (or if you know of anyone using) for AV at the hypervisor level?

    Thanks.

  17. Chris McQ says:

    @Nick Marshall
    Thanks for the heads up on Autolab. It looks pretty cool. Didn’t know it existed and I will definitely be checking it out.

  18. Hack says:

    I bought this exact same set up (with the exception of the nic) and I have been fighting with this thing for days. It came with the latest bios on the website (from May 2012). No matter what keyboard I plug in, the BIOS menu is unusable as hitting one key translates to about 40 on screen. Gave up on installing from USB because of that, installed a DVD drive, and ESXi 5.0 gets to that “Welcome to the VMWare ESXi 5.0 Installation” screen and basically stops there and won’t allow me to continue. I was hoping to use my Synology for the back end storage, but I think I still have some fighting to do. Oh well. I’ll figure something out. 🙂

  19. Hack says:

    Just wanted to post an update that I managed to catch a quick flash of an error in red text before it freezes on that “Welcome” install screen. I actually had to video tape and play it back to see it. Although fuzzy, it looks like it says todays date, my cpu, then something about logs being stored on persistent storage and consulting documentation to configure a scratch partition. Any one ever experience anything like this?

  20. Ryan Birk says:

    Hack, what version of the motherboard do you have by chance? I’m looking into this as well. I see Shuttle released a new version a short time ago. Mine is a v1 motherboard. I’d like to compare and see if the issues might be regarding that new motherboard version. See http://cinlortech.wordpress.com/2012/04/20/ivy-bridge-support-on-the-shuttle-sh67h3/

  21. cinlortech says:

    Ryan, if Hack’s H3 has the BIOS from May 2012 (v.202) then it is the V2 motherboard that supports Ivy Bridge.

  22. Pingback: Shuttle SH67H3 ESXi 5 « CinlorTech's Blog

  23. John says:

    Forgive my ignorance, but why is it better to nest storage in a big VM and share it back to the host instead of using the local storage directly?

    I have set up a Shuttle SH67V7 with twin 1TB drives with ESXi 5 booting from a USB stick using the drives directly for virtual machines. They are not part of a RAID setup, In intend to use a separate NAS box for back-up of the important VM’s and data.

    Again, why would it be better to assign both disks to a VM and share it back to the host? Would it be to support RAID, or are there other reasons?

    Thank you in advance for explaining it …

    • Ryan Birk says:

      John, it allows you to simulate a real SAN environment. If you use just local disk, you don’t get that feel/options. You can get block level disk and do what you want with it. It’s more useful if you’ve got more than 1 host however.

  24. Codera says:

    Have You tried VMware SiteSurvey Report ? Or is it possible You to try it?
    My main problem is, that it reports, that my i7 2600 is not supported by FT!

    CPU type Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz is not supported by FT.

  25. bw says:

    hey ryan,

    will one shuttle with esxi work, say versus two shuttle’s? just ordered a synology nas and disks.

    great info here

    • Ryan Birk says:

      Yes, most definitely. You can put together a lab with one host, just remember you won’t be able to do any clustering features.

  26. bw says:

    when money allows, i’ll add a second shuttle. i’m clearer on advantages of having 2 or more, i.e., clustering. thx

    this i7-2600 3.4GHz supports Vt-d, yes? i believe so.

    have you had luck passing graphics card to a VM? not sure what GPU would work well for this

  27. John says:

    Thank you for responding to my question. I see why it adds value when having more than one host. I am currently happy with one host with local discs plus a QNAP NAS.

    I am investigating a new setup including vCenter to be able to use the web client for the newest features as it looks like vmware do not plan to maintain the vsphere client.

    Is there a good setup including vCenter on one host. or should I invest in a second host?

  28. bw says:

    Ryan Birk :
    Yes, most definitely. You can put together a lab with one host, just remember you won’t be able to do any clustering features.

    the synology arrived. it now has one huge vol (regular files). for home labs do users make more use of mulitple iSCSI LUNS/Targets (Block Level) versus (regular files)?

    i want to store movies/songs on part of this box and also have some VMs running off it. guess you cant have regular files and block level

    • Ryan Birk says:

      You should be able to split it into different volumes so you can do that. You’ll probably need to delete the existing volume that it has and start new. Synology might also be able to re-size the existing volume as well. It should work great for that! Most people will use a mix of both iSCSI and NFS in their labs to do testing. You might want to consider using one of each!

  29. cinlortech says:

    Hey Ryan,

    Just an FYI for you and your readers, Shuttle has their 1st NAS server coming out the end of this month. It’s an 8 TB, only 2 bays and diskless, but looks like a pretty nice system for $229.99. I have a write up on it on my blog cinlortech/wordpress.com titled Shuttle-rolls-out-its-first-low-energy-nas-server-OMNINAS-KD20.

  30. KW says:

    Hey Ryan, may be a dumb question but with running the two ESX hosts in your lab what are you using as the VCenter, DNS, etc. A third system?

    • Ryan Birk says:

      Not a dumb question at all. I have a VM running on one of the ESXi hosts that runs vCenter and another VM is a small little domain controller/DNS.

  31. KW says:

    @Ryan Birk
    Thanks! Purchasing all of the PC’s now. I was thinking I would need another system that would have to manage(VCenter) these since they are ESX hosts

    • Ryan Birk says:

      No, just use the vSphere client to hit them directly and install the VMs, then add the hosts to vCenter. Works good! You could run them on another physical box but what fun is that? 🙂

  32. Jared says:

    My SH67H3 has an I7-3770, 32 GB Corsair RAM and a 256 GB SSD. After a week this drive wasn’t seen by ESXi so I moved my VMs to a 1TB iSCSI LUN hosted by my Synology DS410j NAS. I probably going to repurpose that SSD in my next work laptop. My next purchase is that dual NIC you have.

    Other thoughts, I’ve experienced some general weirdness with the vSphere client, and am leaning (and learning) more of the CLI.

    Thanks for your input on this build.

  33. Ron says:

    All, I wanted to post quickly. I got a v2 motherboard (one that supports Ivy Bridge) and it works just fine with ESXi 5. In fact, I installed it with the 5.1 media to an 8GB usb stick and 32GB of memory. Thanks for the recommendation on these machines Ryan. They’re great! I ALMOST want to run them in production.

  34. Ryan, with an Ivy Bridge CPU, is there any benefit to moving up to the ‘built-for IB’ chipset (Comparing the SH67H3 with the SZ77R5, so the chipset comparison is H67 vs Z77; from what I’ve been able to dig up it means PCI-E 3.0, native USB 3.0, and native DDR3 speed moves from 1066/1333 to 1333/1600) for a $110 step-up per box. Also an (unneeded in my mind) 200W boost to the PSU (300W->500W) for this role.

    Not sure if any of this will really benefit an ESXi stack, aside from the faster RAM. (Been an AMD guy for awhile, and haven’t followed the Intel chipset/CPU featureset)

  35. Pingback: Ryan Birk – Virtual Insanity » Reformatting an ESXi USB Stick

  36. Charles (X3mGroup) says:

    Hi,

    I am about to purchase a Shuttle XH61V Ivy Bridge, i will out an i7+16GB of mem and use it as a ESX 5.1 Lab, can anyone tell me if this setup will run ESXi out of the box.

    Would very much appreciate your input!

    Kind regards
    Charles

    • Ryan Birk says:

      Hey Charles, I’ve not tried it on that platform yet. The Ethernet ports are Realtek 8111E’s so those should probably work and it looks like it will support VT-d. Let us know how it all works out for you. I’m curious!

  37. Pingback: Ryan Birk – Virtual Insanity » The 2013 $350 ryanbirk.com vBook Giveaway

  38. Wim Kestens says:

    Hi,

    First of all, thanks a lot for sharing this information !
    I read from Jared’s post that SSD is not recognized. Is it possible to use an SSD with ESXi 5.1 ? I suppose it’s much better from a performance point of view ? I will have less drive space but this could be solved by using linked clones.

    Best regards,

    Wim

    • Ryan Birk says:

      Yes, you can use SSD with 5.1. In fact ESXi 5 and up will recognize the disk and tag it as an SSD. I do exactly that when I am playing with View in my home lab actually. Linked Clones on SSD!

  39. Scott Fella says:

    Ryan,

    Pretty new to this and have a question. I have the same setup with two Shuttle PC’s running ESXi5.1 and a DS1512+. With three NIC’s in each Shuttle, how should I configure them for management, vMotion and or FT. I have vCenter also and was wondering the best way to setup my lab. Do I dedicate one for management, one for vMotion and the other for FT or combine some?

    Thanks,

    Scott

  40. Ryan Birk says:

    Scott, do you have the ability to use some sort of VLAN on your lab network? What I would do is setup VLANs and run them all on the same cables so you’ve got a bit more flexibility. @Scott Fella

  41. Scott Fella says:

    I do have the ability to setup vlans, but how should I break it apart? Thats what I’m confused on. Should I have management only on one nic, vmotion on another and FT on the 3rd nic?

    @Ryan Birk

  42. Wim Kestens says:

    FYI. Running the following configuration now :

    Shuttle SH67H3, Intel Core i7 3770, 2 x Seagate ST1000DM003 Barracuda 1TB, 4 x Kingston 8 GB DIMM DDR3 1600MHz CL11

    The support list for CPU, memory,… on the Shuttle site is also usefull : http://us.shuttle.com/barebone/Models/SH67H3.html

    Best regards,

    Wim

  43. Mr Rex says:

    Has anyone had experience with the Shuttle SX79R5? This shuttle is based off the X79 chipset. I would think this would be a serious power house for a ESXi lab. Here is a link to the shuttle.

    http://global.shuttle.com/main/productsSpec?productId=1607

    What do you guys think? Is there any reason I should not go this route if I want to create a home lab. I’m thinking of buying two of them. Thoughts.

  44. mrrob2g says:

    Has anyone got the SX79R5 to work with VMWare ESXi or Even ESX?

  45. Nav says:

    Hey Ryan, wicked post, just built my 32gb, i7 3770S shuttle. Was just wondering, have any of you guys been able to get esxi provide temperature information? From what I understand it’s not supported by the hardware and that’s (one reason) why HP/Dell, etc provide custom builds…it’s kind of annoying having to reset and jump into BIOS.

  46. Ryan Birk says:

    @Nav For these boxes, I have not been able to pull temperature. Maybe we need to talk Shuttle into providing us some CIM providers. 🙂 Haha.

  47. Ryan Birk says:

    @Mr Rex I have not, if you go that route, I’d love to have you do a guest post on the blog and send us some feedback. Either you or @mrrob2g . I don’t see a reason they wouldn’t work. Keep me posted!

  48. Ryan Birk says:

    @Scott Fella I enable all the VLANs across all the cables and then setup each port group to override the vswitch. So with 3 NICs in total, I use management on vmnic0 and then vmnic1/2 as standby. For vMotion I do vmnic1 primary and then vmnic0/2 standby. Setup one big vSwitch and then edit each port group itself to override the switch. Hope that makes more sense.

  49. Pingback: vSphere 5.x Whitebox Home Lab | Jason Gass Tech Dashboard

  50. Keir Whitlock says:

    Excellent guide Ryan. Some interesting comments too. Think I will embark on a project like this myself.

  51. William Li says:

    Ryan, this is great post! I’m interested to know how you enabled FT as it didn’t pass site survey and i7-2600 isn’t on the official FT hcl, thanks.

  52. Nav says:

    @Ryan Birk
    Haha, I came to the same conclusion. I’m still looking for other solutions workarounds (other than resetting and entering bios). Would be nice to, at least, access bios whilst esxi is running :/

  53. CaSC says:

    Excellent Guide, Mr Birk !! What do You have for power consumption on this Shuttle boxes? I think it would be nice to know how mutch “juice” the need running WM

  54. Hills says:

    @Mr Rex

    Mr Rex i had the same idea as you and have a Shuttle SX79R5 built it up as ESXi home virtual. Installed esxi 5.1 no problems added to vcenter, everything going well….so you think )) transferring large amounts of data eg migrating vm’s crashes and shuts down the management network to fix reboot and start again and it shut it down again.
    So thought i would put Xen on it and wow same issue…
    begrudgingly i put hyper vomit 2012 on it and is currently running, only problem is there arent any supported drivers for 2012…sounds like vista all over again….
    If you have them already would be interested on your experiences but if you havent got them yet then i would advise against it.
    Happy days 🙂

  55. ben says:

    Great write up, I noticed that several talked about using the new SZ77R5 or SX79R5. My main issue is I don’t believe either has AMT (vPro) Ip-KVM abilities? I would love to use one of these system but require remote power and remote KVM access. Any thought on this?

  56. Thomas says:

    Hello Ryan,

    I am planing to build my first home lab.
    But I have a few questions.

    I want to use the following parts:

    -Shuttle XPC Barebone SH67H7
    -Intel Core i7-3770, CPU “Ivy Bridge”
    -2x Corsair DIMM 16 GB DDR3-1333 Kit, CMX16GX3M2A1333C9, XMS
    -Intel PRO/1000 PT Dual Port Server Adapter or Intel PRO/1000 PT Quad Port Server Adapter
    -Sandisk Cruzer Fit 8 GB, USB-Stick
    -Four-drive 2.5″ SATA cage Icy Dock MB994SP-4SB-1
    -Intel 520series 2,5″ SSD 240 GB
    -HTS721010A9E630 HDD 2,5″ SATA3 1000GB/7200RPM HGST

    On the Sandisk Cruzer Fit Usb-Stick i want to install the Esxi 5.1.
    The SSD is planed to used as datastore for iso-images and 2 or 3 VM’s.
    The harddisk is used as second datastore and all the other VM’s.

    At the moment I don’t know if I should buy a dual port or quad port Intel
    NIC. What do you suggest? How many cards can I install in the shuttle?
    1 is onboard + 2 NIC’s in addition (PCIex) right?

    In another VM-Forum I have read that someone installed 3 NIC’s in the Shuttle SH67H3. Makes a total of 4.

    – NIC: Realtek RTL 8111E (Onboard)
    – NIC: Intel Pro 1000 PT Dual Port (PCI-e 1x)
    – NIC: Intel Pro 1000 PT Dual Port (PCI-e 16x)
    – NIC: Intel Pro 1000 PT Single Port (PCI-e 1x)

    How did he install the 3 NIC’s?

    Specification SH67H3 from shuttle website

    Expansion Slots
    1x PCI-Express x16 v2.0 slot (PEG, for graphics cards only)
    1x PCI-Express x1 v2.0 slot
    1x Mini-PCI-Express x1v2.0 slot (for the optional WLAN module)
    Supports Dual-slot (double-width) graphics cards –
    in this case the second PCI-Express slot will be occupied.
    A 6 pin power connector is available for the graphics card.

    So I can only use the 1x PCI-Express x1 v2.0 slot.
    Right?

    I’m a bit confused… How did he install the 3 Intel cards?

    What do you think about my configuration? Is it ok?

    Hope to hear from you soon to get some answers to my questions.

    Thank you.

    best regards
    Thomas

  57. Express says:

    Looking for a little help I have a shuttle setup for the last year running with no problems until the other day I had a power outage. I boot from a USB now I get fatal error 33 inconsistent data. I recreated a new USB with the same iso but now after passing all the way to the first yellow screens I lose the keyboard. So I get stuck reboot try again no good. Any advice??

  58. Express says:

    @Hack
    Hack, were you ever able to get a fix for the keyboard issue you had, looks like i am having the exact same issue. If you dont mind sharing with me…

    Thanks in advance

  59. Express says:

    I’ll have to double check once i get home but if i am not mistaken it is version 1.15 build date 11/24/2011.

  60. birb says:

    Hi Ryan

    I just installed an esxi5 on top of my sh67h3

    I also configure a raid1 volume with my both sata internal disk and internal raid controller

    But when I boot my esxi5 I still see my both sata disk

    Normally I should see only my raid1 volume0 ?

    Many thanks for your help

    • Ryan Birk says:

      I’ve not tried anything as RAID1 on these boxes. I’m not sure that ESXi would have a driver for the RAID controller actually.

  61. Greg Porter says:

    ESXi Installer and USB 3 ports:

    I built a esxi server using the SH67H3 v2 BIOS 2.02 with a i7-3770. I had a heck of a time getting it to install on a flash drive. The installer would boot but wouldn’t recognize my flash drive. Finally tried doing the install in vmware workstation which would boot but the nic wouldn’t work. It was loading the r8168 driver but it would not work. I played around with the USB settings in the BIOS but still no joy. I finally realized the issue was with the USB 3 drivers. Once I put the flash in one of the USB 2 ports the installer recognized my flash drive. Hope this saves someone some frustration.

    Great article Ryan. Saved me a lot of time and money. One quick question. The Intel PRO/1000 you list is a PCIe x4. The SH67H3 only has x16 and x1 PCIe slots and the x16 (at least in the older postings) only supports video cards. Which bus slot are you using for the dual nic? If you are using the x16 which BIOS version are you using and if it is the x1 slot how’s your bandwidth?

    Thanks again
    GP

  62. Pingback: Ryan Birk – Virtual Insanity » Indy VMUG: My Demo Days Presentation

  63. prashanth says:

    Hi,
    I.m trying to set up a Esxi 5.1.0 at home for testing. After installation is completed keyboard stops responding at the last stage where I have to enter the system management to assign the ip. The same (ps2) keyboard works fine if I install esxi on a workstation. I have tried to change the keyboard used a usb keyboard, but no luck…
    Hardware
    Intel I7 -2600 cpu @ 3.40 ghz
    Corsair ram xms 8 gb, ddr3.
    Motherboard sandy bridge
    Please let me know if anyone has similar issue and was able to fight through… thanks in advance.

    • Ryan Birk says:

      Be sure you’re not plugging it into a USB 3.0 port. Try a USB 2.0 port instead. A PS2 keyboard should work though but I’ve heard people have issues trying to use newer USB 3.0 ports. -Ryan

  64. Brian says:

    @Greg Porter

    Did you get the nic installed without any issues?

  65. Brian says:

    Anyone run into the below error trying to install esxi? It will boot up just fine on a usb I have that already has esxi installed on it, but if I try a fresh installation I get this error. I have tried with 5.0u2 and 5.1u1

    error loading /tools.t00 fatal error 10 (resources exhausted)

  66. Brian says:

    It was an issue with the bootable USB I had created, I created a CD from the ISO and the installation of esxi 5.1U1 completed without any issues. I have it installed on a 2GB USB.

  67. Bill Parrish says:

    BTW, your pic above isnt a computer, its an operting console for a naval nuclear power plant. Recognize the photo from an old text book…. 😉

  68. Raymond says:

    Anyone have any experience with the SH87R6? I’m thinking this along with an i7-4770s would make a nice ESXi box. Ideally I could pass through the onboard video but that might be asking too much?

  69. Valon says:

    Hi Ryan, what do you think of removing the NAS or shared storage and using the money to buy a third Shuttle box and go with VSAN?

    You would need on SSD and one SATA Disk in each system. Should be way faster and you would use the newest technology.

    • Ryan Birk says:

      Actually, I think that’s a great idea! I’ve actually had this lab setup that way for a bit playing with vSAN a few months ago in the beta.

  70. Pingback: Thoughts on VCP-IaaS / VCP-Cloud Exam | Ryan Birk – Virtual Insanity

  71. Pingback: Fixing broken Realtek and Marvell NICs in ESXi 5.5. | Ryan Birk – Virtual Insanity

  72. Juan says:

    Hello.
    I have been working with this configuration since July, 2012:
    vSphere 5.0 1u
    -Shuttle XPC Barebone SH67H7
    -Intel Core i7-3770, CPU “Ivy Bridge”
    -2x Corsair DIMM 16 GB DDR3-1333 Kit, CMX16GX3M2A1333C9, XMS
    -Intel PRO/1000 PT Quad Port Server Adapter
    -Sandisk Cruzer Fit 8 GB, USB-Stick
    – 1 Barracuda 7200 3Tb internal HD for datastore
    – 1 8Gb USB stick for boot.

    I need to upgrade to vSphere 5.5 and I want to know if somebody has the experience. My worry is about datastore.

    I am thinking about:
    Create a CD with your steps to get the Realtek driver.
    Install a fresh copy in another usb stick
    Restore the vSphere configuration from a previous backup.
    Boot.

    Any thoughts or recommendations?

    Thank you.

    • Ryan Birk says:

      If your main worry is not losing data, you can simply just boot from the custom iso and select upgrade/preserve VMFS. It won’t format any datastores you might have.

  73. Pingback: Shuttle SZ87R6 VMware ESXi 5.5 Home Lab | Ryan Birk – Virtual Insanity

  74. David Thompson says:

    Hello Ryan,
    I’ve been following your website for a while and now that I have the cash to build my own VMware home lab I have a few questions to finalize my hardware choices. Even though this is my first dive into VMware my IT experience includes both System Admin and networking.

    1. To save on cost wouldn’t an 8 core AMD processor provide more power than a 4 core i7? I do remember that more memory is better.
    2. With the changes in vSphere 5.5 are we limited in the 60 day trial versions, if so what limitations are there? I wish VMware had an educational version. That’s something I would pay for.
    3. I’ve noticed most companies I’ve talked to for government work are still using vSphere 4.0 and only plan on upgrades to 5.0/5.1 in the future.
    How fast will companies upgrade to 5.5 and require these skills?

    • Ryan Birk says:

      David, you have to remember that an i7 quad core will get you hyper threading as well. So you get 8 “cores” presented to the ESXi box. Although 8 true cores would probably be better if you really get down to it. I’ve not done any testing so I can’t speak to it. For a lab environment, you’re probably fine. I just like the power consumption and size of the Shuttle units. In terms of licensing yes, you are stuck with 60 day evals unless you have other ways of obtaining a license. VMware does not have the VMTN anymore which is similar to what you want with an educational license. As far as SMBs go, 5.5 is being deployed. I don’t see a reason for holding back outside of organization policy basically forcing them to stay a version back. 🙂 Hope that helps.

  75. Jeff Walzer says:

    Ryan,

    I was wondering what your thoughts were regarding using your config to build a virtual ESXi lab using VM Workstation.

    Thx,
    Jeff

  76. Pingback: Home Lab Overview

  77. Pingback: The Shuttle “Ultra Mini” XH61V vSphere 5 Lab | Ryan Birk – Virtual Insanity

  78. Pingback: The Shuttle XH61V Ultra Mini Lab | Ryan Birk – Virtual Insanity

  79. Pingback: The Shuttle XH61V Ultra Mini vSphere Lab | Ryan Birk – Virtual Insanity

  80. Pingback: Build own RAID server rather than buying NAS solutions?

  81. Pingback: Happy New Year! | Ryan Birk – Virtual Insanity

  82. Julien Munar says:

    Hi Ryan,
    I was wondering after reading all your blogs if finally the best solution was to use the SH67H3 boxe(sandy bridge processor) instead of the ZH87R6(haswell processor) boxe because mostly people got a NIC realtek issue ? is it right ?
    could do please give you advice ? I’m preparing a vmware lab for my Cisco ccie voice lab and I don’t want to waste a lot of time …

    thx 🙂

  83. Julien Munar says:

    thanks for your replay !! I will by the way use your solution to built my own boxe 🙂 thanks again..
    Julien.

  84. Julien Munar says:

    Hi Ryan,
    I have two questions about this boxe ..
    I saw on internet that the “Shuttle XPC SH67H3” only accepts 16 gb of DD3 memory, is it sure that I can buy 2x8gb to implement it on two slots on the motherboard as you says on this webpage ?

    other question,
    If I only use the NIC: Realtek RTL 8111E (Onboard), i don’t need therefore the NIC: Intel Pro 1000 PT Dual Port (PCI-e 1x)?

    thanks again 🙂
    and good weekend 🙂

    • Ryan Birk says:

      Julien, not sure where you saw it only took 16GB but check out Shuttles direct website. 32GB is supported. http://us.shuttle.com/barebone/models/sh67h3.html Yea, the internal NIC will work ok, but remember you’ll have a single NIC for all traffic on the host. Not really ideal.

      • Julien Munar says:

        Ryan,
        many thanks for all your advices !!! Regarding the NIC option, I will only create this server to represent the HQ Site… so it will on register some phones and will be connected to a survival router …
        If case I feel the need to get one more ethernet port, I will change the network card ..
        Thanks Ryan, it’s a pleasure to ask you 🙂

        see you =)

  85. Morufudeen says:

    Ryan,

    Thanks for the post and keep up the good work. I have just completed my lab setup based on the Shuttle SH67H3. I bought two of these, i7 3770s processors, 32GB RAM each. I also bought the Synology DS1513+ I see you are reviewing at the moment. I’d be interested to see what your thoughts are on that one. From my experience so far though, it has been amazing. The lab is quick and quiet. In fact, the Synology seems to be the loudest of all the machines, yet quiet.

  86. Pingback: Home Lab for 2014

  87. Nighteyes says:

    Great post Ryan,

    Wondering, can you see any issue with using this memory: http://www.amazon.co.uk/Corsair-CML16GX3M2A1600C10-Vengeance-Profile-Performance/dp/B007TG8QRW

    Thanks

  88. Frank says:

    Hello. Is this lab setup still up-to-date and a good starting point into VMware or is there another article detailing something for today’s tech?

    • Ryan Birk says:

      Hey Frank, this lab still works great with the new versions of vSphere! They’ve came down in price as well recently.

  89. Pingback: Intel NUC: Perfect vSphere 5.5 Home Lab | Ryan Birk – Virtual Insanity

  90. Julien Munar says:

    Hello ryan,

    I have 1 question regarding the Ethernet dual port card…
    Indeed, you recommend via the link ” Intel PRO/1000 PT Dual Port Server Adapter Adaptateur réseau PCI Express x4 Ethernet, Fast Ethernet, Gigabit Ethernet 10Base-T, 100Base-TX, 1000Base-T 2 ports”.

    On which PCIE should we install the card, there are two …
    on the small one or the longest ?

    thanks in advance !!

    Julien.

  91. Pingback: The Perfect vSphere 6 Home Lab | Ryan Birk – Virtual Insanity

  92. mike kolling says:

    Hi
    Is there anything specific that needs to be done to see all the sata ports in the shuttle sh67h3?
    I can only use sata 3 and 4. Ports 1 and 2 on motherboard dont seem to workb?
    Only being able to use two discs is a pain…

    Mike

  93. Bayu Wibowo says:

    Hi Ryan,

    Does this lab (Shuttle XPC SH67H3) support vSphere 6?
    I am planning to buy some hardware for my homelab

    Thanks,
    Bayu

  94. Pingback: Deciding between a few servers for a home lab, need opinions

  95. Pingback: R Homelab | Marry Home

Leave a Reply to Nighteyes Cancel reply

Your email address will not be published. Required fields are marked *