Thought I would pass along a great slide deck about troubleshooting vSphere storage.
Over on VMware blogs, they have released some real jewels lately. This post looks at the protocol purely from a vSphere perspective. It doesn’t cover performance because they state another team inside VMware does that, but this is a great post, so check it out. If you’re looking for performance documents and comparisons the links at the bottom are what you want.
http://blogs.vmware.com/vsphere/2012/02/storage-protocol-comparison-a-vsphere-perspective.html
Here are some VMware whitepapers which cover storage performance comparison:
VMware has released a whitepaper that details some technical information about the VSA. In the release of VMware vSphere 5.0, VMware also released a new software storage appliance called the VMware vSphere Storage Appliance (VSA). VMware VSA provides an alternative shared storage solution to Small to Medium Business (SMB) customers who might not be in a position to purchase a Storage Area Network (SAN) or Network-Attached Storage (NAS) array for their virtual infrastructure. Without shared storage configured in a vSphere environment, customers have not been able to exploit the unique features available in vSphere 5.0, such as vSphere High Availability (HA), vSphere, vMotion and vSphere Distributed Resource Scheduler (DRS). The VSA is designed to provide “Shared Storage for Everyone”.
See the whitepaper here.
I recently read through the “Storage vMotion of a Virtualized SQL Server Database” technical whitepaper that VMware released. While reading I found a great section that’s worth sharing. It contained overall information on how svMotions work. I would recommend reading the entire whitepaper, especially if you plan to virtualize SQL servers. Below is an excerpt:
There are two main factors that can influence the svMotion of virtual disks in a vSphere virtual infrastructure: I/O access patterns on the virtual disk that has to be migrated and the underlying storage infrastructure. While it is hard to expect change to the former, careful considerations given to the latter can help in achieving a better svMotion experience and in minimizing impact to the applications that run in a virtual infrastructure where the VM storage is migrated. Based on the knowledge gained during the tests, the following best practices can help while planning an infrastructure that is capable of allowing live migration of VM storage:
1. Random access patterns on virtual disks interfere with the sequential access of svMotion and negatively affect the I/O throughput of svMotion. This can increase migration time by a significant value. If there is a need to migrate such virtual disks, plan to schedule the migration during periods when there is low or no I/O activity on the virtual disk.
2. Sequential access patterns of a VM on a VM’s virtual disk (for example, writes to log files in a virtual disk) generally don’t affect the sequential access pattern of svMotion. Even with more than one sequential stream on the virtual disk, most modern arrays are capable of utilizing the I/O prefetching logic in the arrays’ firmware to improve the I/O performance. Such virtual disks could be migrated even when there is some I/O activity on them.
However, if the VM’s I/O access to its virtual disk is significant, then the svMotion traffic will have to contend with the VM’s access for the I/O bandwidth. This could reduce the throughput for both the traffic flows. In such situations, it is better to schedule svMotion during periods when the existing I/O load level reduces.
1. svMotion moves data blocks in 64KB chunks. Any I/O operation from the VM that uses a small request size might see higher access latency due to the large sized blocks of svMotion traffic. If the application in the VM is very sensitive to an increase in the access latency, consider scheduling svMotion during periods of low or no I/O activity on the virtual disk.
2. Most applications that rely on random access patterns to fetch data from a physical storage media may not benefit from cache in the storage array. In such situations, administrators tend to configure (if permissible) a significant chunk of array cache for write access patterns. This may limit the amount of data prefetched by the array for svMotion which can potentially impact the disk migration time. Having enough buffer space to hold the prefetched data for svMotion may help in reducing the disk migration time.
3. In certain situations such as the experiments discussed in this paper, moving a virtual disk from an array with newer hardware, firmware, and larger cache to older arrays could be faster than the other way around.
4. Faster storage media such as solid state disks (SSDs) provide a faster access to the smaller blocks even in the presence of requests for larger blocks. By utilizing the services of SSDs (for example, storage pools built on SSDs and SSD-based secondary cache) in the array, the impact on the performance of the application can be reduced when migrating the application’s disks.
Over the years, I’ve built a few ESXi white boxes for my home lab. I really got serious back in 2009 when “BirkleNET” was built. I must add that I didn’t name it that, my colleagues gave it that name and it stuck. By doing so it helps me stay on top of the new offerings from VMware. Through teaching for the Purdue CIT program, I’ve recently inherited an older EMC CX3-10c SAN from the Department of Medicine with 2 storage processors, power cables, 3 shelves of 300GB 10k drives, 2 FC switches, 
their PSUs and 3 FC cards which are PCI-E. Their trash is my treasure! Perfect timing too as I was beating the life out of my home-brew openfiler box I was using there. Big thanks to Kyle Ruddy!
I decided it was also time to get my whiteboxes upgraded there.
I wanted something small that would pack power. It needed to have enough juice so that I could run vCloud Director, vMA, VMware View 5, etc. I ended up getting 2 Shuttle XPC systems. Very similar to Kendrick Colemans home lab.
I did not purchase any disks because I don’t plan on using any local disk immediately, although I will admit that I have a few things with local SSD that I want to try out with VMware View in the near future! I am just going to instal ESXi and boot these from USB sticks for now … and who doesn’t have 20 or so of those laying around??
The goodies:
* 2x Shuttle SH67H3
* 2x 16GB DDR3 1333 Memory
* 2x Intel i7 Sandy Bridge 3.4GHz
* 2x Intel EXPI9402PT PRO/1000 PT Dual Port Server Adapter
Each of these machines cost around $800.00 to build. You can probably save some money shopping around for the dual NIC and memory. It seems like those vary a lot from place to place. If you’re reading this 30 days or so after the original post date, you’ll probably pay half of what I got these for. Haha.
After updating the BIOS to the latest version, I had no issues installing ESXi 5 after making appropriate BIOS tweaks like VT-d, set to boot USB in hard disk mode, etc. All of the NICs, even the integrated one came up and appear to work well.
The BIOS update is important because the 1 x PCI Express 2.0 x16 slot says it’s for graphics cards only and ESXi will PSOD if you try to use a NIC in that slot. After the update, the system no longer PSODs.
These were the easiest of all of the white boxes I’ve ever set up. I plan to do another post on how I plan to configure the SAN in this environment soon. Brian Wuchner over at EnterpriseAdmins.org, Jake Robinson over at http://geekafterfive.com and I plan to play with it sometime next week and will hopefully be able to start work on our next project … building a hybrid cloud!
I was privileged to be able to attend a full day “VDI: Best Practices” session today that EMC hosted here in Indianapolis. Andre Leibovici, a vSpecialist at EMC was the speaker. He blogs at http://myvirtualcloud.net and is probably the best resource online in terms of large scale VDI deployments. He’s also on twitter @andreleibovici, I would highly recommend following him if you want some good info. I really enjoyed the session today, I hope we get more of these technical sessions in Indy soon!
I wanted to post my notes from the session, so others might learn something as well. Andre said he was going to send out the slides, I’ll ask him if it would be ok to post here as well and do so if he allows it.
What’s new in View 5:
PCoIP Optimization – client side caching, default CODEC optimization for fonts, build to lossless.
Unified communications – Better VoIP support for soft phones.
Enhanced clipboard controls
Automatically reconnect disconnected sessions
Android support
Persona management built in – copies only what is required at boot time.
General Tips:
* Use local disk as VM swap locations to save on shared storage. People wonder how this works with vMotion. If you set it up this way, you CAN vMotion. It will create a new swap file on the destination host.
* Use floating pools whenever possible for manageability and ease of administration. With dedicated pools you need to manage your VMs like you used to in terms of patching, etc.
Storage:
* All View .vmdk disks are Thin Provisioned (exception is internal.vmdk (stores users account information to sync with Windows domain)).
* Always add at least 10% storage overhead when sizing
* Learn to play with VM Memory Reservations to reduce storage footprint. (Windows XP (40% reduction from using Transparent Page Sharing), Windows 7 (Not so much, because of ASR)).
* VMware View 5 has a new .vswp file based on RAM assigned for video with hardware v8 which needs to be considered when sizing storage.
* When 3D is enabled a 256MB overhead is added to the secondary .vswp file! The 256MB overhead is independent of vRAM settings.
* Split your storage to use SSD or EFDs for replica disks to provide better read rates on Linked Clone VMs.
An ideal Linked Clone deployment would put:
Replicas on flash disk
Linked Clone on fiber channel disk
User Data on serial ata disk
Datastore Sizing
* With Linked Clones, snapshot delta files grow forever. Use a disposable disk to keep more control of this. (A disposable disk gets wiped on a reset/reboot)
* The size of a replica is the thin provisioned size of your parent VM.
* Maximum VMs per VMFS datastore: 64 VMs for Linked Clones, 32 VMs for Full Clones
IO Sizing:
* Users use VMs differently, there is no right number when it comes to sizing IOPS. You need to pilot, pilot, pilot! See what your workload looks like with real life load. Do not use perfmon inside the guests. ESXi has overhead added as well so you need to get the numbers from esxtop or inside vCenter.
RAID adds a write penalty!
RAID 5 adds a write penalty of 4
RAID 10 adds a write penalty of 2
VM IO = VM Read IO + (VM Write IO * RAID Penalty)
* VMware vStorage API for multi-pathing is a great tool that provides better use of your storage array. (VAAI, VASA)
* VMstore profiles allow you to select what datastores are used for what. (i.e. performance or capacity)
VAAI:
* Offload certain tasks to be run where it makes sense to run from a storage perspective.
* Without VAAI the entire LUN will be reserved and requires several SCSI commands. With VAAI the lock occurs at the VM level. There is no LUN locking. (You can increase your VMs from 64 to 140 Linked Clones VMs per datastore)
Storage Best Practices:
* Misalignment of filesystems results in additional work on storage controller to satisfy I/O requests. This means every protocol and every storage array. Be sure to have properly aligned disks!
* Enable Jumbo Frames across the entire stack if possible.
* Round Robin policy is best policy for storage multipathing on most storage platforms.
* Use Storage I/O Control if you can. If a VM is misbehaving and causing other VMs to perform badly, SIOC will throttle the troublemaker back so it will not suffer because of one bad VM.
* 8 hosts maximum per cluster due to View Composer limitations because there is a limit on reads. Your replica disk is the reason for this limit. VMFS-5 does not change this limitation.
Securing:
* Antivirus can be offloaded using vShield at the hypervisor level.
* VDI can provide better security, but it’s important to make sure your attack surface is minimized. You cannot manage your virtual desktops like physical desktops.
Networking:
* Adjust DHCP Scope from 8 days to 1 hour.
Gotchas:
* You cannot use the “User Data” persistent disk to redirect the Windows profile with Linked Clone pools, because those disks are assigned to certain VMs.
* Do not run vCenter, connection servers or security servers on the same ESXi hosts as your virtual desktops in case of failure.
  |   |
