vSphere Troubleshooting Series: Part 2 – vSphere Troubleshooting Tools

Before you can troubleshoot issues, you need to understand the various tools that are out there. In this section, we will discuss some of the tools that VMware provides and how to identify the log locations for additional troubleshooting.

VMware Command Line Tools

You can run command-line tools on an ESXi host in several ways:

  • The vSphere ESXi shell itself, which includes:
    • esxcli commands (esxcli network, esxcli storage, esxcli vm, etc)
    • A set of esxcfg-* commands: The esxcfg commands are deprecated but you will likely still see some older documentation with them. The recommendation today is to use esxcli.
    • The host shell can be accessed a couple of different ways, either by using the local DCUI (Direct Console User Interface) or via SSH.
      • Local access by using the Direct Console User Interface (DCUI):
        1. Enable the vSphere ESXi Shell service, either in the DCUI or vSphere Web Client. Typically, this is running by default.
        2. Access the ESXi Shell from the DCUI by pressing Alt-F1 after logging in.
        3. When finished, disable the ESXi Shell service when not using it.
      • Remote access by using PuTTY or an SSH client.
        1. Enable the SSH service on your ESXi host, either in the DCUI or through the vSphere Web Client.
        2. Use PuTTY or your preferred SSH Client to access the ESXi host.
        3. Disable the SSH Service when finished.
  • vSphere Management Assistant (This tool has been deprecated. 6.5 is final release):
    • A virtual appliance that includes components for running vSphere commands:
    • esxcli
    • vmware-cmd
    • vicfg-* commands
    • vi-fastpass authentication components for automated authentication to vCenter or ESXi hosts. This saves you from having to type your name and password with every command that is ran.
  • VMware PowerCLI:
    • VMware PowerCLI provides an easy-to-use Windows PowerShell interface for command-line access to administration tasks or for creating executable scripts.

VMware Log Locations for Troubleshooting

VMware stores logs for their products in various locations. It’s important to know where to look when you’re having problems quickly and efficiently.

  • vCenter Log Locations:
    • Location for vCenter Server on Windows 2008/2012:
      • %ALLUSERSPROFILE%\VMWare\vCenterServer\logs
    • Location for VMware vCenter Server Appliance:
      • /var/log/vmware/
        • Includes logs for SSO, Inventory Service and the Web Client.
      • Useful ESXi Host Logs:
        • log: Host management service logs.
        • log: Service initialization, watchdogs, scheduled tasks, DCUI.
        • log: Core VMkernel logs. Storage and Networking device events.
        • log: Warning and alert log messages.
        • log: ESXi host startup and shutdown, uptime details, resource consumption.
      • vCenter vpxd.log
        • This log file is the main vCenter Server log file. If you ever contact VMware for support, it is highly likely that they will ask you for this file. Don’t confuse this with vpxa, that is the vCenter agent and runs on the ESXi hosts.
        • You can monitor and view the logs easily through the vSphere Web Client, under the Monitor tab (Figure 1), with an SSH session at /var/log (Figure 2) or in the DCUI under “View System Logs” under System Customization (Figure 3).

The vSphere Syslog Collector

You can gather logs at the above locations or setup a single location for all of your ESXi hosts to point to. It uses port 514 for TCP and UDP, and port 1514 for SSL. The Syslog collector is installed on both the Windows based vCenter and the vCenter Appliance.

The vm-support Command

In addition to the Syslog Collector, you can also gather logs with the vm-support command. It collects data from the ESXi hosts and compresses the following data:

  • Log files
  • System status
  • Configuration files

The tool does not require any arguments and it create a zip file using the host name and time stamp.

The vCenter Bash Shell

You can also use the vCenter Bash Shell from the vCenter Appliance console under troubleshooting options. From the Bash shell, you can verify the status of a service and start or restart services.

Part 3 of this troubleshooting series can be found here: http://www.ryanbirk.com/vsphere-troubleshooting-series-part-3-vsphere-installation-troubleshooting/

Posted in Troubleshooting, vSphere | 1 Comment

What’s New in Performance: vSphere 6.5

Underlying each release of VMware vSphere are many performance and scalability improvements. The VMware vSphere 6.5 platform continues to provide industry-leading performance and features to ensure the successful virtualization and management of your entire software-defined datacenter.

This whitepaper is broken into various subsections that show increases and improvements around performance.

Posted in vSphere 6.5, Whitepapers | Leave a comment

vSphere Troubleshooting Series: Part 1 – Introduction

vSphere Troubleshooting Introduction

Before we begin, we need to start off with an introduction to a few things that will make life easier. We’ll start with a troubleshooting methodology and how to gather logs. After that, we’ll break this eBook into the following sections: Installation, Virtual Machines, Networking, Storage, vCenter/ESXi and Clustering.

ESXi and vSphere problems arise from many different places, but they generally fall into one of these categories:

  • Hardware issues
  • Resource contention
  • Network attacks
  • Software bugs
  • Configuration problems

A typical troubleshooting process contains several tasks:

  1. Define the problem and gather information.
  2. Identify what is causing the problem.
  3. Fix the problem, implement a fix.

One of the first things you should try to do when experiencing a problem with a host, is try to reproduce the issue. If you can find a way to reproduce it, you have a great way to validate that the issue is resolved when you do fix it. It can be helpful as well to take a benchmark of your systems before they are implemented into a production environment. If you know HOW they should be running, it’s easier to pinpoint a problem.

You should decide if it’s best to work from a “Top Down” or “Bottom Up” approach to determine the root cause. Guest OS Level issues typically cause a large amount of problems. Let’s face it, some of the applications we use are not perfect. They get the job done but they utilize a lot of memory doing it.

In terms of virtual machine level issues, is it possible that you could have a limit or share value that’s misconfigured?

At the ESXi Host Level, you could need additional resources. It’s hard to believe sometimes, but you might need another host to help with load!

Once you have identified the root cause, you should assess the impact of the problem on your day to day operations. When and what type of fix should you implement? A short-term one or a long-term solution? Assess the impact of your solution on daily operations.

  • Short-term solution: Implement a quick workaround.
  • Long-term solution: Reconfiguration of a virtual machine or host.

Next in this series: vSphere Troubleshooting Series: Part 2 – vSphere Troubleshooting Tools

Posted in Troubleshooting, vSphere | Leave a comment

DRS Cluster Management with Reservation and Shares

Reservation and shares are important resource management settings in a vSphere DRS cluster. They can be set on cluster objects like VMs and resource pools to isolate resources, prioritize, and/or guarantee their availability.

To know when to set them for VMs and when to set them on resource pools, we need to understand:

  • What these settings mean.
  • How these settings can impact resource availability for a VM.

In this paper, VMware explains how these settings are different for a VM and resource pool while giving some general guidelines for using them.


Posted in vSphere, Whitepapers | Leave a comment

vSAN 6.6 download links are publicly available.

Today VMware has released vSAN 6.6 to the public! 6.6 is a big step forward. While the base architecture of vSAN stays the same, it becomes more and more efficient. With the price of flash going down, it’s an even more compelling reason to look into HCI these days.

One of my favorite use cases for vSAN is with VMware Horizon Advanced/Enterprise editions. To me, it’s a no brainer. Simple, elastic scalability, much higher performance over traditional tiered storage and lower TCO. Not long ago storage in VDI environments would have given most of us nightmares.

What’s New

  • Unicast – In vSAN 6.6, cluster communication has been redesigned to use unicast traffic. Multicast is no longer required on the physical switches to support the vSAN cluster.
  • Enhanced Stretched Clusters with Local Failure Protection – Previously, vSAN was able to provide a fully active-active, stretched cluster. vSAN 6.6 takes this a step further, allowing for storage redundancy within a site AND across sites at the same time.
  • Encryption – vSAN supports data-at-rest encryption of the vSAN datastore. When encryption is enabled, vSAN performs a rolling reformat of every disk group in the cluster.
  • Site Affinity for Stretched Clusters – A new feature for vSAN 6.6 Stretched Clusters is the ability to configure site affinity.
  • Configuration Assist and Updates – New Configuration Assist and Updates pages allows to check the configuration of your vSAN cluster, and resolve any issues.
  • Resynchronization throttling – IOPS used for cluster resynchronization can be throttled to prevent performance bottlenecks.
  • vSAN Health Command Line Tool – A new esxcli command allows to check vSAN health from the command line (esxcli vsan health).
  • Degraded Device Handling – vSAN 6.6 provides a more proactively stable environment with the detection of degraded and failing devices.

Additional Information

Posted in vSAN, vSphere 6.5 | Leave a comment

New Webinar: VMware Administrator’s Troubleshooting Guide

Altaro is a great blog sponsor of mine and reached out to me recently to see if there was interest in doing a separate project with them. A vSphere troubleshooting webinar. I’ve compiled a list of some of the most popular scenarios and put them together in an eBook. We’ll be discussing them April 25th! You can register at the link below.

Troubleshooting complex virtualization technology is something that system administrators have to face at some point, and it’s not always an easy fix to get things up and running again. We’re running a webinar that will cover the most common problems experiences in VMware vSphere. Register for the webinar here.

Andy Syrewcize and Ryan Birk will bring a wealth of experience in troubleshooting some of the most common issues admins face. Andy has spent the last 12 years providing technology solutions across a variety of industries, and has experience in troubleshooting VMware infrastructures for education, manufacturing, healthcare and other industries. Ryan is a VMware vExpert, and a VMware certified trainer, having consulted and engineered infrastructures for a variety of companies.

Here’s what you’ll learn:

  • Troubleshooting some of the most common vSphere problems
  • Quick and efficient issue fixing practices
  • Maintaining a smooth running vSphere environment to avoid future issues


Date: Tuesday April 25th, 2017
Time: For US registrants: 10am PDT/1pm EDT, for RoW registrants: 2pm CET


PS: We’re releasing an eBook by Ryan on the same topic soon as well. By registering for the webinar you will get early access automatically!

Posted in Troubleshooting, Webinars | Leave a comment