Vembu Webinar: How to address Data Center Challenges

Hi everyone, here is a webinar link that one of my blog sponsors Vembu is running.

The webinar will run on July 4th at 12:00pm GST.

  • How fast can I recover the data center during the disaster?
  • How can I avoid data loss during DR?
  • How can I rely on my backups?
  • Exponential data growth
  • Migration Plans – P2V and V2V

There also will be discussion on use-cases that have been developed with Vembu.

Posted in Backup, Webinars | Leave a comment

vSphere Troubleshooting Series: Part 3 – vSphere Installation Troubleshooting

This part of the series will cover some of the common issues with vSphere deployments. We will split this section into two sections. The first will cover ESXi host troubleshooting during installation, the second will cover vCenter deployments at installation.

ESXi Host Troubleshooting

It’s common to think that ESXi will just install on any hardware, but it’s important to know a few details before you decide to get started. First, VMware only will support hardware that is officially supported on the VMware Hardware Compatibility List. Specific drivers are tested and chosen. If it’s not on the list, don’t expect support. VMware has a large partner eco-system and both hardware and software goes through rigorous testing and is signed off on for official support.

VMware also has various community driver support. What this means is that even though your hardware can work with ESXi, it’s not running in a fully supported mode. This is a nice feature for users who build homelabs for practice.

Another important note to remember during installation is that not all of your drivers might install automatically. It’s possible that your hardware could be newer and you might have to download a vSphere Installation Bundle, also called a VIB. A VIB is somewhat like a tarball or ZIP archive in that it is a collection of files packaged into a single archive to make software deployments easier.

A VIB is comprised of three parts:

  • A file archive
  • An XML descriptor file
  • A signature file

The signature file is the electronic signature used to verify the level of trust. The trust level will be one of the four listed below:

  • VMwareCertified:  VIBs created and tested by VMware.  VMware Certified VIBs undergo thorough testing by VMware.
  • VMwareAccepted:  VIBs created by a VMware partners that are approved by VMware.  VMware relies on partners to perform the testing, but VMware verifies the results.
  • PartnerSupported:  VIBs created and tested by a trusted VMware partner.  The partner performs all testing.  VMware does not verify the results.
  • CommunitySupported:  VIBs created by individuals or partners outside of the VMware partner program.  These VIBs do not undergo any VMware or trusted partner testing and are not supported by VMware or its partners.

If installation was successful and you have all the right VIBs and software configured, but other issues have come up, you should always check the hostd.log file first. The hostd management service is the main communication channel between ESXi hosts and VMkernel. If hostd fails, the ESXi host disconnects from vCenter and cannot be easily managed.

  • Try restarting hostd by running /etc/init.d/hostd restart

Occasionally, an ESXi host will crash and display a purple diagnostic screen. A host can crash for several reasons. CPU exceptions, driver issues, machine check exceptions, hardware fault or a software bug.

To recover from a PSOD, you should try following these steps:


  1. Take a screenshot of the screen.
  2. Restart the host, get the VMs up and running on another host if possible. If using HA, this should happen on its own if configured properly.
  3. Contact VMware support if you can’t find any information online. Occasionally others have the same issue and the fix can be implemented easily through firmware or software updates.

Another possible issue is that the ESXi host simply hangs during the boot process. You never get a PSOD, it just sits there and the entire system becomes unresponsive. Typically hangs happen during a power cycle of a system during the boot process. It’s caused by VMkernel being too busy or a possible hardware lockup.

To determine that the host has locked up:

  1. Ping the VMkernel (Management) network interface.
  2. Try to login to the host with the client.
  3. Monitor network traffic from the ESXi host.

If you can ping the host, that’s a good sign. Next connect to the DCUI to display any messages on the screen. Press Alt-F12 at the host console to do that.

To recover from a host that has hung, try rebooting the ESXi host, review logs and gather performance statistics. If you determine it’s a hardware issue, fix the hardware and if required reinstall or reconfigure ESXi. Lastly update the host with the most recent patches.

vCenter Troubleshooting

When installing the vCSA, VMware has split the install into two different stages. Stage 1 is the appliance deployment. Stage 2 is the configuration of the appliance.

Stage 1 in most cases, is a very straightforward install. Stage 2 is where traditionally, users have had issues with deployments and it generally can be resolved with verifying your DNS settings and NTP.

Some deployments seem successful but upon login, the authentication fails if the NTP server on the ESXi host and the newly created appliance are not synced to the same source.

Occasionally you might run into issues replacing certificates with the Certificate Manager. It can hang at 0% and perform an automatic rollback error. This issue can be caused by using non-Base64 certificates. To resolve, manually publish the full chain to the certificate store.

Nest post in this series: vSphere Troubleshooting Part #4 – Virtual Machine Troubleshooting

Posted in Troubleshooting, vSphere | 1 Comment

vSphere Troubleshooting Series: Part 2 – vSphere Troubleshooting Tools

Before you can troubleshoot issues, you need to understand the various tools that are out there. In this section, we will discuss some of the tools that VMware provides and how to identify the log locations for additional troubleshooting.

VMware Command Line Tools

You can run command-line tools on an ESXi host in several ways:

  • The vSphere ESXi shell itself, which includes:
    • esxcli commands (esxcli network, esxcli storage, esxcli vm, etc)
    • A set of esxcfg-* commands: The esxcfg commands are deprecated but you will likely still see some older documentation with them. The recommendation today is to use esxcli.
    • The host shell can be accessed a couple of different ways, either by using the local DCUI (Direct Console User Interface) or via SSH.
      • Local access by using the Direct Console User Interface (DCUI):
        1. Enable the vSphere ESXi Shell service, either in the DCUI or vSphere Web Client. Typically, this is running by default.
        2. Access the ESXi Shell from the DCUI by pressing Alt-F1 after logging in.
        3. When finished, disable the ESXi Shell service when not using it.
      • Remote access by using PuTTY or an SSH client.
        1. Enable the SSH service on your ESXi host, either in the DCUI or through the vSphere Web Client.
        2. Use PuTTY or your preferred SSH Client to access the ESXi host.
        3. Disable the SSH Service when finished.
  • vSphere Management Assistant (This tool has been deprecated. 6.5 is final release):
    • A virtual appliance that includes components for running vSphere commands:
    • esxcli
    • vmware-cmd
    • vicfg-* commands
    • vi-fastpass authentication components for automated authentication to vCenter or ESXi hosts. This saves you from having to type your name and password with every command that is ran.
  • VMware PowerCLI:
    • VMware PowerCLI provides an easy-to-use Windows PowerShell interface for command-line access to administration tasks or for creating executable scripts.

VMware Log Locations for Troubleshooting

VMware stores logs for their products in various locations. It’s important to know where to look when you’re having problems quickly and efficiently.

  • vCenter Log Locations:
    • Location for vCenter Server on Windows 2008/2012:
      • %ALLUSERSPROFILE%\VMWare\vCenterServer\logs
    • Location for VMware vCenter Server Appliance:
      • /var/log/vmware/
        • Includes logs for SSO, Inventory Service and the Web Client.
      • Useful ESXi Host Logs:
        • log: Host management service logs.
        • log: Service initialization, watchdogs, scheduled tasks, DCUI.
        • log: Core VMkernel logs. Storage and Networking device events.
        • log: Warning and alert log messages.
        • log: ESXi host startup and shutdown, uptime details, resource consumption.
      • vCenter vpxd.log
        • This log file is the main vCenter Server log file. If you ever contact VMware for support, it is highly likely that they will ask you for this file. Don’t confuse this with vpxa, that is the vCenter agent and runs on the ESXi hosts.
        • You can monitor and view the logs easily through the vSphere Web Client, under the Monitor tab (Figure 1), with an SSH session at /var/log (Figure 2) or in the DCUI under “View System Logs” under System Customization (Figure 3).

The vSphere Syslog Collector

You can gather logs at the above locations or setup a single location for all of your ESXi hosts to point to. It uses port 514 for TCP and UDP, and port 1514 for SSL. The Syslog collector is installed on both the Windows based vCenter and the vCenter Appliance.

The vm-support Command

In addition to the Syslog Collector, you can also gather logs with the vm-support command. It collects data from the ESXi hosts and compresses the following data:

  • Log files
  • System status
  • Configuration files

The tool does not require any arguments and it create a zip file using the host name and time stamp.

The vCenter Bash Shell

You can also use the vCenter Bash Shell from the vCenter Appliance console under troubleshooting options. From the Bash shell, you can verify the status of a service and start or restart services.

Part 3 of this troubleshooting series can be found here:

Posted in Troubleshooting, vSphere | 1 Comment

What’s New in Performance: vSphere 6.5

Underlying each release of VMware vSphere are many performance and scalability improvements. The VMware vSphere 6.5 platform continues to provide industry-leading performance and features to ensure the successful virtualization and management of your entire software-defined datacenter.

This whitepaper is broken into various subsections that show increases and improvements around performance.

Posted in vSphere 6.5, Whitepapers | Leave a comment

vSphere Troubleshooting Series: Part 1 – Introduction

vSphere Troubleshooting Introduction

Before we begin, we need to start off with an introduction to a few things that will make life easier. We’ll start with a troubleshooting methodology and how to gather logs. After that, we’ll break this eBook into the following sections: Installation, Virtual Machines, Networking, Storage, vCenter/ESXi and Clustering.

ESXi and vSphere problems arise from many different places, but they generally fall into one of these categories:

  • Hardware issues
  • Resource contention
  • Network attacks
  • Software bugs
  • Configuration problems

A typical troubleshooting process contains several tasks:

  1. Define the problem and gather information.
  2. Identify what is causing the problem.
  3. Fix the problem, implement a fix.

One of the first things you should try to do when experiencing a problem with a host, is try to reproduce the issue. If you can find a way to reproduce it, you have a great way to validate that the issue is resolved when you do fix it. It can be helpful as well to take a benchmark of your systems before they are implemented into a production environment. If you know HOW they should be running, it’s easier to pinpoint a problem.

You should decide if it’s best to work from a “Top Down” or “Bottom Up” approach to determine the root cause. Guest OS Level issues typically cause a large amount of problems. Let’s face it, some of the applications we use are not perfect. They get the job done but they utilize a lot of memory doing it.

In terms of virtual machine level issues, is it possible that you could have a limit or share value that’s misconfigured?

At the ESXi Host Level, you could need additional resources. It’s hard to believe sometimes, but you might need another host to help with load!

Once you have identified the root cause, you should assess the impact of the problem on your day to day operations. When and what type of fix should you implement? A short-term one or a long-term solution? Assess the impact of your solution on daily operations.

  • Short-term solution: Implement a quick workaround.
  • Long-term solution: Reconfiguration of a virtual machine or host.

Next in this series: vSphere Troubleshooting Series: Part 2 – vSphere Troubleshooting Tools

Posted in Troubleshooting, vSphere | Leave a comment

DRS Cluster Management with Reservation and Shares

Reservation and shares are important resource management settings in a vSphere DRS cluster. They can be set on cluster objects like VMs and resource pools to isolate resources, prioritize, and/or guarantee their availability.

To know when to set them for VMs and when to set them on resource pools, we need to understand:

  • What these settings mean.
  • How these settings can impact resource availability for a VM.

In this paper, VMware explains how these settings are different for a VM and resource pool while giving some general guidelines for using them.


Posted in vSphere, Whitepapers | Leave a comment