VMware at re:Invent 2018

Hey everyone, re:Invent this year was huge and VMware presented news about their future plans on AWS infrastructure. Much of my time was spent on Twitter and social media checking to see if any new relevant announcements would come out surrounding VMware on AWS. There’s been a lot – and I mean a lot – of activity since it was announced. VMware Cloud on AWS is the only hybrid cloud solution allowing you to modernize, protect and scale vSphere-based apps to the cloud, leveraging AWS. Together, these services integrate allowing you to rapidly extend and migrate your VMware environment to the AWS public cloud.

I thought I’d put together a quick post to highlight some of my favorite items that were announced at re:Invent. So here goes!

VMware Cloud on AWS Outposts

VMware and AWS are already huge Goliath’s in the virtualization and cloud market but they’ve partnered again to deliver a new as-a-service, on-premise offering that includes the full VMware software stack (think vSphere, vSAN and NSX) that can run on AWS Outposts. After partnering on technology to bring VMware virtualization software to the AWS public cloud last year, they’re now joining up to introduce “Outposts,” hardware that brings the AWS cloud on-premises. It’s a fully managed and configurable server built to run on AWS-designed hardware. It will be a subscription-based service and will support existing VMware payment options.

AWS CEO Andy Jassy said AWS Outposts provides a way to run AWS infrastructure on premises for a “truly consistent” hybrid experience. It’s available in two options, with the first through the VMware Cloud on AWS offering and the second as AWS native.

Option #1: For customers who want to use the same VMware control plane and APIs they’ve been using to run their infrastructure, they will be able to run VMware Cloud on AWS locally on AWS Outposts.

Option #2: For customers who prefer the same exact APIs and control plane they’re used to running in AWS’s cloud, but on-premises, they can use the AWS native variant of AWS Outposts.

AWS Outposts are in private preview, with public general availability in the second half of 2019, according to Amazon.

This offering is the AWS and VMware answer to the hybrid cloud deployment model Microsoft has been pushing with Azure Stack. This provides AWS a hybrid cloud play that they previously lacked, and sets up a rivalry of sorts in an area that Azure has dominated (hybrid deployments). There have been many AWS customers looking for this type of play, as well as many VMware customers wanting a more native hybrid offering. This solution covers both bases, and it will be interesting to see how it evolves over the coming year.

VMware Cloud Foundation for EC2

Another huge announcement from re:Invent was the addition of services that extend datacenter management to the public cloud. They’ve coined it VMware Cloud Foundation for EC2. There are two major components. A mechanism to insert and manage these services on Amazon EC2, as well as networking, security, data, and management services themselves.  It creates a common set of data center services that spans the hybrid cloud. These services support all types of workloads from traditional VM based enterprise applications to modern container-based workloads utilizing platforms like PKS or Red Hat OpenShift.

Wrap up

Hopefully, the above tools will help expand some environments. When they officially go live it will be interesting to see the adoption. I’ll leave you with one of my favorite sessions I watched from re:Invent. I still have several more to catch up on. It’s a great video for anybody wondering about connectivity for VMware Cloud on AWS. If you’re new or even considering, you should check this session out. One day I hope to make it to re:Invent. I hear it’s a great conference to go to!

If you’ve attended and would like to share your experience, let us know in the comments section below!

Enjoy the Video!

Posted in AWS, vSphere | Leave a comment

Altaro VM Backup v8 has been released.

I’ve got Altaro running in my home lab and decided to upgrade recently to the newest version. I wanted to highlight a few updates regarding version 8!

The interface is still the same for the most part. It’s easy to understand, and intuitive.

My friend, Andy Syrewicze over at Altaro has a great What’s New video:

Drastically reduce RTO.

“WAN-Optimized Replication allows businesses to continue accessing and working in the case of damage to their on-premise servers. If their office building is hit by a hurricane and experiences flooding, for instance, they can continue working from their VMs that have been replicated to an offsite location,” explained David Vella, CEO and co-founder of Altaro Software.

“As these are continually updated with changes, businesses using Altaro VM Backup can continue working without a glitch, with minimal to no data loss, and with an excellent recovery time objective, or RTO.”

Centralized, multi-tenant view for MSPs.

Managed Service Providers (MSPs) can now add replication services to their offering, with the ability to replicate customer data to the MSP’s infrastructure. This way, if a customer site goes down, that customer can immediately access its VMs through the MSP’s infrastructure and continue working.

With Altaro VM Backup for MSPs, MSPs can manage their customer accounts through a multi-tenant online console for greater ease, speed and efficiency, enabling them to provide their customers with a better, faster service.

You can check more information out on the Altaro page.

Posted in Backup, Tech Reviews | Leave a comment

What is a VMware VIB file?

For today’s post, I want to cover one of the fundamental aspects of VMware: the VIB file.

When working with VMware over the years, I’ve run into multiple occasions that have required the use of VIB files. I’ve talked with multiple people who still struggle with what a VIB file is. So today, we’re going to talk about them.

What is a VMware VIB?

In short, a VIB is a software package that gets installed on a vSphere ESXi host that contains things like drivers. They have become quite a bit more common in the last few years as the supported hardware base for vSphere has increased over time.

If you want the official description from VMware:

VIB stands for vSphere Installation Bundle, and it’s a software package that gets installed on a vSphere ESXi Host.  A VIB is somewhat similar to a tarball or ZIP archive in that it is a collection of files packaged into a single archive to facilitate distribution.  If we look deeper we will find that a VIB is comprised of three parts:

  • A file archive or payload

  • An XML descriptor file

  • A signature file

The file archive/payload contains the files that make up the VIB. These are essentially the files that will be installed on the ESXi host when it’s instructed to.

The XML descriptor file describes the contents of the VIB. It contains the requirements for installing the VIB and identifies who created the VIB and the amount of testing that’s been done. Often times you will run into various third-party vendors who have their own VIBs so having this content is good.

The signature file is used to verify the level of trust. This is where it can get confusing because, in lab environments like my homelab, I generally use third-party VIBs and community supported VIBs. In a production environment, you might want to use only PartnerSupported VIBs and above. VIBs are distributed in the following levels:

  • VMwareCertified:  The VMwareCertified acceptance level has the most stringent requirements of all of them. VIBs with this level go through rigorous testing by VMware in-house. Today, only IOVP drivers are published at this level. VMware takes support tickets for VIBs with this acceptance level.
  • VMwareAccepted:  VIBs created by VMware partners that are approved by VMware.  VMware relies on partners to perform the testing, but VMware verifies the results. VMware directs support calls for VIBs with this acceptance level to the partner’s support organization.
  • PartnerSupported:  VIBs with the PartnerSupported acceptance level are published by a partner that VMware trusts. The partner performs all testing. VMware does not verify the results. This level is used for a new or nonmainstream technology that partners want to enable for VMware systems. Today, driver VIB technologies such as Infiniband, ATAoE, and SSD are at this level with nonstandard hardware drivers. VMware directs support calls for VIBs with this acceptance level to the partner’s support organization.
  • CommunitySupported:  VIBs created by individuals or partners outside of the VMware partner program.  These VIBs do not undergo any VMware or trusted partner testing and are not supported by VMware or its partners.

Troubleshooting VIBs

You can check to see what your host level acceptance is set to by doing the following:

From the vSphere Client select the ESXi Host and go to Configuration >> Security Profile. Under “Host Image Profile Acceptance Level” edit the acceptance level. You can also check with PowerCLI. From a PowerCLI command prompt while connected to the ESXi host run the following commands:

$esxcli = Get-EsxCli

You can set or change the policy from PowerCLi by running the following:


VIBs are generally installed and upgraded as a normal upgrade process. It’s possible that you might have missing VIBs. To check, look in the /var/log/vmkernel.log file and check for any error entries. Sometimes you can download a VIB and install it manually, which we’ll be covering in more detail in a future segment.

Wrap Up

To wrap up, we’ve covered what a VIB is and the various acceptance levels. It depends on what kind of support you’re looking for but be careful in production environments that you’re not using community supported VIBs. They aren’t officially supported, and if you get into a bind with one of these, VMware will be unable to assist you. Again, as mentioned in the next post, I’ll cover how to work with VIBs. We’ll walk through how you can upgrade your ESXi hosts using the esxcli command as well.

Also, to leave you with a question, have you used many community-supported VIBs? Have you had good luck with them? Not good luck? Let us know in the comments section below!

Posted in vSphere, Whitebox | Leave a comment

What’s New in vSphere 6.7: Whitepaper

VMware vSphere 6.7 delivers key capabilities to enable IT organizations to address the following notable trends that are putting new demands on their IT infrastructure:

  • Explosive growth in quantity and variety of applications, from business-critical applications to new intelligent workloads
  • Rapid increase in hybrid cloud environments and use cases
  • Global expansion of on-premises data centers, including at the edge
  • Heightened importance of security relating to infrastructure and applications

Download the Technical White Paper: What’s New in vSphere 6.7

Posted in vSphere, Whitepapers | Leave a comment

Altaro VM Backup: 7.6 Review

Hi everyone. I wanted to get a quick post out there about one of my blog sponsors, Altaro. They’re a great partner of mine and I also happen to write content for their VMware blog over here. With that little tidbit out of the way, lets get to the good stuff. They have released a new 7.6 version and I thought I’d writer a bit about some of my favorite new features.

  • With Altaro VM Backup 7.6, users can switch from running daily backups to a continuous data protection model yielding an improved Recovery Point Objective (RPO) of up to 5 minutes.
  • Altaro VM Backup 7.6 introduces GFS (Grandfather-Father-Son (GFS) Archiving), enabling users to choose to archive the backup versions over and above their continuous and daily backups instead of deleting them (local backups only). Now you can easily set up separate backup cycles to store a new backup version every week, every month and every year.
  • In previous Altaro VM Backup Versions only one operation could be performed on a Virtual Machine at the same time. This caused the following pain points:
    • If a retention policy takes quite long to complete, then backups and restore operations are queued until retention is complete.
    • If an Offsite Copy to Azure takes days to complete, especially for the initial backup; then backups and restore operations for that VM are queued until it is complete
    • If a Restore, File Level Restore or Boot from Backup operation is active then no backups for that Virtual Machine could take place until they are completed.

    Each of these limitations have been addressed in v7.6 , allowing users to restore and take Offsite Copies without delaying any scheduled or CDP backups whether scheduled or CDP.

Altaro is still very competitive on price for the feature set you get. Per host, unlimited sockets. You can check their pricing calculator here and see for yourself.

I would also recommend checking out the video Andy Syrewicze did that demos some of the new v7 features. Myself, I find the interface very easy to use and to setup. I had no trouble navigating the client and setting things up without having to read the entire manual. 🙂 Within 15 minutes I had multiple hosts and VMs setup and backup jobs running. I have also tested the restore and sandbox functions and they have worked each time I have tried it. My upgrade was very smooth as well.

Posted in Backup | Leave a comment

vSphere Troubleshooting Series: Part 6 – Network Troubleshooting

In vSphere, networking problems can occur at many different levels. It is important to know which level to start with. Is it a virtual machine problem or a host problem? Did the issue arise when you migrated the machine to a new host?

  • Virtual switch connectivity can be managed in two ways:
    • Standard switches
    • Distributed switches

You also must determine if it’s a virtual machine or a host management issue.

Network Troubleshooting Scenario #1 – No network connectivity to other systems.

One of the first things you need to do is a simple ping. Ping a system that is up and that you have tested and should be accessible to the ESXi host.

Starting at the ESXi host, verify these possible configuration problems:

  • Does the ESXi host network configuration appear correct? IP, subnet mask, gateway?
  • Is the uplink plugged in? Yes, that had to be said.
    • esxcli network nic list
  • If using VLANs, does the VLAD ID of the port group look correct?
    • esxcli network vswitch standard portgroup list
  • Check the trunk port configuration on the switch. Have there been any recent changes?
  • Does the physical uplink adapter have all settings configured properly? (speed, duplex, etc.)
    • vicfg-nics –d duplex -s speed vmnic#
  • If using NIC teaming, is it setup and configured properly?
  • Are you using supported hardware? Any driver issues?
  • If all of the above test ok, check that you don’t have a physical adapter failure.

If you recently moved the VM to a new host, also verify that an equivalent port group exists on the host and that the network adapter is connected in the virtual machine settings. The firewall in the guest operating system might be blocking traffic. Ensure that the firewall does not block required ports.

Network Troubleshooting Scenario #2 – ESXi hosts dropping from vCenter

Occasionally an ESXi host is added to the vCenter Server inventory with no issues at all, but disconnects 60 seconds after the task completes.

Typically, this issue is because of lost heartbeat packets between vCenter (vpxd) and an ESXi host (vpxa).

The first thing you should check is that no firewall is in place blocking the vCenter communication ports. Then verify that network congestion is not occurring on the network. This issue is more prevalent with Windows based vCenter systems.

Adjust the Windows Firewall settings:

  • If ports are not configured, disable Windows Firewall.
  • If the firewall is configured with the proper ports, ensure that Windows Firewall is not blocking UDP port 902.

By default vpxa uses UDP port 902, but it is possible to change the ports to something else. Check the /etc/vmware/vpxa/vpxa.cfg file <ServerPort> setting.

When it comes to network congestion, dropped heartbeats can happen as well. Some tools you can use to troubleshoot:

  • You can use the resxtop utility or graphical views to analyze traffic.
  • The pktcap-uw command is an enhanced packet capture and analysis tool.
    • pktcap is unidirectional and defaults to inbound direction only.
    • Direction of traffic is specified using –dir 0 for inbound and –dir 1 for outbound.
    • Two (or more) separate traces can be run in parallel but need to be merged later in wireshark.
  • Wireshark

Network Troubleshooting Scenario #3 – No Management Connectivity on ESXi Host

VMware Management networks are configured using VMkernel port groups. Typically, when a host loses connectivity to vCenter and was working prior, a recent change to that port group has caused the issue.

One feature VMware has, which helps in this case is the Rollback feature. Several different types of events can trigger a network rollback:

  • Updating the speed or duplex of a physical NIC
  • Updating teaming and failover for the management VMkernel adapter
  • Updating DNS and routing settings on the ESXi host
  • Changing the IP settings of a management VMkernel adapter

If any of the above are changed and it fails, the host rolls back to the last known good configuration.

You can also restore the network configuration from the DCUI. Select “Network Restore Options” and you can select to restore either standard switches or distributed switches. The Restore Network Settings option deletes all the current network settings except for the Management network if you’re looking to start with a new configuration.

Posted in Troubleshooting | Leave a comment