Azure VMware Solution is powered by VMware Cloud Foundation: vCenter Server, vSphere, vSAN, and NSX-T. Also included is VMware HCX, the Swiss army knife of workload mobility. Customers can securely extend their networks and migrate workloads from on-premises (vSphere 6.x -7.x) to AVS or between AVS private clouds in different regions using a combination of migration options. Microsoft will handle the billing, lifecycle operations (upgrades), and troubleshooting of the service, allowing customers to focus on their workloads.

Note: Updated 3/18/21 to include changes to the service.

Getting Started

There are a few things you need to have in place before you can deploy an Azure VMware Solution private cloud. First, an Azure account, which you can get for FREE. Next, your account must be associated with a subscription that is either part of a Microsoft enterprise agreement or a Cloud Solution Provider (CSP). Finally, is requesting host quota for your subscription. Once quota is applied to a subscription, search for “Azure VMware Solution” in the Azure portal’s search box. Alright, you don’t need to type out the entire thing; it will appear within the first couple of characters in VMware. An option to deploy your first private cloud in your subscription will be displayed on the next screen. As part of the private cloud creation, there is some basic information needed:

• Subscription – Billing framework that provides entitlement to deploy and consume Azure resources
• Resource Group – Logical way to group Azure services
• Location – Where to deploy the private cloud
• Resource Name – Name of your private cloud
• SKU – Node type used during deployment
• ESXi Hosts – Number of hosts to deploy, min of 3 by default with the option to increase to a max of 16 per cluster
• vCenter Admin Password – password used to log in vCenter with cloudadmin@vsphere.local
• NSX-T Manager Password – password used to log in NSX-T manager with admin
• Address Block – CIDR block used when deploying management components, requires a /22
• Virtual Network – A representation of cloud networking and provides abstraction and logical isolation. An Azure environment can contain multiple VNets.

Note: Providing vCenter Server and NSX-T credentials during a private cloud deployment is no longer required, they are now auto generated. If they need to be rotated you will currently need to open a support request.

[Previous private cloud creation screen]

[Updated private cloud creation screen]

Private Cloud

• ExpressRoute – Is a private and secure connection from a customer’s physical datacenter providing dedicated bandwidth into Microsoft Azure.

• Global Reach – Connects ExpressRoute bi-directionally from a customer’s environment to Azure VMware Solution.

A subscription can have 1-4 private clouds, each with a maximum of 4 clusters per cloud. An initial private cloud deployment starts with a 3-node minimum with the opportunity to add additional nodes during or scale-up later to a maximum of 16 nodes per cluster in the Azure portal. The hardware specification dropdown lists AVS36 as the current selectable node type. Below is a visual representation of the hardware specs for the AVS36, but it does not represent the actual server 🙂 An Azure Virtual Network (VNet) can be created during the initial private cloud deployment or afterward. If an existing VNet exists, it can also be used. A VNet is created to support an ExpressRoute from Azure VMware Solution to connect to other Azure services and allow connectivity back to an on-premises environment via Azure Global Reach. Once ready, click review and create. After you verify everything entered is correct, click the magical create button and wait for roughly 2+hrs for the process to complete. The process is mostly self-service from the Azure portal allowing you to get from zero to a private cloud which includes:

• Provisioning of hardware and backbone networking
• Installation and configuration of ESXi, vCenter Server, vSAN, NSX-T, and HCX
• Creation of initial cluster including vSAN datastore encryption

Note: if you don’t have an Azure ExpressRoute, you can use a site-to-site VPN to connect to Azure VMware Solution private cloud, but you will not be able to use HCX for workload migration as this is not supported.

Post Deployment

When your private cloud is ready, you’ll be redirected to the overview page in the resource menu. This page is handy with valuable information; you can always come back here by searching your private cloud name or simply bookmarking. The first section we’ll want to select is identity. Here is where you’ll find your login information for NSX Manager and vCenter Server. Next is clustering, where you can edit (aka increase/decrease) the number of nodes in a private cloud, with 3 being the magic number of minimum nodes. Keep in mind increasing the number of nodes is tied to your allocation associated with the subscription used.

The NSX T-1 router is where all workload network segments need to be created before VMs are deployed or being migrated to a new segment. These segments can be created in NSX manager or directly in the Azure Portal under segments, including the ability to create DHCP servers to handle DHCP requests and DHCP relay services to relay DHCP traffic to external DHCP servers. Additional workload networking options such as port mirroring and DNS are also available.

I’ve mentioned VMware HCX; the good news, it’s automatically deployed as part of the private cloud provisioning. The HCX Cloud manager is where you’ll get the necessary HCX Connector bits to deploy in your on-premises environment, which can be found under the connectivity section in the Azure portal of your private cloud. The HCX Cloud manager address is provided, and you will use cloudadmin@vsphere.local credentials to login and download the HCX connector bits. Licensing your HCX connector is also part of the self-service offering, allowing you to request up to 3 advanced licenses. Additionally, there is an option to upgrade your HCX advanced license to an HCX enterprise license by opening a support request with Microsoft. HCX enterprise will provide features like mobility groups, replication assisted vMotion, mobility optimized networking, and more. These features make use cases like datacenter extension and evacuation easier to any VMware powered cloud.

Once the HCX connector is deployed and configured on-premises, customers can access and manage directly from the vSphere client, the dedicated HCX Client, or via automation using PowerCLI or REST APIs. A site pairing can then be established with on-premises environments and an Azure VMware Solution private cloud, followed by L2 connectivity for workloads that will retain their IP addresses. Let the migration planning begin!

There are additional support and licensing benefits that come with Azure VMware Solution. Microsoft is providing extended support for Windows Server 2008 and SQL SQL 2008 workloads when they are migrated to Azure, including the Azure VMware Solution. There is also the Azure Hybrid Benefit which is a licensing benefit allowing customers to use their on-premises Windows Server and SQL Server licenses on Azure.

Resources

Here are a few resources help get you started in learning more about Azure VMware Solution and more will be added as they are available. Also please reach out if you know of any others!

• VMware Hands On Lab Azure VMware Solution
• Platform updates for Azure VMware Solution
• FAQ: Common questions about Azure VMware Solution
• VMworld: Azure VMware Solution Networking & Security in a Hybrid Cloud Environment
• Azure VMware Solution Product Page
• Azure VMware Solution Documentation
• Video Playlist from Microsoft Global Blackbelt Trevor Davis
• Virtual Workloads Blog – Trevor Davis
• Microsoft Ignite 2020 Session: Accelerate your Cloud Journey with Azure VMware Solution
• Microsoft Ignite 2020 Session: An introduction to Azure VMware Solution
• Set up vRealize Operations for Azure VMware Solution
• Deploying VMware Horizon on Azure VMware Solution

As part of the GCVE service availability, customers have several pricing options: on-demand (per-hour), 1yr commitment, and 3yr commitment, which includes the cost of infrastructure, licensing, and support. Billing is handled through Google Cloud; for more details, please see the following. The GCVE service allows customers to focus on workloads while Google manages all other aspects of the service, including the lifecycle of the vSphere environment.

GCVE Service

* Project – must be created as part of the Google Cloud service and is a way to group and manage resources.
* Private Cloud – is the equivalent of a VMware Software-Defined Data Center (vCenter Server, ESXi, vSAN, and NSX).

To get started with GCVE first requires enabling the VMware Engine API in the Google Cloud console. This will need to be done on a per-project basis. Next, you’ll need to request a quota which is located under IAM & Admin. A quota is a way to verify, protect against unexpected usage, and prevent misuse. Each project in Google Cloud could have a different quota. Once your quota request has been accepted, we can start getting ready to deploy a GCVE private cloud.

The GCVE service configuration will consist of a minimum of 3 nodes and can scale up to 16 nodes per cluster (roughly 15 mins per node add) or 64 nodes in a single private cloud. Each GCVE node has the following specifications:

• • CPU – 2x Intel Xeon Gold 6240 (Cascade Lake) 2.6 GHz, 36 Cores, 72 Hyper-Threads
    • Memory – 768 GB
    • Storage:
      • 2x 1.6 TB (3.2 total TB) NVMe for cache
      • 6 x 3.2 TB (19.2 TB Total) NVMe for data
    • Network – 4 x Dual Port 25GbE

GA included software specifications (keep in mind these versions will change over time):

[table id=14 /]

Part of the service level agreement (SLA) ensures customers have the needed reliability. GCVE is providing 99.99% of network uptime (less than an hour per year) for its service and has network redundancy across rack configuration. Connectivity from on-premises to GCVE (or between private clouds within GCVE) can be established using a Google Cloud VPN, the quickest method (self-service) to getting started. There is also the option to use Google’s Cloud Interconnect (available from Google or Service Providers), a dedicated, highly available, low latency direct connection. As part of the configuration process, customers will need to create Virtual Private Cloud (VPC) within the Google Cloud that allows them to control how workloads connect within a region or globally, create policies within projects, and expand IP spaces without downtime. After the VPC is created, workloads can communicate internally within a project, keeping traffic local and providing extensibility to Google Cloud Services.

A CIDR block is required to deploy all the vSphere management components within GCVE, but workload networks will be provisioned from NSX-T (NSX segments). There will be DNS servers deployed within a private cloud and are only used for management. Within the network section of the GCVE console, customers can allocate public IP addresses to workloads, which are protected by Google Cloud Edge. The VPN Gateway section provides a point to site VPN that gives access to the management components like vCenter Server within GCVE.

There are a couple of things I want to highlight within the GCVE service. First, when it comes to deploying a private cloud, GCVE has fast provisioning, where it takes approximately 30 minutes to deploy a private cloud with all components running (the clock starts when you click on create). On the back end, the GCVE service has built-in intelligence and optimization magic sauce allowing it to pre-provision capacity on unused nodes before they are requested. These nodes are then re-purposed based on the customer’s specifications and capacity. This ensures deployments are the same as if they were installed from scratch.

Second, is how Role-Based Access Control (RBAC) works in GCVE. It starts with the Google Cloud console, to gain access to GCVE you either have to be an owner or editor of a project in the Google Cloud console or granted the following roles within a project’s IAM roles and permissions:

• • • VMware Engine Service Viewer (read-only access)
      • VMware Engine Service admin (full admin access)

Once access has been granted, the GCVE portal has three RBAC options: CloudOwner, Elevated Privilege, and Service Provider. The CloudOwner account is the default account; it allows customers the access needed to handle their day-to-day operations on their workloads and provides full access to NSX-T networking (including NSX-T manager). Elevated privilege is a temporary and time-based request that allows access to hosts, storage, and management VMs to create and add identity sources and auth management; this level of access is monitored. The GCVE service monitors actions of the elevate privilege and provides notification if an action is disruptive to the service and the customer, potentially preventing those actions from happening. It also allows the creation and addition of accounts necessary for 3rd party integration, such as backup solutions. Customers can request Elevated Privilege within the GVCE portal on-demand. Service Provider access is only available for Google support only and provides full access to the customer’s private cloud within GCVE for troubleshooting.

When first logging into vSphere Client, you will be prompted to change the default CloudOwner@gve.local account password, which must be changed every 365 days afterward. There are also some pre-created roles that can be used, or you can create your own, but no role can be granted privileges higher than CloudOwner. You can find a list of the GCVE vCenter Server roles and privileges here. GCVE also supports adding your identity solution, which allows you to use your on-premises groups in vCenter Server.

All service and access configurations are done from the GCVE console; this includes launching the vSphere Client, network configuration/management, and Private Cloud management (deployment, configuration, and capacity). The GCVE console allows the management of different private clouds from a centralized location. It also provides an activity view that displays alarms, events, tasks, and audits, which tracks all the changes made across private clouds. Customers also have the option to download report information from various sections of the GCVE console in CSV format.

Workload Mobility

Included with the GCVE service is VMware HCX (advanced licensing), which allows customers to migrate workloads as far back as vSphere 5.0 from on-premises to GCVE without re-platforming. It’s the swiss army knife of workload mobility. VMware HCX Cloud Manager is deployed, configured, and licensed out of the box as part of the GCVE private cloud deployment, requiring a CIDR block for the VMware HCX components. Once the HCX Cloud Manager is deployed, customers can download the HCX Connector OVA to deploy in their on-premises data center, allowing them to pair with their GCVE private cloud. With a site pairing established, HCX provides the option to extend workload networks (L2 Stretch) and migrate using different migration types based on your SLAs: Bulk Migration – min downtime, vMotion – no downtime, and Cold Migration – downtime.

Additional functionality and migration types are also included with the HCX enterprise add-on licensing:

• • • Replication Assisted vMotion – the ability to migrate with zero downtime
      • Mobility Groups -create migration groups based on criteria and migrate in waves
      • more

These are a few examples of what’s included with VMware HCX enterprise licensing, contact your VMware/Google representative for more details. This also simplifies uses cases such as data center evacuation, disaster recovery, and hybrid connectivity.

Resources

With the launch of the new Google Cloud VMware Engine service I wanted to share some resources to help get you started and bookmark:

• • Google Cloud VMware Engine Documentation
    • Release Notes
    • Known Issues
    • Service Level Agreement
    • Prerequisites
    • Pricing
    • Ecosystem Partners
    • Google Cloud Next ’20 – Filtered with VMware as a Service

When a VM gets created within a vCenter Server, it generates a MoRef ID that helps uniquely identify it. A MoRef ID gets created not just for VMs but all objects (hosts, folders, datastores, etc.) within a vCenter Server, including the vCenter Server itself. If you’ve heard me talk about using the vCenter Server Migration Tool (Windows to VCSA), I reference the importance of a vCenter Server keeping its MoRef ID, which is how other solutions will identify a vCenter Server. In this case, the Windows vCenter Server MoRef ID gets retained during the migration process, which means any registered solutions will continue to identify the original vCenter Server unknowingly of the migration process 🙂

Another way to uniquely identify a VM within a vCenter Server is by using its instanceUUID, introduced in vSphere 4.0. If you’ve explored inside the VM .vmx file, there is an entry called vc.uuid, which is a 128bit integer and looks something like “50 3e 84 9c 71 56 12 29-a6 a7 00 d3 3c a5 d3 28”. Both MoRef and InstanceUUID information is obtainable using the vSphere API. For more details, I recommend looking at the following blog posts by William Lam. This post builds on the information he provided, focusing on HCX migration types with VM MoRef ID and InstanceUUID.

• Uniquely Identifying Virtual Machines in vSphere and vCloud Part 1: Overview
• Uniquely Identifying Virtual Machines in vSphere and vCloud Part 2: Technical
• Uniquely Identifying VMs in vSphere Part 3: Enhanced Linked Mode & Cross VC-vMotion

HCX VM Migrations

With HCX, VMs can easily migrate using various migration types (Cold, vMotion, Bulk, RAV) between vCenter Servers, whether on-prem vSphere (5.0-7.0) or cloud (VMware Cloud on AWS, Azure VMware Solutions, etc.). In this example, I have a source on-prem vCenter Server using HCX to migrate VMs to the destination vCenter Server in VMware Cloud on AWS (VMC). Each VM name contains the migration type that will be used when its migrated to VMC with HCX. Before starting the migration, I grabbed the MoRef ID, InstanceUUID, and Tag for each VM as well as the source vCenter Server Name and its InstanceUUID.

After the migrations completed successfully to the destination vCenter Server, we can see only one difference. All the VMs received a new MoRef ID, which is as expected since its managed per vCenter Server independently. The VM instanceUUID did not change, and will only change if a VM with the same UUID already exists on the destination vCenter Server.

I was also able to pull the destination vCenter Server UUID, VM Name, and VM MoRef ID (HCX has this labeled as Id) using the Get-HCXVM cmdlet.

HCX removes barriers! These are two distinct vSphere Single Sign-On (SSO) domains between on-prem and VMC, by combining VM instanceUUID with the vCenter Server unique identifier, you can identify VMs across multiple vCenter Servers. Another way to uniquely identify VMs in your environment is the use of vSphere tags. HCX will retain VM tag and category information from the source to the destination vCenter Server, see examples above. Now you can migrate successfully using HCX while understanding the impact on your VMs IDs and solutions that rely on them.

I spent the first three years working alongside a group of highly-skilled technical marketing architects in the Cloud Platform business unit (CPBU), part of the R&D organization at VMware. My first focus area was vCenter Server, a product I was familiar with and used as a customer. I started out helping customers migrate from vCenter Server for Windows to the vCenter Server Appliance (VCSA) and covering the vCenter Server product overall. We did these migrations using various methods with no to minimal downtime, depending on the customer environment. Of course, this was before the release of the first official migration tool, vCenter Server 6.0 U2m, announced during VMworld 2016.

Once the official vCenter Server migration tool was available, it made sense to also focus on vSphere lifecycle. The vCenter Server migration tool not only copied data (inventory and configuration by default) from one platform to another (vCenter Server for Windows to the VCSA), the process was also an upgrade. During this time, I worked on a series of content that included scenarios and later an ebook to help customers with their vSphere upgrades. Adam Eckerle and I also started to create workshops that we delivered at various VMUGs, VMworld US, and VMworld EMEA. These workshops became in such high demand; this led to the creation of a team under the vSphere Tech Marketing managed by Adam to specifically help with vSphere lifecycle.

Shortly after, in addition to vCenter Server, I started working on VMware Cloud on AWS (VMC) with a focus on hybridity and workload mobility. It was exciting being part of something like VMC from the beginning and getting the opportunity not only to influence the service but also work on new technology such as stretched clusters and hybrid linked mode. When it came to workload mobility, this was my first introduction to VMware HCX. An all-in-one platform that simplified workload mobility from any site, whether on-premises or cloud. Around this time, I had been in technical marketing for about 2 1/2 years and began to think about what I wanted to do next in my career.

An opportunity to join the vSphere product management team presented itself working on driving the direction for vCenter Server lifecycle. Towards the end of 2018, I joined the PM team working on new functionality to help simplify vSphere lifecycle. While I can’t go into details, I would recommend watching this VMworld session presented by William Lam and myself and pay attention to the tech preview section. Having been in the PM role for a year, I learned so much about product development, delivery, and the number of meetings one could have on their calendar.

Recently another opportunity presented itself, and I’m excited to announce that I have joined the VMware HCX team as Customer Zero. In this role, I’ll be working closely with VMware HCX product management, engineering, and design on upcoming enhancements/functionality and also providing early feedback on usability and integrations, including on-premises, VMware Cloud on AWS, Google Cloud VMware Solution, and Azure VMware Solution. There is also the bonus of getting to work with a community member and food thief (guard your plate), Amy Lewis. I look forward to this new opportunity and sharing everything coming from the VMware HCX team.

• The first misconception is an embedded deployment is a new deployment model. The embedded deployment model is not a new concept. It has been around since the initial VirtualCenter (2004) deployment. Over time VMware introduced new vCenter Server services including the PSC. With the addition of the PSC services, the embedded deployment model did not support ELM. Greenfield deployments on vSphere 6.5 and 6.7 support embedded deployments with ELM.
• Another misconception is the maximums supported differs between the two deployment models. The embedded deployment supports the same vCenter Server maximums as the external deployment. This includes the same number of hosts, VMs, datastores, etc. there is no difference. Also, the embedded deployment supports a total of 15 vCenter Servers in a vSphere SSO domain.
• A third misconception is with a PSC failure, an embedded deployment is likely to fail. A PSC failure, whether external or embedded, has the same result. Users will no longer be able to authenticate to the registered vCenter Server(s). The probability of PSC service failing when embedded is slim. Using vCenter Server High Availability (VCHA) is the preferred method for protecting against these types of failures. This also removes the overhead of using a load balancer for PSC high availability.

Convergence Tool Overview

[Update: The vCenter Server Appliance convergence tool is now also included in vCenter Server Update 2d. Same steps below apply]

The convergence tool is only available on the vCenter Server Appliance (VCSA) 6.7 Update 1 ISO. If you are familiar with the VCSA lifecycle JSON templates, the converge CLI tool is similar. There are two JSON templates, converge and decommission. In an external deployment, the PSC is running on a separate node. The necessary PSC RPMs are not installed on the registered vCenter Server. The converge tool will handle the installation of the PSC RPMs on the vCenter Server. There is no manual intervention required and it does not make any changes to the external PSC. After the installation of the PSC, it then registers with the vCenter Server.

Next is the replication of the vSphere SSO data from the external PSC to the embedded PSC. The PSC replication partner is part of the converge tool JSON template, more on that in a bit. Only during convergence is replication from an external PSC to embedded PSC supported. Once the replication is complete, it time to decommission the external PSC. Let’s walk through the steps in more detail.

Note: The convergence process can only run on one VCSA at a time.

Prerequisites

There are a few prerequisites and considerations before starting the convergence process:

• The converge tool only supports the VCSA and PSC 6.7 Update 1. All nodes must be on 6.7 Update 1 before converting.
• If you are currently running a Windows vCenter Server or PSC, you must first migrate to the appliance.
• Before converting take a backup of your VCSA(s) and a least one of the PSCs in your vSphere SSO domain.
• Know all other solutions (i.e. SRM, vROPS, NSX, etc.) using the PSC for authentication in your environment. They will need to be re-registered after the convergence completes and before decommissioning.
• A machine on a routable network which can communicate with the VCSA and PSC will be used to run the convergence and decommission process.
• It is recommended to change the DRS Automation Level to partially automated or manual and the Migration Threshold to conservative. There could be issues if the VCSA being converged is moved during the process.
• If VCHA is enabled, it must be disabled prior to running the convergence process.
• The converge process will handle PSC HA load balancers, make sure you point to the VIP in the JSON template.
• All vSphere SSO data is migrated with the exception of local OS users.

Note: If the VCSA File-Based backup is running using the scheduler the convergence process will fail. You must delete the scheduler prior to starting the convergence process. It can be re-enabled after the convergence process is complete.

Convergence Steps

1. Backup all VCSAs and at least one PSC (multi-master) within your vSphere environment.
2. Mount the VCSA 6.7 Update 1 ISO on a machine with routable network access to the VCSA and PSC. The VCSA ISO supports macOS, Linux, and Windows.
3. Expand the vcsa-converge-cli directory and go to templates. Open the converge directory and copy the converge.json file to your local machine.

4. Open the local copy of the coverge.json file with your favorite editor. Fill out the JSON template with the required information. There are two sections to pay close attention to. The first is the “ad_domain_info” section. If your PSC is a member of an active directory domain, this section needs filled out. Otherwise, this entire section is not needed. The second section is the “replication” section of the JSON template. If there is only a single PSC with no replication partners, then this section is not needed. Use the vdcrepadmin command to check the replication agreements if you are unsure. Note: Use ./vcsa-util converge – –template–help for additional help filling out the converge JSON template.

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
   29
   30
   31
   32
   33
   34
   35
   36
   37
   38
   39
   40
   41
   42
   43
   44
   45
    <pre class="lang:default decode:true">{
        "__version": "2.11.0",
        "__comments": "Template for VCSA with external Platform Services Controller converge",
            "vcenter": {
                "description": {
                   "__comments": [
                        "This section describes the vCenter appliance which you want to",
                        "converge and the ESXi host on which the appliance is running. "
                    ]
                },
                "managing_esxi_or_vc": {
                    "hostname": "esx-02.cpbu.corp",
                    "username": "root",
                    "password": "P@ssw0rd"
                },
                "vc_appliance": {
                    "hostname": "vcsa-01.cpbu.corp",
                    "username": "administrator@vsphere.local",
                    "password": "P@ssw0rd",
                    "root_password": "P@ssw0rd"
                },
                "ad_domain_info": {
                    "__comments": [
                        "Important Note: This section is needed only when PSC (Platform Services Controller) appliance is joined to a domain.",
                        "Remove this section if PSC appliance is not joined to a domain.",
                        "Keeping this section without valid values results in JSON validation errors."
                    ],
                    "domain_name": "cpbu.corp",
                    "username": "emad",
                    "password": "P@ssw0rd",
                    "dns_ip": "192.168.13.50"
            },
        "replication": {
                "description": {
                   "__comments": [
                   "Important Note: Make sure you provide the information in this section very carefully, as this changes the replication topology.",
                   "Refer to the documentation for complete details. Remove this section if this is first converge operation in your setup.",
                   "This section provides details of the PSC node which will be set up as a replicated node for a new PSC on the target VCSA node."
                ]
                    },
                "partner": {
                    "hostname": "psc-02.cpbu.corp"
                }
            }
    }
   

   Note: You can leave the passwords for any of the sections blank in the template and will be prompted during the convergence process.

5. After completing the converge JSON template, make sure you save it.
6. Open a terminal window to the vcsa-util application path. Depending on the local operating system in use will determine the directory. In this example, I’m using macOS and using the vcsa-util under the mac directory.
7. Run ./vcsa-util converge – –help to get a list of the supported parameters.
   
8. To start the convergence process, use the correct parameters followed by the path of the converge JSON template you saved earlier. ```

   ./vcsa-util converge --no-ssl-certificate-verification --backup-taken --verbose /Users/eyounis/Documents/converge.json
    ```
   

9. Once the convergence process begins, your VCSA will not be available until the process is complete.
10. After the convergence process has completed, verify the VCSA is now running with an embedded PSC. There are two ways to verify. The first is using the VMware Appliance Management on port 5480. Go to the Summary tab and verify the type is “vCenter Server with an embedded Platform Services Controller”. A second option is using the vSphere Client (HTML5). Look under the Administration menu for System Configuration and verify the type as above.
11. If there are any products using the PSC for authentication they will need to be re-registered. Re-register them with the embedded VCSA deployment before decommissioning the external PSC. Also, do not decommission an external PSC until all VCSAs registered have gone through the convergence process.

Decommissioning Steps

*Verify the steps above have been completed prior to starting the decommissioning process*

1. Expand the vcsa-converge-cli directory and go to templates. Open the decommission directory and copy the decommission_psc.json file to your local machine.
2. Open the local copy of the decommission_psc.json file with your favorite editor. Fill out the JSON template with the required information. If the default port of 443 was changed, make to enter the custom port that was used in the decommission template in the specified sections. ```

   {
        "__comments": "Template for decommissioning PSC node with converge CLI tool.",
        "__version": "2.11.0",
            "psc": {
                "description": {
                   "__comments": [
                       "This section describes the PSC appliance which you want to",
                        "decommission and the ESXi host on which the appliance is running. "
                    ]
                },
                "managing_esxi_or_vc": {
                    "hostname": "esx-02.cpbu.corp",
                    "username": "root",
                    "password": "P@ssw0rd"
                },
                "psc_appliance": {
                    "hostname": "psc-01.cpbu.corp",
                    "username": "administrator@vsphere.local",
                    "password": "P@ssw0rd",
                    "root_password": "P@ssw0rd"
                }
            },
            "vcenter": {
                "description": {
                   "__comments": [
                        "This section describes the embedded vCenter appliance which is in ",
                        "replication with the provided PSC"
                    ]
                },
                "managing_esxi_or_vc": {
                    "hostname": "esx-02.cpbu.corp",
                    "username": "root",
                    "password": "P@ssw0rd"
                },
                "vc_appliance": {
                    "hostname": "vcsa-01.cpbu.corp",
                    "username": "administrator@vsphere.local",
                    "password": "P@ssw0rd",
                    "root_password": "P@ssw0rd"
                }
            }
    }
    ```
   

3. After completing the decommission JSON template, save it.
4. Open a terminal window to the vcsa-util application path, same steps as above.
5. Run ./vcsa-util decommission – –help to get a list of the supported parameters.
6. To start the decommission process, use the correct parameters followed by the path of the decommission JSON template you saved earlier. ```

   ./vcsa-util decommission --no-ssl-certificate-verification --verbose /Users/eyounis/Documents/decommission_psc.json
    ```
   

7. The decommissioning process will begin removing the external PSC from the vSphere SSO domain. There are validations included to ensure there are no registered VCSAs to the external PSC being decommissioned. If any are found, the decommissioning process will fail.
   

Summary

Great job by the vCenter Server engineering team for all their hard work on this tool. As stated earlier VMware’s recommended deployment model going forward is embedded. I would say the writing is on the wall for the external deployment now that the convergence tool is available. The next blog post will cover a few topology convergence examples. Happy converting 🙂

The vSphere SSO domain also is receiving more flexibility with support for embedded repointing. Now a single embedded deployment can move to another vSphere SSO domain. Another feature with enhancements in vSphere 6.7 Update 1 is Content Library. It now supports native vCenter Server templates (.vmtx) and Open Virtual Appliance (OVA). Of course, there are more new features and enhancements in this release. Without further ado, here is what’s new in vCenter Server 6.7 Update 1.

vSphere Client

The wait is finally over! We now have a fully featured vSphere client in vSphere 6.7 Update 1. Delivering a fully featured client to customers was only part of the process. The vSphere team also wanted to optimize the vSphere Client’s performance and workflows. Ensuring the best customer experience possible. This release includes smart porting over and completing the remaining features which included:

• vCenter High Availability (VCHA)
• Auto Deploy
• Host Profiles
• vSphere Update Manager
• Network Topology Diagrams
• Performance Charts
• Improved Searching
• Dark Theme
• more…vSphere Client Youtube Playlist

There are two features I want to highlight from the vSphere Client in vSphere 6.7 Update 1. The first is the improved searching capabilities. Customers can now search objects with a string and filter based attributes. These attributes include Tags, Custom Attributes, and VM power state to name a few. For common searches, there is also an option to save the search for future usage.

The second feature I want to highlight is considered one of the top feature requests. The vSphere team has been listening and now the dark theme option is available. Now customers can switch between the traditional light theme to the new dark theme in a single click. When switching to dark theme a blue information banner appears at the top of the screen. In this release dark theme supports vSphere features and plugins. Keep in mind other features and plugins may not render well until the necessary changes are in place. Glad the vSphere team was able to get this nice surprise in this release.

Convergence Tool

vCenter Server is going back to its roots of a simplified architecture. vSphere 6.0 introduced us to a new vCenter Server service called the PSC. The role of the PSC is to manage the following vSphere SSO domain components:

• Authentication
• Licensing
• Tags & Categories
• Global Permissions
• Custom Roles
• Certificates

The PSC has two deployment models, embedded and external. The only reason to use an external PSC at the time was for Enhanced Linked Mode (ELM). Otherwise, use the embedded deployment model for simplicity. Starting with vSphere 6.7 and 6.5 Update 2 the embedded deployment model supports ELM, but only for greenfield deployments. Until now customers who deployed an external PSC for ELM had no way to change their topology to an embedded PSC. vSphere 6.7 Update 1 introduces a new CLI tool to converge a PSC deployment from external to embedded.

The convergence tool is available in a directory called “VCSA-Converge-CLI” on the VCSA 6.7 Update 1 ISO. Before using the convergence tool there are two important prerequisites. First, a vSphere environment must be running vSphere 6.7 Update 1. This includes all the external PSCs and vCenter Servers in the vSphere SSO domains. Second, the convergence tool does not support Windows PSCs or vCenters Servers. Meaning migrate to the vCenter Server Appliance (VCSA) you should!

There are two JSON template options, converge and decommission. We start by using the converge JSON which will install the necessary PSC RPMs on the external VCSA. After the PSC service gets registered on the external VCSA, its now converted to embedded. But we’re not done, we still need to replicate the vSphere SSO data between the external PSC and the embedded PSC. This is actually taken care of as part of the JSON template and requires no manual intervention. The final step is to decommission the external PSC using the decommission JSON template. Stay tuned, there will be a more detailed blog post with examples coming soon.

Note: Embedded PSC is the recommended deployment model for vCenter Server.

Embedded Domain Repoint

Domain repoint is a feature available in vSphere 6.7 using the cmsso-util CLI command. Customers can repoint an external vCenter Server across a vSphere SSO domain. New in vSphere 6.7 Update 1 is support for embedded deployment domain repoint. Remember the time you selected a new vSphere SSO domain instead of an existing? Leaving you with several single embedded deployment instances. With domain repoint we can now join them to a single vSphere SSO domain. Other use cases are merges or acquisitions, no longer requiring a setup of a new vSphere SSO domain.

vCenter High Availability

vCenter Server native high availability was first introduced in vSphere 6.5. This is an active/passive solution designed to protect against application failure. In this case, it’s protecting the vCenter Server application. In vSphere 6.7 Update 1 vCenter High Availability is an example of a smart port workflow. There is no longer two workflows of basic and advanced, now only one workflow. The new workflow is a three-step configuration:

• vSphere SSO credentials for either management vCenter Server or management credentials
• Resource settings for the Passive and Witness nodes including compute, storage, and network
• IP Settings for the Passive and Witness nodes

Also included is auto-creation of the Active node clones to create the Passive and Witness nodes. A prerequisite for auto clone provisioning is entering the vSphere SSO domain credentials. Regardless of its self-managed or managed by another vCenter Server. Other enhancements including auto-detection of VCHA during an upgrade. Customers no longer have to destroy a VCHA configuration before upgrading to vSphere 6.7 U1. The upgrade wizard is aware of the configuration and will handle it as part of the upgrade process. We’ll save the best for last, REST APIs for VCHA are also included in the VCSA 6.7 Update 1 API Explorer.

Content Library

This feature has been around since vSphere 6.0 and is one of my favorites. As a customer, I had to manage several images, scripts, OVAs, and templates for several remote sites. Having to maintain and update the same content in several locations can be a nightmare. Content Library is a native vCenter Server service which solves this problem. It uses the subscriber/publisher model to distribute content. There are two deployment models when using Content Library:

• Single vCenter Server managing several sites
  • Centralized managed content
  • Uses WAN to sync content, requires publishing vCenter Server to be available
  • Content stored on a datastore at the remote sites
  • When provisioning content at the remote sites, traffic is kept local
• Multi vCenter Server
  • Centralized managed content
  • Uses WAN to sync content
  • Can sustain publishing vCenter Server failure, remote locations continue to operate
  • Content stored on a datastore at the remote sites
  • When provisioning content at the remote sites, traffic is kept local

This release Content Library now supports two more new file formats. This functionality is already available to VMware Cloud on AWS customers. Now it’s making its way to on-premises vSphere customers in vSphere 6.7 Update 1. Customers can now manage their templates on the local vCenter Server Content Library. Replication of template files to other subscriber Content Libraries is not available in this release. The other new supported format is Open Virtual Appliance (OVA). When importing an OVA file format in a Content Library, its content is first extracted. This allows Content Library to validate the OVA manifest and certificate. After the validation process is complete, it creates an OVF item for deployment. Give Content Library a try in your environment and let us know your thoughts.

vSphere Health

This is a new feature with huge potential. When deploying your vCenter Server there is an option to enable CEIP. It stands for Customer experience improvement program. When enabled data gets sent to VMware’s analytics cloud, customer workload data is not part of the data collected. This is all anonymous, data is secure and housed in VMware’s data centers. VMware uses this telemetry data to help improve features and customer experience. Now when enabled this functionality provides even more value to the customers. It provides an assessment of the vSphere environment and returns KBs on how to solve the problem at hand. For example, the warning you get when you only have a single management network for your ESXi hosts. We detect and provide alarms for both existing and potential problems. This helps in protecting customer environments from future problems.

Keep in mind this is a new service and will not have all health checks day one. The good news is this services independent of the vSphere release cycle. It features an out-of-band delivery infrastructure. If you are running vSphere 6.7 you also have the same capabilities as well. The goal here is overtime this service will become more proactive to ensure you have a healthy vSphere environment. If you haven’t enabled CEIP during your vSphere deployment, it’s easy enough to enable and gain the new benefits this service has to offer.

Appliance Management

The appliance management interface (VAMI) continues to gain new functionality with every release. vSphere 6.7 Update 1 continues to add-on from what vSphere 6.5 started. In this release, there is a new tab called “Firewall”. Now customers can manage firewall rules on the VCSA from the VAMI. This was previously only available using the VAMI APIs.

Another new feature which isn’t obvious is the ability to login VAMI with a local SSO user account. This release supports local vSphere SSO users who are members of the “SytemConfiguration.Administrators group. Also, members of the “Systemonfiguration.BashShellAdministrators” can use their SSO account to log in the VCSA bash shell. This makes it easier from a security perspective to track who logged in and what they did.

Summary

As you can see vSphere 6.7 Update 1 is full of new features and enhancements for vCenter Server. We will go into more details on these new vCenter Server features and enhancements in future posts. Also, if you have any feedback on the new features or enhancements, please don’t hesitate to reach out.

• How will the workloads get migrated?
• Does the migration type meet the SLA of the workloads?
• What is the impact of the migration to the workloads (downtime)?
• Are there any required changes pre and post-migration?
• Is the proper networking (secure) in place and is there enough bandwidth?
• Was a discovery of the environment done, including mapping of dependencies?

As you start to ask these questions more will come up. The point is you can’t migrate to the Cloud without understanding the impact. The first blog post of this series was an overview of Hybrid Cloud Extension (HCX). This post will focus on the available migration types HCX offers.

Migration Types

HCX supports four different migration types:

• Cold Migration – offline
• vMotion (live) – no downtime
• Bulk Migration – low downtime
• Cloud Motion with vSphere Replication – no downtime

Cold migration and vMotion should be familiar to those using the built-in vSphere mobility options. Bulk Migration and Cloud Motion with vSphere Replication are new options and only available with HCX. Let’s take a look at how each option works and what is the impact to the workloads.

Cloud Motion with vSphere Replication

The new kid on the block announced during the VMworld 2018 US keynote. This new option provides zero downtime for workload mobility from source to destination. First, the workload disk(s) (VMDK) get replicated to the destination site. The replication is handled using the HCX built-in vSphere replication. This process is dependent on the amount of data and available network bandwidth. Once the data sync is complete the HCX switchover initiates a vMotion. The vMotion migrates the workload to the destination site and synchronizes only the remaining data (delta) and workload memory state. There is an option to schedule a maintenance window for the vMotion data sync switchover otherwise, it happens immediately.

When using the scheduling option, there are a few things to be aware of. First, the switchover can be re-scheduled any time prior to the actual scheduled time. Secondly, if the initial vSphere Replication data sync has not completed prior to the scheduled window, the workload(s) being migrated will fail. In this case, these workload migrations must be restarted in their entirety. Finally, if the vSphere Replication data sync has completed prior to the scheduled switchover window, HCX will continue to sync delta information to keep the migrated data as current as possible prior to the vMotion switchover. The vMotion switchover happens in a serialized fashion, only one at a time. Today Cloud Motion with vSphere Replication is uni-directional from on-premises to VMware Cloud on AWS (VMC). This is due to HCX leveraging changes in the newer vSphere platform available in VMC. As a result, only VMware Cloud on AWS supports Cloud Motion with vSphere Replication today. Stay tuned, more will be coming for on-premises to on-premises migrations using Cloud Motion with vSphere Replication.

Cloud Motion with vSphere Replication Notes

• Uni-directional from on-premises to VMware Cloud on AWS (see details above)
• Supports source vSphere (vCenter Server / ESXi) versions 5.5 and above
• Cross-version migration from vSphere 5.5 to the current release of 6.7 Update 1
• Currently no support for on-premises to on-premises migrations
• Concurrent migrations = 100 (initial data sync using vSphere Replication)

Bulk Migration

Bulk Migration creates a new virtual machine on the destination site. This can either be on-premises or VMC and retains the workload UUID. Then it uses vSphere Replication to copy a snapshot of the workload from source to destination site while the workload is still powered on. In this case, a snapshot is a point of the time of the workload disk state, but not a traditional vSphere snapshot. The Bulk Migration is managed by the HCX interconnect cloud gateway proxy. We’ll go into more details about the HCX interconnect cloud gateway in the deployment post. During the data sync, there is no interruption to the workloads. The data sync is dependent on the amount of data and available bandwidth. There is an option to schedule a maintenance window for the switchover (similar to the schedule options described in the Cloud Motion section above) otherwise, the switchover happens immediately. Once the initial data sync completes, a switchover takes place (unless scheduled). The source site workloads are quiesced and shut down leveraging VMware Tools. If VMware Tools is not available, HCX will prompt you to force power off the workload(s) to initiate the switchover. During the switchover process, a delta sync occurs based on changed block tracking (CBT) to sync the changes since the original snapshot. The workloads on the destination site will begin to power on once the data sync is complete (including delta data changes). There are checks in place to ensure resources are available to power on the workloads. If a destination workload cannot power on due to resources, the source workload will get powered back on.

Unlike cold migration, the downtime incurred after the bulk migration cut over is low. The HCX manager renames the original source workloads by appending a binary timestamp. A migrated VMs folder gets created in the VM and templates view. Finally, the original source workloads get placed in the migrated VMs folder. This also provides an option for easy rollback and quick data seeding since bulk migration supports reverse migrations. One thing worth mentioning is HCX has built-in network resiliency and will pick up where it left off during a data sync if there is any network interruption. On the flip side, if there is an issue powering on workloads on the destination site or missed scheduling a cut over, the entire data sync process will need to be started over. This technology allows a workload migration between different chipset versions (e.g. Sandy Bridge to Skylake) and across different CPU families (e.g. AMD to Intel).

Bulk Migration Notes

• Automatic update of VM compatibility (hardware) version and VMware Tools available during the switchover
• Supports vCenter Server 5.1 and above / ESXi 5.0 and above
• Cross-version migration from vSphere 5.0 / 5.1 to the current release of 6.7 Update 1
• Concurrent migrations = 100
• Bi-directional migration (reverse HCX option)

vMotion “Live Migration”

HCX supports the vMotion we know and love today. The workloads are migrated live with no downtime similar to Cloud Motion with vSphere Replication. vMotion should not be used to migrate hundreds of workloads or workloads with large amounts of data. Instead, use Cloud Motion with vSphere Replication or Bulk Migration. Usually, a vMotion network needs to be configured and routed to the target vSphere host, in this case, the vMotion traffic is handled by the HCX Interconnect cloud gateway for cross-cloud vMotion. vMotion through HCX encapsulates and encrypts all traffic from source to destination removing network complexity of routing to cloud. The vMotion captures workload:

• Active Memory
• Execution State
• IP Address
• MAC address

HCX has a built-in option to retain the workloads MAC address. If this option is not checked, the workloads will have a different MAC on the destination site. Workloads must be at compatibility (hardware) version 9 or greater and 100 Mbps or above of bandwidth must be available. With vMotion and bi-directional migration, it’s important to consider Enhanced vMotion Compatibility (EVC). The good news here is HCX also handles EVC. The workloads can be migrated seamlessly and once rebooted will inherit the CPU features from the target cluster. This allows a cross-cloud vMotion between different chipset versions (e.g. Sandy Bridge to Skylake) but within the same CPU family (e.g. Intel). Also, an important thing to note is vMotion is done in a serialized manner. Only one vMotion occurs at a time and queues the remaining workloads until the current vMotion is complete.

vMotion Notes

• Does not support disk sizes over 2TB and shared VMDK files
• Attached virtual media or ISOs can be removed prior using the HCX global option
• Does not support compatibility (hardware) version 8 or below
• Supports vSphere ESXi Host versions 5.5 and above
• Cross-version migration from vSphere 5.5 to the current release of 6.7 Update 1
• Bi-directional migration (reverse HCX option)

Cold Migration

As the name states, a cold migration occurs when a workload is offline. CPU compatibility checks do not apply during a cold migration. During a cold migration, VMware’s Network File Copy (NFC) protocol is used to transfer the workloads from source to destination which includes:

• Configuration files including BIOS settings (NVRAM)
• Disks associated with the workload
• Log files

Usually, cold migration traffic takes place over the host management network. This traffic type is known as provisioning traffic. Provisioning traffic is not encrypted but uses run-length encoding. Run-length encoding is a simple form of lossless data compression. In the case of HCX, cold migration traffic will get encrypted using Suite B encryption. It leverages the same host management network to migrate the data through the HCX vMotion proxy. HCX only displays the cold migration option if the selected workload is not powered on. Although HCX provides the option to multi-select it is still limited. vCenter Server only handles 8 concurrent vMotions (cold and live) at a time. It places the remaining selected workloads in a queue until ready to migrate.

Cold Migration Notes

• If transferring large amounts of data consider using Bulk Migration or Cloud Motion
• vCenter Server templates are good candidates
• Supports vSphere ESXi Host versions 5.5 and above
• Cross-version migration from vSphere 5.5 to the current release of 6.7 Update 1
• Bi-directional migration (reverse HCX option)

Summary

HCX offers customers several different workload mobility options covering different SLAs and requirements. No reconfiguration of the source or destination sites is necessary to get started. The platform offers built-in encryption, WAN optimization, and cross-version migration through the HCX interconnects. The HCX service is evolving at a rapid rate, the team is making improvements and adding new functionality on almost a monthly basis. Next, we’ll be diving into how to deploy and configure HCX in VMware Cloud on AWS and on-premises.

Think about what’s usually involved when migrating workloads from one location to another. The network is at the top of the list, ensuring adequate bandwidth and routing is in place. Not only from data center to data center or data center to cloud, but also across regions and remote sites. Validating vSphere version compatibility between the source and destination is also important. Other workload migration considerations could include:

• Moving across different vSphere SSO domains independent of enhanced linked mode
• Mapping workload dependencies
• Older hardware preventing from upgrading to vSphere 6.x
• Bi-directional migrations independent of vSphere SSO domains, vSphere versions, hardware, or networks
• Validation of resources (compute and storage) on the destination side
• SLA and downtime involved
• Mobility options correlating to SLAs (downtime, low downtime, no downtime)
• No replatforming of the workloads or applications including no changes to IP or MAC Addresses, VM UUID, certs

These are only a few examples of workload migration considerations, but you get the point. I’ve been working with the HCX team for the past few months and I have to say it’s definitely great technology. They have built-in the necessary tooling within the product and removed several boundaries.

HCX Overview

HCX is the swiss army knife of workload mobility. It abstracts and removes the boundaries of underlying infrastructure focusing on the workloads. A HCX vMotion, for example, requires no direct connectivity to ESXi hosts in either direction compared to a vSphere vMotion. All HCX vMotion traffic gets managed through the HCX vMotion Proxy at each location. The HCX vMotion Proxy resembles an ESXi host within the vCenter Server inventory. It’s deployed at the data center level by default, no intervention is necessary. One thing to mention is the HCX vMotion proxy gets added to the vCenter Server host count by default. The HCX team is aware and will be changing this in the future, but this has no impact on your vSphere licensing.

Another boundary HCX removes is it supports several versions of vSphere going back to vSphere 5.0 to the most current release of vSphere 6.7 Update 1. This provides flexibility in moving workloads across vSphere versions, on-premises locations, and vSphere SSO domains. For on-premises to on-premises migrations, a NSX Hybrid Connect license is required per HCX site pairing. We will cover site pairing in the configuration blog post. Migrating workloads from on-premises to VMC does not require a separate HCX license. By default when deploying a VMC SDDC HCX is included as an add-on and is enabled by default. From the VMC add-ons tab, all that is required is clicking open Hybrid Cloud Extension and then deploy HCX. The deployment of HCX is completely automated within the VMC SDDC.

In order to start migrating workloads, network connectivity between the source and destination needs to be in place. The good news is it’s all built-in to the product. HCX has WAN optimization, deduplication, and compression to increase efficiency while decreasing the time it takes to perform migrations. The minimum network bandwidth required to migrate workloads with HCX is 100 Mbps. HCX can leverage your internet connection as well as direct connect. The established network tunnel is secured using suite B encryption. The on-premises workloads being migrated with no downtime will need to reside on a vSphere Distributed Switch (VDS). It also supports a 3rd party switch in the Nexus 1000v. Cold and bulk HCX migration types are currently the only two options which support the use of a vSphere standard switch but implies downtime for the workload (The HCX team is working on adding support for the vSphere standard switch for other migration types). To minimize migration downtime, HCX has a single click option to extend on-premises networks (L2 stretch) to other on-premises sites or VMware Cloud on AWS. Once the workloads have been migrated there is also an option to migrate the extended network, if you choose. Other built-in functionality includes:

• Native scheduler for migrations
• Per-VM EVC
• Upgrade VM Tools / Compatibility (hardware)
• Retain mac address
• Remove snapshots
• Force Unmount ISO images
• Bi-directional migration support

Summary

HCX provides enhanced functionality on top of the built-in vSphere VM mobility options. Customers can now use HCX to migrate workloads seamlessly from on-premises to other paired on-premises sites (multisite) and VMware Cloud on AWS. Workload mobility can also help with hardware refreshes as well as upgrading from unsupported vSphere 5.x version. The next post will cover the different migration options available within HCX, followed by how to setup and configure the product.

Infrastructure Configuration

The Stretched Clusters feature deploys a single SDDC across two AWS availability zones. This option is only available during the SDDC creation steps under SDDC properties. When enabled the default number of ESXi hosts supported in a Stretched Cluster is six. Additional hosts can be added later but must to be done in pairs across AWS availability zones.

The Stretched Clusters feature requires an AWS VPC with two subnets, one subnet per availability zone. The subnets determine an ESXi host placement between the two availability zones. As an SDDC is deploying, it provisions three hosts in each AWS availability zone and creates a vSAN stretch cluster across the two availability zones. The vSAN stretched cluster allows synchronous writes across the two availability zones. A vSAN witness node appliance, which looks like an ESXi host, will automatically be provisioned. It resides outside the SDDC cluster and in a third AWS availability zone. The vSAN witness node appliance is required in case network communication is lost, assisting to avoid split brain for virtual machines across AWS availability zones. Logical networks are also extended using NSX to support workload mobility across the two AWS availability zones.

Workload Management

Once a stretched clusters SDDC is available, go to the summary tab and view the VMware Cloud on AWS console. The console displays the capacity and usage of the SDDC for the two AWS availability zones.

Now let’s login to vCenter Server. Here the SDDC will appear as a single logic datacenter with two fault domains. The ESXi hosts in the cluster span the two AWS availability zones. Customers can view which availability zone each host resides in from its summary tab. The fault domains listed correlate to the given AWS availability zone name. For a single view of all the ESXi hosts in a cluster and their fault domain, select the cluster and go to the Hosts tab. Within the Hosts tab, add the fault domain column.

When provisioning virtual machines, customers have options when deploying to an availability zone. The first option is auto placement. This occurs at the cluster level, where DRS will handle the placement of the workload. A more granular option is selecting a host in an availability zone to deploy virtual machines to. DRS will honor the virtual machine availability zone placement as a sticky rule. It will only move the virtual machine in case of a failure. vSphere HA will attempt to honor this placement decision if possible. An availability zone failure will have the same behavior as a vSphere HA event. All virtual machines in the failed availability zone will get restarted in another availability zone. vSphere HA restart priority is also taken into account:

• vCenter Server has the highest restart priority. Other management virtual machines have a high restart priority
• Virtual machines migrated (cold or live) from on-premises will keep their restart priority
• New Virtual Machines created on a stretched cluster will restart after other higher priority virtual machines

Customers now have full resiliency for their mission-critical applications across AWS availability zones with zero RPO due to synchronous replication built in natively in their Cloud SDDC. You can use the VMware Hands-On Labs to test drive VMware Cloud on AWS and this new product walkthrough and video to get familiar with the Stretched Clusters feature. Glenn Sizemore will also have a blog post coming with more details on the how storage works with Stretched Clusters, stay tuned.

To simplify the distributed model, VMware introduced the Platform Services Controller (PSC). The role of the PSC is to manage authentication, licensing, tags & categories, global permissions, and custom roles. It is also the certificate authority for the vSphere SSO domain. The PSC is a vCenter Server service which manages the components mentioned above. While the PSC is an improvement over the original SSO introduced in 5.1 and 5.5 it also added complexity. So began another debate of external versus embedded deployments. The reality comes down to enhanced linked mode. At the time (6.0 – 6.5 U1) only external deployments supported enhanced linked mode and external as a standalone. Now (6.5 U2 – 6.7) supports enhanced linked mode for embedded deployments. Now we are back to a more simplified deployment model and questions about how to transition.

Figure above represents the evolution of vCenter Server deployments. VMware has recognized the complexities involved in vCenter Server deployments and is putting efforts into getting back to a simpler deployment model allowing administrators can focus more on day-2 operations.

PSC Basics

Before we actually get into architecture it’s important to understand the basics. This is especially important when it comes to discussing availability. Let’s start with what happens when the PSC service is not available. We cannot authenticate (login) to vCenter Server, doesn’t matter if the PSC is embedded or external. Not only vCenter Server but other VMware products using the PSC for authentication will be affected. Examples are NSX, SRM, and vROPS to name a few. The next time you enter your credentials in the vSphere Client, hit enter, and watch the browser URL. You’ll see the URL change from vCenter Server’s FQDN to the PSC FQDN and back. For external deployments with several PSCs, it means losing a replication partner. This only affects authentication to any vCenter Server registered with the unavailable PSC. Workloads are still running, but the ability to login and centrally manage is not available. Now that you know the impact, think about who does this impact inside your organization. We’ll come back this during the availability discussion in this series.

It’s also important to understand the relationship between PSC and vCenter Server. An embedded PSC is only registered to the vCenter Server on the same node. An external PSC has a one-to-many relationship with vCenter Server.

• vSphere 6.0 – a single external PSC can have up to 10 vCenter Servers registered
• vSphere 6.5 and 6.7 – a single external PSC can have up to 15 vCenter Servers registered
• vSphere 6.0 – supports up to 8 external PSCs within a vSphere SSO domain
• vSphere 6.5 and 6.7 – supports up to 10 external PSCs within a vSphere SSO domain

One of the vSphere maximums you don’t see is one for sites. A site is nothing more than a logic boundary and a way to group PSCs. You can have as many sites as you want as long as it does not exceed the number of supported PSCs in a vSphere SSO domain. Starting with vSphere 6.7 sites are not required for new deployments. During upgrades and migrations, sites will remain. Adding a new PSC to an upgraded or migrated environment will prompt for site information. External deployments can repoint vCenter Servers to any PSC in a vSphere SSO domain. Also now in vSphere 6.7 across different vSphere SSO domains. Currently, embedded deployments cannot repoint across different vSphere SSO domains. Sites are one less boundary to worry about going forward. The recommendation is to keep the default for new deployments from the GUI or CLI.

Example 1: vSphere 6.7 embedded deployment with enhanced linked mode in a vSphere SSO domain. The PSC is running as a vCenter Server service in this example and everything within the vSphere SSO domain is considered a single default site.

Example 2: vSphere 6.7 external deployment with enhanced linked mode in a vSphere SSO domain. Everything within the vSphere SSO domain is considered a single default site.

The key thing to remember is the PSC is multi-master. Meaning there is no difference between the first one or the last one in the vSphere SSO domain. The PSCs replicate to each other every 30 seconds; this means if a PSC is unavailable the same data is available on its replication partners. To mitigate data loss, create a replication agreement between the bookends. This is referring to the first and last PSC in a vSphere SSO domain, creating a ring. The ring is creating a second data path in case you lose a data path. I’ve have seen several blog posts refer to the creating the ring as a requirement which is not true. The ring is a recommendation but not a requirement. Outside of the ring, creating more PSC replication agreements can cause unnecessary overhead. Keep it simple!

Example 3: An additional replication agreement created using vdcrepadmin from the last PSC to the first PSC (bookends) in the vSphere SSO domain, providing a secondary data path.

vCenter Server Basics

Most of the architecture discussions revolve around the vSphere SSO domain. There are some core concepts and considerations for vCenter Server as well. I pose the same question, but now what happens when the PSC is available but vCenter Server is not? The impact is you’ve lost management visibility and can not make any changes. Existing workloads are not affected by the outage. There are other implications to a vCenter Server outage. Some solutions communicating to vCenter Server as an endpoint will also lose access. For example, vROPS will not be able to collect metrics. Your backup window may also be affected. There are other products are resilient and not dependent on vCenter Server being available. Take vSAN for example, it is not dependent on vCenter and will continue to function. The VMware ecosystem has grown over the years and it’s important to know the impact of each solution within your environment when vCenter Server is not available.

We discussed the relationship between PSC to vCenter Server as one-to-many, but vCenter Server to PSC is 1:1. A vCenter Server can only be registered to a single PSC at a time.

• vSphere 6.0 – supports 10 external vCenter Server deployments in enhanced linked mode
• vSphere 6.5 & 6.7 – supports 15 external vCenter Server deployments in enhanced linked mode
• vSphere 6.7 – supports 15 embedded deployments in enhanced linked mode
• vSphere 6.5 U2 – supports 15 embedded deployments in enhanced linked mode

In regards to physical location placement, VMware currently does not have any documented latency numbers. The recommendation is up to 100ms in a multisite configuration. This is the same for the new embedded with enhanced linked mode deployment. For external deployments at least one local PSC per site. Remember communication is continuous between vCenter Server and PSC. Stay tuned as we are trying to get official latency numbers published.

vCenter Server licensing comes up quite a bit. A standard vCenter Server license includes:

• All PSCs when in an external deployment.
• Native vCenter Server high availability (VCHA)
• File-Based Backup / Restore
• Other advanced vCenter Server features

vCenter Server also has a foundation version. It has limitations on available features and functionality compared to the standard version.

There are many questions about the new vCenter Server architecture 6.7, but I felt it was important to go basics. Over the next few posts, I’ll go into more details on both the embedded and external deployment models.