Enable the ESXi Side-Channel-Aware Scheduler (SCAv1) or the ESXi Side-Channel-Aware Scheduler v2 (SCAv2) in ESXi 6.7u2 (13006603) or later

 

Implementing Hypervisor-Specific Mitigations for Microarchitectural Data Sampling (MDS) Vulnerabilities

Before Applying SCAv1 and SCAv2 to ESXi Hypervisor ,Let's discuss about MDS and What is Goal of this Article .

Purpose :

In this Blog  will explain the Hypervisor-Specific Mitigation enablement process required to address Micro-architectural Data Sampling (MDS) Vulnerabilities identified by CVE-2018-12126, CVE-2018-12127, CVE-2018-12130, and CVE-2019-11091 in vSphere.

The release of vSphere 6.7 Update 2 brought with it a new vSphere CPU scheduler option, the Side-Channel Aware Scheduler version 2 (SCAv2) or “Sibling Scheduler.” This new scheduler can help restore some performance lost to CPU vulnerability mitigations, but also carries some risk with it..

CPU vulnerabilities are security problems that exist in processors delivered by major CPU vendors. They’re particularly serious because these problems may allow attackers to bypass all of the other security controls we’ve built up in our environments.

To prevent present in vulnerabilities like Spectre, Meltdown, L1TF, and MDS,we have few new sets of instructions provided by the CPU vendors using microcode that ship as part of ESXi or is shipped in vendor firmware updates.

So What is L1TF and MDS vulnerabilities ?

L1TF :The L1TF vulnerability exposed a problem where processors can leak information between two threads on a core in an Intel CPU.

MDS :The MDS vulnerability can also allow processes to leak information between processes,Intel has disclosed details on a new wave of speculative-execution vulnerabilities known collectively as “Micro-architectural Data Sampling (MDS)" that can occur on Intel micro-architecture prior to 2nd Generation Intel® Xeon® Scalable Processors (formerly known as Cascade Lake). These issues may allow a malicious user who can locally execute code on a system to infer the values of data otherwise protected by architectural mechanisms.

There are three schedulers to choose from in ESXi 6.7 Update 2: the default, SCAv1, and SCAv2

1. Default Scheduler : In default scheduler it should be assumed that all workloads running on a host with the default scheduler can see each other’s data.

2.SCAv1 :Its  implements per-process protections which assist with L1TF and MDS and is the most secure but can be the slowest.

Availability :
- ESXi 5.5,6.0,6.5 and 6.7 release lines
- Does not require Hardware Refresh .

Security :
-Prevents VM to VM Concurrent-Context Attack Vector information leakage.
-Prevents VM to Hypervisor Concurrent-Context Attack Vector information leakage.
-Prevents Intra-VM Concurrent-Context Attack Vector information leakage.

Perfomance:
-Retain up to 100% performance for low to moderately loaded system.
-Retain up to 70% performance for heavily loaded system.
-Does not utilize Hyper-threading.

3.SCAv2 : Its implements per-VM protections but needs considerations for the risk involved in using it.

Availability :

- ESXi 6.7u2.
- Does not require Hardware Refresh .

Security :
-Prevents VM to VM Concurrent-Context Attack Vector information leakage.
-Prevents VM to Hypervisor Concurrent-Context Attack Vector information leakage.
- Does NOT Prevents Intra-VM Concurrent-Context Attack Vector information leakage.

Perfomance:
-Retain up to 100% performance for low to moderately loaded system.
-Retain up to 90% performance for heavily loaded system.
-Utilize Hyper-threading.

Resolution :

Enabling the ESXi Side-Channel-Aware Scheduler (SCAv1) using the vSphere Web Client or vSphere Client

1. Connect to the vCenter Server using either the vSphere Web or vSphere Client.
2. Select an ESXi host in the inventory.
3. Click the Configure tab.
4. Under the System heading, clickAdvanced System Settings.
5. ClickEdit.
6. Click in the Filter box and search forVMkernel.Boot.hyperthreadingMitigation
7. Select the setting by name
8. Change the configuration option to true (default: false).
9. Click in the Filter box and search for VMkernel.Boot.hyperthreadingMitigationIntraVM
10.Change the configuration option to true (default: true).
11.Click OK.
12. Reboot the ESXi host for the configuration change to go into effect.

Please refer the below screenshot for your reference :

Go to ESXi > Configure >Advanced > filter > Mitigation change value in case of SCAv1 

Note : You can also apply this configuration using ESXi Embedded Host Client and ESXCLI for that you can follow the mentioned Kb :

Enabling the ESXi Side-Channel-Aware Scheduler Version 2 (SCAv2) using the vSphere Web Client or vSphere Client

1.Connect to the vCenter Server using either the vSphere Web or vSphere Client.
2.Select an ESXi host in the inventory.
3.Click the Configure tab.
4.Under the System heading, click Advanced System Settings.
5.Click Edit.
6.Click in the Filter box and search for VMkernel.Boot.hyperthreadingMitigation
7.Select the setting by name
8.Change the configuration option to true (default: false).
9.Click in the Filter box and search for VMkernel.Boot.hyperthreadingMitigationIntraVM
10.Change the configuration option to false (default: true).
11.Click OK.
12.Reboot the ESXi host for the configuration change to go into effect.

Go to ESXi > Configure >Advacnced > filter > Mitigation change value in case of SCAv2

Note : You can also apply this configuration using ESXi Embedded Host Client and ESXCLI for that you can follow the mentioned Kb :

Finally below is configuration which we need to apply for ESXi 6.7u2 (and later) Scheduler Configuration :

Thank You ,Hope this Article will help you to decide which vSphere CPU scheduler you have to configure  in production environment as per the requirement .

vRealize Operations Manager -Part 2

Installation and Configuration vRealize Operation manager

Stage2: Configure vRealize Operation Manager .

1.First login to browser using FQDN or using IP. You will get screen where you need to choose the installation type:


Express Installation : (Will only ask about license information and move to cluster configuration)

Expand an existing Installation : (This option is used only when you have an existing cluster and this node will be added into it, can be done for multiple reasons). New Installation: It same as Express but when you are going to deploy primary node you have to provide other information.

New Installation: It same as Express but when you are going to deploy primary node you have to provide other Information.

2.In My case I am going to choose New Installation and it will started with below page.

3.In This Step we need to setup the password for admin credentials of vROPS.

4. Now we must provide certificate details if you want to use the custom certificate then browse file and choose the certificate which you created from certificate server. In my case I am going to continue with Default Certificate .

5. This is the 1st node or primary node of the cluster; it becomes the master node. we need to add the NTP address as well.

6. This step is very important if you want to configure High Availability for Primary node. We have two Option here 1. High Availability and 2. Continuous Availability.

High Availability: As name defined you can choose this option for high availability of node means it will prevent data loss in case of node failure. Once you will use this option it will create secondary copy of master node and create replica node.

Continuous Availability: This mode is also help you in case of node failure and to prevent from data loss. but in this node you need one more node called as witness node. This configuration divide across two fault domain which basically need atleast three data nodes .

7.In this step we need to role of node. As this is 1st node it will be master node but if you are going to deploy 2nd node and want to configure as data node then choose Data node you want to configure as witness node.

8.Final step Click on Finish to complete the configuration part.

9.Once you will click o Finish on Last step it will start configuring the nodes and create cluster.

10.Once configuration will finish, it will highlight option to start vRealieze Operations.

11.When we will click on start vRealieze Operations initial installation will start and Cluster will come up online.

12. In Below picture you can see Cluster is running and online .Using above steps you can also deploy other data node and configure in this cluster .

Setup vRealize Operation and configure Data Source .

Basically, you are getting two options to login vRealize Operation Manager:

1.Admin Login: - When you are login to vROPS with admin, you will land to administrator page from where you can upgrade software, cluster status and some other admin status. https://<FQDN_vROPS/admin>

2.Operational login: When you login to vROPS without admin it will land to operational page from where you can perform you all operation task like dashboard, monitoring infrastructure, configure data soure,add cloud end point and lots of other task which you can explore when you will deploy in your environment .

1.Let me show you when we will login vROPS with admin how it will look like , https://<FQDN_vRops/admin>,In Below screenshot in left plane you can see administrator task when you login with admin .

2.Now let’s login without admin option in url: https://<<Fqdn_vROPS>>

3.Now let’s understand what are the steps we need to perform to setup operation page .

4.Login to browser with https://fqdn_vROPS ,First screen you will et is Welcome to configuration and setting up vRealize Operation .

5.In next screen accept the EULA and click Next .

6.Now we have to provide License key that you can also apply later and you can proceed with Evaluation License.

7.Join the CIEP (Customer Experience Improvement Program) and Click Next.

8.Click on Finish to complete the setup.

9.Once you click on Finish it will complete the setup and you are good to configure the Data sources from where vROPS collect the data.

1.Click On Home page and from Left Plane navigate Data Sources. Expand the Data sources option and Navigate to Integration .

2.Once you will click on Integration you will get different Account Types which you can configure. Like vCenter, NSXT, VMC on AWS, etc..

In My case I am going to configure with my on prem vCenter. So clicked on vCenter from below account types.

3.After Click on vCenter it will ask me to provide admin credentials to configure vROPS with vCenter.

4.Once you will provide the administrator credentials, Click in Validation it will validate that provided details is correct or not and is there any issue between connectivity .

5.When you click on Validate it will ask you to review and Accept the certificate .Review and Click on Accept.

6.In below screenshot you can see connection is successfully validated.

7.After successfully connection we can see vCenter is connected with my vROPS and soon it will start collecting data.

8.Go to Home Page ,Navigate to Operation Overview and you will able to monitor infrastraucture .Cluster ,VMs,any alert,Utilization and lots of other task .For which I will engourge configure your enviorment with vROPS and explore all options .

I hope my blog will help you to undertand vRealize operation Manager and how to setup in your enviorment .

vRealize Operations Manager -Part 1

Deployment of vRealize Operations Manager 8.10.1

Install and configure vRealize Operations Manager

 

Lets understand what is vRealize Operations Manager?

vRealize Operations deliver a unified management platform to optimize, plan, and scale hybrid cloud deployments from applications to infrastructure, powered by AI/ML, as a service.
It is also called as vROPS (vRealize Operations manager).

vRealize Operations Manager comes as a virtual appliance that is to be deployed in your management cluster if you have one. It can be installed in a number of ways, tailored to your environment’s size and complexity.

What is use cae of vROPS in Privare and Hybrid Cloud ?

In simple word , vROps is a tool from VMware that helps IT administrators monitor, troubleshoot, and manage the health and capacity of their virtual environment. The VMware vRealize Operations (vROps) Management Suite provides complete information about performance, capacity, and health of our infrastructure.

What are the some other benifts which you can get with vROPS ?

It enables proactive monitoring, capacity planning, and automation, allowing organizations to manage and scale their virtual infrastructure effectively. Additionally, vROps offers integration with other VMware solutions, such as vSphere and vSAN, to provide a holistic view of the entire virtual environment.Few other benefits which will you recommendation for VM sizing like undersized VMs,Oversized VMs ,High vCPU vonfigured with VMs and like that lots of other recommendations which you can use to manage your infrastructure.

vRealize Operations components and Architecture?

Master Node: Master node we also called as primary nodes , The primary node performs administration for the cluster and must be online before you configure any new nodes. In addition, the primary node must be online before other nodes are brought online.
In a single-node installation, the master node manages itself, has adapters installed on it, and performs data collection and analysis.

Replica Node: Replica Node as name defined is copy of Primary Node. During a failure event of the master node, the master replica DB is promoted to a full read/write master. Although the process of the replica DB's promotion can be done online, the migration of the master role during a failover does require an automated restart of the cluster.

Data Node : It provides the core functionality of collecting and processing data and data queries as well as extending the vROps cluster by being a member of the GemFire Federation, which in turn provides the horizontal scaling of the platform. Specifically, they host adapters performing data collection as well as analysis. Larger deployments usually include adapters only on the data nodes so that master and master replica nodes can be dedicated to cluster management.

Remore Collector Node: The remote collector role is for remote sites or secure enclaves. Remote collectors do not process data themselves; instead, they simply forward metric data to data nodes for analytics processing. remote collector node that can span firewalls, collect data from a remote source, reduce bandwidth across data centers, and/or reduce workload on the vROps cluster.

Witness Node : Witness node is play his role ,If network connectivity between two fault domains is lost, the witness node acts as a decision maker regarding the availability of vROps.

vRealize Operations Large Deployment Profile Architecture:

Sizing tool for deployment of vROPS nodes as per your environment

(https://vropssizer.vmware.com)

You have to choose the version which you are going to deploy then need to provide details about workload like how many datacenter you have ,vCenter ]s ,Cluster ,Object ,etc… 

Below is the sample snap for you:

Deployment of vRealize Operations appliance or nodes:

1.Righit Click on Cluster and Click on deploy OVF .

2. Once you will click on deploy OVF , IT will ask you to choose offline location where you have download OVF from customer connect .Else , If you have internet connectivity with vCenter you can

also provide url.

3.In this Step you have to provide name of vROPS appliance as per your company standard.

4.Next you have to choose the compute resource where you want to deploy vROPS nodes which will utilize the compute resource of cluster.

5.Now Review all the details related to appliance like version, size on disk ,product and etc and click Next.

6. Accept the License Agreement and Click Next.

7. Now her we must choose the size of node. If you remember we have provided you vROPS sizer tool which will help you to determine what kind of node you need to deploy as per infrastructure like how many object, cluster size, vCenter etc.. I am deploying in Lab so choosing small node.

8.Here we have to select the datastore where vROPS appliance will use, You can also change disk format as per your requirement.

9. Now we must choose the port group or network from which vROPS manager will get ip and using which it will communicate will other nodes as well as management VMs (like vCenter, ESXi..)

10.In Customize Template choose the Time zone, gateway network and DNS name and click Next .Make sure DNS entry for all nodes should be on place before deployment .

11.Last Review all the details which you provided in all previous steps and once everything is fine Click on Finish. Deployment of OVF will take 15-20 minutes .

12.After Deployment of OVF will complete ,Go to vCenter and check the vROPS cm console it will give you FQDN which we need to access on browser to start stage 2 and finish the configuration Part .

Now we have completed the deployment of vROPS ovf and In my next blog we will discuss stage 2 , how to configure and setup vROPS .

Configure Hybrid Cloud Extension (HCX)-Part 2

Configure HCX Hybrid Cloud Extension services -Site Pairing ,Compute Profile ,Network Prodile and Service Mesh.

Site Pairing:

A Site Pair establishes the connection needed for management, authentication, and orchestration of HCX services across a source and destination environment.

In HCX Connector to HCX Cloud deployments, the HCX Connector is deployed at the legacy or source vSphere environment. The HCX Connector creates a unidirectional site pairing to an HCX Cloud system. In this type of site pairing, all HCX Service Mesh connections, Migration and Network Extension operations, including reverse migrations, are always initiated from the HCX Connector at the source.

In HCX cloud-to-cloud deployments, site pairing can be unidirectional or bidirectional:

In unidirectional site pairing, the HCX Cloud containing the virtual machine inventory and networks (similar to HCX Connectors) will site pair to the destination HCX Cloud.

In bidirectional site pairing, the HCX Cloud systems are site paired with each other, share a common Service Mesh, and can initiate Migration and Network Extension operations from either HCX Cloud system.

Let us discuss about different network use by HCX service.

1.Management Network:

The HCX Interconnect appliances use this network to communicate with management systems like the HCX Manager, vCenter Server, ESXi Management, NSX Manager, DNS, NTP.

2.Uplink Network:

The HCX Interconnect appliances use this network for WAN communications, like TX/RX of transport packets.

3.vMotion Network:

The HCX Interconnect appliances use this network for the traffic exclusive to vMotion protocol operations.

4.vSphere Replication Network:

The HCX Interconnect appliances use this network for the traffic exclusive to vSphere Replication.

5.Guest Network for OS Assisted Migration

The Sentinel Gateway appliances use this vSphere network to connect with non-vSphere virtual machines.

Network Profile:

The Network Profile is an abstraction of a Distributed Port group, Standard Port group, or NSX Logical Switch, and the Layer 3 properties of that network.

We can create a different network profile based on below use case.

1.Use case 1: if customer wants to use Shared Management, Replication, and Uplink with Dedicated vMotion

2.Use case 2: If customer configured Dedicated Replication Network and want to use dedicated network profile for replication network.

3.Use case 3: If customer configured Dedicated Uplink Network and they want to use dedicated network profile for uplink.

4. Use case 4: Network profile for OS Assisted Migration Using Guest Network.

5.Use Case 5: If customer wants to use OS Assisted Migration Using Management Network

Configure Site Pairing:

1.Login to source vCenter using administrator@vsphere.local and click on Menu.

2.From the Drop-down menu click on HCX .

3.Once you will click on HCX, it will redirect on HCX dashboard, where we need to configure the rest of configuration like site pairing, network profile, service mesh and other services.

Before configuring site, pairing and network profile let discuss about the Site pairing and network profile service mesh.

4.Navigate to Site Pairing and click on connect to remote site:

5.Once you will click on Connect to Remote site, you need to provide Remote HCX Url (https://hcx_fqdn), Username (administrator@vsphere.local) and Password XXXX.

6.Once the connection will establish with remote HCX you will able to see source HCX and destination HCX are in connected state.

7.Now navigate to Interconnect from Left side bar and click on Interconnect .

8.This step in most important step where I am going to create Network profile which i explained you in with different use Case

9.Click on Network Profile and Create Network Profile in this page.

10.Once you will click on create network profile, you need to provide below details .

10.1.Select the vSwitch, vDS, or NSX network (not mandatory for HCX Connector) that you will use for this Network Profile. In this case, it is a vDS.

10.2.Add the IP Pool addresses for for the HCX Management. Uplink, vMotion and Replication. We will need one IP address per Appliance. Since we are only deploying one, then one IP Address. But you can add here a full IP Pool range. HCX will use the necessary IP addresses, and I will add two here.

10.3.In my Lab i am going to user this network profile for MGMT, vMotion and vSphere Replication and HCX uplink.

If you have dedicated network as I explained in above, use case you can create separate Network profile for HCX traffic.

Note : In deployments where ESXi servers use a dedicated Provisioning vmkernel for NFC traffic, the HCX continues to route Cold and vMotion NFC traffic through the Management interface.

11.Once First network profile if create in same manner you can create other network profile .

12.Next we are going to create a Compute Profile. From same page navigate to Compute Profile and Create Network profile.

Compute Profile :

1.Create Compute Profile, Give the Name of Compute Profile and Next.

2.In Next screen choose the HCX service which you want to use for your environment and Click continue.

3.Now we must choose the cluster where you want to use the HCX service if you want to use for all cluster the you can go with datacenter .

4.Next we must choose deployment resource like cluster datastore and folder where you want to deploy the HCX service appliance.

5.Now we must choose port group for each type of traffic. If we have MGMT port group which you are using for MGMT, vMotion, replication, uplink then choose same MGMT network profile in each type of traffic.

If you have dedicated port groups for different type of traffic, then choose separate network profile or use case for MGMT, vMotion, replication, uplink.

In my Lab environment I have only MGMT port group which we can used for vMotion and other traffic.

6.Select port for Management network profile and click continue.

7.Choose port group for uplink network, using which interconnect appliance can communicate with peer appliance.

8.Select vMotion Network profile, as I told mentioned that i am going to use MGMT network profile for all type of traffic.

9.Now select network profile for vSphere replication.

10.Review the topology for all connection and click continue.

If you are using dedicated port group for different type of traffic then make sure firewall rules are allowed to these connections.

11.Once you will review the configuration, Click on Finish to create compute profile.

You have to repeat all above steps to create network profile ,compute profile on destination or (Cloud side ).

Service Mesh:

Service Mesh initiates the deployment of HCX Interconnect virtual appliance on both sites and it will add to connected site pair that has valid compute profile created on both sites

1.Same page where we added network profile and compute profile ,Click on Service Mesh.

2. Select the source and destination sites, if you have two or more that then click on arrow to see the other sites details. Click continue.

3.Select the Compute profile in source and destination where virtual machine can use HCX services.

4.Select the service which we need to activate in your environment.

5.Select the uplink network profile to connect the HCX interconnect appliance. As we already create MGMT network profile and want to use for uplink we will select the same .

You can also change and override if wanted to change network profile for uplink network. Click continue.

6.Advacenced configuration will give you option to create a multiple transport tunnel to establish path diversity to the target.

In my Lab its is not given me option as they are not licensed with this HCX installation.

7.Review the topology and if everything looks good, Click Continue.

8.Provide the name of service mesh and Click Finish for the deployment.

9.Once we will finish the deployment of Interconnect appliance and other appliance start on source and destination.

10. Once all tasks will complete you can see on appliance that service mesh is created, and all appliance deployed.

11.Expand appliance and service mesh you will be able to see that tunnel status is up.

Hope my Blog will help to configure HCX in your client enviorment .