TCP/UDP 389
TCP/UDP 636
TCP 443
TCP 2012
TCP 2014
TCP 2015
TCP/UDP 2020
TCP 5480
TCP 8281
TCP/UDP 902
Thursday, May 2, 2019
PSC Convergence from External to Embedded mode & Network Ports
TCP/UDP 389
TCP/UDP 636
TCP 443
TCP 2012
TCP 2014
TCP 2015
TCP/UDP 2020
TCP 5480
TCP 8281
TCP/UDP 902
Tuesday, April 9, 2019
vMotion and Virtual Switches
vMotion vSwitche N-VDS VDS VSS
---------------- BUG DESCRIPTION: ---------------- Live migration of a VM from one NVDS to another NVDS was failing and Powering off the VM. ----------- ROOT CAUSE: ----------- VC thinks LS with same network name across datacenter to be different networks and send a deviceBackingChange in the migrateSpec even if the vMotion is over same LS. ------- OUTPUT: ------- 1). Live migration of a VM from NVDS to same NVDS within same Datacenter and across Datacenters is now PASSING. 2). Live migration of a VM from one NVDS to another NVDS within same Datacenter and across Datacenters is now BLOCKING. 3). Live migration of a VM from VSS to NVDS and vice-versa within same Datacenter and across Datacenters is now PASSING. ----------------------------- BUILDS USED FOR VERIFICATION: ----------------------------- VC: 12713247 (vSphere67u2) ESXi: 12698103 (vSphere67u2) ------------------ SCENARIOS COVERED: ------------------ (1). Live migration of VM across VSS, DVS and NVDS(OPN) both in single and across VC/Datacenters. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Network-Path Same-DC Across-DC XVC (Across VC) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VSS1 -> VSS1 (different PG) PASS(E1) NA NA VSS1 -> VSS2 (within same host) PASS(E1) NA NA VSS1 -> VSS2 (with different configs) PASS PASS PASS VSS1 -> VSS2 PASS PASS PASS DVS1 -> DVS1 (same port group) PASS NA NA DVS1 -> DVS1 (different port group) PASS NA NA DVS1 -> DVS2 (with different configs) PASS PASS PASS DVS1 -> DVS2 PASS PASS PASS OPN1 -> OPN1 PASS PASS PASS OPN1 -> OPN2 PASS(B1) PASS(B1) PASS(B1) DVS -> VSS PASS(B1) PASS(B1) PASS(B1) VSS -> DVS PASS PASS PASS VSS -> OPN PASS PASS PASS OPN -> VSS PASS PASS PASS DVS -> OPN PASS PASS PASS OPN -> DVS PASS PASS PASS VSS, DVS, OPN -> OPN, DVS, VSS PASS PASS PASS VSS, DVS, OPN -> OPN, OPN, DVS PASS PASS PASS VSS, DVS, OPN -> VSS, OPN, DVS PASS PASS PASS VSS, DVS, OPN -> OPN, OPN, OPN PASS PASS PASS VSS, DVS, OPN -> VSS, OPN, OPN PASS PASS PASS VSS, DVS, OPN -> OPN, DVS, OPN PASS PASS PASS VSS, DVS, OPN -> DVS, DVS, OPN PASS PASS PASS VSS, DVS, OPN -> DVS, OPN, OPN PASS PASS PASS VSS, DVS, OPN -> VSS, OPN, VSS PASS PASS PASS VSS, DVS, OPN -> OPN, DVS, DVS PASS PASS PASS VSS, DVS, OPN -> DVS, OPN, VSS PASS PASS PASS VSS, DVS, OPN -> OPN, OPN, VSS PASS PASS PASS VSS, DVS, OPN -> DVS, OPN, DVS PASS PASS PASS With Disconnected Network Adapter PASS PASS PASS Different NVDS with Same Name PASS PASS PASS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ E1 -> Error stack: Migrating VM standalone-cbd6dd057-esx.2-vm.0 to different network without changing its host is not supported. Please use Reconfigure API to change VM's network. B1 -> "Currently connected network interface" 'Network adapter 1' cannot use network 'LogicalNetwork2 (nsx.LogicalSwitch:00021250-382c-995d-2ae4-56c5c6fbe603)', because "the type of the destination network is not supported for vMotion based on the source network type". See KB article 56991 for more details. (2). Compatibility checks for scenarios where destination switch is without PNIC. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Network-Path Same-DC XVC (Across VC) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VSS1 -> VSS2 (without pnics) BLOCK(CE) BLOCK(CE) VSS1 (without pnics) -> VSS2 PASS PASS VSS -> DVS (without pnics) BLOCK(CE) BLOCK(CE) VSS (without pnics) -> DVS PASS PASS DVS1 -> DVS2 (without pnics) BLOCK(CE) BLOCK(CE) DVS1 (without pnics) -> DVS2 PASS PASS OPN -> VSS (without pnics) BLOCK(CE) BLOCK(CE) VSS (without pnics) -> OPN PASS PASS OPN -> DVS (without pnics) (NE) (NE) DVS (without pnics) -> OPN PASS PASS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Compatibility Error (CE) -> Currently connected network interface 'device' uses network 'network', which is a 'virtual intranet'. Not Expected (NE) -> Allowing migration without CE. Raised Bug – 2289453 (3). Destination network not accessible cases. Delete the destination network before migration process starts. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Network-Path Result ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VSS -> DVS PASS(E1) OPN -> VSS PASS(E1) OPN -> DVS PASS(E1) VSS -> OPN PASS(E2) DVS -> OPN PASS(E2) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Task Error (E1) - "currently connected network interface 'device' uses network 'network', which is not accessible." Task Error (E2) - "A general system error occurred: Invalid fault" (4). Suspended VM migration cases. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Network-Path Same-DC XVC (Across VC) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VSS -> DVS PASS PASS VSS -> OPN PASS PASS DVS -> OPN PASS PASS OPN -> DVS PASS PASS OPN -> VSS PASS PASS DVS -> VSS (CW) (CW) OPN1 -> OPN2 PASS PASS VSS1 -> VSS2 PASS PASS DVS1 -> DVS2 PASS PASS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Compatibility Warning (CW) -> Network interface" 'Network adapter 1' cannot use network 'VM Network', because "the type of the destination network is not supported for vMotion based on the source network type.
Datastore capacity expansion issue
TAM recommended to create new additional 4TB datastore, format it to VMFS 6 and use storage migration from old datastore (ibm103-xx) to new datastore (ibm104-xx).
Old datastore (ibm103-xx) will be empty and next week we will do further troubleshooting of old datastore (ibm103-xx) and will try datastore expansion in Host Client (HTML5).
TROUBLE SHOOTING COMMANDS
Checking VMFS Metadata Consistency with VOMA - https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-6F991DB5-9AF0-4F9F-809C-B82D3EED7DAF.html
voma -m vmfs -f check -d /vmfs/devices/disks/naa.00000000000000000000000000000703:3
Running the esxcli storage core device list command shows the size of the extended LUN.
Running the vmkfstools -Ph /vmfs/volumes<datastore name> command shows the original size of datastore
Running the vmkfstools -Ph /vmfs/volumes<datastore name> command shows the original size of datastore
Command partedUtil - Using the partedUtil command line utility on ESXi and ESX (1036609) - https://kb.vmware.com/kb/1036609
The whole runbook procedure is describe in VMware KB: Growing a local datastore from the command line in vSphere ESXi (2002461) - https://kb.vmware.com/kb/2002461
Wednesday, February 27, 2019
APD - ExtendAPDCondition
Good To Know about this advanced setting.
[root@esx24:~] esxcli system settings advanced list -o /Scsi/ExtendAPDCondition
Path: /Scsi/ExtendAPDCondition
Type: integer
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Trigger APD condition when paths are in unavailable states
[root@esx24:~] esxcli system settings advanced list -o /Scsi/ExtendAPDCondition
Path: /Scsi/ExtendAPDCondition
Type: integer
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Trigger APD condition when paths are in unavailable states
VMware VDS Healthcheck enhancement
Overview
Customer give the feedback for healthcheck about scalability issues. There are two things here:
1. Currently, each uplink will send out broadcast packets for each vlan, if vlan range is big, then that will cause physical switch flushes its port’s lookup table and the normal traffic will be flooding, and cause performance issue.
2. Currently, we send out quite a lot broadcast packets at the same time, and those broadcast will introduce a lot of ACK packets followed, and thus it causes traffic bursted for this healthcheck.
We need to work out a way to reduce multicast packets number sending by healthcheck and resolve the lookup table flush issue.
Scope and requests
The scope is for vSphere-2016 release.
The requests are:
1. Can do one time check for specific vlan.
2. Resolve the physical switch flush lookup table issue.
3. Reduce the broadcast packets as much as possible.
Detail design
In the new design, healthcheck will provide:
● User can specified vlan checking range instead of whole vlan range of DVS.
● User can specified vlan checking at certain host instead of all hosts within same DVS.
● Using unicast packet instead of broadcast to avoid broadcast storm and response packets(same host might not need to send back the ACK packets) when there are more than two physical uplinks connected to this DVS on this host.
● Change the ACK mode: there will be no ACK packets send out from the same host. If the receiving packet is sending from the same host, just mark the session is ACK’ed directly instead of sending back the ACK packets to physical host to the same host.
Provide UI interface in order to do customizing vlan/MTU check:
User can specify the vlan range and select the hosts to run the check, and the results listed per host as well. From the UI side, we need to provide following interface to let customer initiate his customized vlan MTU check.
For the result showing, we can use the current format to show result for both one time checking and periodical checking.
Changes for management plane:
From MP side, the original code get VLAN settings from all DVPorts and DVPortgroups, we need to provide VIMAPI to accept inputted VLAN range from UI side and to initialize the one time checking.
The way to fetch result does not need to change.
Changes in data plane:
Change the way to send out probing packets.
Original vlanMTU check model:
In the original design, each uplink will send out broadcast packets for each configured vlanID. And the ACK’ed packets will received from both same host and other hosts within one same DVS.
New VLANMTUCHECK steps:
In the new design, all uplinks of the same vswitch will be treated as one same checking group instead of sending out packets separately in order to reduce the number of packets sending to the physical switch. Here is the reasons:
- If the unicast packet of specific vlanID sending from uplink0 to uplink1 is ACK’ed by uplink1, it indicates that this vlanID configuration is correct on both uplink0 and uplink1; if it is not ACK’ed by uplink1, but the same vlanID is ACK’ed by another uplink2, it indicates that vlanID is setting correct on uplink0 and uplink2 and is wrong on uplink1.
The overall design is:
- If the vswitch has only one uplink, it will send out broadcast packet as it did in old version.
- If there are more than 1 linkup uplinks, it will choose the first linkup uplink, and make all other linkup uplinks are destination ports and send out unicast packet for each vlanID to each other uplink.
- If the ticket get all ACK’ed packets from all other uplinks, mark all vlanID setting is corret.
- If received only from part of all other uplinks, picked each ACK uplink and mark both vlanID correct for the source uplink and ACK’ed uplink.
- If there are vlanIDs did not receive ACK packet from any other uplinks, choose the next uplink as source uplink, send unicast packets to all following uplinks for all non-ACK’ed vlanIDs. Recording ACK’ed vlanID for each uplink and repeat this until: 1) there is no untrunked vlanID; 2) the last uplink.
- Comparing all trunked vlan bitmap of all uplinks with configured vlan bitmap, if there are still vlanIDs untrunked, trigger another round of broadcast phase for each uplink for each untrunked vlanID just as previous version. Doing this, in order to reduce the chance that the vlanID settings in correct at only one of the uplinks or there is LAG configured at physical switch side. If there is a LAG configured at physical switch side, unicast packets sending among uplinks within one LAG will not be received by the targeted port, but broadcast packets can be responsed by remote hosts. So we need to send out broadcast packets as second round of checking.
Please refer to figure.1 below:
Figure. 1 New model of vlan MTU check
Detailed flowchart is:
ACK model change:
In the new design, if the request packets sending among uplinks belong to same DVS and the same host, will not send back ACK packets and will updated the ticket’s ACK’ed list directly in order to reduce the unicast packet packet amount and reduce the possibility to flush the MAC table of physical switch.
Risk and assumptions:
The new design uses unicast packets replacing the broadcast packets, that makes the ways to sending packets and checking process change totally, vlanmtucheck module will be re-architectured, it will introduce code changes at most of the places. So need QE team to run healthcheck testing for good quality.
Part of this change requests UI and MP resource, without that, customized checking request is not possible to implement for most of that changes is in UI and MP side.
Test cases
For this design will change the way to run vlan MTU check, uplinks with the same DVS at the same host will interact together, so need to design new test cases to cover this.
Subscribe to:
Posts (Atom)