VMware
vSAN 5.5 (and now 6.0) is a software-defined storage solution developed by
VMware and integrated into the kernel of its premier virtualization platform,
allowing for the creation and management of shared object-based storage using
the local solid-state and spinning media in the physical host servers
themselves.
Note
that vSAN is not the same animal as VMware’s vSphere Storage Appliance (vSA),
though the underlying value proposition is the same. The two are implemented
and managed very differently. vSA is now end-of-life/availability, though still
supported through 2018. vSAN has been integrated directly into the kernel, so
it is there whether you use it or not, and no longer requires the deployment of
controller appliances. Storage appears as a single unified ‘datastore’ across
all hosts in the cluster and is managed entirely through vCenter’s web client.
vSAN
is licensed separately from vSphere but in the same familiar fashion, on a per
socket basis. When you enable vSAN on the cluster you are initially allowed a
60-day evaluation period but must assign a proper license to the cluster before
this evaluation period expires.
The
purpose of this email is to provide notes from both field deployments and from
working with VMware support.
General
notes:
1.
All
hosts should be configured to report to a syslog server
2.
All
hosts should be configured to synchronize their time to the same valid time
source
3.
The
minimum number of hosts supported in a vSAN cluster is three
4.
The
maximum number of hosts supported in a vSAN cluster is thirty-two (in 5.5)
5.
The maximum
number of VMs per host is currently limited to 100 (in 5.5)
6.
The
maximum number of VMS per datastore to be protected by HA is 2048.*
7.
The
sweet-spot for cluster sizing is up-to sixteen hosts.
*This
is important since vSAN storage appears as a single ‘datastore’.
On
the host side:
1.
vSAN
hosts must be comprised of certified controllers and disks
Note:
Make sure and verify that the controller and disks in-use in the design appear
on the VMware HCL! This is key to the supportability of the solution and must
be followed. Note that the controller must support pass-through or
pseudo-pass-through disk access modes, furthermore the controller must have
sufficient queue depth. A minimum depth of 256 is required for vSAN (5.5),
though a higher queue depth (>512) is recommended.
2.
You can
have multiple disk groups per host
3.
A disk
group is made up of at-least one SSD and at-least one HDD
4.
A disk
group can contain up-to one SSD and up-to seven HDD each
5.
There is
a maximum of five disk groups per host
6.
Utilize
10GbE interfaces for the best performance
7.
Dedicate
10GbE interfaces if you can, especially if using Broadcom adapters (see note on
Network I/O Control below)
8.
If you
do not have 10GbE interfaces, consider physically dedicated 1GbE interfaces for
vSAN
9.
SSDs are
used for caching – do not count them towards your capacity
10.
When
sizing your vSAN cluster, ensure that you take into account the resiliency
level (replicas) you intend to support and ensure that your SSD to HDD ratio is
at-least 1:10 respectively. SSD capacity should be sized to at-least 10% of the
capacity of HDDs in the disk group. An example would be if you are building a
disk group of four 1.2TB 10K SAS disks, giving you’re a disk group capacity of
4.8TB, your SSD selection should be at-least 480GB.
11.
Keep in
mind that by default 70% of the SSD capacity per disk group will be used as a
read cache and 30% will be used as a write buffer. Using SSDs with the right
bias (Read or Write Intensive) or a non-bias (Mixed Use) will make a
significant difference in performance based on your intended workload so take
this into account. For general purpose virtualization, the recommendation would
be to use Mixed Use SSDs because of their non/even-bias.
12.
Also
note that when sizing your host memory, keep in mind the ideal workload and
consolidation ratios you hope to achieve. Given storage is more finite with
vSAN clusters, large amounts of physical memory (>256GB) are certainly
supported but may be underutilized in many environments. Keep in mind that IF
you are sizing a host with 512GB or more of physical memory, the embedded SD
cards are not supported and ESXi must be installed on physical media.
On
the virtualization side:
1.
Both the
standard and distributed virtual switches are supported.
2.
Use of
the web client is required. You cannot configure vSAN using the thick client.
3.
Use of
vCenter is also required. This will need to be taken into consideration on
green field deployments. You will need to format one of the HDD disks on the
first host and create a local datastore, install vCenter and configure it,
configure vSAN and then use storage vMotion to move the VM to the new vSAN
storage. Once storage vMotion is complete, you can then remove the ‘legacy’
datastore and move the disk into the vSAN disk group.
4.
vSAN
storage is presented as a single common ‘datastore’ but the utilization and
expression of the objects (VMs) on that store are controlled through storage
policies. vSAN storage policies must be defined as they control the resiliency
level (FTT, number of replicas) and other tuning parameters.
5.
When configuring
HA for use with vSAN, choose ‘Power Off’ as your isolation response.
6.
When
configuring HA for use with vSAN, ensure that your ‘host failures to tolerate’
setting aligns with your vSAN availability strategy and settings.
7.
vSAN
does NOT (in 5.5) support FT, DPM, Storage DRS or Storage I/O Control.
8.
vSAN
does support Network I/O Control and if you are using Intel adapters and the
distributed virtual switch, the recommendation would be to enable and configure
it for optimal performance.
On
the (physical and virtual) networking side:
1.
Layer-2
Multicase IS required for vSAN.
2.
It is a
recommended practice to create a separate, segregated, VMkernel for vSAN data
3.
The
VMkernel interface created for vSAN can utilize private IP space
4.
At-least
one VLAN per vSAN cluster. vSAN clusters should NOT share the same broadcast
domain
5.
It is a
recommended practice to create two VLANs per vSAN cluster for maximum
performance. It is however not supported to have a VMkernel for vSAN active on
more than one NIC, therefore the recommendation is to set this up similarly to
iSCSI. It is key that each separate VMkernel have its own IP subnet.
a.
VMkernel
called vSAN0 attached to VLAN 92 with IP 192.168.92.10 and vmnic1 as active and
vmnic3 as standby.*
b.
VMkernel
called vSAN1 attached to VLAN 93 with IP 192.168.93.10 and vmnic3 as active and
vmnic1 as standby.*
Note:
Because of the Active/Standby (as opposed to Active/Unused) and the use of two
different subnets, these physical switch ports must be configured as trunks and
be tagged for both VLANs.
6.
The
current recommended practice from VMware is to avoid the use of Jumbo Frames
with vSAN
Jumbo
Frames are officially supported however there was an issue discovered with
jumbo frames and multicast, which vSAN makes extensive use of, in vSphere 5.5
update 2. Not sure if this has been fixed in update 3 or not but something to
be aware of. The consensus from VMware support is that jumbo frames does not
make a significant difference in performance with vSAN. You may utilize Jumbo
Frames elsewhere in the environment, however the VMkernel(s) for vSAN should be
configured for the default 1500.
7.
IP HASH
link aggregation is supported by vSAN but keep in mind that since traffic will
be flowing to and from the same IPs, it is unlikely that you will drive the
link utilization desired using this method.
8.
For our
physical switches, the same quick configuration guides for EqualLogic can be
used as reference, the cabling recommendations are the same, however do not
enable DCB or iSCSI optimization. You may also need to create additional VLANs
and provision switch ports as trunks instead of access (tagged instead of
untagged) depending on your host and cluster design.
5.5
Reference:
6.0
Reference:
What’s
New in vSAN 6.0?
Configuration
Maximums for vSphere 6:
VMware
Virtual SAN 6.0 Design and Sizing Guide:
Hope
this helps!
