With today’s release of vSphere, VMware has introduced numerous new features and technologies all designed to advance the virtual evolution of our data centers. With this release VMware and their storage alliance partners have delivered Plug-n-Play SAN storage connectivity. As you begin downloading the final bits, refreshing your labs and planning your upgrade path to vSphere I’d like to dig into the considerations of and how one can deploy a robust, reliable, and greatly simplified P-n-P SAN architecture with vSphere and NetApp.
The Pluggable Storage Architecture (PSA)
There are numerous posts covering the VMware’s PSA; however, in order for us to have an in depth conversation regarding the technical details covered later in this post I’d like to briefly cover this new architecture.
In ESX/ESXi the PSA is a VMkernel layer responsible for managing storage paths. It also provides an open modular framework that coordinates the operation of multipathing plugins (MPPs), which include the VMware Native Multipathing Plugin (NMP).
The NMP supports all storage arrays listed on the VMware storage HCL and provides a path selection algorithm based on the array type. The NMP associates a set of physical paths with a specific storage device, or LUN. The specific details of handling path failover for a given storage array are delegated to a Storage Array Type Plugin (SATP). The specific details for determining which physical path is used to issue an I/O request to a storage device are handled by a Path Selection Plugin (PSP).
ESX/ESXi offers a SATP for every type of array that VMware supports. In addition, the PSA provides support for third party SATPs & PSPs. When a storage scan is initiated in ESX or ESXi the storage array is identified and assigned a SATP. The NMP associates the SATP with physical paths to the storage array and the SATP implements the tasks that include the following:
• Monitors health of each physical path.
• Reports changes in the state of each physical path to the NMP.
• Performs array-specific actions necessary for storage fail-over.
PSPs are responsible for choosing the physical path for I/O requests. Each SATP is associated with a PSP that the NMP assigns for every logical device associated with the physical paths for that device. The default PSP to SATP assignment can me modified. The NMP provides the following PSPs:
• Fixed — Uses the first working path discovered at system boot time unless manually configured. If the host cannot use the preferred path, it selects an alternative available path. The host automatically reverts back to the original preferred path when it becomes available.
• Most Recently Used (MRU) — Uses the first working path discovered at system boot time unless manually configured. If the host cannot use the preferred path, it selects an alternative available path. The host remains on the new path even if the original path has become available.
• Round Robin (RR) – Uses an automatic path selection rotating through all available paths and enabling I/O load balancing across all of the paths.
Support for Asymmetric logical unit access (ALUA)
Plug-n-Play FC HBAs & FCoE CNAs
That’s right, more Plug-n-Play goodness. Unlike VI3, vSphere on NetApp FC and FCoE architectures do not require storage adapter configurations. Now I wish it wasn’t the case, but iSCSI HBA implementations will still be required to configure their hardware adapters. As with VI3, these customers have the option of doing so manually or can leverage NetApp’s ESX Host Utilities (EHU) to automate this process.
I would add that the majority of iSCSI customer base leverages the iSCSI software initiator, which does not require any configuration changes with VI3 and vSphere.
Prepare the array and put it together
OK, so we’ve addressed our storage adapters and next let’s take a look at enabling ALUA on our NetApp storage controller. This is a one-time configuration setting that must be completed from the console of the FAS array. This setting is not a global option, it is enabled on a per initiator group (or igroup) basis. If you have followed our best practices you should have one igroup per ESX/ESXi cluster which to configure.
For customers upgrading from VI3 to vSphere they can enable ALUA on existing igroups without impact to connections made by ESX/ESXi 3.x hosts.
Viola! At this time we have achieved our Plug-n-Play SAN architecture as our ESX/ESXi hosts automatically understand the preferred storage paths and are configured with a fault tolerant multipathing policy. With ALUA enabled our ESX/ESXI hosts automatically identify our storage array, assign a SATP and its associated PSP, which for ALUA enabled NetApp arrays is MRU.
Is there more?
Like any solid out-of-the-box solution there is always room for improvement and this case is no exception. While ALUA & MRU do provide a production worthy storage connectivity architecture this design still requires manual path load balancing. I’m don’t know the reasons why, but with ESX/ESX 4.0 the SATPs for all storage vendors are configured with either the Fixed or MRU PSP.
An ESX/ESXi host configured with the Round Robin PSP receives a fault tolerant storage connectivity architecture that will send I/O down every available path. From an I/O perspective each host will receive the aggregated bandwidth available to each LUN. This is especially good news for customers who connect via iSCSI on 1GbE networks as they can combine the new support for multiple TCP sessions with RR to provide aggregated storage bandwidth.
I’ll post a blog on multiple TCP sessions & RR next week for our iSCSI customers.
It is fairly straight forward to change the VMW_SATP_ALUA (the SATP which NetApp arrays are assigned to) to be configured to use the VMW_PSP_RR. This process can be accomplished in one of two ways. For customer who automate ESX/ESXi deployments and upgrades via scripted installs (such as PXE booting) this change can be done manually by editing your config files.
As an example I have done so on the command line of my ESX 4.0 server.
Alternatively NetApp provides an automated means to make this change via our Virtual Storage Console. The VSC is the evolution of our EHU into a vCenter Plugin which will serve as the foundation for our various plugins like the Rapid Cloning Utility and VMIsight. By default the VSC will identify ESX/ESXi hosts connect to NetApp arrays and for host connected to ALUA enabled igroups it can automatically modify their SATP to use the VMware’s Round Robin PSP instead of MRU.
What about 3rd party SATPs & PSPs?
As you may recall the vSphere PSA provides support for third party SATPs & PSPs. At present the only available third party PSA is PowerPath VE from EMC. The goal of PP VE is to provide capabilities beyond what is natively available within ESX/ESXi. These capabilities are touted by EMC as ‘automating path utilization dynamically optimize performance and availability.’
Now, before you jump up and say ‘I need that’ and run off to purchase PP VE (note it is licensed per CPU), I believe it is critical to discuss ‘why’ you may need PP VE.
Traditional legacy storage arrays have a limit in the number of simultaneous commands that can be processed by a LUN. This limit is called a queue depth. A high volume of simultaneous requests is usually desirable, and results in good performance; however, if the queue becomes full the storage array will respond with a queue-full flow control command.
An ESX/ESXi response to a queue-full command is HBA dependant, but it typically results in the suspension of more than one second.
PowerPath provides multipathing intelligence to ensure that I/O is redistributed across all links in an attempt to avoid a queue-full situation. While this level of intelligence is not available in VMware’s Round Robin Path Selection Plugin you still need to understand more before purchasing.
Queue depth impacts VM performance and datastore scaling
VMware has been educating our joint customers around the impact of a LUN queue depth and how it impacts performance and scaling. Below is a chart that is from the VMware Scalable Storage Performance whitepaper. As you can see from this chart, queue depth in combination with the average number of SCSI commands per VM are the two metrics one should consider when sizing datastore scaling.
So does my array need PowerPath VE?
It depends, what is the queue depth of the LUNs on your array?
NetApp storage arrays virtualize storage and as a result we do not have LUN queue depths, so we avoid LUN queue-full conditions. Please note NetApp arrays do have a queue depth per FC & FCoE target of 2,000 per port. As a reference each FC/FCoE target port is capable of greater than 30,000 IOPs.
By contrast a quick reference of the EMC Clariion Performance and Availability Best Practices with Flare 28.5 document states that a Clariion array has a LUN queue depth that is calculated based on the size of the LUN and the type of data protection in use. In addition, Clariion arrays have a target port queue depth of 1,600.
From page 26 of this document the calculation for a RAID 5 protected LUN is:
(14 * (# of data drives) + 32)
This configuration results in a RAID 5 LUN comprised of five disks (4+1) having a queue depth of 88!
If your storage array has a queue depth of 88, I believe we have identified why it is your best interest to purchase PowerPath VE as you will need automated path utilization to dynamically optimize performance and availability. If you don’t and instead use the vSphere Round Robin PSP you risk having poor performance across all of your LUNs should a single LUN issue a queue-full command.
It sure seems that the need for PP VE is a direct result of a less than VMware friendly design within the storage controller.
Maybe a better decision would be to look a NetApp virtualized storage array for your vSphere deployments. By design our arrays don’t have LUN queues. Instead they have target port queues, which are global, very deep, and when combined with RR the queues are aggregated. As I stated earlier each port has a queue of 2,000, or a single dual port target adapter has a queue of 4,000. This design allows NetApp arrays to avoid the issues that can arise with such shallow queues.
The virtualized architecture of a NetApp array is the ideal design for use with VMware’s Round Robin PSP as we don’t have the challenges associated with traditional legacy arrays.
I should have mentioned it earlier in this post, but NetApp has provided the EHU and will provide the VSC at no cost. Maybe EMC will consider doing the same with PP VE as it seems to be required in order to ensure the availability and performance with a Clariion array.
Wrapping it up
Congratulations VMware on the vSphere release, I believe I speak for all of us at NetApp when I say we love they technology and how it is evolving our data centers. The Plug-n-Play SAN architecture is finally here! It is robust, resilient, available at no cost, and with modern storage arrays it is awesome!