Block vs file for Hyper-V and VMware storage: Which is better?
Block vs file for Hyper-V and VMware storage: Which is better?
via Block vs file for Hyper-V and VMware storage: Which is better?.
Original: Chris Evans, Contributor, SearchStorage.co.uk
When it comes to Hyper-V and VMware storage, which is better: block- or file-based access? The rate of adoption of server virtualisation has accelerated over recent years, and virtual server workloads now encompass many production applications, including Tier 1 applications such as databases.
For that reason it is now more important than ever that Hyper-V and VMware storage is well-matched to requirements of the environment. In this article we will discuss the basic requirements for Hyper-V and VMware storage and examine the key question of block vs file storage in such deployments.
Basic requirements for storage in virtual server environments
When selecting storage for virtual server environments, a basic set of requirements must be met, irrespective of the hypervisor or the storage protocol. These include:
- Shared access. Storage connected to hypervisors typically needs to provide access shared among hypervisor hosts. This enables redundant and high-availability configurations. Where shared storage is implemented for multiple hypervisors, guests can be load-balanced across the servers for performance and availability in the event of a server failure.
- Scalability. Virtual server environments can include hundreds of virtual machines. This means any storage solution needs to be scalable to cater for the large volume of data virtual guests create. In addition, scalability is required for shared connectivity, providing for multiple hosts with multiple redundant connections.
- High availability. Virtual server environments can contain hundreds of virtual servers or desktops. This represents a concentration of risk requiring high availability from the storage array. Availability can be quantified in terms of array uptime but also of components that connect the server to the array, such as network or Fibre Channel switching.
- Performance. Virtual environments create a different performance profile for I/O than that of individual servers. Typically, I/O is random in nature, but certain tasks, such as backup and guest cloning, can result in high sequential I/O demands.
Protocol choice: Block vs file?
Virtual servers can be deployed either to direct-attached storage (DAS) or networked storage (NAS or SAN). DAS does not provide the shared access required of highly available virtual clusters because it is physically associated with a single virtual server. Enterprise-class solutions, therefore, use networked storage and this means protocols such as NFS, CIFS, iSCSI, Fibre Channel and Fibre Channel over Ethernet (FCoE).
File-level access: NAS
Network-attached storage encompasses the NFS and CIFS protocols and refers specifically to the use of file-based storage to store virtual guests. VMware ESXi supports only NFS for file-level access; Hyper-V supports only CIFS for file access. This difference is perhaps explained by the fact that CIFS was developed by Microsoft from Server Message Block(SMB) and NFS was originally developed by Sun Microsystems for its Solaris operating system — both Solaris and ESXi are Unix variants.
For VMware, NFS is a good choice of protocol as it provides a number of distinct benefits.
- Virtual machines are stored in directories on NFS shares, making them easy to access without using the hypervisor. This is useful for taking virtual machine backups or cloning an individual virtual guest. VMware configuration files can also be directly created or edited.
- Virtual storage can easily be shared among multiple virtual servers; VMware uses a locking file on the share to ensure integrity in a clustered environment.
- No extra server hardware is required to access NFS shares, which can be achieved over standard network interface cards (NICs).
- Virtual guests can be thinly provisioned, if the underlying storage hardware supports it.
- Network shares can be expanded dynamically, if the storage filer supports it, without any impact on ESXi.
There are, however, some disadvantages when using NFS with VMware.
- Scalability is limited to eight NFS shares per VMware host (this can be expanded to 64 but also requires TCP/IP heap size to be increased).
- Although these NFS shares can scale to the maximum size permitted by the storage filer, the share is typically created from one group of disks with one performance characteristic; therefore, all guests on the share will experience the same I/O performance profile.
- NFS does not support multipathing, and so high availability needs to be managed at the physical network layer with bonded networks on ESXi and virtual interfaces on the storage array — if it supports it.
For Hyper-V, CIFS allows virtual machines (stored as virtual hard disk, or VHD, files) to be stored and accessed on CIFS shares specified by a Uniform Naming Convention (UNC) or a share mapped to a drive letter. While this provides a certain degree of flexibility in storing virtual machines on Windows file servers, CIFS is an inefficient protocol for the block-based access required by Hyper-V and not a good choice. It is disappointing to note that Microsoft currently doesn’t support Hyper-V guests on NFS shares. This seems like a glaring omission.
Block-level access: Fibre Channel and iSCSI
Block protocols include iSCSI, Fibre Channel and FCoE. Fibre Channel and FCoE are delivered over dedicated host adapter cards (HBAs and CNAs, respectively). iSCSI can be delivered over standard NICs or using dedicated TOE (TCP/IP Offload Engine) HBAs. For both VMware and Hyper-V, the use of Fibre Channel or FCoE means additional cost for dedicated storage networking hardware. iSCSI doesn’t explicitly require additional hardware but customers may find it necessary to gain better performance.
VMware supports all three block storage protocols. In each case, storage is presented to the VMware host as a LUN. Block storage has the following advantages.
- Each LUN is formatted with Virtual Machine File System, or VMFS, which is specifically written for storing virtual machines.
- VMware supports multipath I/O for iSCSI and Fibre Channel/FCoE.
- Block protocols support hardware acceleration through vStorage APIs for Array Integration (VAAI). These hardware-based instructions improve the performance of data migration and locking to increase throughput and scalability.
- ESXi 4.x supports “boot from SAN†for all protocols, enabling stateless deployments.
- SAN environments can use RDM (Raw Device Mapping), which enables virtual guests to write non-standard SCSI commands to LUNs on the storage array. This feature is useful on management servers.
For VMware, there are some disadvantages to using block storage.
- VMFS is proprietary to VMware, and data on a VMFS LUN can be accessed only through the hypervisor. This process is cumbersome and slow.
- Replication of SAN storage usually occurs at the LUN level; therefore, replicating a single VMware host is more complex and wasteful in resources where multiple guests exist on the same VMFS LUN.
- iSCSI traffic cannot be encrypted and so passes across the network in plain view.
- iSCSI security is limited to CHAP (Challenge Handshake Protocol), which isn’t centralised and has to be managed through the storage array and/or VMware host. In large deployments this may be a significant overhead in management.
Hyper-V is deployed either as part of Windows Server 2008 or as Windows Hyper-V Server 2008, both of which are Windows Server variants. Therefore virtual guests gain all the benefits of the underlying operating system, including multipathing support. Individual virtual machines are stored as VHD files on LUNs mapped to drive letters or Windows mount points, making them easy to back up or clone.
Summary
NFS storage is suitable only for VMware deployments and is not supported by Hyper-V. Typically, NAS filers are cheaper to deploy than Fibre Channel arrays, and NFS provides better out-of-band access to guest files without the need to use the hypervisor. In the past NFS had been used widely for supporting data like ISO installation files, but today it has wider deployments where the array architecture supports the random I/O nature of virtual workloads.
CIFS storage is supported by Hyper-V but is probably best avoided in preference of iSCSI, even in test environments; Microsoft has now made its iSCSI Software Target freely available.
Block-based storage works well on both virtualisation platforms but can require additional hardware. Directly accessing data is an issue for iSCSI/Fibre Channel/FCoE, making data cloning and backup more complex.
Overall, the choice of platform should be considered against your requirements. There are clearly pros and cons with either a file- or block-based approach, each of which can coexist in the same infrastructure. There’s no doubt that both will find homes with server virtualisation for many years to come.
Comparing ESXi 4.1 and ESXi 5.0 Scaling Performance
In previous articles on VROOM! we used VMmark 2 to investigate the effects of altering a single hardware component, such as a storage array or server model, in a vSphere cluster. In contrast to these earlier studies, we now examine the effects of upgrading the hosts’ software from ESXi 4.1 to ESXi 5.0 on the performance of a VMmark 2 cluster. vSphere 5 includes many new features and virtual machine enhancements, the details of which can be found here. To the IT professional weighing the costs and benefits of upgrading their existing infrastructure to vSphere 5, an often important question is whether ESXi 5.0 can outperform ESXi 4.1 in the same environment. VMmark 2 is an ideal tool for answering this question with measurable results. We used VMmark 2.1.1 to see how ESXi 5.0 stacked up to ESXi 4.1 on an identically configured cluster. VMmark 2 is a multi-host virtualization benchmark that models application performance as well as the effects of common infrastructure operations such as vMotion, Storage vMotion, and virtual machine deployments. Each VMmark tile contains a set of VMs running diverse application workloads as a unit of load. VMmark 2 scores are computed as a weighted average of application workload throughput and infrastructure operation throughput. For more details, see the overview, release notes for VMmark 2.1, and for 2.1.1. Testing Methodology All VMmark 2 tests were conducted on a cluster of four identically configured entry-level Dell Power Edge R310 servers. To determine the impact of the vSphere 5 environment on performance, a series of tests was conducted with these hosts running ESXi 4.1, then with ESXi 5.0. In addition, for the vSphere 5 environment, the virtual machine hardware and VMware Tools were upgraded on all workload VMs, and LUNs were reformatted as VMFS5. All other components in the environment remained unchanged during testing. Configuration Results To characterize cluster performance at multiple load levels, we increased the number of tiles until the cluster reached saturation, defined as when the run failed to meet Quality of Service (QoS) requirements. Scaling out the number of tiles until saturation allows us to determine the maximum VMmark 2 load the cluster could support and to compare the ESXi 4.1 and ESXi 5.0 configurations at each level of load. The graph below shows the results of the VMmark 2 testing as described above with identically configured clusters running ESXi 4.1 and ESXi 5.0. All data points are the mean of three tests in each configuration. Â
The ESXi 4.1 cluster reached saturation at 3 tiles, but ESXi 5.0 was able to support 4 tiles while still meeting workload Quality of Service requirements. The ESXi 5.0 cluster also outperformed ESXi 4.1 by 3% and 4% on the two and three-tile runs, respectively. Differences in CPU utilization were negligible. The results show that, in an equivalent environment, vSphere 5 handled greater load than ESXi 4.1 before reaching saturation, and showed increased performance at lower levels of load as well. At saturation, vSphere 5 showed a 22% increase in overall VMmark 2 scores over ESXi 4.1. In this cluster, vSphere 5 supported 33% more VMs and twice the number of infrastructure operations while meeting Quality of Service requirements. VMmark 2 scores are based on application and infrastructure workload throughput, while application latency reflects Quality of Service. For the Mail Server, Olio, and DVD Store 2 workloads, latency is defined as the application’s response time. The completion time for vMotion, Storage vMotion, and VM Deploy is used as the latency measurement for the infrastructure operations. Latency can be very informative about the functioning of the environment and how the cluster as a whole performs under increasing loads. Examining latency at a 3-tile load, as seen in the figure below, reveals significant differences between the hypervisor versions. Latencies were normalized to the ESXi 4.1 results. We saw decreases in latency for all VMmark 2 workloads with vSphere 5. The latency decreases were most striking in Olio, Storage vMotion, and DVD Store 2, with decreases of 20%, 19%, and 15%, respectively. These improvements to vMotion and Storage vMotion are consistent with publicized improvements in vMotion and Storage vMotion latency for vSphere 5 (details here). A VMmark 2 run passes when all of its application QoS metrics, or latencies, remain below a specified threshold. These decreases in latency with ESXi 5.0 are directly related to why ESXi 5.0 was able to support an additional tile relative to ESXi 4.1. Our comparison has shown that upgrading an ESXi 4.1 cluster to vSphere 5 had two high-level effects on performance. The vSphere 5 cluster supported 33% more VMs at saturation than the ESXi 4.1 cluster, and it also exhibited improved latency and throughput at lower levels of load, showing that ESXi 5.0 does outperform ESXi 4.1. |
|
 |
|
Download VMware Products | Privacy | Update Feed Preferences Copyright © 2010 VMware, Inc. All rights reserved. |
|
VMware KB: ESXi 5.x boot delays when configured for Software iSCSI
To list the current number and configuration of an ESX host’s network portals, run the command:
esxcli iscsi networkportal list
The output is similar to:
vmhba34:
Adapter: vmhba34
Vmknic: vmk6
MAC Address: 00:1b:21:59:16:e8
MAC Address Valid: true
IPv4: 192.168.1.206
IPv4 Subnet Mask: 255.255.255.0
IPv6:
MTU: 1500
Vlan Supported: true
Vlan ID: 10
Reserved Ports: 63488~65536
TOE: false
TSO: true
TCP Checksum: false
Link Up: true
Current Speed: 10000
Rx Packets: 656558
Tx Packets: 111264
NIC Driver: ixgbe
NIC Driver Version: 2.0.84.8.2-10vmw-NAPI
NIC Firmware Version: 0.9-3
Compliant Status: compliant
NonCompliant Message:
NonCompliant Remedy:
Vswitch: dvSwitch0
PortGroup: DvsPortset-0
VswitchUuid: 26 46 30 50 c0 cf df 1e-52 ef ab d7 a2 ab 96 f9
PortGroupKey: dvportgroup-78003
PortKey: 1731
Duplex:
Path Status: active
Note: This is an example of one network portal (HBA34).
To list currently running targets, run the command:
VMware KB: ESXi 5.x boot delays when configured for Software iSCSI.
Go Que vs The World