SAP HANA TDI on Cisco UCS and VMware vSphere - Part 3
Consult the SAP Guidelines as well as the VMware Best Practices for scale-up or scale-out deployments.
CPU and Memory Sizing
This document is not a sizing instrument but delivers a guideline of technically possible configurations. For proper sizing, refer to BW on HANA (BWoH) and Suite on HANA (SoH) sizing documentation. The ratio between CPU and memory can be defined as 6.4 GB per vCPU for Westmere-EX, 8.53 GB per Ivy Bridge-EX and 10.66 GB per Haswell-EX vCPU, while the numbers for SoH can be doubled. Since HANA SPS11, the ratio is 14.22 GB per Haswell-EX vCPU for BWoH, while the number for SoH don't change.
Since Broadwell-EX and SPS12, things get even more complicated. This processor family has two processors that are certified for HANA, but with different amount of physical cores. Additionally, SoH can be sized differently with from SPS12 on. If you adhere to this matrix, you're pretty much on the safe side:
See that the ESXi host and every virtual machine produce some memory overhead. For example, an ESXi server with 512 GB physical RAM can not host a virtual machine with 512 GB vRAM because the server would need some static memory space for its kernel and some dynamic memory space for each virtual machine. A virtual machine with eg. 500 GB vRAM would most likely fit into a 512 GB ESXi host.
There's also a blog on sizing a virtual machine for HANA.
Sizing E5 Entry Level Systems
Source: TDI Overview, April 2016, SAP SE
The sizing for E5 systems works different. There's no specific CPU : RAM ratio nor a specific E5 CPU model that is supported, therefore there can be no sizing guideline that compares to E7 systems. The E5 systems were meant to be a cost-optimized alternative to the regular E7 systems - that's why it is called "Entry Level System" in the context of HANA. Usually they are used for non-production (see SAP Note 2271345), but also productive systems with E5 are possible following these standards:
8 cores per socket minimum
up to 1.5 TB
homogenous symmetric assembly of DIMMs
maximum utilization of all available memory channels
local or remote
fulfill TDI storage requirements
pass KPI test
This gives a lot of freedom for customers to build their Entry Level Systems. The CPU model can range from 8 to 22 cores per socket and from 1.6 to 3.2 GHz per core, with a maximum RAM size of 1.5 TB. Defining a fixed vCPU : vRAM ratio for such a variety of processors is simply not possible. Additionally, if multiple VMs with production systems should run on one host, the restriction from SAP Note 2024433 also applies for Entry Level Systems, which limit the amount of allowed VMs to the number of two:
The vCPUs of a single VM must be pinned to physical cores, so the CPU cores of a socket get exclusively used by only one single VM. A single VM may span more than one socket, however. If a VM running SAP HANA as workload uses physical cores of a certain socket there shall not be any other VM (used for SAP HANA or any other workload) making use of any of the physical cores of this same socket. CPU and Memory overcommitting must not be used.
With all this freedom, how to configure productive VMs on E5, then? One thing we can derive from SAP's guidelines is a ratio of 1 socket : 768 GB RAM, but still this doesn't say anything about the actual compute power that is behind this number. If you use E5 2699 v4, then it's a totally different performance than with E5 2667 v4. It's not more or less performance, it's different performance: You would get more cores at the cost of less clockspeed. HANA usually prefers cores over clockspeed, but there's a reason why in the E7 systems only the top-notch CPU models are chosen: to get a lot of cores and still have decent clockspeed. So when you run a 1.5 TB HANA system with low-end E5 CPUs (8 cores with 1.7 GHz for example), there's a good chance that the user experience might suffer. Good E5 CPUs come also with a price, see the comparison between an E7 CPU supported for HANA and the E5 CPU with the highest accumulated throughput:
|E5 2699 v4||22||2.2 GHz||55 MB||4,115 $|
|E7 8880 v4||22||2.2 GHz||55 MB||5,895 $|
Of course, an E7 CPU has a greater memory throughput and because of its enhanced capabilities it is integrated in systems with greater reliability, availability, and serviceability. But the whole point about E5 systems is to save money, and using expensive CPUs in low-cost servers mitigate the whole purpose.
In the end, there's no answer to the question of how to size HANA VMs on an E5 server, because there's no definite guideline on how to size HANA on E5 servers at all. It's all about optimizing price : performance ratio, nothing else. If you want adequate systems for your HANA database, virtualized or not, use E7. If you want to save money, use E5, and depending on how far you want to go dumping the price, use a corresponding processor.
An example configuration with good price : performance could be following:
E5 2683 v4, 16 cores, 2.1 GHz
1 TB RAM
First VM (production): 32 vCPUs (2 vSockets), 500 GB vRAM, reserve vCPU and vRAM
Second VM (non-production): 16 vCPUs (1 vSocket), 250 GB vRAM, reserve vRAM
Third VM (non-production): 16 vCPUs (1 vSocket), 250 GB vRAM, reserve vRAM
vRAM size of VMs deviate from standard sizings to satisfy memory overhead
Configure Numa.PreferHT=1 on the ESXi host
Virtual Machine Creation
Create virtual machine with hardware version 11 (ESXi 6.0)
Select the appropriate Guest OS
Configure cores per socket
It is recommended to configure the cores per socket according to the actual hardware configuration, which means:
- 10 cores per socket on HANA certified Westmere-EX processors
- 15 cores per socket on HANA certified Ivy Bridge-EX processors
- 18 cores per socket on HANA certified Haswell-EX processors
- 22 cores per socket on HANA certified Broadwell-EX processor E7-8880v4
- 24 cores per socket on HANA certified Broadwell-EX processor E7-8890v4
The total amount of vCPUs will then be determined by setting the appropriate amount of sockets. The cores per socket should never be changed and always reflect the underlying hardware.
Virtual SCSI controller configuration
Configure 4 virtual SCSI controllers of the type "VMware Paravirtual".
Configure virtual disks
To fully utilize the virtual resources, a disk distribution is recommended where the disks are connected to different virtual SCSI controllers. This improves parallel IO processing inside the guest OS.
Consult the SAP HANA TDI - Storage Requirements to get the disk sizes right. The size of the disks can initially be chosen lower than for the HANA Appliance models. This is based on the capability to increase the virtual disk size online.
Size (old measure)
Size (new combined measure)
1 x vRAM
1 x vRAM
3 x vRAM
1.5 x vRAM**
1 x vRAM
vRAM < 512 GB: 0.5 x vRAM
vRAM ≥ 512 GB: min. 512 GB
** The official recommendation is: 1.2 * net disk space for data (use Quick Sizer HANA Version to determine)
Configure virtual network adapters
Configure one VMXNET3 adapter for each network and connect it to the corresponding port group. See that some networks are only configured on ESXi level, such as storage and backup network.
CPU and Memory Reservation
For productive virtual machines, it is recommended to reserve CPU and memory. While the CPU reservation can vary, the memory reservation has to be set to 100 %. Check the box "Reserve all guest memory (All locked)". This also has the positive side effect that the size of the vm swap file is 0 bytes.
Apply SAP Note 1606643 - Linux: VMware vSphere host monitoring interface.
After creating a VM, the guest OS installation can begin.