Solution For Sap Hana Platform in Scale Up Configuration Using Advanced Server ds7000 Second Generation Intel Xeon Scalable Processors
Solution For Sap Hana Platform in Scale Up Configuration Using Advanced Server ds7000 Second Generation Intel Xeon Scalable Processors
MK-SL-147-06
June 2021
© 2021 Hitachi Vantara LLC. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including copying and recording,
or stored in a database or retrieval system for commercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Vantara LLC
(collectively “Hitachi”). Licensee may make copies of the Materials provided that any such copy is: (i) created as an essential step in utilization of the
Software as licensed and is used in no other manner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials.
“Materials” mean text, data, photographs, graphics, audio, video and documents.
Hitachi reserves the right to make changes to this Material at any time without notice and assumes no responsibility for its use. The Materials contain
the most current information available at the time of publication.
Some of the features described in the Materials might not be currently available. Refer to the most recent product announcement for information about
feature and product availability, or contact Hitachi Vantara LLC at https://support.hitachivantara.com/en_us/contact-us.html.
Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicable Hitachi agreements. The use of Hitachi
products is governed by the terms of your agreements with Hitachi Vantara LLC.
By using this software, you agree that you are responsible for:
1. Acquiring the relevant consents as may be required under local privacy laws or otherwise from authorized employees and other individuals; and
2. Verifying that your data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws.
Notice on Export Controls. The technical data and technology inherent in this Document may be subject to U.S. export control laws, including the
U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations in other countries. Reader agrees to
comply strictly with all such regulations and acknowledges that Reader has the responsibility to obtain licenses to export, re-export, or import the
Document and any Compliant Products.
Hitachi and Lumada are trademarks or registered trademarks of Hitachi, Ltd., in the United States and other countries.
AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eServer, FICON, FlashCopy, GDPS, HyperSwap, IBM, Lotus, MVS, OS/
390, PowerHA, PowerPC, RS/6000, S/390, System z9, System z10, Tivoli, z/OS, z9, z10, z13, z14, z/VM, and z/VSE are registered trademarks or
trademarks of International Business Machines Corporation.
Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft, the Microsoft Corporate Logo, MS-DOS,
Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQL Server, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo,
Windows Azure, Windows PowerShell, Windows Server, the Windows start button, and Windows Vista are registered trademarks or trademarks of
Microsoft Corporation. Microsoft product screen shots are reprinted with permission from Microsoft Corporation.
All other trademarks, service marks, and company names in this document or website are properties of their respective owners.
Copyright and license information for third-party and open source software used in Hitachi Vantara products can be found at https://
www.hitachivantara.com/en-us/company/legal.html.
Feedback
Hitachi Vantara welcomes your feedback. Please share your thoughts by sending an email message to [email protected]. To assist the
routing of this message, use the paper number in the subject and the title of this white paper in the text.
Revision history
MK-SL-147-06 Add support for Hitachi Virtual Storage Platform G200 (VSP G200) June 30, 2021
MK-SL-147-05 ■ Add support for Hitachi Advanced Server DS7000 series CLX with RHEL 8.1 July 15, 2020
MK-SL-147-04 Add support for Hitachi Advanced Server DS7000 series with Intel CLX with May 5, 2020
SUSE Linux Server 15 SP1
Hitachi Solution for SAP HANA Platform Scale-up using Hitachi Advanced Server DS7000 2
Reference Architecture Guide
Use this reference architecture guide to implement SAP HANA in a scale-up configuration for
Hitachi Solution for the SAP HANA platform on Hitachi Advanced Server DS7000 series
using Intel Xeon Scalable Processors. This document covers following deployments:
■ SAP HANA appliance — With a SAP HANA appliance deployment, the hardware vendor
pre-assembles, pre-installs, and pre-configures the hardware and software, with the
complete package certified by SAP.
■ SAP HANA tailored data center integration (TDI) — With a SAP HANA TDI deployment,
each installation is customized by assembling hardware, operating system, and hypervisor
(optional) from SAP-certified components. SAP HANA Tailored Data Center Integration —
Overview has the details about SAP HANA TDI Phase 5 and the TDI overview.
These solutions use the following components:
■ Hardware
● One Hitachi Advanced Server DS7000 series
● External subsystem storage Hitachi Virtual Storage Platform G200 (VSP G200)
● External subsystem storage Hitachi Virtual Storage Platform F350 (VSP F350) or VSP
G350
■ Software
● Preconfigured with SAP HANA to provide a converged solution for applications using
SAP HANA Platform
The validation of this environment with external storage uses Hitachi Virtual Storage Platform
F350 or VSP G350 and VSP G200 with DS7000 servers using first generation Intel Xeon
Scalable Processors. Your needs may require other storage options and fall under a Tailored
Datacenter Integration (TDI) solution. Contact your account representative for details and
implementation services whenever you require using external storage.
The following figure shows the topology of this reference solution using external drives on a
Hitachi Virtual Storage Platform F350 or VSP G350 storage subsystem.
This technical paper assumes that you have familiarity with the following:
■ Storage area network (SAN)-based storage systems
■ General storage concepts
■ Common IT storage practices
■ SAP HANA
Note: Testing of this configuration was in a lab environment. Many things affect
production environments beyond prediction or duplication in a lab environment.
Follow the recommended practice of conducting proof-of-concept testing for
acceptable results in a non-production, isolated test environment that otherwise
matches your production environment before your production implementation of
this solution.
Memory configurations
This section shows the available memory configurations for systems with first- or second-
generation Intel Xeon Scalable Processors, including the use of Intel Optane DC persistent
memory for systems with second generation processors.
Note: M and L type processors are required for 128 GB DIMM use.
The lower DIMM slot number in the table below is closer to the CPU.
Controller 0 1 Capacity
Number (GB) per
2 CPUs
Channel 0 1 2 3 4 5
Number
DIMM Slot 1 2 3 4 5 6 7 8 9 10 11 12
Number
1DPC 32 32 32 32 32 32 32 384
GB DIMM
2DPC 32 32 32 32 32 32 32 32 32 32 32 32 32 768
GB DIMM
1DPC 64 64 64 64 64 64 64 768
GB
DIMM*
2DPC 32 64 32 64 32 64 32 64 32 64 32 64 32 1152
or 64 GB
Mixed
DIMM*
2DPC 64 64 64 64 64 64 64 64 64 64 64 64 64 1536
GB
DIMM*
2DPC 128 128 128 128 128 128 128 128 128 128 128 128 128 3072
GB
DIMM*
Memory configurations with Intel Optane DC Persistent Memory are listed in the following
table, with the DIMM slots shown accordingly in the next table.
** According to the Intel PCN 117365-00, the discontinuance of the M series began on
January 15, 2020.
controller 0 1 Capacity
# (GB) per
2 CPUs
channel # 0 1 2 3 4 5
DIMM slot 1 2 3 4 5 6 7 8 9 10 11 12
#*
2DPC 128 128 128 128 128 128 128 128 128 128 128 128 128 3072
GB + 128
GB
DRAM
2DPC 256 256 128 256 128 256 128 256 128 256 128 256 128 4608
GB + 128
GB
DRAM
2DPC 512 512 128 512 128 512 128 512 128 512 128 512 128 7680
GB + 128
GB
DRAM
* Odd numbered slots are for Intel Optane DC Persistent Memory; Even numbered slots
are for DRAM.
Note: Do not change the layout of any of the components in this environment
without consulting your Hitachi Vantara account representative. Changing this
layout can require manual configuration of the network and/or using different
components.
Hardware elements
This section details the hardware used to deploy this specific scale-up configuration of Hitachi
Solution for SAP HANA for the different sized solutions.
Hitachi Advanced Server DS7000 series uses an external storage subsystem, as indicated in
the following table.
Hitachi Advanced 1 CPUs per SAP HANA node: SAP HANA server
Server DS7020 ■ One of the following:
● 2 Intel Xeon Platinum
8176/8176M Processor
28-core, 2.1GHz, 165W
● 2 Intel Xeon Platinum
8180/8180M Processor
28-core, 2.5GHz, 205W
● 2 Intel Xeon Platinum
8276*/8276M**/8276L
processors, 28-core, 2.20
GHz, 152 W
● 2 Intel Xeon Platinum
8280*/8280M**/8280L
processors, 28-core, 2.70
GHz, 205 W
■ 2 heat sinks CPU 0/1
■ For the RAM per SAP HANA
node, see the previous
section
*Not covered by Intel Select Solutions. Also refer to the supported configurations with and
without Intel Optane DC Persistent Memory described above for 4 socket, 6 socket, and 8
socket CPU information that is covered by Intel Select Solutions.
**According to the Intel PCN 117365-00, the discontinuance of the M series began on
January 15, 2020.
Software elements
The following table describes the software products used to deploy this solution.
Purpose Software
Solution design
The detailed design for this scale-up configuration of Hitachi Solution for the SAP HANA
Platform for this reference solution includes the following:
■ Hitachi Advanced Server DS7000 series Configuration
■ Network Architecture
■ Storage Architecture Configuration
■ SAP HANA Configuration
The following figure shows the locations of the memory modules. Only insert the Intel Optane
DC Persistent Memory DIMMs in the slots with slot name CH-<X>1. For example, use the
CH-C1 slot. The numbers of DCPMM DIMMs must match with number of DRAM DIMMs.
The following figure is an example for mix-and-match DRAM and Intel Optane DC Persistent
Memory DIMMs on an Advanced Server DS7000 series 2S module:
The following figure is an example of DRAM DIMMs without Intel Optane DC Persistent
Memory DIMMs on an Advanced Server DS7000 series 2S module:
Hitachi Advanced Server DS7000 series systems scale from 2 sockets to 8 sockets. See the
following figure for the locations of the network and HBA cards.
Make the following 10 GbE network connections for SAP HANA nodes as shown in the table .
■ Connect the following to Cisco Nexus 93180YC-FX or 93180YC-EX switches, or to any
other external switches:
● Port 0 of 10 GbE Mellanox Connect-4x LX PCIe card to two different Cisco Nexus
93180YC-FX or 93180YC-EX switches.
● Bond the corresponding two ports, eth9901 and eth9902, as bond0 at the operating
system level using active-active network bond mode with the following options:
mode= 802.3ad miimon=100 xmit_hash_policy=2 updelay=5000 lacp_rate=fast
This acts as the client network for the SAP HANA node.
■ If additional 10 GbE network connections are required, connect Port 1 of both 10 GbE
Mellanox Connect-4x LX PCIe cards to the Cisco Nexus 93180YC-FX or 93180YC-EX
switches or to any other external switches.
■ For all sizes, PCIe Slots 1 and 3 of the master module (see above) are used to create a
bond. All other slots and ports can be used at the customer’s discretion.
Note: The management network and client network can be on the same network
switch or a separate network switch, depending on the network environment.
For a direct connection between Hitachi Virtual Storage Platform F350, VSP G350, or VSP
G200 and the Emulex HBA on Advanced Server DS7000 series, use System Mode 847 on
the storage to enable the following host mode options (HMO) for the corresponding storage
port connected to the server:
■ HMO 94— Mandatory for direct connection between Hitachi Virtual Storage Platform and
the Emulex Lpe31002 HBA to work. Without setting up Host Mode Option 94, SAN
storage cannot be identified from the Emulex HBA.
■ HMO109— Fixes the SAN boot issue where sometimes the system cannot find the boot
LUN after a reboot. Host Group ID must be 00 for SAN Boot.
The following figure shows the connections between the storage and the Hitachi Advanced
Server DS7000 series system.
Also, use the following port properties as listed in the following table:
Fabric OFF
Dynamic
Provisioning Pool
Name Purpose Parity Group RAID Level and Disks
The example used for SAP HANA certification is minimal storage configuration on Hitachi
Virtual Storage Platform F350 or VSP G350 in a scale-up appliance solution for SAP HANA
with sizes ranging from 384 GB to 2304 GB memory, as follows:
■ Logical devices layout
■ Virtual volume for SAP HANA node
■ LUN path assignment
The following table shows the logical devices layout.
Parity
Group
RAID
Dynamic Parity Level MPU
Provisioning Group and LDEV Assignm
Pool ID Disks LDEV ID LDEV Name Size (GB) ent
The following table shows the virtual volume information for SAP HANA nodes.
Virtual MPU
Dynamic Provisioning Virtual Volume Assignme
Pool Volume ID Virtual Volume Name Size nt
File system
The implementation of this solution uses an external storage subsystem:
■ BTRFS Filesystem
● Operating system volume
■ XFS Filesystem
● For DRAM with Intel Optane DC Persistent Memory
■ SAP HANA shared volume
■ Data volume
■ Log volume
■ Persistent memory volume
Device-mapper multipath
This solution uses Device-Mapper Multipath to consolidate the multiple connections coming
from external storage subsystems.
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 64
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = auto
max_parallel_io_requests[log] = 64
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = auto
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 128
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = on
max_parallel_io_requests[log] = 128
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = on
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 64
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = auto
max_parallel_io_requests[log] = 64
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = auto
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
RHEL 8.2
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 128
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = on
max_parallel_io_requests[log] = 128
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = on
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 64
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = auto
max_parallel_io_requests[log] = 64
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = auto
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
RHEL 8.2
[communication]
tcp_backlog = 2048
[fileio]
max_parallel_io_requests[data] = 128
max_submit_batch_size[data] = 64
size_kernel_io_queue[data] = 512
async_read_submit[data] = on
async_write_submit_blocks[data] = all
min_submit_batch_size[data] = 16
async_write_submit_active[data] = on
max_parallel_io_requests[log] = 128
max_submit_batch_size[log] = 64
size_kernel_io_queue[log] = 512
async_read_submit[log] = on
async_write_submit_blocks[log] = all
min_submit_batch_size[log] = 16
async_write_submit_active[log] = on
[multidb]
mode = multidb
database_isolation = low
singletenant = yes
[persistence]
basepath_datavolumes = /hana/data/HIT
basepath_logvolumes = /hana/log/HIT
Product descriptions
The following information describes the hardware and software components used in this
reference architecture.
SAP HANA
SAP HANA converges database and application platform capabilities in-memory to transform
transactions, analytics, text analysis, predictive and spatial processing so businesses can
operate in real-time. This combines database, data processing, and application platform
capabilities in a single in-memory platform. Also, the platform provides libraries for predictive,
planning, text processing, spatial, and business analytics — all on the same architecture.
This architecture comes from leading hardware partners of SAP, including Hitachi Vantara.
For more information, see https://www.sap.com/products/hana.html.
By eliminating the divide between transactions and analytics, SAP HANA allows you to
answer any business question anywhere in real time.
As a SAP customer, you can download more information, including the following:
■ SAP HANA Master Guide
This is the central starting point for the technical implementation of SAP HANA. Use this
for basic concepts and for planning.
■ SAP HANA Server Installation and Update Guide
Use the various installation guides to install the required SAP In-Memory Database and
the other software components for the different replication technologies.
■ SAP HANA Administration Guide
This provides the central operations documentation for the on-premises deployment of the
SAP HANA Platform.
SAP HANA hardware directory provides information about SAP HANA appliances certified by
SAP hardware partners.
Changing the configuration settings is only supported along the guidelines of SAP and the
operating system distributor and may otherwise cause significant performance problems. The
following SAP Notes for SUSE Linux Enterprise Server and Red Hat Enterprise Linux are a
good starting point for information on this topic:
■ 1944799 - SAP HANA Guidelines for SLES Operating System Installation
■ 2009879 - SAP HANA Guidelines for Red Hat Enterprise Linux (RHEL) Operating System
For more details, see "Updating and Patching the Operating System" by searching in the
"View SAP HANA document" from Technical Information and Best Practices.