Hitachi Solution for Databases - Virtualized Oracle

Real Application Cluster (RAC) on VMware vSphere

using Hitachi Virtual Storage Platform E990 and
Hitachi Advanced Server DS220

Reference Architecture Guide

By Hitachi Vantara

November 2020
Feedback
Hitachi Vantara welcomes your feedback. Please share your thoughts by sending an email message to
[email protected]. To assist the routing of this message, use the paper number in the subject and the title of
this white paper in the text.

Revision History

Revision Changes Date

MK-SL-209-00 Initial release November 17, 2020

Table of Contents
Solution Overview 2
Business Benefits 2
High Level Infrastructure 3

Key Solution Components 5

Solution Design 7
Storage Architecture 7
Oracle ASM Configurations 11
Management Server Configurations 12
Server and Application Architecture 12
SAN Architecture 13
Network Architecture 14
Physical Network Configuration 15

Engineering Validation 18
Database Configuration 18
Testing Environment 18
Test Methodology 19
Test Results 20

Conclusion 22

Product Descriptions 22
Hitachi Storage Virtualization Operating System RF 23
Hitachi Advanced Server DS220 23
Hitachi Advanced Server DS120 23
Hitachi Ops Center 23
Oracle Linux 24
Oracle Database With the Real Application Clusters Option 24
Oracle Automatic Storage Management 24
VMware ESXi 24
VMware vCenter Server Appliance 24
Brocade Switches 25
Cisco Switches 25
1

Hitachi Solution for Databases - Virtualized Oracle Real

Application Cluster (RAC) on VMware vSphere using Hitachi
Virtual Storage Platform E990 and Hitachi Advanced Server DS220
Reference Architecture Guide

The use of Hitachi Solution for Databases with Oracle Real Application Cluster (RAC) virtualized on VMware vSphere
provides significant dynamic scalability, high availability, and reliability benefits while processing small to large Oracle
workloads and maintaining predictable performance capabilities for varying deployment types. This reference architecture
provides a validated virtualization solution for Oracle Database 19c running on VMware ESXi 6.7 while keeping
virtualization overhead to a minimum, with virtual machines using Oracle Linux 7.8 as Oracle RAC instances.

Hitachi Solution for Databases with virtualized Oracle RAC on VMware vSphere is a fast, agile offering from Hitachi Vantara
that makes efficient use of deployed resources. Oracle on VMware aims for simplifying IT transformation, as it offers
increased efficiency, quality of service, predictable performance, flexible journey to minimize cost, dynamic scalability,
disaster recovery with VMware vSphere and-most importantly-it is intuitive to implement.

This solution integrates many innovative technologies from Hitachi Vantara, Oracle, and VMware. To prepare virtualized
Oracle RAC on VMware vSphere, this environment includes the following:

 Hitachi Virtual Storage Platform (VSP E990)

 Hitachi Advanced Server DS220 for storage and computing resources

 Hitachi Advanced Server DS120 for management servers

 Oracle Real Application Cluster 19c

 VMware vCenter Server Appliance

 VMware ESXi
 VMware vSphere

 Running Oracle databases on VMware vSphere provides the following benefits:

 Dynamic scalability

 Predictable performance
 High availability
This solution provides the flexibility to select storage and compute resources based on unique requirements. Deploy small
databases as well as large databases, depending on resource availability.

This document is for the following audiences:

 Database administrators
 Storage administrators

 System administrators

1
2

To use this reference architecture guide, you need familiarity with the following:

 Hitachi Virtual Storage Platform E990

 Hitachi Advanced Server DS220 servers

 Hitachi Advanced Server DS120 servers

 Storage Area Networks
 Oracle Database administration

 Oracle Database 19c Release 3

 Oracle Linux

 VMware vCenter Server Appliance

 VMware ESXi

Note — Testing of this configuration was in a lab environment. Many things affect production environments beyond
prediction or duplication in a lab environment. Follow the recommended practice of conducting proof-of-concept
testing for acceptable results in a non-production, isolated test environment that otherwise matches your production
environment before your production implementation of this solution.

Solution Overview
Use this reference architecture to implement Hitachi Solution for Databases with virtualized Oracle Real Application Cluster
(RAC) on VMware vSphere using DS220 as VMware ESXi hosts and Hitachi Virtual Storage Platform E990 (VSP E990) with
VMware vCenter Server Appliance. This solution is engineered, pre-tested, and qualified to provide high performance, high
availability, and high reliability in demanding, dynamic Oracle environments. This reference architecture implements
Hitachi Solution for Databases for Oracle Real Application Clusters database on two VMware ESXi hosts using VSP E990,
which addresses high availability, performance, and scalability requirements for OLTP and OLAP workloads.

Business Benefits
The Oracle RAC solution for VMware vSphere virtualization provides numerous benefits to database administrators
because it creates a layer of abstraction between application, operating system, and the underlying hardware. VMware
vCenter Server Appliance manages the node resources jointly such that the cluster owns all the CPU and memory of all
nodes.

The following are some of the benefits of this reference architecture:

 Achieve high Oracle Real Application Clusters database performance with VSP E990 storage.

 Provide a solution for customers who are looking for converged Oracle products – not covered by physical servers for
Oracle RAC.
 Create a VMware ESXi cluster with multiple Hitachi Advanced Server DS220 servers to deploy virtual machines for an
Oracle Real Application Clusters database deployment.
 Add or remove ESXi hosts in the VMware ESXi cluster based on business requirements.
 Using the Oracle solution with VMware ESXi reduces space, power consumption, and IT administrative requirements
while driving high-speed performance.

2
3

High Level Infrastructure

Hitachi Solution for Databases with virtualized Oracle Real Application Cluster (RAC) on VMware vSphere includes the
following components:

 Hitachi Advanced Server DS220 servers

 Hitachi Advanced Server DS120 servers

 Hitachi Virtual Storage Platform E990

 Brocade G620 32 Gb/s SAN infrastructure

 Cisco 10/25 GbE LAN infrastructure
The configuration of Virtual Storage Platform E990 and Advanced Server DS220 has the following characteristics:

 Fully redundant hardware

 Dual fabric connectivity between hosts and storage

Figure 1shows the high-level infrastructure for this solution.

3
4

Figure 1

To avoid any performance impact to the production database, Hitachi Vantara recommends using a configuration with the
following characteristics:

 A dedicated storage system for the production database

 A dedicated storage system for storing backup data, if needed

The uplink speed to the corporate network depends on the customer environment and requirements. The Cisco Nexus
93180YC-EX switches used in this reference architecture can support uplink speeds of 40 GbE or 100 GbE if higher
bandwidth is required.

4
5

Key Solution Components

The key solution components for this solution are listed in Table 1, Table 2, and Table 3. Detailed component information is
provided in the Product Descriptions section.

TABLE 1. HARDWARE COMPONENTS

Hardware Detail Description Version Quantity

Hitachi Virtual Two Controllers: 8 × 32Gb/s Fibre Channel 93-01-01-60/01 1
Storage Platform ports
VSP E990
1024 GB cache memory

16 × 1.9 TB Backend NVMe SDDs

Hitachi Virtual Two Controllers 88-04-02-60/00 1
Storage Platform
VSP G900 4 × 32 Gbps Fibre Channel ports

16 × 12 Gbps backend SAS ports

512 GB cache memory

4 × 6.0 TB 7.2K RPM SAS drives

Hitachi Advanced 2 × Intel Xeon Platinum 8276L 28C CPU @ BIOS: 2
Server DS220 2.20GHz S5BH3B16.H00

768 GB (64GB × 12) DIMM DDR4 BMC: 4.64.06

Synchronous Registered (Buffered) 2666
MHz CPLD: 11

2 Intel XXV710 dual port 25 GbE NIC cards Driver: i40e

Driver Version:
1.7.17

Firmware: 6.8
Intel X722 Dual Port 10 GbE SFP+ Driver: i40e

Driver Version:
1.7.17

Firmware: 4.0
2 × Emulex LightPulse LPe31002-M6 2-Port Driver: lpfc
16Gb Fibre Channel Adapters
Driver Version:
12.0.193.14.

Firmware:
12.0.193.13

5
6

TABLE 1. HARDWARE COMPONENTS (CONTINUED)

Hardware Detail Description Version Quantity

Hitachi Advanced 2 × Intel Xeon Gold Processor 6240, 18-core, BIOS: 2
Server DS120 2.60GGHz, 85W S5BH3B16.H00

256 GB (32 GB × 8) DIMM DDR4 Synchronous BMC: 4.64.06

Registered (Buffered) 2666 MHz
CPLD:11
1 × 64 GB MLC SATADOM for boot

1 × Dual Port 25 GbE NIC Intel XXV710 PCIe Driver Version:

card 1.7.17

Firmware: 6.80

(1.2007.0)
1 × Emulex LightPulse LPe31002-M6 2-Port Firmware:
16 Gb Fibre Channel Adapters 12.0.193.13

Driver version:
12.0.193.14
Brocade G620 Fibre 24 × 48 port Fibre Channel switches Kernel: 2.6.34.6 2
Channel switches
32 Gb/s SFPs Fabric OS: v8.2.0b
Cisco Nexus 48 × 10/25 GbE fiber ports BIOS: version 2
93180YC-EX 07.61
6 × 40/100 Gbps Quad SFP (QSFP28) ports
NXOS: version
7.0(3)I7(6)
Cisco Nexus 1 GE 48-Port Gb Ethernet Switch BIOS: version 1
3048TP 4.0.0

NXOS: version
7.0(3)I4(7)

Note — The solution was tested with PCIe and OCP mezzanine NIC cards. Using all PCIe cards is recommended for
consistency and better NIC bonding options. SATADOM, SAN boot, or local boot can be used for the boot option.

6
7

TABLE 2. SOFTWARE COMPONENTS FOR COMPUTE NODES

Software Version Function

Oracle Linux 7.8 with UEK kernel update Operating system for Oracle VM
5.4.17-2011.4.6.el7uek.x86_64
Oracle Database 19c (19.3.0.0) Database software
Oracle Grid Infrastructure 19c (19.3.0.0) Volume management, file system software, and
Oracle automatic storage management

TABLE 3. SOFTWARE COMPONENTS FOR MANAGEMENT NODES

Software Version Function

VMware ESXi 6.7.0 U3 Hypervisor for management server

VMware vCenter Server Appliance 6.7.0 U3 VMware cluster management server

Hitachi Device Manager- Storage Microcode dependent Storage management software
Navigator
Hitachi Ops Center 10.2.0 Hitachi infrastructure management software
Swingbench 2.6.0 Oracle database stress test tool

Solution Design
This describes the reference architecture environment to implement Hitachi Solution for Databases with virtualized Oracle
RAC on VMware vSphere. The environment uses Hitachi Virtual Storage Platform E990.

The infrastructure configuration includes the following:

 VMware ESXi server – There are two hosts configured in an Oracle database environment.

 Management Node – One VMware vCenter Server Appliance is used to manage all ESXi hosts.
 Storage System – There are virtual volumes mapped to each port that are presented to the server as LUNs.
 SAN Connection – There are SAN connections to connect the Fibre Channel HBA ports to the storage through Brocade
G620 switches.

Storage Architecture
This describes the storage architecture for this solution.

Storage Configuration
The storage configuration takes into consideration Hitachi Virtual Storage Platform, Oracle, and VMware recommended
best practices for the design and deployment of database storage.

Figure 2 illustrates the high-level storage configuration for this solution.

7
8

Figure 2

8
9

Table 4 shows the storage pool configuration used for this solution.

TABLE 4. DYNAMIC PROVISIONING POOL FOR HITACHI VIRTUAL STORAGE PLATFORM E990

Pool ID VMware-ESXi-Cluster
Pool Type Dynamic Provisioning Pool

RAID Group 1-1 to 1-4

RAID Level RAID-10 (2D+2D)
Drive Type 1.9 TB SSD
Number of Drives 16

Number of LDEVs 70
LDEV Size(s) 880 GB
Pool Capacity 13.74 TB

In this solution, servers use SAN boot with RAID 10 protection. Table 5 shows the logical storage configuration used in this
solution.

TABLE 5. VSP E990 LOGICAL STORAGE CONFIGURATION

ASM disk Total number Dynamic Purpose Storage Port

group of dynamic provisioning
provision volume sizes
volumes

DATA 16 200 GB OLTP application 1A, 2A, 1B, 2B, 1C, 2C, 1D,
tablespaces, system sysaux, 2D
undo, temp
REDO 8 10 GB Online redo logs and control
files
TEMP 4 40 GB Temp logs

FRA 2 200 GB Incremental backups,

archived redo logs, control
file and auto backups
OCR 3 60 GB Oracle cluster registry and
voting disk
N/A 2 200 GB SAN Boot host OS volumes
N/A 2 200 GB SAN Boot Oracle RAC VM OS
volumes

9
10

Table 6 shows the VSP G900 configuration for management servers.

TABLE 6. VSP G900 CONFIGURATION FOR MANAGEMENT SERVERS

Item Value or Description

Purpose VMware datastores

CCI device
RAID Level RAID-10 (2D+2D)
Drive Type 6.0 TB 7.2K RPM SAS
Number of Drives 4

Number of Spare Drives 0

Number of LDEVs 3
LDEV Size(s) 3000 GB

Number and Size of CCI device 1 × 100 MB

Storage Port for Management Servers 7A, 7B, 8A, 8B

There is an additional RAID group consisting of four 3 TB 7.2k RPM SAS drives configured as RAID-10 (2D+2D). This is used
as shared storage for the management server cluster.

Additional LUNs can be mapped if required. While the test environment was configured using a dedicated SAS RAID group
for the management server cluster, this can be configured as a dedicated SSD RAID group, a dedicated dynamic
provisioning pool, and capacity on the dynamic provisioning pool configured for the Oracle environment.

Database Layout
The database layout design uses recommended practices from Hitachi Vantara for Hitachi Virtual Storage Platform E990 for
small random I/O traffic, such as OLTP transactions. The layout also considers Oracle ASM best practices when using
Hitachi Vantara storage.

Base the storage design for database layout needs on the requirements of the specific application implementation. The
design can vary greatly from one implementation to another, based on the RAID configuration type and number of drives
used in the implementation.

The components in this solution have the flexibility for use in various deployment scenarios to provide the right balance
between performance and ease of management for a given scenario.

The database layout has two nodes with Oracle RAC instances. The Oracle Linux 7.8 operating system is installed on two
vCenter VMs with Oracle RAC database installed on them.

Shared LUNs from storage are assigned to two VMs as shared virtual machine disks (VMDK) for the Oracle RAC database.
The shared LUN is mapped to a datastore cluster, and virtual machine disks (VMDK) are utilized for OS SAN boot of VMs.

10
11

Oracle ASM Configurations

 Data and Indexes Tablespace - Assign an ASM diskgroup with external redundancy for the data and index tablespaces.
 TEMP Tablespace - Place the TEMP tablespace in this configuration in the TEMP ASM diskgroup.

 Undo Tablespace - Create an UNDO tablespace in this configuration within the Oracle Data ASM diskgroup.
 Online Redo Logs - Create an ASM diskgroup with external redundancy for Oracle online redo logs.
 Oracle Cluster Registry and Voting Disk - Create an ASM diskgroup with normal redundancy to contain the OCR and
voting disks and to protect against single disk failure to avoid loss of cluster availability. Place each of these files in the
OCR ASM diskgroups.

Oracle Initial Parameters

Table 7 lists the Oracle Database settings.

TABLE 7. ORACLE DATABASE SETTINGS

Environment Value
RAC Yes
ASM Yes – to support Oracle RAC Database

Oracle ASM Disk Mappings

Table 8 lists the details of the disk mappings from the LUNs to the ASM disk groups for Oracle Database tablespaces for the
2 TB database size. This is an example with a single instance database virtual machine. Adjust parameters accordingly
when multiple virtual machine pairs are used.

TABLE 8. LUN DISK MAPPINGS TO ASM GROUPS

ASM Disk ASM Disk UDEV Rules LUN Details Purpose

Group
N/A N/A /dev/xvd[a-b]1 2 × 200 GB OS and Oracle Database
OCR N/A /dev/xvd[c-e]1 3 × 60 GB Oracle cluster registry and
voting disk
DATA N/A /dev/xvd[f-u]1 16 × 200 GB Application data

REDO N/A /dev/xvd[v-z]1, /dev/ 8 × 10 GB Online REDO log group

xvda[a-c]1
FRA N/A /dev/xvda[d-e]1 4 × 200 GB Flash recovery area

TEMP N/A /dev/xvda[f-i]1 4 × 40 GB Temp

11
12

Management Server Configurations

Table 9 lists management server virtual machine (VM) configuration details.

TABLE 9. MANAGEMENT SERVER VIRTUAL MACHINE CONFIGURATION

Virtual Machine vCPU Virtual Memory Disk capacity IP Address

vCenter Server Appliance 2 10 GB 300 GB 192.168.242.xx

Hitachi Ops Center Administrator 4 16 GB 100 GB 192.168.242.xx

Hitachi Ops Center Analyzer 4 32 GB 800 GB 192.168.242.xx

Hitachi Ops Center Analyzer detail

4 10 GB 110 GB 192.168.242.xx
view

Server and Application Architecture

This reference architecture uses two Hitachi Advanced Server DS220 servers for each storage system. This provides the
compute power for the VMware infrastructure that is used for the Oracle RAC database deployment.

Two Hitachi Advanced Server DS120 servers are used for VMware ESXi management server configuration as well as
additional virtual machines to manage the environment.

Table 10 lists the details of the server configurations for this solution.

TABLE 10. HITACHI ADVANCED SERVER DS220 AND DS120 SPECIFICATIONS

Server Server Host Role CPU Core RAM

Name and Name
Model

DS220 VMware ESXi VMware ESXi server 56 (2 × 28C) 768 GB (64 GB × 12)
server 1 for Oracle RAC
database VM
VMware ESXi VMware ESXi server 56 (2 × 28C) 768 GB (64 GB × 12)
server 2 for Oracle RAC
database VM
DS120 VMware ESXi 1 Management server 16 256 GB (32 GB × 8)
for Hitachi
VMware ESXi 2 management 16 256 GB (32 GB × 8)
applications

12
13

SAN Architecture
Map the provisioned LDEVs to multiple ports on Hitachi Virtual Storage Platform E990 (VSP E990). These LDEV port
assignments provide multiple paths to the storage system from the host for high availability. This reference architecture
uses two dual port Emulex HBAs per Advanced Server DS220.

 Compute Servers
 8 SAN switch connections are used for VSP E990 Fibre Channel ports.

 8 SAN switch connections are used for server HBA ports.

 Management Servers

 4 SAN switch connections are used for VSP G900 Fibre Channel ports.
 4 SAN switch connections are used for management server HBA ports.
Table 11 lists details of the Fibre Channel switch connect configuration on Hitachi Virtual Storage Platform E990 SAS ports.

TABLE 11. SAN HBA CONNECTION CONFIGURATION TO HITACHI VIRTUAL STORAGE PLATFORM E990

Server HBA Host Host Name Switch Zone Storage Storage Brocade
Group System Port G620
Name Switch

DS220 HBA1 CN31 CN31_HBA1_1 CN31_HBA1_1_ ASE44_66_1A VSP E990 1A SAN-

Server 1 switch 1

HBA2 CN31 CN31_HBA1_2 CN31_HBA1_2_ ASE44_66_2A 2A SAN-

switch 2
HBA3 CN31 CN31_HBA2_1 CN31_HBA2_1_ ASE44_66_1B 1B SAN-
switch 1
HBA4 CN31 CN31_HBA2_2 CN31_HBA2_2_ ASE44_66_2B 2B SAN-
switch 2
DS220 HBA1 CN32 CN32_HBA1_1 CN32_HBA1_1_ ASE44_66_3A 3A SAN-
Server 2 switch 1
HBA2 CN32 CN32_HBA1_2 CN32_HBA1_2_ ASE44_66_4A 4A SAN-
switch 2
HBA3 CN32 CN32_HBA2_1 CN32_HBA2_1_ ASE44_66_3B 3B SAN-
switch 1
HBA4 CN32 CN32_HBA2_2 CN32_HBA2_2_ASE44_66_4B 4B SAN-
switch 2

13
14

TABLE 11. SAN HBA CONNECTION CONFIGURATION TO HITACHI VIRTUAL STORAGE PLATFORM E990 (CONTINUED)

Server HBA Host Host Name Switch Zone Storage Storage Brocade
Group System Port G620
Name Switch

DS120 HBA1 MN33 MN33_HBA1 MN33_HBA1_ASE42_43_1A VSP G900 1A SAN-

Server1 switch 1
HBA2 MN33 MN33_HBA2 MN33_HBA2_ASE42_43_3A 3A SAN-
switch 2
DS120 HBA1 MN34 MN34_HBA1 MN34_HBA1_ASE42_43_2A 2A SAN-
Server2 switch 1
HBA2 MN34 MN34_HBA2 MN34_HBA2_ASE42_43_4A 4A SAN-
switch 2

Note — In a production environment, use separate storage ports for the management servers to avoid impact on
database performance. While shared storage ports can be used, monitor port utilization to avoid performance issues in
high performance environments.

Network Architecture
Do the following when configuring networks in your environment:

 Use NIC bonding to provide failover and load balancing of interconnections within a server.
 Set all NICs to full duplex mode.
Configure each VMware ESXi server node with at least the bonding interfaces for the following:

 Management network

 Public network
 Private network
Each virtual machine has public, private and management vNICs. Use separate VLANs for the following:

 VMware vCenter server management network

 Private Oracle database network
 Public network

14
15

Table 12 describes the network configuration for VMware vCenter virtual machines installed with Oracle RAC database.

TABLE 12. NETWORK CONFIGURATION FOR VMWARE VCENTER VIRTUAL MACHINES

VMs vSwitch Network IP Configuration Uplinks Bandwidth

Oracle RAC IDSE-DSwitch Management 10.76.32.xx Uplink 1 10 GbE
VM 1 and Network
10.76.32.xx Uplink 2
Oracle RAC
VM 2 Public Network 10.76.32.xx Uplink 4 25 GbE

10.76.32.xx Uplink 6

Private Network 192.168.100.xx Uplink 3 25 GbE

192.168.100.xx Uplink 5

Physical Network Configuration

Figure 3 shows the IP network switch connection.

Figure 3

15
16

Table 13, Table 14, and Table 15 list the network configurations for this solution.

TABLE 13. HITACHI ADVANCED SERVER DS220 AND DS120 NETWORK CONFIGURATION

Server NIC Ports Subnet NIC IP Address Network Bandwidth Cisco Nexus
Bond (Gb/s) 93180YC-EX
Switch
Switch Port
Number
DS220 Server1 NIC 1-Port 0 242 Bond0 192.198.242.204 Management/ 10 1 41
Public
NIC 1-Port 1 10 2

NIC 2-Port 0 242 Bond1 192.198.242.205 Public 25 1 42

NIC 3-Port 0 25 2
NIC 2-Port 1 100 Bond2 192.168.100.10 Private/Public 25 1

NIC 3-Port 1 25 2
DS220 Server2 NIC 1-Port 0 242 Bond0 192.198.242.206 Management/ 10 1 43
Public
NIC 1-Port 1 10 2

NIC 2-Port 0 242 Bond1 192.198.242.207 Public 25 1 44

NIC 3-Port 0 25 2
NIC 2-Port 1 100 Bond2 192.168.100.11 Private/Public 25 1
NIC 3-Port 1 25 2

DS120 NIC 1-Port 0 242 Bond0 192.198.242.101 Management/ 25 1 49

Management Public
Server1 NIC 1-Port 1 25 2

DS120 NIC 1-Port 0 242 Bond0 192.198.242.102 Management/ 25 1 50

Management Public
Server2
NIC 1-Port 1 25 2

Note — When creating NIC Bonding pairs, ports should be used on different cards to avoid single points of failure (SPoF).

16
17

TABLE 14. NETWORK CONFIGURATION FOR SERVERS AND HITACHI VIRTUAL STORAGE PLATFORM E990

Name IP Address
VMware ESXi Server 1 192.168. 242.xx
VMware ESXi Server 2 192.168. 242.xx

Management Server 1 192.168.242.xx

Management Server 2 192.168.242.xx
VSP E990 192.168.242.xx
VSP E990 CTL1 192.168.242.x

VSP E990 CTL2 192.168.242.x

TABLE 15. NETWORK CONFIGURATION FOR SWITCHES

Switch Type Model Switch Name IP Address for MGMT

Port
Cisco 1GbE Management Cisco Nexus 3048TP Cisco3048TP-1 192.168.242.95
Network Switch
Cisco 10G/25GbE Cisco Nexus N9K- Cisco C93180YC-EX-1 192.168.242.97
Network Switch C93180YC-EX
Cisco 10G/25GbE Cisco Nexus N9K- Cisco C93180YC-EX-2 192.168.242.98
Network Switch C93180YC-EX
Brocade Fibre Channel G620 SAN-switch 1 192.168.242.67
SAN Switch
Brocade Fibre Channel G620 SAN-switch 2 192.168.242.68
SAN Switch

17
18

Engineering Validation
This summarizes the key lab verification tests performed on Hitachi Solution for Databases - Virtualized Oracle on Hitachi
Unified Compute Platform CI using Hitachi Virtual Storage Platform E990 and Hitachi Advanced Server DS220.

Database Configuration
Table 16 lists parameter details for a two node Oracle ASM database.

TABLE 16. ORACLE ASM DATABASE PARAMETERS

Oracle Database Parameter Value

Compatible 19.3.0.0.0

Oracle Database size 2 TB

Database storage type ASM
Database fill factor 80%

Table 17 shows the details of Oracle Database environment parameters for peakmarks.

TABLE 17. ORACLE DATABASE ENVIRONMENT PARAMETERS

Category Item Value

Oracle Database environment SGA_TARGET 32 GB
parameters
PGA_AGGREGATE_TARGET 13 GB

DB_KEEP_CACHE_SIZE 6 GB
DB_RECYCLE_CACHE_SIZE 2 GB
INMEMORY_SIZE 800 MB

Testing Environment
Table 18 lists configuration details for Hitachi Virtual Storage Platform E990 with SAS backend with one node for testing.

TABLE 18. TEST ENVIRONMENT

Item Value

Operating System OL 7.8

Workload Type OLTP/OLAP

Database Size 2 TB
Number of vCPUs 16

18
19

TABLE 18. TEST ENVIRONMENT (CONTINUED)

Virtual Memory 128 GB

Host Cluster VM Network 2 × 25 Gbps NIC Bonding

1 × 10 Gbps NIC Bonding

Test Methodology
Oracle Orion and peakmarks were used for performance testing of this solution.

Orion
Oracle Orion is a tool for predicting the performance of an Oracle database without having to install Oracle or create a
database. Unlike other I/O calibration tools, Oracle Orion is expressly designed for simulating Oracle database I/O
workloads using the same I/O software stack as Oracle. Orion can also simulate the effect of striping performed by Oracle
Automatic Storage Management.

For more information about Orion, see “I/O Configuration and Design” in the Oracle Database Performance Tuning Guide.

peakmarks
The test results are demonstrated using peakmarks OLTP test cases.

peakmarks is benchmark software for Oracle platforms. It was used in our tests for the following:

 Performance verification (quality assurance)

 Evaluation of different infrastructure products, technologies, and solutions (price/performance comparison)
 Performance optimization (improvement in efficiency)
This provides transparency and comparability in price versus performance considerations for Oracle infrastructures. The
peakmarks 9.4 tool is used to validate this solution.

Failover and Failback tests using VMware vSphere vMotion

VMware vSphere vMotion allows the migration of virtual machines to different hosts and datastores simultaneously with no
downtime. The migration process is completely transparent to users. Processed virtual machines continue to run with no
service disruption.

Test scenarios performed using vSphere vMotion:

 Virtual machine migration with compute resources. In this case, only the virtual machines and its compute resources
are migrated to a specified VMware ESXi host.
 Virtual machine migration with compute and storage resources. In this case, the virtual machine with its compute and
storage resources are migrated to a specified ESXi host.
 ESXi host failover. In this case, due to ESXi host failover, the virtual machines connected to a failed ESXi host migrate to
any other active ESXi hosts using vSphere vMotion.

19
20

Test Results
Table 19 provides Orion test result details.

TABLE 19. ORION TEST RESULTS

Test Category Test Case Test Result

Throughput RT (ms)

Orion 8 KB Random Read 602,025 IOPS 1.3

8 KB Random Write 327,078 IOPS 0.8

1 MB Sequential Read 13.783 GB/s N/A

1 MB Sequential Write 4.292 GB/s N/A

20
21

Table 20 provides peakmarks test results.

TABLE 20. PEAKMARKS TEST RESULTS

Test Category Test Case Test Results

Throughput RT (ms)

peakmarks STO-READ 13.022 GB/s N/A

STO-Random 100% read 455,618 IOPS 1.8

SRV-SCAN 29,936 GB/s N/A

DA-LOW/ ROW 29.779 GB/s N/A

DA-LOW/STO 12.510 GB/s N/A

TP-LIGHT 1 RPT 0 %update 115,672 TPS 0.4

TP-LIGHT 1 RPT 20 %update 47,076 TPS 0.6

TP-LIGHT 1 RPT 30%update 44,817 TPS 0.6

TP-MEDIUM 25 RPT 0%update 13,535 TPS 0.2

TP-MEDIUM 25 RPT 20%update 1,238 TPS 0.4

TP-MEDIUM 25 RPT 30%update 1,437 TPS 0.3

Figure 4 shows swingbench data with all transactions that occurred while the virtual machine was migrated with compute
and storage resource information.

21
22

Figure 4

Conclusion
Hitachi Virtual Storage Platform E990 has been tested and validated as an ideal platform for virtualized workloads. It was
tested on an Oracle Real Application cluster database as well as with multiple environments running at the same time.

Product Descriptions
Hitachi Virtual Storage Platform E990
Hitachi Virtual Storage Platform E990 supercharges business application performance with all-NVMe storage. It uses
Hitachi Ops Center, so you can improve IT operations with the latest AI and ML capabilities. Advanced data reduction in
Virtual Storage Platform E990 enables you to run data reduction with even the most performance-hungry applications.

The all-NVMe architecture in Virtual Storage Platform E990 delivers consistent, low-microsecond latency to reduce latency
costs for critical applications. This predictable performance optimizes storage resources.

With Virtual Storage Platform E990 and the rest of Hitachi’s midrange storage family, you have agile and automated data
center technology. These systems allow you to cost-effectively meet your current digital expectations and give you the
ability to address future challenges, as your application data needs, and service levels evolve. With time-tested, proven
availability and scalability, Hitachi Vantara delivers infrastructure solutions that help you maximize your data center
advantage.

Hitachi Virtual Storage Platform E990, used in this reference architecture, supports Oracle Real Application Clusters.

22
23

Hitachi Storage Virtualization Operating System RF

Hitachi Storage Virtualization Operating System RF (SVOS RF) spans and integrates multiple platforms. It integrates
storage system software to provide system element management and advanced storage system functions. Used across
multiple platforms, Storage Virtualization Operating System includes storage virtualization, thin provisioning, storage
service level controls, dynamic provisioning, and performance instrumentation.

SVOS RF integrates with Hitachi’s Base and Advanced software packages to deliver superior availability and operational
efficiency. You gain active-active clustering, data-at-rest encryption, insights via machine learning, and policy-defined
data protection with local and remote replication.

Hitachi Advanced Server DS220

Hitachi Advanced Server DS220 is a high-performance two-socket rackmount server designed for optimal performance
and power efficiency. This allows owners to upgrade computing performance without overextending power consumption
and offers non-latency support to virtualization environments that require the maximum memory capacity. Hitachi Advanced
Server DS220 provides flexible I/O scalability for today’s diverse data center application requirements.

Hitachi Advanced Server DS120

Optimized for performance, high density, and power efficiency in a dual-processor server, Hitachi Advanced Server DS120
delivers a balance of compute and storage capacity. This rack mounted server has the flexibility to power a wide range of
solutions and applications.

The highly scalable memory supports up to 3 TB RAM using 24 slots of 2666 MHz DDR4 RDIMM. DS120 is powered by the
Intel Xeon scalable processor family for complex and demanding workloads. Flexible OCP and PCIe I/O expansion card
options are available.

The following applications were installed in individual virtual machines in this architecture and would be used in most
cases:

 VMware vCenter Server

 Hitachi Ops Center
Other management applications can be installed on additional virtual machines depending on customer needs and
requirements.

Hitachi Ops Center

Manage, optimize, orchestrate and protect your data with advanced IT analytics and automation using Hitachi Ops Center.
Achieve new insights, accelerate resource delivery, eliminate risks, and speed innovation to modernize your data center
operations.

23
24

Use the power of AI operations with the following:

 Analyzer. Improve IT operations with machine learning (ML) to drive resource service levels, utilization, and
automation at lower costs. Obtain operational visibility from virtual machines, servers, SAN switches to shared
storage resources to optimize an application’s full data path.
 Automator. Deliver resources up to 70% faster than manual processes. Free staff to focus on strategic initiatives.

 Protector. Meet tight service level requirements when protecting critical data and applications. Automatically support
secondary business functions with data copies staff need to do their jobs. Make better use of backup data for activities,
such as e-discovery and analysis. Simplify administration and replication management. Do it all with no disruption to
production application availability and performance.

Oracle Linux
Oracle Linux is a Linux distribution packaged and freely distributed by Oracle, available partially under the GNU General
Public License. It is compiled from Red Hat Enterprise Linux source code, replacing Red Hat branding with Oracle branding.

Oracle Database With the Real Application Clusters Option

Oracle Database has a multi-tenant architecture so you can consolidate many databases quickly and manage them as a
cloud service. Oracle Database also includes in-memory data processing capabilities for analytical performance.
Additional database innovations deliver efficiency, performance, security, and availability. Oracle Database comes in two
editions: Enterprise Edition and Standard Edition 2.

Oracle Automatic Storage Management

Oracle Automatic Storage Management (Oracle ASM) is a volume manager and a file system for Oracle database files. This
supports single-instance Oracle Database and Oracle Real Application Clusters configurations. Oracle ASM is the
recommended storage management solution that provides an alternative to conventional volume managers, file systems,
and raw devices.

VMware ESXi
VMware ESXi is the next-generation hypervisor, providing a new foundation for virtual infrastructure. This innovative
architecture operates independently from any general-purpose operating system, offering improved security, increased
reliability, and simplified management. This reference architecture uses VMware ESXi for the management servers only.

VMware vCenter Server Appliance

The VMware vCenter Server Appliance is a preconfigured Linux virtual machine, which is optimized for running VMware
vCenter Server and the associated services on Linux.

vCenter Server Appliance comes as an Open Virtualization Format (OVF) template. The appliance is imported to an ESXi
host and configured through the web-based interface. It comes pre-installed with all the components needed to run a
vCenter Server, including vCenter SSO (Single Sign-on), Inventory Service, vSphere Web Client and the vCenter Server
itself.

This reference architecture uses VMware vCenter Server Appliance for the management server cluster only.

24
25

Brocade Switches
Brocade and Hitachi Vantara partner to deliver storage networking and data center solutions. These solutions reduce
complexity and cost, as well as enable virtualization and cloud computing to increase business agility.

SAN switches are optional and direct connect is also possible under certain circumstances, but customers should check the
support matrix to ensure support prior to implementation.

The solution uses Brocade G620 Fibre Channel switches.

Cisco Switches
The Cisco Nexus switch product line provides a series of solutions that attempt to make it easier to connect and manage
disparate data center resources with software-defined networking (SDN). Leveraging the Cisco Unified Fabric, which
unifies storage, data, and networking (Ethernet/IP) services, the Nexus Switches create an open, programmable network
foundation built to support a virtualized data center environment.

The solution uses the following Cisco products:

 Nexus 93180YC-EX, 48-port 10/25 GbE switch

 Nexus 3048TP, 48-port 1GbE switch

25
For More Information
Hitachi Vantara Global Services offers experienced storage consultants, proven methodologies and a comprehensive
services portfolio to assist you in implementing Hitachi products and solutions in your environment. For more information,
see the Services website.

Demonstrations and other resources are available for many Hitachi products. To schedule a live demonstration, contact a
sales representative or partner. To view on-line informational resources, see the Resources website.

Hitachi Academy is your education destination to acquire valuable knowledge and skills on Hitachi products and solutions.
Our Hitachi Certified Professional program establishes your credibility and increases your value in the IT marketplace. For
more information, see the Hitachi Vantana Training and Certification website.

For more information about Hitachi products and services, contact your sales representative, partner, or visit the Hitachi
Vantara website.
1

Hitachi Vantara
Corporate Headquarters Regional Contact Information
2535 Augustine Drive USA: 1-800-446-0744
Santa Clara, CA 95054 USA Global: 1-858-547-4526
www.HitachiVantara.com | community.HitachiVantara.com HitachiVantara.com/contact

© Hitachi Vantara LLC, 2020. All rights reserved. HITACHI is a trademark or registered trademark of Hitachi, Ltd. VSP is a trademark or registered trademark of Hitachi Vantara
LLC. Microsoft, Windows Server, and Microsoft Office are trademarks or registered trademarks of Microsoft Corporation. All other trademarks, service marks, and company
names are properties of their respective owners.
Notice: This document is for informational purposes only, and does not set forth any warranty, expressed or implied, concerning any equipment or service offered or to be offered
by Hitachi Vantara LLC.
MK-SL-209-00, November 2020