OceanStor Dorado

Best Practices Oriented to Oracle

Database Deployment

Issue 01
Date 2020-12-10

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior
written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.

Notice
The purchased products, services and features are stipulated by the contract made between Huawei and
the customer. All or part of the products, services and features described in this document may not be
within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees
or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China

Website: https://e.huawei.com

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. i

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment Contents

Contents

1 Overview....................................................................................................................................1
1.1 Overview.................................................................................................................................................................................... 1
1.2 Conventions............................................................................................................................................................................... 2
1.3 Change History......................................................................................................................................................................... 3

2 OceanStor Dorado V6 Features and Oracle Databases.................................................. 4

2.1 High Performance................................................................................................................................................................... 4
2.2 High Availability...................................................................................................................................................................... 4
2.3 High Efficiency.......................................................................................................................................................................... 5

3 Best Practices About Storage................................................................................................7

3.1 Overview.................................................................................................................................................................................... 7
3.2 Storage Pools............................................................................................................................................................................ 8
3.2.1 Disk Selection........................................................................................................................................................................ 8
3.2.2 RAID Technology.................................................................................................................................................................. 9
3.2.3 Hot Spare Policies..............................................................................................................................................................10
3.3 LUNs.......................................................................................................................................................................................... 11
3.3.1 Application Types...............................................................................................................................................................11
3.4 Mapping Views...................................................................................................................................................................... 11
3.4.1 LUN Groups......................................................................................................................................................................... 11
3.4.2 Host Groups.........................................................................................................................................................................12
3.4.3 Port Groups.......................................................................................................................................................................... 12

4 Best Practice About Networking....................................................................................... 13

4.1 Overview.................................................................................................................................................................................. 13
4.2 Logical Topology................................................................................................................................................................... 14
4.3 Physical Topology..................................................................................................................................................................15
4.4 Number of Paths................................................................................................................................................................... 17

5 Best Practices About Hosts................................................................................................. 20

5.1 Overview.................................................................................................................................................................................. 20
5.2 Multipathing........................................................................................................................................................................... 22
5.3 HBA............................................................................................................................................................................................ 23
5.3.1 HBA Parameters................................................................................................................................................................. 23
5.3.2 Maximum Number of Concurrent Connections Supported by an HBA..........................................................24
5.4 Udev Configuration.............................................................................................................................................................. 25

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. ii

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment Contents

5.4.1 Key Technologies............................................................................................................................................................... 25

5.4.2 Parameter Description..................................................................................................................................................... 26
5.4.3 Configuration Example.................................................................................................................................................... 27
5.5 I/O Scheduler.......................................................................................................................................................................... 28
5.5.1 Principles.............................................................................................................................................................................. 28
5.5.2 Configuring the I/O Scheduler...................................................................................................................................... 30
5.6 Kernel Parameters................................................................................................................................................................ 30
5.6.1 Virtual Memory.................................................................................................................................................................. 31
5.6.2 Shared Memory.................................................................................................................................................................. 32
5.6.3 SEM........................................................................................................................................................................................ 34
5.6.4 Ephemeral Network Ports.............................................................................................................................................. 35
5.6.5 I/O Requests........................................................................................................................................................................ 35
5.6.6 File Handles......................................................................................................................................................................... 36
5.7 HugePages.............................................................................................................................................................................. 36
5.7.1 Principle................................................................................................................................................................................ 36
5.7.2 Configuring HugePages...................................................................................................................................................38
5.8 Optimization of Other Parameters................................................................................................................................. 39

6 Best Practices About Oracle Databases........................................................................... 41

6.1 Overview.................................................................................................................................................................................. 41
6.2 Data Block............................................................................................................................................................................... 43
6.2.1 DB_BLOCK_SIZE..................................................................................................................................................................43
6.2.2 Nonstandard Block Sizes................................................................................................................................................. 44
6.2.3 Block Size of Redo Log Files.......................................................................................................................................... 45
6.2.4 Block Size of Control Files.............................................................................................................................................. 46
6.3 ASM........................................................................................................................................................................................... 46
6.3.1 ASM Advantages................................................................................................................................................................ 47
6.3.2 Disk Group........................................................................................................................................................................... 47
6.3.3 AU Size.................................................................................................................................................................................. 49
6.3.4 ASM Striping....................................................................................................................................................................... 49
6.3.5 ASM LUN.............................................................................................................................................................................. 51
6.3.6 Multiplex............................................................................................................................................................................... 53
6.3.7 Dynamic Rebalancing...................................................................................................................................................... 53
6.3.8 Capacity Expansion of the ASM Disk Group............................................................................................................ 54
6.3.9 Oracle ASM Storage Restriction................................................................................................................................... 56
6.3.10 Oracle ASM Processes................................................................................................................................................... 57
6.3.11 Oracle ASM Heartbeat Timeout................................................................................................................................ 58

7 Summary................................................................................................................................. 59
8 References............................................................................................................................... 67
9 Glossary................................................................................................................................... 68
10 Appendixes............................................................................................................................69

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. iii

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment Contents

10.1 Appendix A — Udev Configuration.............................................................................................................................. 69

10.2 Appendix B — I/O Scheduler Configuration............................................................................................................. 70
10.3 Appendix C — HugePages Configuration.................................................................................................................. 73

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. iv

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 1 Overview

1 Overview

1.1 Overview
1.2 Conventions
1.3 Change History

1.1 Overview
Oracle databases are enterprises' core applications. Typically, Oracle databases are
used for online transaction processing (OLTP), online analytical processing (OLAP),
and data warehouse (DSS). However, many enterprises' database systems run
slowly for the following possible causes: The configuration and deployment mode
of the upper-layer application, database, host, network parameter, and storage
device are not optimal, or the plan and design are not proper. To meet the unique
requirements of Oracle workloads, Huawei provides the best practices of the
OceanStor Dorado V6 storage solution.

This document describes the best practices of a storage solution where the
OceanStor Dorado V6 is used with Oracle databases, with the focus on how to use
the OceanStor Dorado V6 to efficiently deploy the Oracle database service. This
document aims to help you obtain higher service deployment efficiency and
service operation quality as well as ensure the performance and availability of
storage systems and Oracle databases. This document also aims to help database
administrators, application architects, system administrators, network
administrators, storage administrators, and system architects easily and quickly
deploy stable, reliable, high-performance Oracle database systems.

NOTE
The recommended configurations are applicable to general-purpose scenarios. In specific
scenarios, you need to verify whether the configurations are appropriate.

This document consists of the following best practices:

● 2-OceanStor Dorado V6 Features and Oracle Databases

● 3-Best Practices About Storage
● 4-Best Practice About Networking

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 1

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 1 Overview

● 5-Best Practices About Hosts

● 6-Best Practices About Oracle Databases
NOTE

If you want to view the overall configurations, see 7-Summary.

1.2 Conventions
Symbol Conventions
Symbol Description

Indicates a potentially hazardous situation which, if not

avoided, may result in minor or moderate injury.

Calls attention to important information, best practices,

and tips.
Note is used to address information not related to personal
injury, equipment damage, or environment deterioration.

General Conventions
Convention Description

Times New Normal paragraphs are in Times New Roman.

Roman

Boldface Names of files, directories, folders, and users are in

boldface. For example, log in as the root user.

Italic Book titles are in italics.

Courier New Examples of information displayed on the screen are in

Courier New. In addition, user input contained in terminal
display is in boldface.

Boldface File paths are in boldface, for example, C:\Program Files

\Huawei.

[......] The subsequent content is omitted.

Command Conventions
Format Description

Boldface The keywords of a command line are in boldface.

Italic Command arguments are in italics.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 2

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 1 Overview

Applicable Versions
The best practices provided in this document are applicable to the products
described in the following table.

Product Type Product Model/Version

Database Oracle 11g, 12c, 18c, and 19c

Operating system RHEL 6.x / 7.x

Storage device OceanStor Dorado V6 series storage products

1.3 Change History

Updates between document issues are cumulative. Therefore, the latest document
issue contains all updates made in previous issues.

Issue Date Description

1.0 2020-11-15 This issue is the first

official release.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 3

OceanStor Dorado
Best Practices Oriented to Oracle Database 2 OceanStor Dorado V6 Features and Oracle
Deployment Databases

2 OceanStor Dorado V6 Features and

Oracle Databases

2.1 High Performance

2.2 High Availability
2.3 High Efficiency

2.1 High Performance

The OceanStor Dorado V6 series uses the FlashLink® technology designed for flash
storage to provide high IOPS concurrency capability and ensure low latency.
FlashLink uses a series of algorithm optimization technologies for flash storage
media to associate the onboard CPU of controllers with the dedicated onboard
CPU of SSDs, ensuring collaboration of SSD algorithms between different CPUs
and achieving high system performance and reliability. This ensures low latency of
Oracle database applications under high IOPS.

NOTE

For details about FlashLink of the OceanStor Dorado V6 series, click here.

2.2 High Availability

HyperSnap
Challenges such as quick backup and restoration and offline backup exist in the
use of Oracle databases. HyperSnap can address these challenges and reduce the
costs in IT system deployment, maintenance, and upgrade. HyperSnap can
generate data copies at different time points. By generating a snapshot of a LUN
at a specific point in time, you can save all data on the LUN at that time. This
snapshot serves as a backup copy. If a fault occurs, the snapshot can be used to
restore the source data to the state at the time when the snapshot was generated.
In addition, the snapshot can be used to deploy a new application system or
perform offline backup and remote replication. In this way, the preceding
challenges can be addressed.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 4

OceanStor Dorado
Best Practices Oriented to Oracle Database 2 OceanStor Dorado V6 Features and Oracle
Deployment Databases

HyperCDP
Generally, the data volume of a database in an Oracle data center can reach 100
TB. The traditional backup and restoration mode can meet the basic backup
requirements, but its long duration affects the progress of deploying development
and test environments, delaying product delivery. In addition, it is difficult to use
production data at any point in time to deploy development and testing
environments as required.

HyperCDP can address the challenges brought by the traditional backup mode. As
a high-density snapshot technology, HyperCDP can generate fully available copies
of specified data sets. A data copy at a certain point in time is generated through
redirect-on-write (ROW) and the mapping table, instead of physical data copying.
Data centers and enterprises can use such data copies for data protection and
recovery. In addition, the data copies can also be used to quickly deploy
development and test environments, with no more than 5% of the impact on
production service performance. HyperCDP generates data copies within seconds.
This shortens data backup and recovery durations, promotes development and test
environment delivery, and ensures the product delivery progress compared with
Oracle DataGuard.

HyperMetro
DR solutions are crucial to protecting customers' production systems against
natural disasters or misoperations and ensuring service continuity, recoverability,
and high availability. Conventional Oracle databases generally use DataGuard and
GoldenGate for DR. However, the recovery process is complex and may cause
exceptions. On the other hand, the traditional active-passive DR solution of
storage systems has disadvantages such as low resource utilization and high total
cost of ownership (TCO). In the event of a fault, customers must manually switch
over services and data recovery takes a long time, resulting in long service
interruption time. To address these challenges, the active-active solution is
proposed. Huawei's active-active data center solution allows two data centers to
carry service loads concurrently, improving service performance and resource
utilization. Both data centers back up each other. If either one fails, the other one
automatically takes over services to guarantee service continuity.

When the active-active data center solution is deployed for an Oracle cluster, the
hosts in the cluster run on two active-active storage systems. The hosts and
storage systems are deployed in different data centers for remote DR, providing
the optimal RPO and RTO for database cluster applications.

2.3 High Efficiency

SmartDedupe and SmartCompression
SmartDedupe and SmartCompression release space through data deduplication
and compression. In Oracle database scenarios, SmartDedupe and
SmartCompression can be used independently or together to improve storage
efficiency and reduce investment and O&M costs. In addition, the service life of
SSDs is prolonged by reducing the number of data write operations to SSDs and
the amount of written data. Data compression is most applicable to Oracle

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 5

OceanStor Dorado
Best Practices Oriented to Oracle Database 2 OceanStor Dorado V6 Features and Oracle
Deployment Databases

databases, and the little impact on services brings more than 65% of storage
space saving.
For details about SmartDedupe and SmartCompression of the OceanStor Dorado
V6 series, click here.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 6

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

3 Best Practices About Storage

3.1 Overview
3.2 Storage Pools
3.3 LUNs
3.4 Mapping Views

3.1 Overview
Parameter/ Recommended Value/Method Default Value/ Location
Operation Method

Disk Manually select disks of the None 3.2.1 Disk

selection same controller enclosure. Selection

Disk Select disks that have the same None

capacity capacity.

Number of Configure at most 100 disks for None

disks each tier in a storage pool.

RAID level RAID-TP RAID 6 3.2.2 RAID

Technolog
y

Hot spare High or above Low 3.2.3 Hot

policy Spare
Policies

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 7

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

Parameter/ Recommended Value/Method Default Value/ Location

Operation Method

LUN For a single database, select Default 3.3.1

application Oracle_OLAP, Oracle_OLTP, or application type Applicatio
type Oracle_OLAP&OLTP based on (block size = 8 n Types
actual services. KB)
In multi-database scenarios,
select an application type based
on actual services of each
database.

LUN group Plan multiple disk groups of a None 3.4.1 LUN

Real Application Cluster (RAC) Groups
as one LUN group.

Host group Add hosts in a cluster to the None 3.4.2 Host

same host group. Groups

Port group Create a port group to enable None 3.4.3 Port

LUNs in the LUN group and Groups
hosts in the host group to
communicate with each other
using ports in the port group.

3.2 Storage Pools

Create storage pools to provide storage space for Oracle databases. To properly
use storage resources and functions, you need to plan storage tiers, hot spare
policies, and RAID levels for storage pools based on site requirements. The
following describes the best practices of configuring storage pools in the Oracle
database scenario.

3.2.1 Disk Selection

Method
The best practice of creating a disk domain is to manually select disks that come
from the same controller enclosure. This reduces the probability of disk failures
and improves the read and write performance of disks.

Disk Capacity
The best practice about a storage tier is to use the same disk type and capacity. If
member disks vary in capacity, larger-capacity disks cannot be fully used, causing
a waste of capacity. If member disks vary in rotational speed, the performance
may deteriorate.
The disk capacity defined by disk manufacturers is different from that calculated
by operating systems. As a result, the nominal capacity of a disk is different from
that displayed in the operating system.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 8

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

NOTE

● Disk capacity defined by disk manufactures: 1 GB = 1,000 MB, 1 MB = 1,000 KB, 1 KB =

1,000 bytes
● Disk capacity calculated by operating systems: 1 GB = 1,024 MB, 1 MB = 1,024 KB, 1 KB
= 1,024 bytes

Number of Disks
The best practice is to configure no more than 100 disks for each tier in a storage
pool. Assume that the number of disks at a tier is D (D/100 is rounded to an
integer N and the remainder is M). In this case, refer to the following rules:
● If D ≤ 100, configure all disks of this tier in one disk domain.
● If D > 100, create N+1 disk domains and evenly distribute all disks to these
disk domains. That is, the number of disks in each disk domain is D/(N+1). In
addition, it is recommended that disk enclosures be fully populated with disks.
● For the SmartTier feature, it is recommended that a maximum of 100 disks be
configured for each tier of a disk domain. Follow the preceding rules to
configure the specific number of disks at each tier.
Example 1: The total number (D) of SSDs of a storage system is 328. (328/100
rounded up to 3, which is the value of N. The remainder is 28, which is the value
of M). It is recommended that four (N+1) disk domains be configured and the
number of disks in each disk domain be 82 (328/4).
Example 2: If the number (D) of SSDs in a storage system is 223 (223/100
rounded up to 2, which is the value of N. The remainder is 23, which is the value
of M). It is recommended that three (N+1) disk domains be configured and the
number of disks in each disk domain is 74.3 (223/3). Therefore, 74 disks are
configured for each of two disk domains and 75 disks are configured for the
remaining disk domain.

NOTE

A larger number of disks in a disk domain results in a higher reconstruction speed. If the
number of disks in a disk domain reaches 100, the disk domain reliability reaches
99.9999%. However, if the number of disks is too large, the RAID 2.0+ computing overhead
increases and uneven disk use occurs, affecting storage performance to some extent.

3.2.2 RAID Technology

OceanStor Dorado V6 storage systems use dynamic RAID for redundancy and
provide different levels of protection based on the number of parity bits in a RAID
group. The following table describes RAID 5, RAID 6, and RAID-TP when hot spare
space is not considered.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 9

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

RAID Number Redundancy and Data Recovery Maximum

Level of Parity Capability Number of
Bits Tolerable Faulty
Disks

RAID 5 1 Relatively high. Parity data is 1

distributed on different chunks. In
each CKG, the parity data occupies
one chunk. The failure of only one
chunk is tolerated. If two or more
chunks fail, the RAID group fails.

RAID 6 2 High. Parity data is distributed on 2

(default) different chunks. In each CKG, the
parity data occupies two chunks. A
failure of two chunks is tolerated. If
three or more chunks fail, the RAID
group fails.

RAID-TP 3 High. Parity data is distributed on 3

different chunks. In each chunk
group, the parity data occupies the
space of three chunks. RAID-TP is
able to tolerate simultaneous
failures on three chunks. If four or
more chunks fail, RAID-TP
protection can no longer be
provided.

Different RAID levels use different RAID algorithms, but the read/write
performance is similar. The performance of RAID 5, RAID 6, or RAID-TP decreases
slightly as the redundancy increases. The performance of different I/O models
(random/sequential read/write) is different:
● The performance of random read is equivalent to that of sequential read.
● The performance of sequential write is higher than that of random write.
You are advised to use RAID-TP for important Oracle database services.

3.2.3 Hot Spare Policies

When you create a storage pool, the recommended hot spare policy is Low (1
disk). The value can be None, Low (1 disk), High (2 disks), Custom (3 disks),
Custom (4 disks), Custom (5 disks), Custom (6 disks), Custom (seven disks),
or Custom (8 disks).
You are advised to select High (2 disks) or a higher value for important Oracle
database services.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 10

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

NOTE

● RAID 2.0+ allows all member disks in a storage pool to provide hot spare capacity. For
ease of understanding, the hot spare capacity is expressed in the number of hot spare
disks on DeviceManager.
● If the total number of disks in a storage pool is less than 13, the number of hot spare
disks must meet the following requirement: Total number of disks – The number of hot
spare disks ≥ 5.

3.3 LUNs
Block storage uses LUNs as basic units for services. All LUNs created on OceanStor
Dorado V6 storage systems are thin LUNs. Before using a storage system of Oracle
database services, select a proper application type for LUNs based on actual
storage needs.

3.3.1 Application Types

The following preset application types are provided for typical applications:
Default, Oracle_OLAP, Oracle_OLTP, Oracle_OLAP&OLTP, SQL_Server_OLAP,
SQL_Server_OLTP, SQL_Server_OLAP&OLTP, SAP_HANA, Vmware_VDI, Hyper-
V_VDI, and FusionAccess_VDI.
For a single database, select Oracle_OLAP, Oracle_OLTP, or Oracle_OLAP&OLTP
based on actual services.
In multi-database scenarios, select an application type based on actual services of
each database.

NOTE

● Each preset application type has a default application request size: 32 KB for
Oracle_OLAP and SQL_Server_OLAP, and 8 KB for the remaining types.
● The application type of a LUN cannot be changed after being set.
● For details about the number and size of LUNs, see 6.3 ASM.

3.4 Mapping Views

Before creating a mapping view, properly plan LUN groups and host groups to
facilitate management and maintenance. The following describes the best practice
of configuring mapping views in the Oracle database scenario.

3.4.1 LUN Groups

The rules for creating LUN groups oriented to the Oracle database service are as
follows:
● A LUN group is an object designed to facilitate LUN resource management.
Typically, LUNs that serve the same service should be added to the same LUN
group.
● It is recommended that a group of database services, such as multiple disk
groups in a RAC cluster, be planned as one LUN group to form a management
object.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 11

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 3 Best Practices About Storage

NOTE

● For details about the number and size of LUNs, see 6.3 ASM.

3.4.2 Host Groups

A host group is a set of hosts that share storage resources. Each host contains
multiple initiators (host ports). You are advised to create a host for each server
and add all initiators of the server to the host. Add hosts in a cluster to the same
host group. For example, if a RAC cluster is deployed among 2+1 hosts across data
centers, the host group must contain the three RAC hosts that reside in the two
data centers.

3.4.3 Port Groups

A port group is a logical combination of multiple physical ports. A storage system
uses specified ports to establish a mapping relationship between storage resources
(LUNs) and servers. You can create a port group and add it to a mapping view, so
that LUNs in the LUN group and hosts in the host group can communicate with
each other using ports in the port group. If no port group is added to the mapping
view, available ports that are randomly allocated will be used.
The best practice is to create a port group to enable LUNs in the LUN group and
hosts in the host group to communicate with each other using ports in the port
group.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 12

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

4 Best Practice About Networking

To ensure the storage system's security and the connectivity between the storage
system and the database server, you need to plan the networking and security
authentication. The following describes the best practice of general-purpose
networking configuration in the Oracle database scenario.
4.1 Overview
4.2 Logical Topology
4.3 Physical Topology
4.4 Number of Paths

4.1 Overview
Parameter Recommended Value/Method Default Location
/ Value/
Operation Method

Logical Use the Fibre Channel network to None 4.2

topology ensure that there is no single point of Logical
failure on the entire network. Topolog
y

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 13

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

Parameter Recommended Value/Method Default Location

/ Value/
Operation Method

Physical Cross networking: None 4.3

topology Connect at least two Fibre Channel Physical
ports from different SmartIO interface Topolog
modules of each controller to two Fibre y
Channel switches.
Configure at least two dual-port host
bus adapters (HBAs) for the host and
connect the HBAs to different switches.
16 Gbit/s or higher-rate Fibre Channel
interface modules are recommended.
Point-to-point zone planning:
Each zone can contain only one initiator
and one target.

Number of At least 8 paths are required for a dual- None 4.4

paths controller storage system and 16 paths Number
are required for a four-controller storage of Paths
system to ensure full redundancy.

4.2 Logical Topology

In the Oracle database scenario, the best practice is to use a Fibre Channel
networking model. Ensure that there is no single point of failure on the entire
network and design a network based on site requirements. The following figure
shows the logical topology.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 14

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

Figure 4-1 Logical topology of the Fibre Channel network

NOTE

● Oracle RAC is deployed among multiple servers.

● The OceanStor Dorado V6 storage system provides the storage space required by the
database.
● The 16 Gbit/s FC SAN networking model is used between the servers and the storage
system.
● The Oracle cluster uses a 10GE private network.
● The Oracle database provides external services using a 10GE service network.
● A GE network is used for out-of-band management of all devices.

This document describes only one networking model. For more networking
models, click here.

4.3 Physical Topology

The OceanStor Dorado V6 storage system supports two controllers and multiple
controllers (more than two controllers). The two-controller configuration and the
multi-controller configuration vary in networking model. To ensure reliability and
load balancing, you are advised to observe the following rules:
● Cross networking
The best practice is to connect at least two Fibre Channel ports on each
controller to two Fibre Channel switches, and ensure that the connected ports
on a controller are from different SmartIO modules. Configure at least two

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 15

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

dual-port HBAs for the host and connect the HBAs to different switches. 16
Gbit/s or higher-rate Fibre Channel interface modules are recommended.
● Point-to-point zone planning
In a point-to-point zoning plan, each zone contains only one initiator and one
target. Point-to-point zoning is the most reliable and secure zoning rule.
Configuring zones effectively ensures at the network layer that access
permissions are allocated to nodes on a network strictly according to the
service plan. In other zoning rules, devices in zones interfere with each other.
However, the practice of making a zone contain only one initiator and one
target minimizes the possibility of this problem.
NOTE
If zones are incorrectly configured, link contention exists, leading to a decrease in host
performance.
To facilitate demonstration, zones are planned based on ports. In an actual project, you are
advised to plan zones based on WWPNs. If port-based zones are used, subsequent port
changes will interrupt services and require users to reconfigure zones.

Two-Controller Storage System

The following figure (where a two-controller storage system is used) shows the
connections between a host and a storage system on a Fibre Channel network,
where Huawei OceanStor Dorado 5000 V6 is used as an example. The table
(where a two-controller storage system is used) shows the zoning plan.

NOTE

The physical networking planned in this document provides theoretical support for zone
planning and multipathing quantity calculation. In an actual project, plan physical
networking based on site requirements.

Figure 4-2 Fibre Channel physical networking where a two-controller storage system is used

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 16

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

Multi-Controller Storage System

The following figure (where a multi-controller storage system is used) shows the
connections between a host and a storage system on a Fibre Channel network,
where Huawei OceanStor Dorado 18000 V6 is used as an example. The table
(where a four-controller storage system is used) shows the zoning plan.

NOTE

The physical networking planned in this document provides theoretical support for zone
planning and multipathing quantity calculation. In an actual project, plan physical
networking based on site requirements.

Figure 4-3 Fibre Channel physical networking where a multi-controller storage system is used

4.4 Number of Paths

Calculate the number of host links based on the physical connections and zone
plan in section 4.3 Physical Topology. If networks recommended in this document
are used, at least 8 paths are required for a dual-controller storage system and at
least 16 paths are required for a four-controller storage system to ensure full
redundancy. The following describes how to calculate the number of active paths
to a storage system:
The number of active paths to a storage system varies with the zone
configuration. For example, a server is equipped with two PCIe HBAs, each
providing two ports. A storage system has four controllers, each providing two
ports. Based on the point-to-point zoning rule, the number of active paths to a
storage system can be calculated using either of the following methods:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 17

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

Diagrammatic Illustration
As shown in the following figure, each PCIe HBA on the Oracle server connects to
a port on each switch (four ports involved in total), and each controller connects
to a port on each switch (eight ports involved in total). Therefore, there are four
paths from a port on the server to a storage controller. The server has four ports
connected. Therefore, there are 16 paths from the server to the storage system. As
shown in Figure Diagrammatic illustration of multipathing, there are 2 x 4 + 2 x 4
= 16 storage paths.

Figure 4-4 Diagrammatic illustration of multipathing

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 18

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 4 Best Practice About Networking

Combination Enumeration
One controller port corresponds to one host port. Therefore, you can list all paths,
starting from the host side, as shown in the following table. There are 16 storage
paths in total.

Table 4-1 Path enumeration

Host Port Controller Port

PCIE01_Port01 A_Controller_P0

PCIE01_Port01 B_Controller_P0

PCIE01_Port01 C_Controller_P0

PCIE01_Port01 D_Controller_P0

PCIE01_Port02 A_Controller_P1

PCIE01_Port02 B_Controller_P1

PCIE01_Port02 C_Controller_P1

PCIE01_Port02 D_Controller_P1

PCIE02_Port01 A_Controller_P0

PCIE02_Port01 B_Controller_P0

PCIE02_Port01 C_Controller_P0

PCIE02_Port01 D_Controller_P0

PCIE02_Port02 A_Controller_P1

PCIE02_Port02 B_Controller_P1

PCIE02_Port02 C_Controller_P1

PCIE02_Port02 D_Controller_P1

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 19

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

5 Best Practices About Hosts

5.1 Overview
5.2 Multipathing
5.3 HBA
5.4 Udev Configuration
5.5 I/O Scheduler
5.6 Kernel Parameters
5.7 HugePages
5.8 Optimization of Other Parameters

5.1 Overview
Parameter/ Recommended Value/ Default Value/ Location
Operation Method Method

Multipathing Huawei UltraPath MultipathI/O 5.2

tool Multipathing

HBA port retry If QLogic HBAs are used: If QLogic HBAs are 5.3.1 HBA
interval Port Down Retry Count: used: Parameters
10 Port Down Retry
Link Down Timeout: 10 Count: 30
If Emulex HBAs are used: Link Down
Timeout: 30
lpfc_devloss_tmo: 10
If Emulex HBAs are
used:
lpfc_devloss_tmo:
10

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 20

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter/ Recommended Value/ Default Value/ Location

Operation Method Method

Maximum Small-sized database: 32 32 5.3.2

number of Large database: 128 Maximum
concurrent Number of
connections Concurrent
supported by Connections
an HBA Supported by
an HBA

ASM disk Configure Oracle ASM None 5.4 Udev

group disks using udev rules. Configuratio
configuration n

I/O scheduler Deadline Deadline 5.5 I/O

Scheduler

SWAPPINESS 1 to 20 60 5.6.1 Virtual

Memory
DIRTY_RATIO 40 to 80 20

DIRTY_BACKG 3 10
ROUND_RATI
O

DIRTY_EXPIRE_ 500 3000

CENTISECS

DIRTY_WRITEB 100 500

ACK_CENTISEC
S

SHMMAX Red Hat Enterprise Linux Red Hat Enterprise 5.6.2 Shared
7.0 x86_64: Linux 7.0 x86_64: Memory
18446744073692774399 4294967295 bytes
bytes Red Hat Enterprise
Red Hat Enterprise Linux Linux 7.1 and later:
7.1 and later: Default 184467440736927
value 74399 bytes

SHMALL Red Hat Enterprise Linux Red Hat Enterprise

7.0 x86_64: Linux 7.0 x86_64:
18446744073692774399 268435456 pages
pages Red Hat Enterprise
Red Hat Enterprise Linux Linux 7.1 and later:
7.1 and later: Default 184467440736927
value 74399 pages

SHMMNI 4095 4095

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 21

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter/ Recommended Value/ Default Value/ Location

Operation Method Method

SEMMSL Set this parameter based 250 5.6.3 SEM

on the objective of
minimizing the use of
semaphores. For details
about the calculation
method, see 5.6.3 SEM.

SEMMNS The value must be 32000

higher than the result of
SEMMNI x SEMMSL.

SEMOPM Set SEMOPM to the 100

value of SEMMSL. The
minimum value is 100.

SEMMNI Formula: SEMMNI = 128

SEMMNS/SEMMSL. The
result is rounded up to
the nearest power of 2.

fs.aio-max-nr The minimum value is 1048576 5.6.5 I/O

1048576. Requests

fs.file-max It is recommended that 1024 5.6.6 File

at least 512 x processes Handles
be allocated to each
Oracle database
instance. In addition, the
number of allocated
processes must be
reserved for the
operating system.

HugePages On None 5.7.2

Configuring
Transparent Off None HugePages
HugePages

Automatic Off None

Memory
Management
(AMM)

5.2 Multipathing
Windows hosts generally have built-in multipathing software Multi-Path I/O
(MPIO). MPIO can only implement basic failover and load balancing, but cannot
meet the application requirements of systems with higher reliability. Huawei
UltraPath provides basic failover and load balancing functions as well as the
following advanced functions: routine path test, protection against transient path

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 22

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

interruption, path isolation, path alarm push, and path performance monitoring.
UltraPath is more compatible with Huawei storage systems. In addition, UltraPath
for OceanStor Dorado provides the partition alignment function. When I/Os are
delivered, UltraPath detects the partition alignment attributes on the storage
system and calculates the optimal path for forwarding I/Os based on the LBA
information carried by the I/Os. This avoids extra resource overheads caused by
frequent non-aligned I/O forwarding on the storage system, improving the overall
storage performance. UltraPath can meet the requirements of the entire IT system
for reliability, performance, maintainability, and storage adaptation. You are
advised to use Huawei UltraPath for your Huawei storage systems.
If Huawei UltraPath is used, no extra configuration on the storage system or host
is required. By default, UltraPath provides the optimal configuration for Huawei
storage to achieve the best performance and reliability. The best practice is to use
Huawei UltraPath.

NOTE

The following best practices are all based on Huawei UltraPath. If multipathing
software provided by the system is used, refer to the OceanStor Dorado V6 Host
Connectivity Guide for Windows.

5.3 HBA
Before mounting storage resources to a host, check whether the HBA on the host
can be identified and work properly. The commands for querying HBAs vary
according to vendors. For details, see the help documents of the corresponding
vendors. This document describes only the best practice of general HBA
configuration in Oracle database scenarios.

5.3.1 HBA Parameters

QLogic
The following table lists the key parameters that need to be optimized of some
QLogic HBAs. For details about other parameters, see the documentation of the
corresponding QLogic HBA version.

Table 5-1 QLogic HBA parameters of the HBA management tool

Parameter Description

Port Down Specifies the number of times the software retries a

Retry Count command to a port returning port down status.
● Value range: 0 to 255
● Default value: 30
● Recommended value: 10

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 23

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter Description

Link Down Specifies the time when the driver waits for the link to come
Timeout up after link down before returning I/Os. Valid values for the
(seconds) Link Down Timeout setting are ranging from 0 to 240. A value
of 0 indicates that no timeout is used.
● Value range: 0 to 240
● Default value: 30
● Recommended value: 10

Emulex
The following table lists the key parameters that need to be optimized of some
Emulex HBAs. For details about other parameters, see the documentation of the
corresponding Emulex HBA version.

Table 5-2 Emulex HBA parameters

Parameter Description

lpfc_devloss_tmo Specifies the time when the driver waits for the link to
come up after link down before returning I/Os.
● Value range: 1 to 255
● Default value: 10
● Recommended value: 10

NOTE

You are advised to use Huawei UltraPath to configure HBA parameters. UltraPath can
automatically modify the Qlogic-qla2xxx.conf or elx-lpfc.conf configuration file on the
system for optimal HBA parameter configuration. If the multipathing tool provided by the
system is used, refer to the tool instruction provided by the corresponding vendor.

5.3.2 Maximum Number of Concurrent Connections Supported

by an HBA
For Linux operating systems, the HBA queue parameters vary depending on the
HBA type and driver. For details, see the specifications provided by the HBA
vendor. For example, the QLogic dual-port 8 Gbit/s Fibre Channel HBA allows the
maximum queue depth of each LUN to be 32. If the latency on the host side
differs greatly from that on the storage side, run the iostat command to check
whether the concurrency bottleneck is reached, as shown in the following figure.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 24

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Figure 5-1 Running the iostat command to check the disk running status

In the command output, avgqu-sz indicates the average queue depth of the block
devices where LUNs are created. If the value is greater than 10 for a long time, the
problem is probably caused by concurrency limits. As a result, I/O operations are
piled up at the block device layer of the host instead of being delivered to the
storage side. In this case, you can modify the HBA concurrency.
Run the following command to check the HBA driver concurrency:
#cat /sys/module/qla2xxx/parameters/ql2xmaxqdepth
128

Alternatively, you can run the following command:

#lsscsi -l

You are advised to set the same queue depth for hosts in the Oracle cluster. For
small- and medium-sized databases, set the queue depth to 32. For large-sized
databases, you are advised to set the queue depth to 128.

5.4 Udev Configuration

Before deploying Oracle databases or configuring Oracle ASM, you must configure
the name and permission of a storage device to ensure that the storage path and
permission do not change after the system restarts. The following describes the
best practice of the persistent configuration in Oracle database scenarios.

5.4.1 Key Technologies

The following technologies are used to configure device path persistence on
Oracle ASM:
● Udev rules
● Oracle ASMLib
● Oracle Filter Driver
The following table describes udev rules, Oracle ASMLib, and Oracle ASM Filter
Driver (ASMFD).

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 25

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Table 5-3 Key technologies

Technology Pros Cons

Udev rules No proprietary user space Cannot stop an accidental I/O

utilities; native to OS; write done by a program or user
standard device manager on error.
Linux distributions; same
performance as Oracle
ASMLib and ASMFD.

Oracle No pros as it is slowly being Requires additional kernel

ASMLib deprecated in favor of module on OS; disks managed by
ASMFD. Oracle instead of native OS;
errors loading Oracle ASMLib
may cause losing access of Oracle
ASM disks until the module can
be reloaded; no performance
benefit over native udev rules.

Oracle ASM Filters out all non-Oracle Requires additional kernel

Filter Driver I/Os that may cause module on OS; disks managed by
accidental overwrites to Oracle instead of native OS;
managed disks. errors loading ASMFD may cause
losing access of Oracle ASM disks
until the module can be reloaded;
no performance benefit over
native udev rules.

The best practice in this document uses udev rules to configure Oracle ASM
disks. Udev has the same performance as ASMLib and Filter Driver, and is an
RHEL native tool that does not occupy extra space.

NOTE

For details about Oracle ASMLib, click here.

For details about the Oracle ASM Filter Driver, click here.

5.4.2 Parameter Description

Udev Best Practice Remarks

Persistence technology Udev rules No dedicated user space

is required. Without the
need for installing the
Linux device manager,
udev provides the same
performance as ASMLib
and ASMFD.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 26

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Udev Best Practice Remarks

Match keys ● KERNEL: Kernel device name, for example, dm-2.

● BUS: bus name of a device in devpath, for
example, scsi.
● SUBSYSTEM: subsystem name of a device. For
example, the subsystem of sda is block.
● PROGRAM: invokes an external command, for
example: /usr/lib/udev/scsi_id -g -u /dev/dm-2
or /sbin/scsi_id --whitelisted --device=/dev/
$name.
● RESULT: returned result of the external command
PROGRAM. For example,
36207969100f4a3810efc24f70000001a (WWID).

Assignment keys ● NAME: device file name generated in /dev. The

NAME assignment to a device takes effect only
for the first time, and the subsequent NAME
assignment due to the matched rule is ignored. If
there is no rule to assign NAME to a device, udev
will use the kernel device name to generate the
device file. The value of NAME can be planned
according to the naming rule.
● SYMLINK: generates symbolic links for device files
in /dev/. Udev generates only one device file for
a device. Therefore, you are advised to use
symbolic links to prevent the files generated by
default udev rules from being overwritten. The
value of SYMLINK can be planned according to
the naming rule.
● OWNER, GROUP, MODE: sets rights for the
device.

SYMLINK or NAME You are advised to use The device files are
SYMLINK to create a stored in different
directory in /dev for directories from the
storing device files, for original files. This
example, raw. prevents the ASM from
reporting the ORA-15020
error when the original
device and connected
device are discovered at
the same time.

5.4.3 Configuration Example

For details about how to configure udev, see 10.1 Appendix A — Udev
Configuration.
When configuring udev, you are advised to create a directory (for example, raw)
in the /dev directory to store the SYMLINK device file. This prevents the SYMLINK

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 27

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

device from being in the same directory as the original device during the creation
of a disk group. The recommended udev configuration is described in the
following sections.

NOTE
If Disk Discovery Path is set incorrectly when you create an ASM disk group, the ORA-15020
error is reported.

The following is a configuration example for RHEL 7.x:

KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u /dev/$name",
RESULT=="36c4ff1f100ee3d7501948ec2000002c5", SYMLINK+="raw/ORA-ndata-01", OWNER="grid",
GROUP="asmadmin", MODE="0660"

The following is a configuration example for RHEL 6.x:

KERNEL=="sd*",BUS=="scsi",PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/
$name",RESULT=="36c4ff1f100ee3d7501948ec2000002c5",SYMLINK+="raw/ORA-
ndata-01",OWNER="grid",GROUP="asmadmin",MODE="0660"

NOTE

ORA-ndata-01: sets this parameter as planned. Using SYMLINK is recommended. Do not

use NAME to rename the device allocated by the kernel.

5.5 I/O Scheduler

The I/O scheduler, also referred to as disk scheduling, is used by the operating
system to determine the order of submitting I/O operations on a block device. The
following describes the best practices of the I/O scheduler in Oracle database
scenarios.

5.5.1 Principles
Each block device or its partition corresponds to a request queue for which an I/O
scheduler is selected to coordinate submitted requests. The basic purpose of the
I/O scheduler is to arrange the requests according to their sector numbers on the
block device, to reduce disk head movement and improve efficiency. Each
scheduler maintains a different number of queues for sequencing the submitted
requests and moves the first requests in a queue to the request queue for
response in sequence. The following figure shows the position of the I/O scheduler
in the kernel stack.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 28

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Figure 5-2 Position of the I/O scheduler in the kernel stack

The I/O scheduler is to increase I/O throughput and reduce I/O response time,
which is generally contradictory. To balance them, the I/O scheduler offers
multiple scheduling algorithms to adapt to different I/O request scenarios. The
following describes the scheduling algorithms provided by the Linux kernel.

CFQ
CFQ allocates a request queue and a time slice to each process for accessing the
block device. A process sends read/write requests to the underlying block device
within the time slice, and the request queue of the process is suspended and wait
for scheduling after the time slice is used up.

NOTE

In Kernel 2.6.18 (20th, September, 2006), CFQ becomes the default scheduling program.

Deadline
The deadline I/O scheduler is aimed to ensure each I/O request is responded
within a certain period of time. An expiration date is set for all I/O operations to
ensure that requests are scheduled timely based on CFQ.

● The default validity period of a read request is 500 ms.

● The default validity period of a write request is 5s.
NOTE

Tests show that the deadline I/O scheduler is superior to the CFQ I/O scheduler and is
suitable for certain multithreaded workloads and database workloads.
The deadline I/O scheduler is used in RHEL 7.x systems by default while CFQ is used for
SATA disks by default.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 29

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Noop
Noop is short for the No Operation I/O scheduler. It inserts all incoming I/O
requests into a simple first in first out (FIFO) queue and merges adjacent I/O
requests.
Assume that the I/O request sequence is as follows:
● 100, 500, 101, 10, 56, 1000
Noop will process in the following order:
● 100(101), 500, 10, 56, 1000
Noop has obvious advantages in the following scenarios:
● There are more intelligent I/O scheduler devices than the I/O scheduler. If the
block device drivers use RAID, or are SAN or NAS storage devices, these
devices better organize I/O requests and do not need the I/O scheduler to
perform extra scheduling.
● Upper-layer applications have closer contact with underlying devices than the
I/O scheduler. In other words, the I/O request sent by the upper-layer
applications to I/O scheduler has been optimized. Therefore, the I/O scheduler
only needs to execute the I/O request sent by the upper-layer application in
sequence.
● For certain disks without rotating magnetic heads, Noop is a better choice.
For disks with rotating magnetic heads, reassembling requests by the I/O
scheduler consumes certain CPU time. However, for SSDs, the CPU time can
be saved because SSDs provide a more intelligent request scheduling
algorithm.
NOTE

In the preceding scenarios, Noop is not necessarily preferred for all tuning including
performance tuning is based on actual workloads.

5.5.2 Configuring the I/O Scheduler

You can use either of the following methods to modify the I/O scheduler. For
details, see 10.2-Appendix B — I/O Scheduler Configuration.
● Use udev to modify the disk information.
● All disk devices are modified by recreating GRand Unified Bootloader (GRUB).
To obtain the optimal performance of Oracle ASM:

NOTE

● The deadline I/O scheduler is recommended by Oracle. For official explanation, click
here.
● The deadline I/O scheduler is recommended by RHEL. For official explanation, click here.

5.6 Kernel Parameters

This section describes kernel parameters. For production database systems, you
are advised to adjust these parameters to optimize the database systems. The
following describes the best practices of kernel parameters in Oracle database
scenarios.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 30

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

The following describes the functions of virtual memory, shared memory,

semaphores, network ports, I/O synchronous requests, file handles, and kernel
panic on OOPS parameters in RHEL 7.x during Oracle database deployment. You
are advised to read each parameter carefully to better understand how to
optimize the performance based on specific workloads.

5.6.1 Virtual Memory

The following table describes the virtual memory parameters and the definition of
dirty data.

Table 5-4 Virtual Memory description

Parameter Description

SWAPPINESS Controls the degree to which the system favors anonymous

memory or the page cache. A high value improves file
system performance, while aggressively swapping less active
processes out of memory. A low value avoids swapping
processes out of memory, which usually decreases latency, at
the cost of I/O performance.
NOTE
Since Red Hat Enterprise Linux 6.4, setting swappiness to 0 will even
more aggressively avoid swapping out which increases the risk of
out-of-memory (OOM) killing under strong memory and I/O
pressure. To achieve the same behavior of swappiness as previous
versions of Red Hat Enterprise Linux 6.4 in which the
recommendation was to set swappiness to 0, set swappiness to the
value between 1 and 20. The recommendation of swappiness for Red
Hat Enterprise Linux 6.4 or higher running Oracle databases is now a
value between 1 to 20.

DIRTY_RATIO Contains, as a percentage of total system memory, the

number of pages at which a process that is generating disk
writes will itself start writing out dirty data. The default
value is 20. The recommended value is between 40 and 80.
The reasoning behind increasing the value from the standard
Oracle 15 recommendation to a value between 40 and 80 is
because dirty ratio defines the maximum percentage of total
memory that be can be filled with dirty pages before user
processes are forced to write dirty buffers themselves during
their time slice instead of being allowed to do more writes.
All processes are blocked for writes when this occurs due to
synchronous I/O, not just the processes that filled the write
buffers. This can cause what is perceived as unfair behavior
where a single process can hog all the I/O on a system. As
the value of dirty_ratio is increased, it is less likely that all
processes will be blocked due to synchronous I/Os. However,
this allows for more data to be sitting in memory that has
yet to be written to disk.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 31

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter Description

DIRTY_BACKGR Contains, as a percentage of total system memory, the

OUND_RATIO number of pages that the background write back daemon
will start writing out dirty data. The Oracle recommended
value is 3.
NOTE
An example with the dirty_background_ratio set to 3 and dirty_ratio
set to 80, the background write back daemon will start writing out
the dirty data when it hits the 3% threshold asynchronously,
however, none of that data is written synchronously until the
dirty_ratio is 80% full which is what causes for all processes to be
blocked for writes when this occurs.

DIRTY_EXPIRE_C Defines when dirty in-memory data is old enough to be

ENTISECS eligible for writeout. The default value is 3000, expressed in
hundredths of a second. The Oracle recommended value is
500.

DIRTY_WRITEBA Defines the interval of when writes of dirty in-memory data

CK_CENTISECS are written out to disks. The default value is 500, expressed
in hundredths of a second. The Oracle recommended value is
100.

NOTE

DIRTY DATA: Data that has been modified and saved in the cache to gain performance
advantages. After the data is stored to disks, it becomes clean.

5.6.2 Shared Memory

The following table describes the shared memory (SHMMAX, SHMALL, and
SHMMNI) parameters. To run the Oracle database on the system, you need to
allocate proper shared memory pages and segments to data. The table lists the
parameters that must be set.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 32

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Table 5-5 Shared Memory description

Parameter Description

SHMMAX A default installation of Red Hat Enterprise Linux 7.0 x86_64

provides a maximum size of a single shared memory segment,
SHMMAX, to 4294967295 bytes, equivalent to 4 GB to 1 byte. This
value is important since it regulates the largest possible size of one
single Oracle SGA shared memory segment. If the Oracle system
global area (SGA) is larger than the value specified by SHMMAX (4
GB to 1 byte by default), then Oracle is required to create multiple
smaller shared memory segments to completely fit Oracle's SGA.
This can cause a significant performance penalty, especially in
NUMA environments. In an optimal NUMA configuration, a single
shared memory segment for Oracle's SGA is created on each
NUMA node. If SHMMAX is not properly sized and creates multiple
shared memory segments, SHMMAX limitations may keep the
system from evenly distributing the shared memory segments
across each NUMA node.
Starting with Red Hat Enterprise Linux 7.1 and above, SHMMAX
default value is set to 18446744073692774399 bytes, equivalent to
roughly 18 petabytes. Due to this, there is no need to calculate
SHMMAX because of the very large size already provided. You are
advised to use the value set in Red Hat Enterprise Linux 7.1 and
above because the value is purposely set higher than the
architectural memory limits to ensure that any Oracle SGA value
set within an Oracle database instance may fit in one single shared
memory segment.

SHMALL A default installation of Red Hat Enterprise Linux 7.0 x86_64

provides an SHMALL value of 268435456 pages, the equivalent of
1 TB in system pages. This is determined by the following formula:
SHMALL IN BYTES x PAGE_SIZE
Starting with Red Hat Enterprise Linux 7.1 and above, SHMALL
default value is 18446744073692774399 pages, the same value set
to SHMMAX. To ensure an adequate number of memory pages are
allocated to a single Oracle SGA, it is recommended that the value
of SHMALL be set to at least the value using the following
formula:
SHMMAX IN BYTES/PAGE_SIZE
Since the default value of SHMALL in Red Hat Enterprise Linux 7.1
and above is 18446744073692774399 pages, and the minimum
recommended value by Oracle for SHMALL is 1073741824, the
larger default value is kept.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 33

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter Description

SHMMNI SHMMNI is the maximum total number of shared memory

segments. A default installation of Red Hat Enterprise Linux 7
x86_64 provides an SHMMNI default value of 4096. By Red Hat
Enterprise Linux 7 optimizing the SHMMAX value with one shared
memory segment per Oracle SGA, this parameter reflects the
maximum number of Oracle and ASM instances that can be
started on a system. Oracle recommends the value of SHMMNI to
be left at the default value of 4096.

5.6.3 SEM
The following table describes the semaphore kernel parameters comprised of the
SEMMSL, SEMMNI, SEMMNS, and SEMOPM.

Table 5-6 SEMMSL, SEMMNI, SEMMNS, and SEMOPM

Parameter Description

SEMMSL Defined as the maximum number of semaphores per semaphore

set. The default value is 250.
Semaphores are used by Oracle for internal locking of SGA
structures. Sizing of semaphores directly depends on only the
PROCESSES parameter of the instance(s) running on the system.
The number of semaphores to be defined in a set should be set to
a value that minimizes the waste of semaphores.
For example, say our environment consists of two Oracle instances
with the PROCESSES set to 300 for database one and 600 for
database two. With SEMMSL set at 250 (default), the first
database requires 2 sets. The first set is 250 semaphores but an
additional 50 semaphores are required thus an additional SEMMSL
set is required thus wasting 200 semaphores. Our 2nd instance
requires 3 sets, set one 250 semaphores, set two 250 semaphores,
giving us a total of 500, but an additional 100 semaphores are
required thus adding an additional SEMMSSL set wasting 150
semaphores. A better value of SEMMSL in this particular case
would be 150. With SEMMSL set at 150, the first database requires
two sets (wasting zero semaphores), our second instance requires
four sets (wasting zero semaphores). This is an ideal example, and
most likely some semaphore wastage is expected and okay as
semaphores in general consume small amounts of memory. As
more databases are created in an environment, these calculations
may get complicated. In the end, the goal is to limit semaphore
waste.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 34

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter Description

SEMMNS Defined as the total number of semaphores for the entire system.
The default value is 32000.
Oracle requires 2x value of PROCESSES in the init.ora parameter
for semaphores (SEMMNS value) on startup of the database. Then
half of those semaphores are released. To properly size SEMMNS,
one must know the sum of all PROCESSES set across all instances
on the host. SEMMNS should best be set higher than SEMMNI x
SEMMSL value (this is how we get 32000 for default value
250*128)

SEMOPM Defined as the total number of semaphore operations performed

per semop system call. SEMOPM is calculated using the total
SEMMNI divided by SEMMSL. In the default scenario that is
3200/250 = 128. The default value is 100.

SEMMNI Defined as the maximum number of semaphore sets for the entire
system. The default value is 128.
Regarding SEMMNI, SEMMNI should be set high enough for
proper number of sets to be available on the system. Using the
value of SEMMSL, one can determine max amount of SEMMNI
required. Round up to the nearest power of 2.
SEMMNI = SEMMNS/SEMMSL

5.6.4 Ephemeral Network Ports

The following table describes the ephemeral network port parameters.

Table 5-7 Ephemeral network port parameter description

Parameter Description

net.core.rmem_default Optimizing the network settings for the default and

maximum buffers for the application sockets in Oracle
net.core.rmem_max is done by setting static sizes to RMEM and WMEM.
net.core.wmem_defaul The RMEM parameter represents the receive buffer
t size, while the WMEM represents the send buffer size.

net.core.wmem_max

5.6.5 I/O Requests

The following table describes the I/O request parameters.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 35

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Table 5-8 I/O request parameter description

Parameter Description

fs.aio-max-nr The kernel parameter FS.AIO-MAX-NR sets the maximum

number of current asynchronous I/O requests. Oracle
recommends setting the value to 1048576.

5.6.6 File Handles

The following table describes the file handle parameters.

Table 5-9 File handle parameter description

Parameter Description

fs.file-max The kernel parameter FS.FILE-MAX sets the maximum number

of open file handles assigned to the Red Hat Enterprise Linux 7
operating system. Oracle recommends that for each Oracle
database instance found within a system, allocate 512 x
PROCESSSES in addition to the open file handles already
assigned to the Red Hat Enterprise Linux 7 operating system.
PROCESSES within a database instance refers to the maximum
number of processes that can be concurrently connected to the
Oracle database by the oracle user.

5.7 HugePages
HugePages is a feature integrated into the Linux kernel. Enabling HugePages
makes it possible for the operating system to support memory pages greater than
the default (usually 4 KB). Using very large page sizes can improve system
performance by reducing the amount of system resources required to access page
table entries. The size of HugePages varies from 2 MB to 256 MB, depending on
the kernel version and hardware architecture. For Oracle databases, using
HugePages reduces the operating system maintenance of page states, and
increases Translation Lookaside Buffer (TLB) hit ratio. The following describes the
best practices of HugePages in Oracle databases.

5.7.1 Principle
Page Scheduling Process
The following table shows how an operating system converts a virtual address to
a physical address by using a processor (CPU).

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 36

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Figure 5-3 Conversion process of a virtual memory system

Advantages of HugePages
● Reduction of the page table size
The operating system page table (mapping from the virtual memory to the
physical memory) is smaller, because each page table entry is pointing to
pages from 2 MB to 256 MB. For example, if you use HugePages with 64-bit
hardware, and you want to map 256 MB of memory, you may need one page
table entry (PTE). If you do not use HugePages, and you want to map 256 MB
of memory, then you must have 65536 (256 MB x 1024 KB/4 KB) PTEs.
● The memory of HugePages is intelligently locked in the physical memory and
is never swapped out to the swap partition, avoiding the swap impact on the
performance.
● As the number of page tables decreases, the TLB hits in the CPU (CPU cache
for the page table) increases.
● Contiguous pages are preallocated and can only be used for the shared
memory.

Parameters of HugePages
The following table describes the parameters related to HugePages.

Table 5-10 Parameters related to HugePages

Parameter Description

HugePages_Total Size of the pool of huge pages

HugePages_Free Number of huge pages in the pool that are not yet
allocated

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 37

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Parameter Description

HugePages_Rsvd Short for "reserved". It is the number of huge pages for

which a commitment to allocate from the pool has been
made, but no allocation has yet been made. Reserved
huge pages guarantee that an application will be able to
allocate a huge page from the pool of huge pages at
fault time.

HugePages_Surp Short for "surplus". It is the number of huge pages in the

pool above the value in /proc/sys/vm/nr_hugepages.
The maximum number of surplus huge pages is
controlled by /proc/sys/vm/nr_overcommit_hugepages.

Hugepagesize Default size of HugePages (in KB)

Hugetlb Total amount of memory (in KB), consumed by huge

pages of all sizes.
If huge pages of different sizes are in use, this number
will exceed HugePages_Total x Hugepagesize.

5.7.2 Configuring HugePages

You are advised to use HugePages to improve database performance and disable
Transparent HugePages. For details about how to configure HugePages, see 10.3-
Appendix C — HugePages Configuration.

NOTE
Start the Oracle database instance and then calculate the recommended value using a
script. After the configuration is complete, restart the Oracle database instance.

HugePages has the following limitations:

● You must unset both the MEMORY_TARGET and MEMORY_MAX_TARGET
initialization parameters. To unset the parameters for the database instance,
use the ALTER SYSTEM RESET command.
● Automatic Memory Management (AMM) and HugePages are not compatible.
When you use AMM, the entire SGA memory is allocated by creating files
under /dev/shm. When Oracle Database allocates SGA with AMM,
HugePages are not reserved. To use HugePages on Oracle databases, you
must disable AMM.
● Ensure that HugePages is configured properly as the system may run out of
memory if excess HugePages are not used by the application.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 38

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

Table 5-11 Best Practices

The following Best Description

table lists the Practice
best practices of
HugePages.

HugePages Enabled For Oracle databases, using HugePages

reduces the operating system maintenance of
page states, and increases the database
performance by increasing the TLB hit ratio.

Transparent Disabled Transparent HugePages may lead to memory

HugePages allocation delays at runtime, which cause
nodes and instances to be removed from the
cluster.

AMM Disabled AMM and HugePages are incompatible. To

use HugePages on Oracle databases, you
must disable AMM.

Based on the lab test, the service delay decreases and transactions per second
(TPS) increases by about 15% under the same load after HugePages is enabled.
You are advised to enable HugePages when deploying Oracle databases.
Performance improvement varies with service loads.

For details, see the following:

NOTE

● Oracle official document

● RHEL official document

5.8 Optimization of Other Parameters

Block Device's Queue Depth
The queue depth determines the maximum number of concurrent I/Os that can be
written to block devices. In Linux, the default value is 128. You are advised not to
change the value. You can run the cat command to query the queue depth of the
current block device.
# cat /sys/block/sdc/queue/nr_requests
128

I/O Alignment
If master boot record (MBR) partitions are created in earlier versions of RHEL, the
first 63 sectors of a disk are reserved for the MBR and partition table. The first
partition starts from the 64th sector by default. As a result, misalignment occurs
between data blocks (database or file system) of hosts and data blocks stored in
the disk array, causing poor I/O processing efficiency.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 39

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 5 Best Practices About Hosts

In a Linux operating system, you can use either of the following methods to
resolve I/O misalignment:
Method 1: Change the start location of partitions.
When creating MBR partitions in Linux, you are advised to run the fdisk command
in expert mode and set the start location of the first partition as the start location
of the second extent on a LUN (the default extent size is 4 MB). The following is a
quick command used to create an MBR partition in /dev/sdb. The partition uses
all space of /dev/sdb. The start sector is set to 8192, namely, 4 MB.
printf "n\np\n1\n\n\nx\nb\n1\n 8192\nw\n" | fdisk /dev/sdb

Method 2: Use GUID Partition Table (GPT) partitions.

The following is a quick command used to create a GPT partition in /dev/sdb. The
partition uses all space of /dev/sdb. The start sector is set to 8192, namely, 4 MB.
parted -s -- /dev/sdb "mklabel gpt" "unit s" "mkpart primary 8192 -1" "print"

NOTE
If the database is installed using udev, formatting partitions, I/O alignment in other words,
is not involved.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 40

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

6 Best Practices About Oracle Databases

6.1 Overview
6.2 Data Block
6.3 ASM

6.1 Overview
Parameter/ Recommended Value/ Default Value/ Location
Operation Method Method

DB_BLOCK_SI Most applications: 8 KB 8KB 6.2.1

ZE DSS system/LOB data DB_BLOCK_SIZE
processing: 32 KB

Nonstandard When the service model None 6.2.2

Block Sizes of some tablespaces is Nonstandard
different from that of Block Sizes
most tablespaces, set
this parameter based
on the special service
model.

Block Size of 512 bytes 512 bytes 6.2.3 Block Size

Redo Log of Redo Log Files
Files

Block Size of 16 KB 16 KB 6.2.4 Block Size

Control Files of Control Files

Disk Use an ASM disk group. None 6.3 ASM

management
mode

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 41

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

Parameter/ Recommended Value/ Default Value/ Location

Operation Method Method

ASM Set this parameter to NORMAL 6.3.2 Disk Group

redundancy NORMAL for ORAGRID
type and ORAMGMT disk
groups.
Other disk groups are
protected by RAID
groups in external
redundancy mode.

AU SIZE For a large database, 4 MB 6.3.3 AU Size

the recommended AU
size is 4 MB or larger.

ASM striping Most Oracle files None 6.3.4 ASM

(including data files, Striping
archive logs, and
backup sets): coarse
striping
For redo log files and
control files: fine
striping

Number of The number of LUNs None 6.3.5 ASM LUN

ASM LUNs (Oracle ASM disks) in
each disk group must
be at least four times
the number of active
I/O paths.

ORAGRID and Set at least three LUNs None

ORAMGMT and one QUORUM.
In versions earlier than
12c, the capacity of
each LUN must be at
least 10 GB. In 12c and
later versions, the
capacity of each LUN
must be at least 50 GB.

Log Plan at least seven log None 6.3.6 Multiplex

multiplexing groups and two
member files for each
log group.
For higher security, you
are advised to place
group members in
different disk groups.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 42

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

Parameter/ Recommended Value/ Default Value/ Location

Operation Method Method

Oracle ASM If the ASM instance is Default value 6.3.10 Oracle

Processes connected to multiple ASM Processes
(n) databases:
If n is less than or equal
to 10: PROCESSES = 50
x n + 50
If n is larger than 10:
PROCESSES = 10 x n
+ 450

_asm_hbeatio Oracle database Oracle database 6.3.11 Oracle

wait 11.2.0.3, 11.2.0.4, and 11.2.0.3, ASM Heartbeat
12.1.0.1: 120s 11.2.0.4, and Timeout
Oracle database 12.1.0.1: 15s
12.1.0.2 and later Oracle database
versions: Default value 12.1.0.2: 120s
Versions later
than 12.2.0.1:
229s

6.2 Data Block

6.2.1 DB_BLOCK_SIZE
DB_BLOCK_SIZE specifies the size (in bytes) of an Oracle database block. To
achieve good performance, the size of an Oracle database block must be greater
than or equal to a multiple of the size of an operating system block. The following
table describes the DB_DLOCK_SIZE parameter.

Table 6-1 Description of parameter DB_DLOCK_SIZE

Attribute Description

Type Integer

Default value 8192

Value range General: 2048 to 32768 (KB)

The block size of 8 KB is the best choice for most systems. The block size of 8 KB is
the best choice for most systems. However, the online transaction processing
(OLTP) system occasionally uses blocks of a smaller size, while the data storage
subsystem (DSS) occasionally uses blocks of a larger size. Pay attention to the
following when selecting the database block size to achieve the best performance:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 43

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

● Read
The purpose is to minimize the number of read times during data retrieval
regardless of the block size.
– If there are small rows and random accesses at most, select a smaller
block size.
– If there are small rows and sequential accesses at most, select a larger
block size.
– If there are small rows, and random and sequential accesses, a larger
block size may be a better choice.
– If there are large rows, for example, rows that contain the data of a large
object (LOB), select a larger block size.
● Write
In OLTP systems with highly concurrent I/Os, the database block size of 8 KB
is usually the best for most systems that process a large number of
transactions. Only the system that processes LOB data requires a block size
larger than 8 KB.
● Advantages and disadvantages of the block size
The following table lists the advantages and disadvantages of different block
sizes.

Table 6-2 Advantages and disadvantages of block sizes

Block Size Advantage Disadvantage

Relatively small Applies to small rows with a Metadata has relatively

large number of random large space overheads.
accesses to reduce block Therefore, a smaller
competition. block size is not
recommended for large
rows.

Relatively large ● Lower overheads and larger If small rows are

data storing space randomly accessed and
● Allows multiple rows to be the block size is large,
read into the buffer cache the buffer cache space is
using a single I/O wasted. For example, for
(depending on the row size a block size of 8 KB and
and block size). a row size of 50 bytes,
7950 bytes are wasted in
● Applies to sequential a buffer cache during
accesses or very large rows random access.
(such as LOB data).

NOTE

For details, see DB_BLOCK_SIZE Initialization Parameter on the official Oracle website.

6.2.2 Nonstandard Block Sizes

The block size cannot be changed after the database is created using
DB_BLOCK_SIZE, but the table space of a non-standard block size can be created.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 44

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

The size of a non-standard block can be set to 2 KB, 4 KB, 8 KB, 16 KB, or 32 KB.
The ability to specify multiple block sizes for a database is especially useful if you
want to transfer table spaces between databases. For example, a table space that
uses a 4 KB block size can be transferred from an OLTP system to a database that
uses a standard block size of 8 KB.

The following describes how to create a table space with a non-standard block
size of 16 KB.

1. Check whether the value of db_16k_cache_size is empty.

If you want to use a non-standard block size of 16 KB for a table space, assign
a value to db_16k_cache_size. For details about DB_nK_CACHE_SIZE, see
Setting the Buffer Cache Initialization Parameters.
SQL> show parameter db_16k_cache_size

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------
db_16k_cache_size big integer 0

SQL> alter system set db_16k_cache_size=1m scope=both;

System altered.

SQL> show parameter db_16

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------
db_16k_cache_size big integer 256M

The default value of db_16k_cache_size is 0, indicating that the function is

disabled. You can specify the value and the system automatically generates a
minimum value after the modification.
NOTE
In Oracle RAC, you need to use the following command to change the value:
SQL> alter system set db_16k_cache_size=1m scope=spfile;
NOTE
Restart the instance for the modification to take effect.
2. Add a table space whose non-standard block size is 16 KB.
SQL> create tablespace TABLESPACENAME datafile '+ORADATA' SIZE 256M blocksize 16k;

Tablespace created.

6.2.3 Block Size of Redo Log Files

Different from the database block size (between 2 KB and 32 KB), that of the redo
log file is equal to the physical sector size of the disk, usually 512 bytes.

Some of the newer high-capacity disk drives provide 4 KB sectors to add the Error
Correcting Code (ECC) function and improve format efficiency. Most Oracle
database platforms can detect sectors in a larger size and the databases
automatically create redo log files whose block size is 4 KB on these disks.

NOTE

The cause for the change from the default 512 bytes to 4 KB is that some drives use 4 KB
as their local block size and reduce metadata overhead by increasing their block size.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 45

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

When using Huawei OceanStor Dorado V6 storage systems, you are advised
not to change the redo log block size due to the following reasons:

● If the block size is 4 KB, the redo waste increases.

The amount of redo waste in 4 KB blocks is much larger than that in 512-byte
blocks. You can determine the amount of redo waste by looking at statistics
stored in the V$SESSTAT and V$SYSSTAT views.
SQL> SELECT name, value FROM v$sysstat WHERE name = 'redo wastage';
NAME VALUE ----------------------------------------------------------------
---------- redo wastage 219744
● Database data is not directly written into storage disks.
Data is preferentially written to the cache and then to disks after a period of
time. Therefore, changing the block size to 4 KB does not improve
performance.

Check the disk sector size.

#fdisk -l
Disk /dev/sdkf: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

To avoid waste, you can specify a 512-byte block on the disk drive who has 4 KB
sectors to overwrite the default 4 KB block of redo logs. You can run the following
command to add a redo log file group whose block size is 512 bytes:

NOTE
For a disk with 4 KB sectors, BLOCKSIZE 512 overwrites the default 4 KB block.
ALTER DATABASE orcl ADD LOGFILE GROUP 4 ('+ORAREDO') SIZE 100M BLOCKSIZE 512 REUSE;

Run the following command to check the block size of the redo log file:
SQL> select blocksize from v$log;
BLOCKSIZE
----------
512
512
512
512

6.2.4 Block Size of Control Files

The block size of control files is irrelevant to that of databases and always equals
16 KB.
SQL> select block_size from v$controlfile ;

BLOCK_SIZE
----------
16384

6.3 ASM

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 46

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

6.3.1 ASM Advantages

Compared with traditional Linux disk management, automatic storage
management (ASM) has the following advantages:
● Automatic file management
● When ASM disks are added or deleted online, Oracle ASM automatically
reallocates file content without stopping hosts.
● The file content stored in the disk group is evenly distributed to eliminate hot
spots and provide unified disk performance that is equivalent to that of the
original device.
● ASM store files in all disk stripes of a disk group, improving I/O performance.
● Advanced Huawei OceanStor V5 features, such as snapshot and clone,
support ASM disk protection and recovery.

6.3.2 Disk Group

Oracle ASM uses disk groups to store data files. An Oracle ASM disk group is a
collection of disks managed by Oracle ASM as a unit. When creating the disk
group, you need to consider the following:

Redundancy
The following table lists the redundancy types supported by ASM disk groups.

Table 6-3 Redundancy type

Disk Group Tyep Description

NORMAL Oracle ASM provides two-way mirroring by default, which

means that all files are mirrored so that there are two
copies of every extent. A loss of one Oracle ASM disk is
tolerated. You can optionally choose three-way or
unprotected mirroring. A file specified with HIGH
redundancy (three-way mirroring) in a NORMAL
redundancy disk group provides additional protection from
a bad disk sector in one disk, plus the failure of another
disk. However, this scenario does not protect against the
failure of two disks.

HIGH Oracle ASM provides three-way (triple) mirroring by

default. A loss of two Oracle ASM disks in different failure
groups is tolerated.

EXTERNAL Oracle ASM does not provide mirroring redundancy and

relies on the storage system to provide RAID functionality.
Any write error causes a forced unmount of the disk group.
All disks must be located to successfully mount the disk
group.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 47

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

Disk Group Tyep Description

FLEX Oracle ASM provides two-way mirroring by default for

newly-created flex disk groups. For migrated flex disk
groups, the default values are obtained from the template
values in the normal or high redundancy disk groups before
migration. For migration from normal redundancy, if the
template defaults were not changed, then the flex defaults
are two-way mirroring. For migration from high
redundancy, if the template defaults were not changed,
then the flex defaults are three-way mirroring.

EXTENDED Oracle ASM provides two-way mirroring by default. The

redundancy setting describes redundancy within a data site.
For example: If there is a two-way mirrored file in a two-
data-site extended disk group, then there are four copies of
the file, two in each data site.

NOTE
The FLEX or EXTENDED disk groups can have different levels of mirroring redundancy.

Disk Group Planning

As shown in the following table, you are advised to configure disk groups for the
following: data files, log files, archive files, and temporary files, for separate
storage, and easy management and maintenance.

Table 6-4 Disk group planning

Disk Group Name Redundancy Description

ORAGRID NORMAL Oracle Cluster Registry (OCR) and

voting files

ORAMGMT NORMAL Grid Infrastructure Management

Repository (GIMR) files(for Oracle
12.2 and later)

ORADATA EXTERNAL Database data files

ORAFRA EXTERNAL Fast recovery area

ORAREDO EXTERNAL Redo log files

ORATEMP EXTERNAL Temp data files

You are advised to use the EXTERNAL redundancy mode to protect other disk
groups in RAID groups except the ORAGRID and ORAMGMT disk groups. The OCR
and voting files occupy less space, data mirroring does not affect database
performance, and the disk group files are not backed up in normal backup

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 48

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

scenarios. To improve cluster software reliability, you are advised to use the
NORMAL redundancy mode for the ORAGRID and ORAMGMT disk groups.
Other data disk groups use external redundancy to provide a large amount of
storage space. As the disk array already provides the mirroring function, data does
not need to be mirrored in disk groups, providing better I/O performance and
larger capacity.

NOTE

ASM stores the files on the disk stripes of the disk groups that use the EXTERNAL
redundancy mode. The mirroring function is processed and managed by the storage array.

6.3.3 AU Size
ASM allocates space by block called the allocation unit (AU). An AU is the
fundamental unit of allocation within a disk. All disks in an ASM disk group are
partitioned by AUs of the same size. In most ASM deployment scenarios, the
default AU settings, such as 1 MB for Oracle 11g or 4 MB for Oracle 12cR2, are
helpful to provide excellent performance for Oracle databases. For data
warehouses with large sequential reads, a larger AU is recommended, which can
reduce the metadata quantity for describing files in a disk group and avoid
hotspot data to give full play to performance advantages. The value can be 1 MB,
2 MB, 4 MB, 8 MB, 16 MB, 32 MB, or 64 MB.
For large databases whose size is 10 TB or larger, the AU size of 4 MB or larger is
recommended. The advantages are as follows:
● The SGA size in the relational database management system (RDBMS)
instance is decreased.
● The file capacity is limited.
● Less time is taken to open the large database which usually has a large
number of big data files.
NOTE
The AU attribute can be specified only during disk group creation. Once a disk group is
created, the AU size cannot be changed.

6.3.4 ASM Striping

ASM file striping is classified into coarse striping and fine striping. The ASM coarse
striping is always equal to the disk group AU size, but fine striping size always
remains 128 KB in any configuration. The following table describes striping.

NOTE

Does ASM striping conflict with the striping provided by storage arrays? The answer is no.
ASM striping is a supplement to storage array striping.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 49

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

Table 6-5 Striped description

Element Description

Striped The striping attribute of templates applies to all disk group types
(normal redundancy, high redundancy, and external redundancy).
Permitted values for the striping attribute of ASM templates are
the following:
● FINE. Striping in 128 KB chunks
● COARSE. Striping in 1 MB chunks (4 MB starting with Oracle
Release 12.2)
If no striping attribute is specified, the default value is set to
COARSE.

Oracle ASM striping is used to balance loads across all disks in a disk group and
reduce I/O latency.
● Coarse striping
By default, ASM stripes data in a disk group in the AU size, which is named
coarse striping. The stripe size is always equal to the AU size. That is, a stripe
lies in only one disk and does not cross disks. Data is evenly distributed to all
disks in the disk group in the AU size, ensuring logical data continuity.
● Fine striping
By default, the fine striping size always remains 128 KB, that is, data is read
and written as 128 KB stripes. ASM selects eight devices (if any) from the disk
group, divides them into AUs, and stripes the AUs in the size of 128 KB.
There are two striping methods because of the differences between Oracle files.
Most Oracle files (including data files, archive logs, and backup sets) are large in
size and involve a large amount of data in each I/O operation. For these file
operations, the actual disk data read/write time is much longer than the
addressing time. Therefore, in this mode, using large stripes (coarse striping) can
increase the throughput.
Online log files and control files are small in size. Each I/O operation involves a
small amount of data and requires shorter read and write latency. Therefore, fine
striping is used to distribute data to multiple disks to provide concurrent access
with less latency. In addition, less time is taken to synchronize data to disks
because of small I/Os.

NOTE

During disk group creation, only control files use fine striping by default. You can modify
the template for a specified file type to change the default settings.
● ASM stripe template
Oracle ASM configures an ASM stripe template for each file type. The
following figure shows how to view the ASM stripe template.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 50

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

Figure 6-1 Viewing asm_template

User the ALTER command plus the default file stripe type of ORAREDO disk
groups to change the database redo log type to FINE, as shown in the
following:
SQL> ALTER DISKGROUP +ORAREDO ALTER TEMPLATE onlinelog
ATTRIBUTES(FINE);
NOTE

In the preceding command, +ORAREDO indicates the created disk group. Change it based
on the site requirements.

6.3.5 ASM LUN

● Number of LUNs
The number of LUNs (Oracle ASM disks) in each disk group must be at least
four times the number of active I/O paths. For example, if a disk group has
four active I/O paths, at least 16 LUNs must be used. The LUN size and
performance of each disk group must be the same.
● LUN capacity and type
Ensure that all disks in the ASM disk group have similar storage performance
and availability. If SSDs and HDDs exist, I/O performance is limited by the
HDDs. Because Oracle ASM data distribution is based on capacity, all disks in
the ASM disk group must have the same capacity to balance data. There is no

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 51

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

fixed capacity for all databases. Therefore, you are advised to provide
sufficient concurrency and capacity based on the service scale.
● Advanced LUN features
Ensure that the LUNs in the same disk group have the same advanced
features and add the LUNs in the cluster to the consistency group.
● Example of LUN planning
This document assumes that there are four active I/O paths and Oracle
database 12cR2. The following table describes the LUN plan for Oracle ASM.

Table 6-6 Example of LUN planning

Disk Group AU Size Redundancy LUN

ORAGRID 4 MB NORMAL 50 GB x 3

ORAMGMT 4 MB NORMAL 50 GB x 3

ORADATA 4 MB EXTERNAL 500 GB x 16

ORAFRA 4 MB EXTERNAL 500 GB x 16

ORAREDO 4 MB EXTERNAL 100 GB x 16

ORATEMP 4 MB EXTERNAL 100 GB x 16

● ORAGRID and ORAMGMT

The number of LUNs in the disk group must be at least three and the size of
each LUN depends on the database version. For versions earlier than 12c, the
size of each LUN must be at least 10 GB. For 12c and later, the size of each
LUN must be at least 50 GB.
Three LUNs are planned to ensure the disk group has at least three failure
groups and one quorum disk. The odd number of disks helps the quorum disk
to determine the node fault, as shown in the following figure.

Figure 6-2 Creating an ORAGRID disk group

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 52

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

● Other disk groups

The LUN capacity is determined by service requirements. For better
performance, the number of LUNs in the data file disk group is recommended
to be four times the number of active I/O paths to provide concurrency. For
example, if the service requires 8 TB space, you are advised to create 16 LUNs,
each with 500 GB capacity, for the disk group, other than creating four 2 TB
LUNs.
NOTE

The LUN capacity is calculated based on the actual service volume. You are advised to plan
the capacity of Oracle data files, log files, archive files, and temporary files based on the
multiple of the number of active I/O paths. For details, see Recommendations for Storage
Preparation.

6.3.6 Multiplex
For better performance, you are advised not to reuse redo logs and the reuse
affects performance. You are advised to plan at least seven log groups and two
member files for each log group.
For higher security, you are advised to place group members in different disk
groups. For example, if seven log groups are planned and each log group has two
member files, group members are allocated to different disk groups, as shown in
the following table.

Table 6-7 Group members in different disk groups

Log Group Log File Disk Group

group 1 redo01_01.log ORAREDO1

redo01_02.log ORAREDO2

group 2 redo02_01.log ORAREDO1

redo02_02.log ORAREDO2

group 3 redo03_01.log ORAREDO1

redo03_02.log ORAREDO2

[...]

6.3.7 Dynamic Rebalancing

Rebalancing a disk group moves data between disks to ensure that every file is
evenly spread across all of the disks in a disk group. ASM automatically initiates a
rebalance after storage configuration changes, such as when you add, drop, or
resize disks. ASM does not move contents in hotspot areas. ASM implements the
balanced distribution of partitions among disks, and the database cache helps to
avoid the storage of hot data on disks. Therefore, the concept of hot disks or hot
partitions is eliminated.
The power setting parameter determines the speed at which rebalancing
operations occur. You can manually start a rebalance to change the power setting

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 53

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

of a running rebalance. A rebalance is automatically restarted if the instance on

which the rebalance is running stops. Databases can remain operational during
rebalancing operations. You can set the ASM_POWER_LIMIT initialization
parameter to minimize the impact on the database performance, as shown in the
following table.

Table 6-8 ASM_POWER_LIMIT

Attribute Description

Type Integer

Default value 1

Modification option ALTER SESSION and ALTER SYSTEM

Value range 0 to 11 (versions earlier than 11.2.0.2)

Value range 0 to 1024 (11.2.0.2 and later versions)

OceanStor V5 storage systems can use the rebalancing feature of ASM to take
over or replace devices of different storage vendors to implement data migration.
For example, the DATA disk group can be migrated from an old storage device to
OceanStor V5 storage systems. To implement such migration, ensure that the
storage devices are mounted to hosts and discovered by ASM during the
migration. When the rebalancing is complete, all data is removed from the old
framework.

NOTE

● The ASM_POWER_LIMIT parameter can be specified only in the ASM instance.

● ASM_POWER_LIMIT specifies the maximum rate of the automatic storage management
instance for disk rebalancing. A larger value indicates faster rebalancing. A smaller value
will take longer time, but consume less processing and I/O resources.
● You are advised not to set ASM_POWER_LIMIT to 0.

6.3.8 Capacity Expansion of the ASM Disk Group

With the increase of ASM disk usage, the storage capacity of the existing disk
group needs to be increased, and the impact on online services needs to be
minimized. The best practice is to add disks with the same capacity to the disk
group. However, in some scenarios, you need to reconfigure the disk capacity
instead of adding disks of the same capacity. In this case, you need to reset the
capacity of all disks in the disk group at the same time. This section describes how
to expand and reset the capacity of LUNs on ASM disks.

Adding a LUN to Expand the Capacity of Oracle ASM Storage

You can expand the capacity by adding disks. For example, you can add one or
more new LUNs to an ASM disk group. This method is simple and secure, and
does not change existing LUNs.

The capacity expansion process is as follows:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 54

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

1. Use DeviceManager to create a LUN.

2. Configure the same LUN size and features as those of the LUN to be added to
the disk group.
3. Add the new LUN to the specified LUN group and ensure that the LUN has
been mapped to the corresponding host.
4. Scan for new devices using SCSI. For details, see the content about identifying
LUNs.
5. Configure multipathing for the newly added LUN. For details, see 5.2
Multipathing.
6. Configure the udev rule file to add the permission and owner for the new
LUN. For details, see 5.4 Udev Configuration.
7. Add the new LUN to the ASM disk group.
ASM automatically balances disk group data to ensure that each file is evenly
distributed on the LUNs in the disk group.
8. Query disk group status.
# asmcmd lsdsk -gk -G ORADATA
# asmcmd lsdg -g ORADATA

Expanding the Existing LUN Capacity to Expand the Capacity of Oracle ASM
Storage
The following uses DM Multipath as an example to describe how to expand the
LUN capacity when the entire LUN partition is used.
1. Use DeviceManager to expand the capacity of the original LUN, as shown in
the following figure.

Figure 6-3 Expanding the LUN capacity

2. Run the following command to rescan for disks on the host again. The disk
capacity remains unchanged.
# rescan-scsi-bus.sh
3. Obtain the drive letter of the LUN on the host and run the following
command to rescan for disks:
After the scan is complete, the disk capacity becomes 50 GB.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 55

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

# echo 1 > /sys/block/sdx/device/rescan

NOTE

sdx indicates the drive letter of the LUN to be expanded on the application server. For
details about how to view the drive letter, see Step 4. Replace it with the actual drive
letter.
4. View the LUN capacity after expansion.
# multipath -ll ORA-DATA4M-01 | awk '/sd/ {print $(NF-4)}' | xargs -i fdisk -l /dev/{} | egrep "^Disk"
Disk /dev/sdyn: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Disk /dev/sdzd: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Disk /dev/sdyv: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Disk /dev/sdzl: 859.0 GB, 858993459200 bytes, 1677721600 sectors
5. Reload the multipath device.
# multipathd -k "resize map ORA-DATA4M-01"
ok

NOTE

In the preceding command, ORA-DATA4M-01 is the LUN alias in the multipath file.
Change it based on the site requirements.
6. Run the asmcmd lsdsk command to check the capacity change of the ASM
disk.
If the capacity of OS_MB does not change, do not go to the next step.

NOTE

Total_MB: capacity of the LUN in the current disk group before capacity expansion.
OS_MB: maximum size of an ASM disk that can be expanded to.
7. Run the ALTER DISKGROUP RESIZE DISK command to adjust the ASM disk
size.
SQL> ALTER DISKGROUP ORADATA RESIZE DISK ORADATA_0000 SIZE 819200M REBALANCE POWER
10;
Diskgroup altered.
8. Check the size of the new ASM disk.

NOTE
Adjusting the LUN capacity on the operating system may cause data loss. You are advised
to back up all LUN data before adjusting the capacity.

6.3.9 Oracle ASM Storage Restriction

Oracle ASM has the following restrictions on the number of disk groups, disks, and
files:
● A maximum of 511 disk groups can be configured in Oracle Database 12c
Release 1 or later.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 56

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

● A maximum of 10,000 Oracle ASM disks can be configured in a disk group.

● A maximum of 65,530 Oracle ASM disks can be configured in the storage
system.
● Each disk group has 1 million files.

Oracle ASM has the following storage restrictions if the COMPATIBLE.ASM and
COMPATIBLE.RDBMS parameter values of disk groups are smaller than 12.1:

● Maximum 2 TB per Oracle ASM disk

● Maximum 20 PB of the storage system

If the attributes of the COMPATIBLE.ASM and COMPATIBLE.RDBMS parameter

values of disk groups are greater than or equal to 12.1:

● When the AU size is equal to 1 MB, the maximum storage space of each
Oracle ASM disk can be set to 4 PB.
● When the AU size is equal to 2 MB, the maximum storage space of each
Oracle ASM disk can be set to 8 PB.
● When the AU size is equal to 4 MB, the maximum storage space of each
Oracle ASM disk can be set to 16 PB.
● When the AU size is equal to 8 MB, the maximum storage space of each
Oracle ASM disk can be set to 32 PB.
● The maximum storage space of the storage system can be set to 320 EB.
NOTE

● Maximum size of a disk group = Maximum disk size x Maximum number of disks in the
disk group (10, 000)
● The maximum number of disks in all disk groups is 10,000. The 10,000 disks can be in
one disk group or in a maximum of 511 disk groups.

For details, see Oracle Administrator's Guide.

6.3.10 Oracle ASM Processes

The initialization parameter of the number of processes affects Oracle ASM.
Generally, it is proper to use the default value. If the Oracle ASM instance is
connected to multiple databases, you can use the formula shown in the following
table to calculate the value.

Table 6-9 PROCESSES formula

10 or Less Instances per Node More than 10 Instances per Node

Number of processes = 50 x n + 50 Number of processes = 10 x n + 450

NOTE

n indicates the number of Oracle databases connected to Oracle ASM instances. The
formula is based on experience.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 57

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 6 Best Practices About Oracle Databases

6.3.11 Oracle ASM Heartbeat Timeout

In Oracle Database 11.2.0.3, 11.2.0.4, and 12.1.0.1, if ASM disk groups use the
normal or high redundancy configuration, the default ASM heartbeat timeout
interval (_asm_hbeatiowait) is 15 seconds. To prevent database breakdown
caused by ASM heartbeat timeout, you are advised to set _asm_hbeatiowait to
120 seconds. In Oracle Database 12.1.0.2, the default value of this parameter is
changed to 120 seconds. In Oracle Database 12.2.0.1 and later, the default value is
changed to 229 seconds.
The modification process is as follows:
[root@dbn01]#su - grid
[grid@dbn01]#sqlplus / as sysasm
SQL> select ksppinm,ksppstvl,ksppdesc from x$ksppi x,x$ksppcv y where x.indx = y.indx and
ksppinm='_asm_hbeatiowait'; //Check the current ASM heartbeat. The default value is 15s.
KSPPINM -------------------------------------------------------------------------------
KSPPSTVL -------------------------------------------------------------------------------
KSPPDESC ------------------------------------------------------------------------------- _asm_hbeatiowait
15
number of secs to wait for PST Async Hbeat IO return
SQL> create pfile='/tmp/asm_spfile.ora' from spfile; //Back up the spfile configuration of the ASM
database.
SQL> alter system set "_asm_hbeatiowait"=120 scope=spfile sid='*'; //Perform this operation only on the
first database node. SQL> exit;
[grid@dbn01]#echo $ORACLE_HOME /u01/app/11.2.0/grid
[grid@dbn01]#exit
[root@dbn01]#/u01/app/11.2.0/grid/bin/crsctl stop crs //Disable the CRS of the first database node.
[root@dbn01]#/u01/app/11.2.0/grid/bin/crsctl start crs //Enable the CRS of the first database node, and
restart the CRS of other database nodes after the first node is restarted.
[root@dbn01]#su - grid
[grid@dbn01]#sqlplus / as sysasm
SQL> select ksppinm,ksppstvl,ksppdesc from x$ksppi x,x$ksppcv y where x.indx = y.indx and
ksppinm='_asm_hbeatiowait'; //Check whether the ASM heartbeat is modified to 120s.
KSPPINM --------------------------------------------------------------------------------
KSPPSTVL --------------------------------------------------------------------------------
KSPPDESC------------------------------------------------------------------------
_asm_hbeatiowait120number of secs to wait for PST Async Hbeat IO return

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 58

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

7 Summary

The following table lists the best practices of all operation objects and
configurations mentioned in the preceding sections. For specific details, see the
corresponding section.

Table 7-1 Best practices

Object Parameter/ Recommended Default Location
Operation Value/Method Value/
Method

Storage Disk Manually select disks None 3.2.1 Disk

selection of the same Selection
method controller enclosure.

Disk capacity Select disks that have None

the same capacity.

Number of Configure at most None

disks 100 disks for each
tier in a storage pool.

RAID level RAID-TP RAID 6 3.2.2 RAID

Technology

Hot spare High or above Low 3.2.3 Hot

policy Spare
Policies

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 59

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

LUN For a single Default 3.3.1

application database, select application Application
type Oracle_OLAP, type Types
Oracle_OLTP, or (block size =
Oracle_OLAP&OLTP 8 KB)
based on actual
services.
In multi-database
scenarios, select an
application type
based on actual
services of each
database.

LUN group Plan multiple disk None 3.4.1 LUN

groups of a RAC as Groups
one LUN group.

Host group Add hosts in a cluster None 3.4.2 Host

to the same host Groups
group.

Port group Create a port group None 3.4.3 Port

to enable LUNs in Groups
the LUN group and
hosts in the host
group to
communicate with
each other using
ports in the port
group.

Network Logical Fibre Channel None 4.2 Logical

topology network without Topology
single points of
failure

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 60

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

Physical Cross networking: None 4.3 Physical

topology Connect at least two Topology
Fibre Channel ports
from different
SmartIO interface
modules of each
controller to two
Fibre Channel
switches.
Configure at least
two dual-port host
bus adapters (HBAs)
for the host and
connect the HBAs to
different switches. 16
Gbit/s or higher-rate
Fibre Channel
interface modules are
recommended.
Point-to-point zone
planning:
Each zone can
contain only one
initiator and one
target.

Number of At least 8 paths are None 4.4 Number

paths required for a dual- of Paths
controller storage
system and 16 paths
are required for a
four-controller
storage system to
ensure full
redundancy.

5-Best Multipathing Huawei UltraPath MultipathI/O 5.2

Practices tool Multipathin
About g
Hosts

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 61

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

HBA port If QLogic HBAs are If QLogic 5.3.1 HBA

retry interval used: HBAs are Parameters
Port Down Retry used:
Count: 10 Port Down
Link Down Timeout: Retry Count:
10 30
If Emulex HBAs are Link Down
used: Timeout: 30
lpfc_devloss_tmo: 10 If Emulex
HBAs are
used:
lpfc_devloss_
tmo: 10

Maximum Small-sized database: 32 5.3.2

Number of 32 Maximum
Concurrent Large database: 128 Number of
Connections Concurrent
Supported by Connection
an HBA s Supported
by an HBA

ASM disk Configure Oracle None 5.4 Udev

group ASM disks using udev Configurati
configuration rules. on

I/O scheduler Deadline Deadline 5.5 I/O

Scheduler

SWAPPINESS 1 to 20 60 5.6.1
Virtual
DIRTY_RATIO 40 to 80 20 Memory
DIRTY_BACK 3 10
GROUND_RA
TIO

DIRTY_EXPIR 500 3000

E_CENTISECS

DIRTY_WRIT 100 500

EBACK_CENT
ISECS

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 62

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

SHMMAX Red Hat Enterprise Red Hat 5.6.2

Linux 7.0 x86_64: Enterprise Shared
18446744073692774 Linux 7.0 Memory
399 bytes x86_64:
Red Hat Enterprise 4294967295
Linux 7.1 and later: bytes
Default value Red Hat
Enterprise
Linux 7.1 and
later:
18446744073
692774399
bytes

SHMALL Red Hat Enterprise Red Hat

Linux 7.0 x86_64: Enterprise
18446744073692774 Linux 7.0
399 pages x86_64:
Red Hat Enterprise 268435456
Linux 7.1 and later: pages
Default value Red Hat
Enterprise
Linux 7.1 and
later:
18446744073
692774399
pages

SHMMNI 4095 4095

SEMMSL Set this parameter 250 5.6.3 SEM

based on the
objective of
minimizing the use of
semaphores. For
details about the
calculation method,
see 5.6.3 SEM.

SEMMNS The value must be 32000

higher than the result
of SEMMNI x
SEMMSL.

SEMOPM Set SEMOPM to the 100

value of SEMMSL.
The minimum value
is 100.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 63

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

SEMMNI Formula: SEMMNI = 128

SEMMNS/SEMMSL.
The result is rounded
up to the nearest
power of 2.

fs.aio-max-nr The minimum value 1048576 5.6.5 I/O

is 1048576. Requests

fs.file-max It is recommended 1024 5.6.6 File

that at least 512 x Handles
processes be
allocated to each
Oracle database
instance. In addition,
the number of
allocated processes
must be reserved for
the operating system.

HugePages Enabled None 5.7.2

Configuring
Transparent Disabled None HugePages
HugePages

AMM Disabled None

6-Best DB_BLOCK_SI Most applications: 8 8 KB 6.2.1

Practices ZE KB DB_BLOCK_
About DSS system/LOB data SIZE
Oracle processing: 32 KB
Database
s Nonstandard When the service None 6.2.2
Block Sizes model of some Nonstandar
tablespaces is d Block
different from that of Sizes
most tablespaces, set
this parameter based
on the special service
model.

Block Size of 512 bytes 512 bytes 6.2.3 Block

Redo Log Size of
Files Redo Log
Files

Block Size of 16 KB 16 KB 6.2.4 Block

Control Files Size of
Control
Files

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 64

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

Disk Use an ASM disk None 6.3 ASM

management group.
mode

ASM Set this parameter to NORMAL 6.3.2 Disk

redundancy NORMAL for Group
type ORAGRID and
ORAMGMT disk
groups.
Other disk groups are
protected by RAID
groups in external
redundancy mode.

AU size For a large database, 4 MB 6.3.3 AU

the recommended Size
AU size is 4 MB or
larger.

ASM Striping Most Oracle files None 6.3.4 ASM

(including data files, Striping
archive logs, and
backup sets): coarse
striping
For redo log files and
control files: fine
striping

Number of The number of LUNs None 6.3.5 ASM

ASM LUNs (Oracle ASM disks) in LUN
each disk group must
be at least four times
the number of active
I/O paths.

ORAGRID Set at least three None

and LUNs and one
ORAMGMT QUORUM.
In versions earlier
than 12c, the
capacity of each LUN
must be at least 10
GB. In 12c and later
versions, the capacity
of each LUN must be
at least 50 GB.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 65

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 7 Summary

Object Parameter/ Recommended Default Location

Operation Value/Method Value/
Method

Log Plan at least seven None 6.3.6

multiplexing log groups and two Multiplex
member files for
each log group.
For higher security,
you are advised to
place group members
in different disk
groups.

Oracle ASM If the ASM instance is Default value 6.3.10

Processes connected to Oracle ASM
multiple (n) Processes
databases:
If n is less than or
equal to 10:
PROCESSES = 50 x n
+ 50
If n is larger than 10:
PROCESSES = 10 x n
+ 450

_asm_hbeati Oracle database Oracle 6.3.11

owait 11.2.0.3, 11.2.0.4, and database Oracle ASM
12.1.0.1: 120s 11.2.0.3, Heartbeat
Oracle database 11.2.0.4, and Timeout
12.1.0.2 and later 12.1.0.1: 15s
versions: Default Oracle
value database
12.1.0.2: 120s
Versions
later than
12.2.0.1: 229s

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 66

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 8 References

8 References

References
OceanStor Dorado V6 storage system product documentation
Oracle ASM Administrator's Guide
Product Documentation for RHEL 7

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 67

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 9 Glossary

9 Glossary

Acronym or Description
Abbreviation

AMM Automatic Memory Management

ASM Automatic Storage Management

DBA Oracle Database Administrator

FC Fibre Channel

HBA Host Bus Adapter

LUN Logical Unit Number

OLAP Online Analytical Processing

OLTP Online Transaction Processing

PGA Process Global Area

RAID Redundant array of independent disks

RHEL Red Hat Enterprise Linux

SAS Serial Attached SCSI

SGA System Global Area

ASRU Oracle ASM Storage Reclamation Utility

RAC Oracle Real Application Cluster

WWN World Wide Name

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 68

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

10 Appendixes

10.1 Appendix A — Udev Configuration

10.2 Appendix B — I/O Scheduler Configuration
10.3 Appendix C — HugePages Configuration

10.1 Appendix A — Udev Configuration

This section describes how to use udev rules to configure mapping permissions
and persistence for each disk device in the scenario where UltraPath is used. The
content is for reference only.

1. Identify and obtain WWIDs. In the example, sdb is used.

Method 1: Use UltraPath.
ltraPath CLI #0 >show vlun
----------------------------------------------------------------------------------------------------------
Vlun ID Disk Name Lun WWN Status Capacity Ctrl(Own/Work)
0 sdb LUN_200_grid0000 6c4ff1f100ee3d7501948ec2000002c5 Normal 10.00GB 0B/
0B

NOTE

The CLI commands may vary depending on the UltraPath version. For details, click
here.
Method 2:
# cd /dev/disk/by-id
[root@oracle1 by-id]# ll -lh
lrwxrwxrwx 1 root root 9 Mar 12 17:08 wwn-0x6c4ff1f100ee3d7501948ec2000002c5 -> ../../sdb

2. Create the 99-oracle-asmdevices.rules file in the /etc/udev/rules.d/ directory.

3. In the 99-oracle-asmdevices.rules file, create a similar rule for each disk
device.
If the operating system is RHEL 7.x, refer to the following configuration:
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u /dev/$name",
RESULT=="36c4ff1f100ee3d7501948ec2000002c5", SYMLINK+="raw/LUN_200_DATA0000",
OWNER="grid", GROUP="asmadmin", MODE="0660"

If the operating system is RHEL 6.x, refer to the following configuration:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 69

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

KERNEL=="sd*",BUS=="scsi",PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace

--device=/dev/$name",RESULT=="36c4ff1f100ee3d7501948ec2000002c5",SYMLINK+="raw/
LUN_200_DATA0000",OWNER="grid",GROUP="oinstall",MODE="0660"
4. Test the created udev rules. The test process is as follows:
# udevadm test /sys/block/sdb
udevadm_test: UDEV_LOG=6
udevadm_test: DEVPATH=/devices/up_primary/up_adapter/host15/target15:0:0/15:0:0:1/block/sdb
udevadm_test: MAJOR=8
udevadm_test: MINOR=16
udevadm_test: DEVNAME=/dev/oracle/LUN_200_grid0000
udevadm_test: DEVTYPE=disk
udevadm_test: ACTION=add
udevadm_test: SUBSYSTEM=block
udevadm_test: DEVLINKS=/dev/block/8:16 /dev/disk/by-id/
scsi-36c4ff1f100ee3d7501948ec2000002c5 /dev/disk/by-path/scsi-0:0:0:1 /dev/disk/by-id/
wwn-0x6c4ff1f100ee3d7501948ec2000002c
5. Confirm the permission required by the device.
# ls -lh /dev
brw-rw---- 1 grid oinstall 8, 16 Mar 12 19:35 sdb

NOTE

If the required permission is incorrect, restart the node or the UDEV service. The
following are examples:
RHEL7.x
udevadm control --reload-rules
udevadm trigger
or
udevadm trigger --type=devices --action=change
RHEL6.x
udevadm control --reload-rules
start_udev

10.2 Appendix B — I/O Scheduler Configuration

Querying the Scheduler Algorithm
Query the I/O scheduler algorithm of the current system disk.
● In RHEL 7.x, the result is deadline.
[root@oracle1 ~]# cat /sys/block/${ASM_DISK}/queue/scheduler
noop [deadline] cfq
● In RHEL 6.x, the result is cfq.
[root@oracle1 ~]# cat /sys/block/${ASM_DISK}/queue/scheduler
noop anticipatory deadline [cfq]

NOTE

${ASM_DISK} indicates a system disk, for example, sdb.

Modifying the Scheduling Algorithm

Method 1: Configure an I/O scheduler for each disk non-disruptively.
● (Optional) After the system is started, you can modify the configuration file
to enable each disk to invoke a unique I/O scheduler.
Change the default cfg algorithm to deadline. The modification takes effect
immediately without the need to restart the system. In both RHEL 6 and 7,
you can change the default cfg algorithm to deadline as follows:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 70

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

# echo 'deadline' > /sys/block/${ASM_DISK}/queue/scheduler

# cat /sys/block/${ASM_DISK}/queue/scheduler
noop anticipatory [deadline] cfq

NOTE
The echo configuration will become invalid after the system is restarted.
● Run the udev rules to configure I/O scheduler.
RHEL creates a file in the following location. The Linux operating system uses
the udev rules to set the I/O scheduler algorithm after each restart.
#vi /etc/udev/rules.d/99-huawei-storage.rules

In RHEL 6 and 7, add the following content to the udev rules file. Then,
restart the operating system. The following is an example:
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g
-u /dev/$name", RESULT=="36207969100f4a3810efc24f70000001a",ATTR{queue/
scheduler}="deadline"
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g
-u /dev/$name", RESULT=="36207969100f4a3810efc253b0000001b",ATTR{queue/
scheduler}="deadline"
ACTION=="add|change", KERNEL=="sd*" ,SUBSYSTEM=="block",PROGRAM=="/usr/lib/udev/scsi_id -g -
u /dev/$name", RESULT=="36207969100f4a3810efc25a60000001c",ATTR{queue/
scheduler}="deadline"
ACTION=="add|change", KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g
-u /dev/$name", RESULT=="36207969100f4a3810efc25e70000001d",ATTR{queue/
scheduler}="deadline"

The following table describes the parameters.

Table 10-1 Parameter description

Parameter Description

ACTION Event (uevent) behavior, for example, add (to add a

device), remove (to delete a device), or change.

ATTR {file}, SYSCTL{kernel parameter}

● It matches the attribute value of the device in the
sysfs.
The trailing space contained in the attribute value will be
ignored unless the specified value contains trailing space.
The file in the braces refers to the file in the device path
(devpath). For example, for /dev/sdb, ATTR{queue/
scheduler} refers to the content of the /sys/block/sdb/
queue/scheduler file.
● It matches the value of the kernel parameter.
The kernel parameter refers to the one contained in /
proc/sys/, for example, noop, cfg, and deadline.

NOTE

After the configuration is completed, only the changes that apply to the sd * device are
displayed. Changes related to the dm-* device are not directly displayed. However, the dm-*
device inherits the I/O scheduler algorithm from the sd * device that forms its path.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 71

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

NOTE
In some virtual environments (such as VMs) and special devices, the output of the
preceding command may be none. In this case, the operating system or VM bypasses kernel
I/O scheduler and directly submits all I/O requests to the device. Do not change the I/O
scheduler algorithm in such an environment, for example, swap space.

Method 2: Modify the grub file to change the scheduling algorithm for all
devices in a unified manner.
● In RHEL 6, add elevator=deadline to the end of the kernel line in the /etc/
grub.conf file.
– After modifying the /etc/grub.conf file, restart the system for the
modification to take effect.
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=3333d6ac-
b29b-400a-8ccf-4b93e7b232da rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD
SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
rhgb quiet elevator=deadline
initrd /initramfs-2.6.32-431.el6.x86_64.img

● In RHEL7, add elevator=deadline to the /etc/default/grub file and rebuild

the grub2 configuration file.
– Modify the /etc/default/grub file.
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet
numa=off transparent_hugepage=never elevator=deadline"
GRUB_DISABLE_RECOVERY="true"

– Rebuild /boot/grub2/grub.cfg and restart the system for the

modification to take effect.

▪ On a BIOS-based host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-112.16.4.el7uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-112.16.4.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-6976e35301344914acfe829b6e8ec911
Found initrd image: /boot/initramfs-0-rescue-6976e35301344914acfe829b6e8ec911.img
done

▪ On a UEFI-based host:
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-537dacc8159a4d4caaa419342da0b820
Found initrd image: /boot/initramfs-0-rescue-537dacc8159a4d4caaa419342da0b820.img
done

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 72

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

NOTE

Check whether UEFI or Legacy only Boot is installed on the operating

system as follows:
Run the following command on the CLI:
#[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
If it is BIOS boot, the following output is displayed:
#[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
BIOS
If it is UEFI boot, the following output is displayed:
#[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
UEFI

10.3 Appendix C — HugePages Configuration

You are advised to use HugePages to improve database performance and disable
Transparent HugePages. The following describes how to enable HugePages and
disable Transparent HugePages. The procedures are for reference only.

Enabling HugePages
To configure and enable HugePages on a computer, perform the following steps:

1. Run the following command to determine whether the kernel supports

HugePages.
#grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

2. Open the /etc/security/limits.conf file and set memlock. The memlock

parameter is expressed in KB. If HugePages is enabled, the maximum locked
memory must be at least 90% of the current RAM. If HugePages is disabled,
the maximum locked memory must be at least 3145728 KB (3 GB). For
example, if the host memory is 256 GB RAM, add the following content to
increase the maximum locked memory:
#cat /proc/meminfo
MemTotal: 263754792 kB
#cat /etc/security/limits.conf
oracle soft memlock 241591910
oracle hard memlock 241591910

3. Log in to the operating system as user oracle and run the ulimit -l command
to verify that the value of memlock is changed.
$ ulimit -l
241591910

4. Run the following command to query the value of the Hugepagesize

variable.
# grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 73

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

5. Create a script and use it to calculate the recommended value of the current
shared memory segment of HugePages.
a. Create a text file named hugepages_settings.sh.
b. Add the following content in the file:
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
NUM_PG=1
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
do
MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
if [ $MIN_PG -gt 0 ]; then
NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
fi
done
# Finish with results
case $KERN in
'2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
'2.6' | '3.8' | '3.10' | '4.1' | '4.14' ) echo "Recommended setting: vm.nr_hugepages =
$NUM_PG" ;;
*) echo "Unrecognized kernel version $KERN. Exiting." ;;
esac
# End

NOTE

If the host's kernel information is not contained in the script, add it manually. If
the kernel information is not contained in the script, the following error message
is displayed:
Unrecognized kernel version 4.1. Exiting.

a. Run the following command to change the file permission:

#chmod +x hugepages_settings.sh
6. Run the hugepages_settings.sh script to calculate the value of HugePages.
#./huge_pages_settings.sh
Recommended setting: vm.nr_hugepages = 39173
7. Set the following kernel parameter, where value indicates the HugePages
value determined in Step 6.
#sysctl -w vm.nr_hugepages = value
8. To assign HugePages after the system restarts, add the following entry to
the /etc/sysctl.conf file, where value indicates the HugePages value
determined in Step 6.
vm.nr_hugepages = value
9. Confirm the kernel parameter modification.
# /sbin/sysctl --system
# /sbin/sysctl -a
10. Run the following command to check whether HugePages is enabled. The
following command output indicates that HugePages is disabled.

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 74

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

# grep Huge /proc/meminfo

AnonHugePages: 0 kB
HugePages_Total: 39173
HugePages_Free: 39173
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

11. Restart the instance.

12. Run the following command to check whether HugePages is enabled. The
following command output indicates that HugePages is enabled.
[oracle@oracle2 ~]$ grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 39173
HugePages_Free: 516
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

Disabling Transparent HugePages

To disable HugePages on a computer, perform the following steps:

Transparent HugePages may cause memory allocation delay during the runtime.
To prevent performance problems, you are advised to disable Transparent
HugePages on all Oracle database servers.

1. Run one of the following commands as user root to check whether

Transparent HugePages is enabled:
In the event of Red Hat Enterprise Linux kernel:
# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

In the event of other kernels:

# cat /sys/kernel/mm/transparent_hugepage/enabled

The following is an example output indicating that Transparent HugePages is

being used, because the [always] flag is enabled and the [never] flag is
disabled.
[always] never

NOTE

If Transparent HugePages is deleted from the kernel, neither /sys/kernel/mm/

transparent_hugepage nor /sys/kernel/mm/redhat_transparent_hugepage exists.
2. Disable Transparent HugePages.
If the operating system is RHEL 6, add the following content to the kernel
boot line in the /etc/grub.conf file and restart the system:
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=3333d6ac-b29b-400a-8ccf-4b93e7b232da
rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
transparent_hugepage=never
initrd /initramfs-2.6.32-431.el6.x86_64.img

If the operating system is RHEL 7 or a later version:

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 75

OceanStor Dorado
Best Practices Oriented to Oracle Database
Deployment 10 Appendixes

– Modify the /etc/default/grub file.

# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet
numa=off transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"

– Rebuild /boot/grub2/grub.cfg and restart the system for the

modification to take effect.

▪ On a BIOS-based host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-112.16.4.el7uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-112.16.4.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-6976e35301344914acfe829b6e8ec911
Found initrd image: /boot/initramfs-0-rescue-6976e35301344914acfe829b6e8ec911.img
done

▪ On a UEFI-based host:
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-537dacc8159a4d4caaa419342da0b820
Found initrd image: /boot/initramfs-0-rescue-537dacc8159a4d4caaa419342da0b820.img
done

Issue 01 (2020-12-10) Copyright © Huawei Technologies Co., Ltd. 76