Objective : Implementing Huge pages in Linux

Created By : Mahaveer Jain

Date : 06-Jun-2015

HugePages on Oracle Linux 64-bit (Doc ID 361468.1)

Newer version : Oracle Linux: Shell Script to Calculate Values Recommended Linux
HugePages / HugeTLB Configuration (Doc ID 401749.1)

(To include kernel setting as well)

Linux OS - Version Enterprise Linux 4.0 to Oracle Linux 6.0 with Unbreakable Enterprise
Kernel [2.6.32] [Release RHEL4 to OL6]
Oracle Database - Enterprise Edition
Linux x86-64
Oracle Linux
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)

Purpose
This document aims to provide.

 Basic configuration of HugePages on 64-bit Linux

 Fundemental reasons to use HugePages on Linux
 References to known problems
 References to technical background on HugePages

Scope
Information in this document is useful for Linux system administrators and Oracle database
administrators working with system administrators.

This document covers information about Linux HugePages for 64-bit architectures. For more
generic and uses on 32-bit and for references please see Document 361323.1

The configuration steps provided here is primarily for Oracle Linux. Still the same concepts and
configurations should apply to other Linux distributions.

Details
Introduction
HugePages is a feature of the Linux kernel which allows larger pages to manage memory as the
alternative to the small 4KB pagesize. For a detailed introduction, see Document 361323.1

Why Do You Need HugePages?

HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM
and SGA. If your combined database SGAs is large (like more than 8GB, can even be important
for smaller), you will need HugePages configured. Note that the size of the SGA matters.
Advantages of HugePages are:

 Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB
size is 2048K. That means the system would need to handle 512 times less pages.
 Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual
address range than a regular sized page, a probability of getting a TLB hit per TLB entry
with HugePages are higher than with regular pages. This reduces the number of times
page tables are walked to obtain physical address from a virtual address.
 Less Overhead for Memory Operations: On virtual memory systems (any modern OS)
each memory operation is actually two abstract memory operations. With HugePages,
since there are less number of pages to work on, the possible bottleneck on page table
access is clearly avoided.
 Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux
kernel will use less memory to create pagetables to maintain virtual to physical mappings
for SGA address range, in comparison to regular size pages. This makes more memory to
be available for process-private computations or PGA usage.
 No Swapping: We must avoid swapping to happen on Linux OS at all Document
1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is
no page replacement mechanism overhead. HugePages are universally regarded as
pinned.
 No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be
paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible
amount of CPU resource. When HugePages are used, kswapd is not involved in
managing them. See also Document 361670.1

There is a general misconception where the HugePages is a feature specific to 32-bit

Linux. HugePages is a generic feature available to all word-sizes and architectures. Just that
there are some specifics with 32-bit platforms. Please see Document 361323.1 for further
references.

How to Configure

The configuration steps below will guide you to do a persistent system configuration where you
would need to do a complete reboot of the system. Please plan your operations accordingly:
Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB)
slightly smaller than installed RAM. e.g. If you have 64GB RAM installed, you may set:

* soft memlock 60397977

* hard memlock 60397977

There is no harm in setting this value larger than your SGA requirements.

The parameters will be set by default on:

 Oracle Linux with oracle-validated package (See Document 437743.1) installed.

 Oracle Exadata DB compute nodes

Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit

$ ulimit -l
60397977

Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic
Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM
before proceeding. To disable, set the initialization parameters MEMORY_TARGET and
MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for further information
in case you encounter the error below:

ORA-00845: MEMORY_TARGET not supported on this system

Step 4: Make sure that all your database instances are up (including ASM instances) as they
would run on production. Use the script hugepages_settings.sh in Document 401749.1 to
calculate the recommended value for the vm.nr_hugepages kernel parameter. e.g.:

$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
$

Note: You can also calculate a proper value for the parameter yourself but that is not advised if
you do not have extensive experience with HugePages and concepts.

Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:
...
vm.nr_hugepages = 1496
...

This will make the parameter to be set properly with each reboot.

Step 6: Stop all the database instances and reboot the server

(Although settings could have been done dynamically they would not be effective to the extent
we require before a reboot. The best practice is to do a persistent system configuration and
perform a reboot to complete the configuration as presented through the steps above)

What If the Database / SGA Configurations Change?

The performed configuration is basically based on the RAM installed and combined size of SGA
of database instances you are running. Based on that when:

 Amount of RAM installed for the Linux OS changed

 New database instance(s) introduced
 SGA size / configuration changed for one or more database instances

you should revise your HugePages configuration to make it suitable to the new memory
framework. If not you may experience one or more problems below on the system:

 Poor database performance

 System running out of memory or excessive swapping
 Database instances cannot be started
 Crucial system services failing

Check and Validate the Configuration

After the system is rebooted, make sure that your database instances (including the ASM
instances) are started. Automatic startup via OS configuration or CRS, or manual startup
(whichever method you use) should have been performed. Check the HugePages state from
/proc/meminfo. e.g.:

# grep HugePages /proc/meminfo

HugePages_Total: 1496
HugePages_Free: 485
HugePages_Rsvd: 446
HugePages_Surp: 0

The values in the output will vary. To make sure that the configuration is valid, the
HugePages_Free value should be smaller than HugePages_Total and there should be some
HugePages_Rsvd. HugePages_Rsvd counts free pages that are reserved for use (requested for an
SGA, but not touched/mapped yet).

The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your total combined
SGA as instances allocate pages dynamically and proactively as needed.

Troubleshooting

Some of the common problems and how to troubleshoot them are listed in the following table:

Symptom Possible Cause Troubleshooting Action

Not enough HugePages to cover the
Review your HugePages
System is running out of SGA(s) and therefore the area reserved
configuration to make sure
memory or swapping for HugePages are wasted where SGAs
that all SGA(s) are covered.
are allocated through regular pages.
Make sure the settings in
Databases fail to start memlock limits are not set properly limits.conf apply to database
owner account.
One of the database fail The SGA of the specific database Make sure that the RAM and
to start while another is could not find available HugePages HugePages are enough to
up and remaining RAM is not enough. cover all your database SGAs
Cluster Ready Services Make sure the total SGA is
HugePages configured too large
(CRS) fail to start less than the installed RAM
(maybe larger than installed RAM)
and re-calculate HugePages.
HugePages_Total = HugePages are not used at all. No Disable AMM and make sure
HugePages_Free database instances are up or using that the database instances are
AMM. up. See Doc ID 1373255.1
Database started The SGA of the specific database Make sure that the HugePages
successfully and the could not find available HugePages are many enough to cover all
performance is slow and therefore the SGA is handled by your database SGAs
regular pages, which leads to slow
performance