Red Hat Enterprise Linux 7

Kernel Crash Dump Guide

Kernel Crash Dump Configuration and Analysis

Jaromr Hradlek

Petr Boko

Red Hat Enterprise Linux 7 Kernel Crash Dump Guide

Kernel Crash Dump Configuration and Analysis

Jaro mr Hradlek
Red Hat Engineering Co ntent Services
[email protected] m
Petr Bo ko
Red Hat Engineering Co ntent Services
pbo ko [email protected] m

Legal Notice
Co pyright 20 14 Red Hat, Inc.
This do cument is licensed by Red Hat under the Creative Co mmo ns Attributio n-ShareAlike 3.0
Unpo rted License. If yo u distribute this do cument, o r a mo dified versio n o f it, yo u must pro vide
attributio n to Red Hat, Inc. and pro vide a link to the o riginal. If the do cument is mo dified, all Red
Hat trademarks must be remo ved.
Red Hat, as the licenso r o f this do cument, waives the right to enfo rce, and agrees no t to assert,
Sectio n 4 d o f CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shado wman lo go , JBo ss, MetaMatrix, Fedo ra, the Infinity
Lo go , and RHCE are trademarks o f Red Hat, Inc., registered in the United States and o ther
co untries.
Linux is the registered trademark o f Linus To rvalds in the United States and o ther co untries.
Java is a registered trademark o f Oracle and/o r its affiliates.
XFS is a trademark o f Silico n Graphics Internatio nal Co rp. o r its subsidiaries in the United
States and/o r o ther co untries.
MySQL is a registered trademark o f MySQL AB in the United States, the Euro pean Unio n and
o ther co untries.
No de.js is an o fficial trademark o f Jo yent. Red Hat So ftware Co llectio ns is no t fo rmally
related to o r endo rsed by the o fficial Jo yent No de.js o pen so urce o r co mmercial pro ject.
The OpenStack Wo rd Mark and OpenStack Lo go are either registered trademarks/service
marks o r trademarks/service marks o f the OpenStack Fo undatio n, in the United States and o ther
co untries and are used with the OpenStack Fo undatio n's permissio n. We are no t affiliated with,
endo rsed o r spo nso red by the OpenStack Fo undatio n, o r the OpenStack co mmunity.
All o ther trademarks are the pro perty o f their respective o wners.

Abstract
The Kernel Crash Dump Guide do cuments ho w to co nfigure, test, and use the kdump crash
reco very service o n Red Hat Enterprise Linux 7, and pro vides a brief o verview o f ho w to analyze
the resulting co re dump using the crash debugging utility. It is o riented to wards system
administrato rs with a basic understanding o f the Red Hat Enterprise Linux system.

T able of Cont ent s

T able of Contents
. .hapt
C
. . . .er
. .1. .. Int
. . .roduct
. . . . . .ion
. . .t.o. kdump
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. . . . . . . . . .
1.1. Ab o ut kd ump and kexec
2
1.2. Memo ry Req uirements
2
. .hapt
C
. . . .er
. .2. .. Inst
. . . .alling
. . . . .and
. . . .Configuring
. . . . . . . . . . .kdump
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. . . . . . . . . .
2 .1. Ins talling kd ump
4
2 .2. Co nfig uring kd ump at the Firs t Bo o t
4
2 .3. Co nfig uring kd ump o n the Co mmand Line
5
2 .4. Co nfig uring kd ump in the G rap hic al Us er Interfac e
8
2 .5. Tes ting the kd ump Co nfig uratio n
11
2 .6 . Ad d itio nal Res o urc es
12
. .hapt
C
. . . .er
. .3.
. .Analyz
. . . . . .ing
. . . a. .Core
. . . . Dump
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. 3. . . . . . . . . .
3 .1. Ins talling the c ras h Utility
13
3 .2. Running the c ras h Utility
13
3 .3. Dis p laying the Mes s ag e Buffer
14
3 .4. Dis p laying a Bac ktrac e
15
3 .5. Dis p laying a Pro c es s Status

16

3 .6 . Dis p laying Virtual Memo ry Info rmatio n

3 .7. Dis p laying O p en Files
3 .8 . Exiting the Utility

16
17
17

. . . . . . . . ly
Frequent
. . Asked
. . . . . .Q
. .uest
. . . .ions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 8. . . . . . . . . .
.Support
. . . . . . .ed
. . kdump
. . . . . . .Configurat
. . . . . . . . . ions
. . . . .and
...T
. .arget
....s
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 9. . . . . . . . . .
B .1. Memo ry Req uirements fo r kd ump
19
B .2. Minimum Thres ho ld fo r Auto matic Memo ry Res ervatio n
19
B .3. Sup p o rted kd ump Targ ets
B .4. Sup p o rted kd ump Filtering Levels

20
20

B .5. Sup p o rted Default Ac tio ns

21

. . . . . . . . .Hist
Revision
. . . ory
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. 2. . . . . . . . . .

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

Chapter 1. Introduction to kdump

1.1. About kdump and kexec
K d u mp is a kernel crash dumping mechanism that allows you to save the contents of the system's
memory for later analysis. It relies on kexec, which can be used to boot a Linux kernel from the
context of another kernel, bypass BIOS, and preserve the contents of the first kernel's memory that
would otherwise be lost.
In case of a system crash, kdump uses kexec to boot into a second kernel (a capture kernel). This
second kernel resides in a reserved part of the system memory that is inaccessible to the first kernel.
The second kernel then captures the contents of the crashed kernel's memory (a crash dump) and
saves it.

Important
UEFI-based systems are not supported in the current version of kexec. However, kd u mp itself
is still able to collect the crash dump and its reliability is not impacted.

1.2. Memory Requirement s

In order for kdump to be able to capture a kernel crash dump and save it for further analysis, a part
of the system memory has to be permanently reserved for the capture kernel. When reserved, this part
of the system memory is not available to main kernel.
The memory requirements vary based on certain system parameters. One of the major factors is the
system's hardware architecture. To find out the exact name of the machine architecture (such as
x86 _6 4 ) and print it to standard output, type the following command at a shell prompt:
uname -m
Another factor which influences the amount of memory to be reserved is the total amount of installed
system memory. For example, on the x86_64 architecture, the amount of reserved memory will be
160 MB + 2 bits for every 4 KB of RAM. On a system with 1 TB of total physical memory installed, this
means 224 MB (160 MB + 64 MB). For a complete list of memory requirements for kdump based on
the system architecture and the amount of physical memory, see Section B.1, Memory Requirements
for kdump .
On many systems, kdump can estimate the amount of required memory and reserve it automatically.
This behavior is enabled by default, but only works on systems that have more than a certain amount
of total available memory, which varies based on the system architecture. See Section B.2, Minimum
Threshold for Automatic Memory Reservation for a list of minimum requirements for automatic
memory reservation based on the system architecture.
If the system has less than the minimum amount of memory required for the automatic allocation to
work or if your use case requires a different value, you can configure the amount of reserved memory
manually. For information on how to do so on the command line, see Section 2.3.1, Configuring the
Memory Usage . For information on how to configure the amount of reserved memory in the graphical
user interface, see Section 2.4.1, Configuring the Memory Usage .

Chapt er 1 . Int roduct ion t o kdump

Important
It is highly recommended to test the configuration after setting up the kdump service, even
when using the automatic memory reservation. For instructions on how to test your
configuration, see Section 2.5, Testing the kdump Configuration .

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

Chapter 2. Installing and Configuring kdump

2.1. Inst alling kdump
In order use the kd ump service on your system, make sure you have the kexec-tools package
installed. To do so, type the following at a shell prompt as ro o t:
yum i nstal l kexec-to o l s
Additionally, if you wish to configure kd ump using a graphical user interface instead of the command
line, install the system-config-kdump utility using the following comand as ro o t:
yum i nstal l system-co nfi g -kd ump
For more information on how to install new packages in Red Hat Enterprise Linux, see the Red Hat
Enterprise Linux 7 System Administrator's Guide.

Important
A limitation in the current implementation of the Intel IO MMU driver can occasionally prevent
the kd ump service from capturing the core dump image. To use kdump on Intel architectures
reliably, it is advised that the IOMMU support is disabled.

2.2. Configuring kdump at t he First Boot

When the system boots for the first time, the f irst b o o t application is launched to guide the user
through the initial configuration of the freshly installed system. To configure kd ump, navigate to the
Kd ump section and follow the instructions below.

2.2.1. Configuring t he Memory Usage

By default, the amount of memory reserved for the kd ump kernel is determined automatically. If you
want to configure the amount of reserved memory manually, set the Kd ump Memo ry R eservati o n
option to Manual and then use the Memo ry T o Be R eserved (MB) field to set the amount of
reserved memory in megabytes. Notice that the Usabl e System Memo ry field changes accordingly,
showing you the remaining memory that will be available to the system.

2.2.2. Advanced kdump Configurat ion

The lower part of the kdump configuration screen contains a large text field labelled Ad vanced
kd ump co nfi g urati o n. This field shows the contents of the /etc/kd ump. co nf configuration
file, which allows you to configure advanced settings such as the target location, default action and
the core collector.
The first part of the file contains a short introduction and descriptions of the various possible
options. The second, shorter part contains a list of actual options, which are commented out by
default. If you want to change any of the options, remove the hash sign (" #" ) from the beginning of
that line and change the option to a desired value.

Chapt er 2 . Inst alling and Configuring kdump

See Section 2.3, Configuring kdump on the Command Line for more details on configuring kd u mp
using the /etc/kd ump. co nf configuration file.

2.2.3. Enabling t he kdump Service

To allow the kd ump daemon to start at boot time, select the Enabl e kd ump? checkbox. This will
enable the service for mul ti -user. targ et. Similarly, unselecting the checkbox will disable the
service.
After you finish making changes to the initial configuration, press the Fo rward button in the lower
right hand corner. Some changes in K d u mp settings, such as changing the amount of reserved
memory, will result in a prompt informing you that a reboot is be necessary before the service will be
able to capture kernel crash dumps.

2.3. Configuring kdump on t he Command Line

2.3.1. Configuring t he Memory Usage
To change the amount of memory reserved for kdump, it is necessary to edit the
/etc/d efaul t/g rub configuration file. Open this file in a text editor as ro o t and find a line
beginning with G R UB_C MD LINE_LINUX. On this line, you will find the following:
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 rd.lvm.lv=rhel/swap $([ -x
/usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :)
rd.luks=0 vconsole.keymap=us rd.lvm.lv=rhel/root rhgb quiet"
The highlighted part references a script (/usr/sbi n/rhcrashkernel -param) which automatically
generates the amount of reserved memory. If you wish to set the reserved memory to a fixed amount,
replace the entire highlighted string with a crashkernel = x parameter, where x is the amount of
memory you want to reserve for kdump. For example, to reserve 128 MB of memory, use the following:
crashkernel=128M
You can also set the amount of reserved memory to be variable, depending on the total amount of
installed memory. The syntax is for variable memory reservation is
crashkernel = <range1>: <size1>,<range2>: <size2>. For example:
crashkernel=512M-2G:64M,2G-:128M
The above example will reserve 64 MB of memory if the total amount of system memory is 512 MB or
higher and lower than 2 GB. If the total amount of memory is more than 2 GB, 128 MB will be reserved
for kdump instead.
On some systems, it might be necessary to reserve memory with a certain fixed offset. If the offset is
set, the reserved memory will begin there. To offset the reserved memory, use the following syntax:
crashkernel=128M@ 16M
The example above means that kdump will reserve 128 MB of memory starting at 16 MB (physical
address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump will offset the
reserved memory automatically. This syntax can also be used when setting a variable memory
reservation as described above; in this case, the offset is always specified last (for example,
crashkernel = 512M-2G : 6 4 M,2G -: 128M@ 16 M).

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

After entering the desired values and saving the file, update the actual GRUB2 configuration file. This
file's location depends on whether the system uses BIO S or UEFI. On systems using BIOS, use the
following command:
g rub2-mkco nfi g -o /bo o t/g rub2/g rub. cfg
On UEFI-based systems, use the following command instead:
g rub2-mkco nfi g -o /bo o t/efi /EFI/red hat/g rub. cfg
After the next system reboot, the reserved memory will be equal to the amount set in the
/etc/d efaul t/g rub configuration file.

2.3.2. Configuring t he kdump T ype

When a kernel crash is captured, the core dump can be either stored as a file in a local file system,
written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure
Shell) protocol. Only one of these options can be set at the moment, and the default option is to store
the vmco re file in the /var/crash/ directory of the local file system. To change this, as ro o t, open
the /etc/kd ump. co nf configuration file in a text editor and edit the options as described below.
To change the local directory in which the core dump is to be saved, remove the hash sign ( # ) from
the beginning of the #path /var/crash line, and replace the value with a desired directory path.
path /usr/local/cores
Optionally, if you wish to write the file to a different partition, follow the same procedure with the one
of the lines beginning with #ext4 . Here, you can use either a device name (the #ext4
/d ev/vg /l v_kd ump line), a file system label (the #ext4 LABEL= /bo o t line) or an UUID (the
#ext4 UUID = 0 3138356 -5e6 1-4 ab3-b58e-2750 7ac4 19 37 line). Change the file system type as
well as the device name, label or UUID to the desired values. For example:
ext4 /dev/sda3
To write the dump directly to a device, remove the hash sign ( # ) from the beginning of the #raw
/d ev/vg /l v_kd ump line, and replace the value with a desired device name. For example:
raw /dev/sdb1
To store the dump to a remote machine using the NFS protocol, remove the hash sign ( # ) from the
beginning of the #nfs my. server. co m: /expo rt/tmp line, and replace the value with a valid
hostname and directory path. For example:
nfs penguin.example.com:/export/cores
To store the dump to a remote machine using the SSH protocol, remove the hash sign ( # ) from the
beginning of the #ssh user@ my. server. co m line, and replace the value with a valid username
and hostname. To include your SSH key in the configuration as well, remove the hash sign ( # ) from
the beginning of the #sshkey /ro o t/. ssh/kd ump_i d _rsa line and change the value to the
location of a key valid on the server you are trying to dump to. For example:
ssh john@ penguin.example.com
sshkey /root/.ssh/mykey

Chapt er 2 . Inst alling and Configuring kdump

For information on how to configure an SSH server and set up a key-based authentication, see the
Red Hat Enterprise Linux 7 System Administrator's Guide.
For a complete list of currently supported and unsupported targets sorted by type, see Table B.3,
Supported kdump Targets .

2.3.3. Configuring t he Core Collect or

To reduce the size of the vmco re dump file, kd ump allows you to specify an external application (that
is, a core collector) to compress the data, and optionally leave out all irrelevant information.
Currently, the only fully supported core collector is maked umpfi l e.
To enable the core collector, as ro o t, open the /etc/kd ump. co nf configuration file in a text editor,
remove the hash sign ( # ) from the beginning of the #co re_co l l ecto r maked umpfi l e -c -messag e-l evel 1 -d 31 line, and edit the command line options as described below.
To enable the dump file compression, add the -c parameter. For example:
core_collector makedumpfile -c
To remove certain pages from the dump, add the -d value parameter, where value is a sum of
values of pages you want to omit as described in Table B.4, Supported Filtering Levels . For
example, to remove both zero and free pages, use the following:
core_collector makedumpfile -d 17 -c
See the manual page for maked umpfi l e for a complete list of available options.

2.3.4 . Configuring t he Default Act ion

By default, when kd ump fails to create a core dump at the target location specified in Section 2.3.2,
Configuring the kdump Type , the root file system is mounted and kd ump attempts to save the core
locally. To change this behavior, as ro o t, open the /etc/kd ump. co nf configuration file in a text
editor, remove the hash sign ( # ) from the beginning of the #d efaul t shel l line, and replace the
value with a desired action as described in Table B.5, Supported D efault Actions .
For example:
default reboot

2.3.5. Enabling t he Service

To start the kd ump daemon at boot time, type the following at a shell prompt as ro o t:
systemctl enabl e kd ump. servi ce
This will enable the service for mul ti -user. targ et. Similarly, typing systemctl sto p kd ump will
disable it. To start the service in the current session, use the following command as ro o t:
systemctl start kd ump. servi ce
For more information on systemd and configuring services in general, see the Red Hat
Enterprise Linux 7 System Administrator's Guide.

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

2.4 . Configuring kdump in t he Graphical User Int erface

To start the K ern el D u mp C o n f ig u rat io n utility, select Act ivit ies O t h er K ern el crash
d u mp s from the panel, or type system-co nfi g -kd ump at a shell prompt. You will be presented
with a window as shown in Figure 2.1, Basic Settings .
The utility allows you to configure kd ump as well as to enable or disable starting the service at boot
time. When you are done, click Appl y to save the changes. Unless you are already authenticated,
you will be prompted to enter the superuser password. The utility will also remind you that you must
reboot the system in order to apply any changes you have made to the configuration.

2.4 .1. Configuring t he Memory Usage

The Basi c Setti ng s tab enables you to configure the amount of memory that is reserved for the
kd ump kernel. To do so, select the Manual setti ng s radio button, and click the up and down
arrow buttons next to the New kd ump Memo ry field to increase or decrease the amount of memory to
be reserved. Notice that the Usabl e Memo ry field changes accordingly showing you the remaining
memory that will be available to the system. See Section 1.2, Memory Requirements for more
information on kdump's memory requirements.

Fig u re 2.1. B asic Set t in g s

2.4 .2. Configuring t he kdump T ype

The T arg et Setti ng s tab allows you to specify the target location for the vmco re dump. The
dump can be either stored as a file in a local file system, written directly to a device, or sent over a
network using the NFS (Network File System) or SSH (Secure Shell) protocol.

Chapt er 2 . Inst alling and Configuring kdump

Fig u re 2.2. T arg et Set t in g s

To save the dump to the local file system, select the Lo cal fi l esystem radio button. Optionally,
you can customize the settings by choosing a different partition from the P arti ti o n drop-down list
and a target directory using the P ath field.
To write the dump directly to a device, select the R aw d evi ce radio button, and choose the desired
target device from the drop-down list next to it.
To send the dump to a remote machine over a network connection, select the Netwo rk radio button.
To use the NFS protocol, select the NFS radio button, and fill the Server name and P ath to
d i recto ry fields. To use the SSH protocol, select the SSH radio button, and fill the Server name,
P ath to d i recto ry, and User name fields with the remote server address, target directory, and a
valid user name respectively.
For information on how to configure an SSH server and set up a key-based authentication, see the
Red Hat Enterprise Linux 7 System Administrator's Guide. For a complete list of currently supported
targets, see Table B.3, Supported kdump Targets .

2.4 .3. Configuring t he Core Collect or

The Fi l teri ng Setti ng s tab enables you to select the filtering level for the vmco re dump.

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

Fig u re 2.3. Filt erin g Set t in g s

To exclude the zero pag e, cache pag e, cache pri vate, user d ata, or free pag e from the
dump, select the checkbox next to the appropriate label.

2.4 .4 . Configuring t he Default Act ion

To choose what action to perform when kd ump fails to create a core dump, select an appropriate
option from the D efaul t acti o n drop-down list. Available options are d u mp t o ro o t f s an d
reb o o t (the default action which attempts to save the core locally and then reboots the system),
reb o o t (to reboot the system), sh ell (to present a user with an interactive shell prompt), h alt (to halt
the system), and p o wero f f (to power the system off).

10

Chapt er 2 . Inst alling and Configuring kdump

Fig u re 2.4 . Filt erin g Set t in g s

To customize the options that are passed to the maked umpfi l e core collector, edit the C o re
co l l ecto r text field; see Section 2.3.3, Configuring the Core Collector for more information.

2.4 .5. Enabling t he Service

To start the kd ump service at boot time, click the Enabl e button on the toolbar and then click the
Appl y button. This will enable and activate the service for mul ti -user. targ et. Clicking the
D i sabl e button and confirming by clicking the Appl y button will disable the service immediately.
For more information on systemd targets and configuring services in general, see the Red Hat
Enterprise Linux 7 System Administrator's Guide.

2.5. T est ing t he kdump Configurat ion

Warning
The commands below will cause the kernel to crash. Use caution when following these steps,
and by no means use them on a production system.
To test the configuration, reboot the system with kd ump enabled, and make sure that the service is
running:
~]# systemctl i s-acti ve kd ump
active
Then type the following commands at a shell prompt:

11

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

echo 1 > /pro c/sys/kernel /sysrq

echo c > /pro c/sysrq -tri g g er
This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmco re file will
be copied to the location you have selected in the configuration (that is, to /var/crash/ by default).

2.6. Addit ional Resources

2.6.1. Inst alled Document at ion
kd u mp .co n f (5) a manual page for the /etc/kd ump. co nf configuration file containing the
full documentation of available options.
maked u mp f ile(8) a manual page for the maked umpfi l e core collector.
kexec(8) a manual page for kexec.
crash (8) a manual page for the crash utility.
/usr/share/d o c/kexec-to o l s-version/kexec-kd ump-ho wto . txt an overview of the
kd ump and kexec installation and usage.

2.6.2. Online Document at ion

h t t p s://access.red h at .co m/sit e/so lu t io n s/6 038
The Red Hat Knowledgebase article about the kexec and kd ump configuration.
h t t p s://access.red h at .co m/sit e/so lu t io n s/223773
The Red Hat Knowledgebase article about supported kd ump targets.
h t t p ://p eo p le.red h at .co m/an d erso n /
The crash utility homepage.

2.6.3. See Also

stuff

12

Chapt er 3. Analyz ing a Core Dump

Chapter 3. Analyzing a Core Dump

To determine the cause of the system crash, you can use the crash utility, which provides an
interactive prompt very similar to the GNU D ebugger (GD B). This utility allows you to interactively
analyze a running Linux system as well as a core dump created by netd ump, d i skd ump, xend ump,
or kd ump.

3.1. Inst alling t he crash Ut ilit y

To install the crash analyzing tool, execute the following command from a shell prompt as ro o t:
yum i nstal l crash
In addition to crash , it is also necessary to install the kernel-debuginfo package, which provides the
data necessary for dump analysis. To install this package, you will first need to enable the relevant
repository. To do this, execute the following command from the command line as ro o t:
yum --enabl erepo = \*d ebug i nfo
After enabling the repository, install the kernel-debuginfo package by executing the following
command as ro o t:
d ebug i nfo -i nstal l kernel
For more information on how to install new packages in Red Hat Enterprise Linux, see the Red Hat
Enterprise Linux 7 System Administrator's Guide.

3.2. Running t he crash Ut ilit y

To start the utility, type the command in the following form at a shell prompt:
crash /var/crash/<timestamp>/vmco re
/usr/l i b/d ebug /l i b/mo d ul es/<kernel>/vml i nux
Note that the <kernel> version should be the same that was captured by kd ump. To find out which
kernel you are currently running, use the uname -r command.

Examp le 3.1. R u n n in g t h e crash u t ilit y

~]# crash /usr/l i b/d ebug /l i b/mo d ul es/2. 6 . 32-6 9 . el 6 . i 6 86 /vml i nux \
/var/crash/127. 0 . 0 . 1-20 10 -0 8-25-0 8: 4 5: 0 2/vmco re
crash 5.0.0-23.el6
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.

13

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

This program is free software, covered by the GNU General Public

License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL:
DUMPFILE:
DUMP]
CPUS:
DATE:
UPTIME:
LOAD AVERAGE:
TASKS:
NODENAME:
RELEASE:
VERSION:
MACHINE:
MEMORY:
PANIC:
PID:
COMMAND:
TASK:
CPU:
STATE:

/usr/lib/debug/lib/modules/2.6.32-69.el6.i686/vmlinux
/var/crash/127.0.0.1-2010-08-25-08:45:02/vmcore [PARTIAL
4
Wed Aug 25 08:44:47 2010
00:09:02
0.00, 0.01, 0.00
140
hp-dl320g5-02.lab.bos.redhat.com
2.6.32-69.el6.i686
#1 SMP Tue Aug 24 10:31:45 EDT 2010
i686 (2394 Mhz)
8 GB
"Oops: 0002 [#1] SMP " (check log for details)
5591
"bash"
f196d560 [THREAD_INFO: ef4da000]
2
TASK_RUNNING (PANIC)

crash>

3.3. Displaying t he Message Buffer

To display the kernel message buffer, type the l o g command at the interactive prompt.

Examp le 3.2. D isp layin g t h e kern el messag e b u f f er

crash> l o g
... several lines omitted ...
EIP: 0060:[<c068124f>] EFLAGS: 00010096 CPU: 2
EIP is at sysrq_handle_crash+0xf/0x20
EAX: 00000063 EBX: 00000063 ECX: c09e1c8c EDX: 00000000
ESI: c0a09ca0 EDI: 00000286 EBP: 00000000 ESP: ef4dbf24
DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process bash (pid: 5591, ti=ef4da000 task=f196d560 task.ti=ef4da000)

14

Chapt er 3. Analyz ing a Core Dump

Stack:
c068146b c0960891 c0968653 00000003 00000000 00000002 efade5c0
c06814d0
<0> fffffffb c068150f b7776000 f2600c40 c0569ec4 ef4dbf9c 00000002
b7776000
<0> efade5c0 00000002 b7776000 c0569e60 c051de50 ef4dbf9c f196d560
ef4dbfb4
Call Trace:
[<c068146b>] ? __handle_sysrq+0xfb/0x160
[<c06814d0>] ? write_sysrq_trigger+0x0/0x50
[<c068150f>] ? write_sysrq_trigger+0x3f/0x50
[<c0569ec4>] ? proc_reg_write+0x64/0xa0
[<c0569e60>] ? proc_reg_write+0x0/0xa0
[<c051de50>] ? vfs_write+0xa0/0x190
[<c051e8d1>] ? sys_write+0x41/0x70
[<c0409adc>] ? syscall_call+0x7/0xb
Code: a0 c0 01 0f b6 41 03 19 d2 f7 d2 83 e2 03 83 e0 cf c1 e2 04 09 d0
88 41 03 f3 c3 90 c7 05 c8 1b 9e c0 01 00 00 00 0f ae f8 89 f6 <c6> 05
00 00 00 00 01 c3 89 f6 8d bc 27 00 00 00 00 8d 50 d0 83
EIP: [<c068124f>] sysrq_handle_crash+0xf/0x20 SS:ESP 0068:ef4dbf24
CR2: 0000000000000000

Type hel p l o g for more information on the command usage.

3.4 . Displaying a Backt race

To display the kernel stack trace, type the bt command at the interactive prompt. You can use bt
<pid> to display the backtrace of a single process.

Examp le 3.3. D isp layin g t h e kern el st ack t race

crash> bt
PID: 5591
TASK: f196d560 CPU: 2
COMMAND: "bash"
#0 [ef4dbdcc] crash_kexec at c0494922
#1 [ef4dbe20] oops_end at c080e402
#2 [ef4dbe34] no_context at c043089d
#3 [ef4dbe58] bad_area at c0430b26
#4 [ef4dbe6c] do_page_fault at c080fb9b
#5 [ef4dbee4] error_code (via page_fault) at c080d809
EAX: 00000063 EBX: 00000063 ECX: c09e1c8c EDX: 00000000 EBP:
00000000
DS: 007b
ESI: c0a09ca0 ES: 007b
EDI: 00000286 GS:
00e0
CS: 0060
EIP: c068124f ERR: ffffffff EFLAGS: 00010096
#6 [ef4dbf18] sysrq_handle_crash at c068124f
#7 [ef4dbf24] __handle_sysrq at c0681469
#8 [ef4dbf48] write_sysrq_trigger at c068150a
#9 [ef4dbf54] proc_reg_write at c0569ec2
#10 [ef4dbf74] vfs_write at c051de4e
#11 [ef4dbf94] sys_write at c051e8cc
#12 [ef4dbfb0] system_call at c0409ad5

15

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

EAX:
DS:
SS:
CS:

ffffffda
007b
007b
0073

EBX: 00000001 ECX: b7776000 EDX: 00000002

ESI: 00000002 ES: 007b
EDI: b7776000
ESP: bfcb2088 EBP: bfcb20b4 GS: 0033
EIP: 00edc416 ERR: 00000004 EFLAGS: 00000246

Type hel p bt for more information on the command usage.

3.5. Displaying a Process St at us

To display status of processes in the system, type the ps command at the interactive prompt. You can
use ps <pid> to display the status of a single process.

Examp le 3.4 . D isp layin g t h e st at u s o f p ro cesses in t h e syst em

crash> ps
PID
PPID CPU
TASK
>
0
0
0 c09dc560
>
0
0
1 f7072030
0
0
2 f70a3a90
>
0
0
3 f70ac560
1
0
1 f705ba90
... several lines omitted ...
5566
1
1 f2592560
5567
1
2 ef427560
5587
5132
0 f196d030
> 5591
5587
2 f196d560

ST
RU
RU
RU
RU
IN

%MEM
0.0
0.0
0.0
0.0
0.0

IN
IN
IN
RU

0.0
0.0
0.0
0.0

VSZ
0
0
0
0
2828
12876
12876
11064
5084

RSS COMM
0 [swapper]
0 [swapper]
0 [swapper]
0 [swapper]
1424 init
784
784
3184
1648

auditd
auditd
sshd
bash

Type hel p ps for more information on the command usage.

3.6. Displaying Virt ual Memory Informat ion

To display basic virtual memory information, type the vm command at the interactive prompt. You can
use vm <pid> to display information on a single process.

Examp le 3.5. D isp layin g virt u al memo ry in f o rmat io n o f t h e cu rren t co n t ext

crash> vm
PID: 5591
TASK: f196d560 CPU: 2
MM
PGD
RSS
TOTAL_VM
f19b5900 ef9c6000 1648k
5084k
VMA
START
END
FLAGS
f1bb0310
242000
260000 8000875
f26af0b8
260000
261000 8100871
efbc275c
261000
262000 8100873
efbc2a18
268000
3ed000 8000075
efbc23d8
3ed000
3ee000 8000070
efbc2888
3ee000
3f0000 8100071
efbc2cd4
3f0000
3f1000 8100073
efbc243c
3f1000
3f4000 100073
efbc28ec
3f6000
3f9000 8000075

16

COMMAND: "bash"

FILE
/lib/ld-2.12.so
/lib/ld-2.12.so
/lib/ld-2.12.so
/lib/libc-2.12.so
/lib/libc-2.12.so
/lib/libc-2.12.so
/lib/libc-2.12.so
/lib/libdl-2.12.so

Chapt er 3. Analyz ing a Core Dump

efbc2568
3f9000
3fa000 8100071 /lib/libdl-2.12.so
efbc2f2c
3fa000
3fb000 8100073 /lib/libdl-2.12.so
f26af888
7e6000
7fc000 8000075 /lib/libtinfo.so.5.7
f26aff2c
7fc000
7ff000 8100073 /lib/libtinfo.so.5.7
efbc211c
d83000
d8f000 8000075 /lib/libnss_files-2.12.so
efbc2504
d8f000
d90000 8100071 /lib/libnss_files-2.12.so
efbc2950
d90000
d91000 8100073 /lib/libnss_files-2.12.so
f26afe00
edc000
edd000 4040075
f1bb0a18
8047000
8118000 8001875 /bin/bash
f1bb01e4
8118000
811d000 8101873 /bin/bash
f1bb0c70
811d000
8122000 100073
f26afae0
9fd9000
9ffa000 100073
... several lines omitted ...

Type hel p vm for more information on the command usage.

3.7. Displaying Open Files

To display information about open files, type the fi l es command at the interactive prompt. You can
use fi l es <pid> to display files opened by only one selected process.

Examp le 3.6 . D isp layin g in f o rmat io n ab o u t o p en f iles o f t h e cu rren t co n t ext

crash> fi l es
PID: 5591
TASK: f196d560 CPU: 2
ROOT: /
CWD: /root
FD
FILE
DENTRY
INODE
0 f734f640 eedc2c6c eecd6048
1 efade5c0 eee14090 f00431d4
2 f734f640 eedc2c6c eecd6048
10 f734f640 eedc2c6c eecd6048
255 f734f640 eedc2c6c eecd6048

COMMAND: "bash"
TYPE PATH
CHR
/pts/0
REG
/proc/sysrq-trigger
CHR
/pts/0
CHR
/pts/0
CHR
/pts/0

Type hel p fi l es for more information on the command usage.

3.8. Exit ing t he Ut ilit y

To exit the interactive prompt and terminate crash , type exi t or q .

Examp le 3.7. Exit in g t h e crash u t ilit y

crash> exi t
~]#

17

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

Frequently Asked Questions

Q:
H o w d o I u p lo ad a larg e d u mp f ile t o R ed H at Su p p o rt Services?
A:

In some cases, it might be necessary to send a kernel crash dump file to Red Hat Global
Support Services for analysis. However, the dump file can be very large, even after being
filtered. Since files larger than 250 MB cannot be uploaded directly through the Red Hat
Customer Portal when opening a new support case, an FTP server is provided by Red Hat for
uploading large files.
The FTP server's address is d ro pbo x. red hat. co m and the files are to be uploaded in the
/i nco mi ng / directory. Your FTP client needs to be set into passive mode; if your firewall does
not allow this mode, you may use the o ri g i n-d ro pbo x. red hat. co m server using active
mode.
Make sure that the uploaded files are compressed using a program such as g z ip and properly
and descriptively named. Using your support case number in the file name is recommended.
After successfuly uploading all necessary files, provide the engineer in charge of your support
case with the exact file name and its SHA1 or MD 5 checksum.
For more specific instructions and additional information, see
https://access.redhat.com/site/solutions/2112.

18

Support ed kdump Configurat ions and T arget s

Supported kdump Configurations and Targets

B.1. Memory Requirement s for kdump
In order for kdump to be able to capture a kernel crash dump and save it for further analysis, a part
of the system memory has to be permanently reserved for the capture kernel. The table below
contains a list of minimum memory requirements for kdump based on the system's architecture and
total available physical memory.
For information on how to change memory settings on the command line, see Section 2.3.1,
Configuring the Memory Usage . For instructions on how to set up the amount of reserved memory
in the graphical user interface, see Section 2.4.1, Configuring the Memory Usage .
T ab le B .1. Min imu m Amo u n t o f R eserved Memo ry R eq u ired f o r kd u mp
Arch it ect u re

Availab le
Memo ry

Min imu m R eserved Memo ry

AMD 64 and Intel 64

(x86 _6 4 )

2 GB and more

IBM POWER (ppc6 4 )

2 GB to 4 GB
4 GB to 32 GB
32 GB to 64 GB
64 GB to 128 GB
128 GB and more
2 GB and more

160 MB + 2 bits for every 4 KB of RAM. For a

system with 1 TB of memory, 224 MB is the
minimum (160 + 64 MB).
256 MB of RAM.
512 MB of RAM.
1 GB of RAM.
2 GB or RAM.
4 GB of RAM.
160 MB + 2 bits for every 4 KB of RAM. For a
system with 1 TB of memory, 224 MB is the
minimum (160 + 64 MB).

IBM System z (s39 0 x)

B.2. Minimum T hreshold for Aut omat ic Memory Reservat ion

On some systems, it is possible to allocate memory for kdump automatically, either by using the
crashkernel = auto parameter in the bootloader's configuration file, or by enabling this option in
the graphical configuration utility. For this automatic reservation to work, however, a certain amount
of total memory needs to be available in the system. This amount differs based on the system's
architecture.
The table below lists the thresholds for automatic memory allocation. If the system has less memory
than specified in the table, memory will have to be reserved manually.
For information on how to change these settings on the command line, see Section 2.3.1,
Configuring the Memory Usage . For instructions on how to change the amount of reserved memory
in the graphical user interface, see Section 2.4.1, Configuring the Memory Usage .
T ab le B .2. Min imu m Amo u n t o f Memo ry R eq u ired f o r Au t o mat ic Memo ry R eservat io n
Arch it ect u re

R eq u ired Memo ry

AMD 64 and Intel 64

(x86 _6 4 )
IBM POWER (ppc6 4 )
IBM System z (s39 0 x)

2 GB
2 GB
4 GB

19

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

B.3. Support ed kdump T arget s

When a kernel crash is captured, the core dump can be either written directly to a device, stored as a
file on a local file system, or sent over a network. The table below contains a complete list of dump
targets that are currently supported or explicitly unsupported by kdump.
For information on how to configure the target type on the command line, see Section 2.3.2,
Configuring the kdump Type . For information on how to do so in the graphical user interface, see
Section 2.4.2, Configuring the kdump Type .
T ab le B .3. Su p p o rt ed kd u mp T arg et s
T yp e

Su p p o rt ed T arg et s

U n su p p o rt ed T arg et s

Raw device

All locally attached raw disks and

partitions.
ext2, ext3, ext4 , mi ni x, btrfs and
xfs file systems on directly attached
disk drives, hardware RAID logical
drives, LVM devices, and md rai d
arrays.
Remote directories accessed using the
NFS or SSH protocol over IP v4 .

Local file system

Remote directory

Any local file system not explicitly

listed as supported in this table,
including the auto type (automatic file
system detection).

Remote directories on the ro o tfs file

system accessed using the NFS
protocol.
Remote directories accessed using the Remote directories accessed using the
i SC SI protocol over software
i SC SI protocol using i BFT .
initiators, unless i BFT (iSCSI Boot
Firmware Table) is utilized.
Multipath-based storages.
Remote directories accessed using the
i SC SI protocol over hardware
initiators.

Remote directories accessed over

IP v6 .
Remote directories accessed using the
SMB/C IFS protocol.
Remote directories accessed using the
FC o E (Fibre Channel over Ethernet)
protocol.
Remote directories accessed using
wireless network interfaces.

B.4 . Support ed kdump Filt ering Levels

To reduce the size of the dump file, kdump uses the maked umpfi l e core collector to compress the
data and optionally leave out irrelevant information. The table below contains a complete list of
filtering levels that are currently supported by the maked umpfi l e utility.
For instructions on how to configure the core collector on the command line, see Section 2.3.3,
Configuring the Core Collector . For information on how to do so in the graphical user interface, see
Section 2.4.3, Configuring the Core Collector .
T ab le B .4 . Su p p o rt ed Filt erin g Levels

20

Support ed kdump Configurat ions and T arget s

O p t io n

D escrip t io n

1
2
4
8
16

Z ero pages
Cache pages
Cache private
User pages
Free pages

B.5. Support ed Default Act ions

By default, when kdump fails to create a core dump, it mounts the root file system and attempts to
save the core locally. You can, however, configure kdump to perform a different operation in case it
fails to save the core dump to the primary target. The table below lists all default actions that are
currently supported by kdump.
For detailed information on how to set up the default action on the command line, see Section 2.3.4,
Configuring the D efault Action . For information on how to do so in the graphical user interface, see
Section 2.4.4, Configuring the D efault Action .
T ab le B .5. Su p p o rt ed D ef au lt Act io n s
O p t io n

D escrip t io n

d ump_to _ro o tf
s

Attempt to save the core dump to the root file system. This option is especially
useful in combination with a network target: if the network target is
unreachable, this option configures kdump to save the core dump locally. The
system is rebooted afterwards.
Reboot the system, losing the core dump in the process.
Halt the system, losing the core dump in the process.
Power off the system, losing the core dump in the process.
Run a shell session from within the initramfs, allowing the user to record the
core dump manually.

rebo o t
hal t
po wero ff
shel l

21

Red Hat Ent erprise Linux 7 Kernel Crash Dump G uide

Revision History
R evisio n 1.0- 2
Fri 05 D ec 2014
Pet r B o ko
Updating to implement new sort_order on the RHEL 7 splash page
R evisio n 1.0- 0
Mo n 02 Ju n 2014
Jaro mr H rad lek
Red Hat Enterprise Linux 7.0 GA release of the Kernel Crash D ump Guide.
R evisio n 0.0- 8
Initial creation of the book.

22

T h u Jan 17 2013

Jaro mr H rad lek