Linux Kernel Programming

Linux Kernel programming

1
 Agenda
• Configuring kernel
• Boot Sequence
• Debugging

2
 Development Environment
 When doing embedded development, there is always a split between
 The host, the development workstation, which is typically a powerful PC
 The target, which is the embedded system under development
 They are connected by various means: almost always a serial line for debugging purposes, frequently an
Ethernet connection, sometimes a JTAG interface for low-level debugging

3
 Development Environment
 An essential tool for embedded development is a serial line communication program, like HyperTerminal in
Windows.

 There are multiple options available in Linux: Minicom, Picocom, Gtkterm, Putty, screen and the new tio (
https://github.com/tio/tio).

 The simplest of them: Picocom

Installation with sudo apt install picocom
Run with picocom -b BAUD_RATE /dev/SERIAL_DEVICE.
Exit with [Ctrl][a] [Ctrl][x]

 SERIAL_DEVICE is typically
ttyUSBx for USB to serial converters
ttySx for real serial ports

 Most frequent command: picocom -b 115200 /dev/ttyUSB0 4

Development Environment

5
Development Environment

6
 Development Environment
 Three machines must be distinguished when discussing toolchain creation
 The build machine, where the toolchain is built.
 The host machine, where the toolchain will be executed.
 The target machine, where the binaries created by the toolchain are executed.

7
Development Environment

8
 Building UML Kernel Image
 Downloading the Kernel and extract
www.kernel.org
# tar -xvzf linux-5.1.16.tar.gz

 Configuring the Kernel

# make ARCH=um menuconfig

 Build Kernel
# make ARCH=um

 Boot system
# ./bin/linux rootfstype=hostfs rootflags=/home/test/uml/rootfs/ rw mem=64M init=/bin/sh

9
 Debugging UML Kernel Image
 The UML kernel can be debugged by using gdb similar to normall application
# ./linux-5.1.16/vmlinux

Setup breakpoint at start_kernel

(gdb) break start_kernel

Start linux kernel with commandline options

(gdb) r rootfstype=hostfs rootflags=/home/test/uml/rootfs/ rw mem=64M init=/bin/sh

Now kernel stops at start_kernel. Later we can setup breakpoint at desired places and debug.

10
 Building Qemu Kernel Image
 Downloading the Kernel and extract
www.kernel.org
# tar -xvzf linux-5.1.16.tar.gz
# cd linux-5.1.16/

 copy configuration file

# cp arch/arm/configs/vexpress_defconfig .config

 configure kernel
# make menuconfig ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

 Build kernel
# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

11
 Building Qemu Kernel Image
 Build kernel modules
# make ARCH=arm CROSS_COMPILE=CROSS_COMPILE=arm-linux-gnueabihf- modules

 Boot kernel image with GUI

# qemu-system-arm -M vexpress-a9 -m 512M -dtb linux-5.1.16/arch/arm/boot/dts/vexpress-v2p-
ca9.dtb -kernel linux-5.1.16/arch/arm/boot/zImage -initrd rootfs.img.gz -append "root=/dev/ram
rdinit=/linuxrc"

 Boot kernel image without GUI

# qemu-system-arm -M vexpress-a9 -m 512M -dtb linux-5.1.16/arch/arm/boot/dts/vexpress-v2p-
ca9.dtb -nographic -kernel linux-5.1.16/arch/arm/boot/zImage -initrd rootfs.img.gz -append
"root=/dev/ram console=ttyAMA0 rdinit=/linuxrc"

12
 Debugging Qemu Kernel Image
 Start qemu kernel with –S –s option this would enable to start gdbserver and on localhost port number
1234
# qemu-system-arm –S –s -M vexpress-a9 -m 512M -dtb linux-5.1.16/arch/arm/boot/dts/vexpress-
v2p-ca9.dtb -kernel linux-5.1.16/arch/arm/boot/zImage -initrd rootfs.img.gz -append "root=/dev/ram
rdinit=/linuxrc"

 Now start gdb-multiarch with arm linux kernel

#gdb-multiarch ./linux-5.1.16/vmlinux

 Setup breakpoint at start_kernel

(gdb)start_kernel

 Connect to gdbserver start debugging either by single stepping or continue if you want to continue.
(gdb) target remote :1234
13
 Building Kernel Image with inbuilt rootfs
 Downloading the Kernel and extract
www.kernel.org
# tar -xvzf linux-5.1.16.tar.gz
# cd linux-5.1.16/

 copy configuration file

# cp arch/arm/configs/vexpress_defconfig .config

 configure kernel
# make menuconfig ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
select path to CONFIG_INITRAMFS_SOURCE . The initramfs should be created using the command
# find . | cpio -o -H newc | gzip > ../initramfs_data.cpio.gz
 Build kernel
# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
14
Setting up Raspberry pi Board

15
Setting up Raspberry pi Board

16
 Setting up Raspberry pi Board – SD Card Setup
 On a Windows PC, the best way to burn the image to your SD card is to use the Win32 Disk Imager
utility. This can be downloaded from http://sourceforge.net/projects/win32diskimager.

 It doesn't have an installer, and launches directly from the EXE file. Now, it's time to create your SD
card image:
 Insert your SD card into the PC and launch the Win32 Disk Imager.
 Select the SD card device drive letter (make sure it's right!).
 Choose the Raspbian image file you've just downloaded.
 Click on the Write button to create the SD card image.

17
 Setting up Raspberry pi Board – SD Card Setup
 On a Linux PC, you'll need to use the gparted and dd utilities to burn the image on your SD card. Carry
out the following steps to create your SD card image:
 Extract 2015-09-24-raspbian-jessie.img to your Home folder.
 Insert your SD card into the PC.
 If you're not already in a shell terminal window, open one (you can use Ctrl + Alt + T on most
graphical-based desktop systems).
 Type the following command in the shell terminal:
$ sudo fdisk -l
 In the list check, your SD card appears as a drive device (for example, /dev/sdb). It's crucial that you
ensure you use the right device in the next step. We'll assume that your device is /sdb.
 To burn the image to the SD card, type the following command:
$ sudo dd if=2015-09-24-raspbian-jessie.img of=/dev/sdb
 Hit Enter and go make a cup of tea or coffee as this will take a while. You'll know that it's finished
when the command ($) prompt re-appears.

18
 Setting up Raspberry pi Board – SD Card Setup
 When the command prompt does re-appear, type the following command:
$ sudo sync
 Once that command has finished, you can remove the SD card from the PC.

19
Setting up Raspberry pi Board – SD Card Setup

20
Setting up Raspberry pi Board – SD Card Setup

21
Setting up Raspberry pi Board

22
Setting up Raspberry pi Board

23
 Setting up Raspberry pi Board – Enabling Serial Console
You can enable/disable the serial console with either editing /boot/config.txt or raspi-config (which will
edit /boot/config.txt for you)

 Option 1. Enabling in /boot/config.txt

You can pop your SD card into a computer and edit config.txt with a text editor like SimpleText, WordPad
or whatnot. You can also edit on a pi with sudo nano /boot/config.txt

At the bottom, last line, add enable_uart=1 on it's own line

24
 Setting up Raspberry pi Board – Enabling Serial Console
 Option 2. Enabling via Raspi-Config
Using a monitor and keyboard, log into the shell and run

# sudo raspi-config

go down to Advanced Options

25
 Setting up Raspberry pi Board – Enabling Serial Console
Hit enter and then go down to Serial

26
 Setting up Raspberry pi Board – Enabling Serial Console
Select Yes

27
 Setting up Raspberry pi Board – Enabling Serial Console
It should now be enabled

28
 Setting up Raspberry pi Board – Enabling Serial Console
Hit return then select Finish

When it asks you to reboot, go to Yes and hit return

OK the serial console is now enabled!

29
 Setting up Raspberry pi Board – Enabling Serial Console
https://www.raspberrypi.com/documentation/computers/config_txt.html

30
 Building Raspberry pi Kernel Image
 Install dependency
# sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev

 Install 32-bt toolchain for 32 bit kernel

# sudo apt install crossbuild-essential-armhf

 Get kernel sources

# git clone --depth=1 https://github.com/raspberrypi/linux

 Configure kernel
# cd linux
# KERNEL=kernel7
# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig
# make menuconfig ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
31
 Building Raspberry pi Kernel Image
 Build kernel
# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs

 Setup SDcard
sdb
sdb1
sdb2

# mkdir mnt
# mkdir mnt/fat32
# mkdir mnt/ext4
# sudo mount /dev/sdb1 mnt/fat32
# sudo mount /dev/sdb2 mnt/ext4

32
 Building Raspberry pi Kernel Image
 Install Kernel modules
# sudo env PATH=$PATH make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
INSTALL_MOD_PATH=mnt/ext4 modules_install

 Copy kernel and dtbs

# sudo cp mnt/fat32/$KERNEL.img mnt/fat32/$KERNEL-backup.img
# sudo cp arch/arm/boot/zImage mnt/fat32/$KERNEL.img
# sudo cp arch/arm/boot/dts/*.dtb mnt/fat32/
# sudo cp arch/arm/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
# sudo cp arch/arm/boot/dts/overlays/README mnt/fat32/overlays/
# sudo umount mnt/fat32
# sudo umount mnt/ext4

33
OpenOCD Debugger
OpenOCD Debugger
OpenOCD Debugger
 OpenOCD Configuration
/usr/local/share/openocd/scripts/
Raspberry for Debugging
 Raspberry for Debugging
 Use the revision table to look up the board revision. In this tutorial we will use Model B v2.0
(revision 0xe), however the same schematics also works for Raspberry Pi 3.
 In order to find which pins on the board are JTAG pins we need to first find the GPIO
numbers that correspond to them. JTAG requires 4 pins: TDI, TDO, TMS and TCK plus
TRST and RTCK pins. Get the peripheral datasheet for BCM2835, the microcontroller used
by the Raspberry PI board. Search for “_TCK” to find the GPIO number table:
Raspberry for Debugging
Raspberry for Debugging
 Enable JTAG on Raspberry pi
enable_jtag_gpio
Setting enable_jtag_gpio=1 selects Alt4 mode for GPIO pins 22-27, and sets up some internal SoC
connections, thus enabling the JTAG interface for the ARM CPU. It works on all models of Raspberry Pi.

42
 Raspberry for Debugging
# sudo openocd -f interface/ftdi/olimex-arm-usb-ocd-h.cfg -f board/rpi3.cfg

# sudo openocd -f interface/ftdi/olimex-arm-usb-ocd-h.cfg -f target/at91sam9g45.cfg

# telnet 127.0.0.1 4444

# target extended-remote :3333

 Bootup Sequence - Bootloader
 The bootloader is a piece of code responsible for
 Basic hardware initialization
 Loading of an application binary, usually an operating system kernel, from flash storage, from the
network, or from another type of non-volatile storage.
 Possibly decompression of the application binary
 Execution of the application
 Besides these basic functions, most bootloaders provide a shell with various commands implementing
different operations.
 Loading of data from storage or network, memory inspection, hardware diagnostics and testing, etc.

44
 Linux Bootup Sequence
 The x86 processors are typically bundled on a board with a non-volatile
memory containing a program, the BIOS.
 On old BIOS-based x86 platforms: the BIOS is responsible for basic
hardware initialization and loading of a very small piece of code from non-
volatile storage.
 This piece of code is typically a 1st stage bootloader, which will load the
full bootloader itself.
 It typically understands filesystem formats so that the kernel file can be
loaded directly from a normal filesystem.

45
Linux Bootup Sequence

46
 Linux Bootup Sequence
Stages of Linux Boot Process:
• BIOS Stage
• Boot Loader Stage
• Kernel Stage

47
 Linux Bootup Sequence - BIOS Stage
 When the machine is powered on BIOS is the first one to be called to verify
if the hardware is present in the machine and if it is functioning.
 This is done by performing a Power On Self Test (POST)
 In a multi-processor or multi-core system one CPU is dynamically chosen
to be the bootstrap processor (BSP) that runs all of the BIOS and kernel
initialization code, others are called application processors(AP)
 After a successful test, BIOS checks the MBR (Master Boot Record) in the
hard disk to check if it refers to the location of the boot loader.

48
Linux Bootup Sequence – BIOS Stage

49
 Linux Bootup Sequence - Boot loader Stage
 The boot loader(GRUB) will present the user with a list of menu entries, each of which
corresponds to different operating system
 The boot loader(GRUB) will then start to boot the operating system
 When you select the the option to start Linux, it decompresses the Linux kernel in memory
 After that Linux kernel (which you selected to boot from) loads initrd (Initial ramdisk)
 The initrd is used by the Linux kernel as a temporary filesystem in the memory
 It contains tools and kernel modules which will continue the boot process including
mounting a virtual root file system temporarily.
 Instead of using initrd, some Linux filesystem will also use initramfs.
 It serves the same purpose of initrd, it is just that it is a successor of initrd
 linuxrc is an executable file that is next spawn, it probes the mass storage hardware and
finds a suitable kernel module to drive the mass storage hardware
 This is required to prepare the real root filesystem to be mounted by the Linux kernel

50
Linux Bootup Sequence – Boot Loader Stage

51
 Linux Bootup Sequence - Kernel Stage
 In the kernel stage of the Linux boot sequence, the Linux kernel based on the result of linuxrc can then
mount the real root file system
 The real root file system in Linux is referenced as "/" and it is where all other sub directory and files
visible when Linux is running exist
 The kernel will then spawn the init process, this process always has the process identifier (PID) as "1"
because it is the first background process or daemon started by the kernel upon boot
 All other background daemons are spawned from the init process
 So the init process will load other system daemons depending upon the configuration of different
runlevel

52
Linux Bootup Sequence – Kernel Stage

53
 Linux Bootup Sequence – ARM Case 1
 When powered, the CPU starts executing code at a fixed address
 There is no other booting mechanism provided by the CPU
 The hardware design must ensure that a NOR flash chip is wired so that it
is accessible at the address at which the CPU starts executing instructions
 The first stage bootloader must be programmed at this address in the
NOR
 NOR is mandatory, because it allows direct access from the CPU (just
like RAM), which NAND doesn’t allow (external storage that needs to be
copied to RAM before executing).
 Not very common anymore (unpractical, and requires NOR flash)

54
 Linux Bootup Sequence – ARM Case 2
 The CPU has an integrated boot code in ROM
BootROM on AT91 CPUs, “ROM code” on OMAP, etc.
Exact details are CPU-dependent
 This boot code is able to load a first stage bootloader from a storage device into an internal
SRAM (DRAM not initialized yet)
Storage device can typically be: MMC, NAND, SPI flash, UART (transmitting data over the
serial line), etc.
 The first stage bootloader is
Limited in size due to hardware constraints (SRAM size)
Provided either by U-Boot (called Secondary Program Loader - SPL), or by the CPU vendor
(usually open-source).
 This first stage bootloader must initialize DRAM and other hardware devices and load a
second stage bootloader into DRAM
55
 Linux Bootup Sequence – ARM Case 2
 RomBoot: tries to find a valid bootstrap image from various
storage sources, and load it into SRAM (DRAM not initialized
yet). Size limited to 64 KB. No user interaction possible in
standard boot mode.
 U-Boot SPL: runs from SRAM. Initializes the DRAM, the NAND
or SPI controller, and loads the secondary bootloader into DRAM
and starts it. No user interaction possible.
 U-Boot: runs from DRAM. Initializes some other hardware devices
(network, USB, etc.). Loads the kernel image from storage or
network to DRAM and starts it. Shell with commands provided.
 Linux Kernel: runs from DRAM. Takes over the system
completely (the bootloader no longer exists).

56
Linux Bootup Sequence – ARM UBOOT

57
Linux Overall Bootup Sequence

58
Linux Overall Bootup Sequence with initramfs

59
 Changing rootfs
chroot
The basic syntax of the chroot command is:

# chroot <mount_point> <shell>

where <mount_point> is the name of the directory you want to use as the root filesystem – in this
context, the new root filesystem is the NFS-exported RFS, and <shell> is the path to the shell
on the new filesystem, typically /bin/sh or /bin/bash. Thus, when the NFS root filesystem is
mounted on the /mnt directory, the following command makes this the new root filesystem:

# chroot /mnt /bin/bash

60
 Changing rootfs
chroot
The basic syntax of the chroot command is:

# chroot <mount_point> <shell>

where <mount_point> is the name of the directory you want to use as the root filesystem – in this
context, the new root filesystem is the NFS-exported RFS, and <shell> is the path to the shell
on the new filesystem, typically /bin/sh or /bin/bash. Thus, when the NFS root filesystem is
mounted on the /mnt directory, the following command makes this the new root filesystem:

# chroot /mnt /bin/bash

61
 Changing rootfs
switch_root
Root filesystems based on initramfs cannot use the pivot_root utility. The switch_root utility is a
widely-used alternative, and is included in BusyBox. The basic syntax of switch_root is:

# exec switch_root <new_root> <new_init>

62
Thank you

63