Linux Kernel and Driver Development Training Lab Book: Free Electrons
Linux Kernel and Driver Development Training Lab Book: Free Electrons
Linux Kernel and Driver Development Training Lab Book: Free Electrons
Free Electrons
Free Electrons
Training setup
Download les and directories used in practical labs Install lab data
For the different labs in the training, your instructor has prepared a set of data (kernel images, kernel congurations, root lesystems and more). Download and extract its tarball from a terminal: cd sudo wget sudo sudo apt-get install xz-utils http://free-electrons.com/labs/labs.tar.xz tar Jxf labs.tar.xz chown -R <user>.<user> felabs
Note that using root permissions are required to extract the character and block device les contained in this lab archive. This is an exception. For all the other archives that you will handle during the practical labs, you will never need root permissions to extract them. If there is another exception, we will let you know. Lab data are now available in an felabs directory in your home directory. For each lab there is a directory containing various data. This directory will also be used as working space for each lab, so that the les that you produce during each lab are kept separate. You are now ready to start the real practical labs!
More guidelines
Can be useful throughout any of the labs Read instructions and tips carefully. Lots of people make mistakes or waste time because they missed an explanation or a guideline. Always read error messages carefully, in particular the rst one which is issued. Some people stumble on very simple errors just because they specied a wrong le path and didnt pay enough attention to the corresponding error message. Never stay stuck with a strange problem more than 5 minutes. Show your problem to your colleagues or to the instructor. You should only use the root user for operations that require super-user privileges, such as: mounting a le system, loading a kernel module, changing le ownership, congurc 2004-2012 Free Electrons, CC BY-SA license 3
Free Electrons
ing the network. Most regular tasks (such as downloading, extracting sources, compiling...) can be done as a regular user. If you ran commands from a root shell by mistake, your regular user may no longer be able to handle the corresponding generated les. In this case, use the chown -R command to give the new les back to your regular user. Example: chown -R myuser.myuser linux-3.4
Free Electrons
Setup
Go to the /home/<user>/felabs/linux/modules directory. Download and extract the Linux 3.5 kernel sources from http://kernel.org.
Apply patches
Install the patch command, either through the graphical package manager, or using the following command line: sudo apt-get install patch Now, download the two Linux patches corresponding to versions 3.6 and 3.6.x (if such a version exists). Apply these patches, check the Makefile le to double check that you have the right version, and rename the source directory to reect the version change.
Free Electrons
Now, just let this command run, from 30 minutes to several hours according to your workstation and network speed.
Free Electrons
After this lab, you will be able to: Cross-compile a kernel for the ARM platform Boot this kernel on an NFS root lesystem, which is somewhere on your development workstation1
Lab implementation
While developing a kernel module, the developer wants to change the source code, compile and test the new kernel module very frequently. While writing and compiling the kernel module is done the development workstation, the test of the kernel module usually has to be done on the target, since it might interact with hardware specic to the target. However, ashing the root lesystem on the target for every test is time-consuming and would use the ash chip needlessly. Fortunately, it is possible to set up networking between the development workstation and the target. Then, workstation les can be accessed through the network by the target, using NFS.
Setup
Stay in the /home/<user>/felabs/linux/modules directory. Install packages needed for this lab: sudo apt-get install libqt4-dev g++ u-boot-tools libqt4-dev and g++ are needed for make xconfig. u-boot-tools is needed to build the uImage le for U-boot (mkimage utility).
1 NFS root lesystems are particularly useful to compile modules on your host, and make them directly visible on the target. You longer have to update the root lesystem by hand and transfer it to the target (requiring a shutdown and reboot).
Free Electrons
Kernel conguration
Set the ARCH and CROSS_COMPILE denitions for the arm platform and to use your crosscompiler. Congure this kernel with the ready-made conguration for boards with the AT91SAM9263 CPU. Make sure that this conguration has CONFIG_ROOT_NFS=y (support booting on an NFS exported root directory). Compile your kernel and generate the uImage kernel image that U-boot needs (the U-boot bootloader needs the kernel zImage le to be encapsulated in a special container and the kernel Makefile can generate this container for you by running the mkimage tool found in the uboot-mkimage package): make uImage
Then, restart the NFS server: sudo /etc/init.d/nfs-kernel-server restart If there is any error message, this usually means that there was a syntax error in the /etc/ exports le. Dont proceed until these errors disappear.
If you run ls -l /dev/ttyUSB0, you can also see that only root and users belonging to the dialout group have read and write access to this le. Therefore, you need to add your user to the dialout group: sudo adduser <user> dialout You now need to log out and log in again to make the new group visible everywhere. Now, you can run picocom -b 115200 /dev/ttyUSB0, to start serial communication on /dev/ttyUSB0, with a baudrate of 115200. If you wish to exit picocom, press [Ctrl][a] followed by [Ctrl][x]. You should now see the U-Boot prompt: U-Boot> You may need to reset the board (using the tiny reset button close to the USB host connectors). You can now use U-Boot. Run the help command to see the available commands.
Select the new wired network connection: c 2004-2012 Free Electrons, CC BY-SA license 9
Free Electrons
In the IPv4 Settings tab, make the interface use a static IP address, like 192.168.0.1 (of course, make sure that this address belongs to a separate network segment from the one of the main company network). You will also need to specify the local network mask (netmask, often 255.255.255.0). You can keep the Gateway eld empty (dont click put the cursor inside the corresponding text box, otherwise it will ask for a legal value) or set it to 0.0.0.0:
Now, congure the network on the board in U-Boot by setting the ipaddr and serverip environment variables: setenv ipaddr 192.168.0.100 setenv serverip 192.168.0.1 In case the board was previously congured in a different way, we also turn off automatic booting after commands that can be used to copy a kernel to RAM: setenv autostart no 10 c 2004-2012 Free Electrons, CC BY-SA license
Free Electrons
Also check that the ethaddr environment variable is dened. Thats the MAC address that U-boot sees. If it isnt dened, set it: set ethaddr 00:00:00:11:22:33 To make these settings permanent, save the environment: saveenv If you changed ethaddr, also power off your board, and power it on again (pressing the reset button is not enough). You can then test the TFTP connection 3 . First, put a small text le in the directory exported through TFTP on your development workstation. Then, from U-Boot, do: tftp 0x21000000 textfile.txt This should download the le textfile.txt from your development workstation into the boards memory at location 0x21000000 (this location is part of the board DRAM). You can verify that the download was successful by dumping the contents of the memory: md 0x21000000
Of course, you need to adapt the IP addresses to your exact network setup. Save the environment variables (with saveenv). Now, download the kernel image through tftp: tftp 0x21000000 uImage Now, boot your kernel: bootm 0x21000000 If everything goes right, you should reach a shell prompt. Otherwise, check your setup or ask your instructor for details. If the kernel fails to mount the NFS lesystem, look carefully at the error messages in the console. If this doesnt give any clue, you can also have a look at the NFS server logs in /var/ log/syslog.
11
Free Electrons
We could also copy the uImage le to NAND ash and avoid downloading it over and over again. However, handling NAND ash is outside of the scope of this course. See our Embedded Linux system development course and its on-line materials for details.
12
Free Electrons
Writing modules
Objective: create a simple kernel module
After this lab, you will be able to: Compile and test standalone kernel modules, which code is outside of the main Linux sources. Write a kernel module with several capabilities, including module parameters. Access kernel internals from your module. Setup the environment to compile it Create a kernel patch
Setup
Stay inside the /home/<user>/felabs/linux/modules directory. Boot your board again, as you did in the previous lab.
Writing a module
Go to the nfsroot/root directory. All the les you generate there will also be visible from the target. Thats great to load modules! Create a hello_version.c le implementing a module which displays this kind of message when loaded: Hello Master. You are currently using Linux <version>. ... and displays a goodbye message when unloaded. You may just start with a module that displays a hello message, and add version information later. Caution: you must use a kernel variable or function to get version information, and not just the value of a C macro. Otherwise, you will only get the version of the kernel you used to build the module. Suggestion: you can look for les in kernel sources which contain version in their name, and see what they do.
Free Electrons
Run a command to check that your module is on the list of loaded modules. Now, try to get the list of loaded modules with only the cat command.
Free Electrons
Run the make command and make sure that the code of your new driver is getting compiled. Then, install your kernel module using make modules_install. Beware, the modules should be installed in the root lesystem of the target, not in the root lesystem of your development workstation!
15
Free Electrons
Setup
Go to the /home/<user>/felabs/linux/character directory. As in the Module development environment lab, we will use a the CALAO board booted from NFS. As in the previous labs, the target IP address will be 192.168.0.100, and the host address will be 192.168.0.1. Extract the latest Linux 3.6.x kernel sources in the current directory, and congure them with the default conguration for at91sam9263 boards. In this lab, we will develop our own driver for the boards serial port. The consequence is that we will have to disable the standard AT91 serial port driver and will thus lose the serial console. Instead of running commands through a shell on the serial line, we will access our target through SSH, a secure shell over the network. To replace the serial console and still get kernel messages (in particular kernel fault messages) we will use a netconsole, a mechanism that allows to get console messages over the network. So, congure your kernel with: Root over NFS support Loadable module support Netconsole support (CONFIG_NETCONSOLE) You will also have to update the kernel command line so that Linux loads the root lesystem over NFS from /home/<user>/felabs/linux/character/nfsroot. The Dropbear SSH server is already installed on your target, and is started automatically. 16 c 2004-2012 Free Electrons, CC BY-SA license
Free Electrons
Boot tests
We are rst going to make sure that your new kernel boots ne and that the SSH connection works well, before disabling the serial driver and the serial console. NFS conguration issues are frequent, and they would be more difcult to x if we have no console left. So, boot your new kernel and try to connect to your board with ssh: ssh [email protected] The root password is empty, just press Enter. Good job!
Device initialization
In the module initialization function, start by reserving the I/O memory region starting at address (AT91_BASE_DBGU1), for a size of SZ_512 (512 bytes). The AT91_* constants are already dened in Linux kernel headers. Compile your module, load it and make sure that this memory region appears in /proc/ iomem. Now, obtain a virtual address corresponding to the start of this memory area. Dont forget to undo all the above in the module exit function!
Free Electrons
1. Wait until the ATMEL_US_TXRDY bit gets set in the ATMEL_US_CSR register (ATMEL_US_ CSR is an offset in the I/O memory region previously remapped). You can busy-wait for this condition to happen. In the busy-wait loop, you can call the cpu_relax() kernel function to relax the CPU during the wait. 2. Write the character to the ATMEL_US_THR register. Note that all the I/O registers of the AT91 processor are 32 bits wide. Add a call to this routine from your module init function. Recompile your module and load it on the target. You should see the corresponding character in picocom, still showing what was written to the serial line by the board.
18
Free Electrons
Setup
You must have completed the previous lab before. Stay in the /home/<user>/felabs/linux/character directory.
Free Electrons
Ioctl operation
We would like to maintain a counter of the number of characters written through the serial port. So we need to implement two unlocked_ioctl() operations: SERIAL_RESET_COUNTER, which as its name says, will reset the counter to zero SERIAL_GET_COUNTER, which will return in a variable passed by address the current value of the counter. Two already-compiled test applications are already available in the nfsroot/root/ directory, with their source code. They assume that SERIAL_RESET_COUNTER is ioctl operation 0 and that SERIAL_GET_COUNTER is ioctl operation 1.
20
Free Electrons
During this lab, you will: Register an interrupt handler for the serial controller of the Calao board See how Linux handles shared interrupt lines Implement the read() operation of the serial port driver to put the process to sleep when no data are available Implement the interrupt handler to wake-up the sleeping process waiting for received characters Handle communication between the interrupt handler and the read() operation.
Setup
This lab is a continuation of the Output-only character driver lab, so well re-use the code in /home/<user>/felabs/linux/character. Your Calao board should boot over NFS and mount /home/<user>/felabs/linux/character/nfsroot/ as the root lesystem. For demonstration purposes, we need to ensure that the tick system will use the shared IRQ. So congure your kernel and disable TC Block Clocksource (CONFIG_ATMEL_TCB_CLKSRC). Then compile and boot your new kernel.
Free Electrons
Write ATMEL_US_RXRDY to the ATMEL_US_IER register (IER stands for Interrupt Enable Register). Now, in our interrupt handler we want to lter out the interrupts that come from the serial controller. To do so, read the value of the ATMEL_US_CSR register and the value of the ATMEL_ US_IMR register. If the result of a binary and operation between these two values is different from zero, then it means that the interrupt is coming from our serial controller. If the interrupt comes from our serial port controller, print a message and return IRQ_HANDLED. If the interrupt doesnt come from our serial port controller, just return IRQ_NONE without printing a message. Compile and load your driver. Have a look at the kernel messages. You should no longer be ooded with interrupt messages. Start picocom on /dev/ttyUSB0. Press one character (nothing will appear since the target system is not echoing back what were typing). Then, in the kernel log, you should see the message of our interrupt handler. If not, check your code once again and ask your instructor for clarication!
Free Electrons
In the read() operation, if the serial_buf_rd value is different from the serial_buf_wr value, it means that one character can be read from the circular buffer. So, read this character, store it in the userspace buffer, update the serial_buf_rd variable, and return to userspace (we will only read one character at a time, even if the userspace application requested more than one). Now, what happens in our read() function if no character is available for reading (i.e, if serial_buf_wr is equal to serial_buf_rd)? We should put the process to sleep! To do so, declare a global wait queue in our driver, named for example serial_wait. In the read() function, use wait_event_interruptible() to wait until serial_buf_wr is different from serial_buf_rd. And in the interrupt handler, after storing the received characters in the circular buffer, use wake_up() to wake up all processes waiting on the wait queue. Compile and load your driver. Run cat /dev/serial on the target, and then in Picocom on the development workstation side, type some characters. They should appear on the remote side if everything works correctly! Dont be surprised if the keys you type in Picocom dont appear on the screen. This happens because they are not echoed back by the target.
23
Free Electrons
Locking
Objective: practice with basic locking primitives
During this lab, you will: Practice with locking primitives to implement exclusive access to the device.
Setup
Stay in the /home/<user>/felabs/linux/character directory. You need to have completed the previous two labs to perform this one. Boot your board with the same NFS environment as before, and load your serial module.
24
Free Electrons
debugfs
Since you have enabled debugfs to control the dynamic debug feature, we will also use it to add a new debugfs entry. Modify your driver to add: A directory called serial in the debugfs lesystem And le called counter inside the serial directory of the debugfs lesystem. This le should allow to see the contents of the counter variable of your module. Recompile and reload your driver, and check that in /sys/kernel/debug/serial/counter you can see the amount of characters that have been transmitted by your driver. c 2004-2012 Free Electrons, CC BY-SA license 25
Free Electrons
26
Free Electrons
Setup
Go to the /home/<user>/felabs/linux/character directory. It contains the root lesystem that you will mount over NFS to work on this lab. Re-use the setup instructions of the lab on Character Device Drivers to get a kernel without the serial port driver and with Network Console support. For this lab, we advise you to work with Linux 3.0, as this lab and the solution that we will provide have been tested with this version. Therefore, you should download the Linux 3.0 sources.
Basic module
The serial core cannot be compiled inside the kernel without an in-tree kernel driver. Therefore, for this lab, we will work directly inside the kernel source tree and not using an external module. To do so: Create a basic module in drivers/tty/serial/fedrv.c with just the init and exit functions ; Add a new conguration option in drivers/tty/serial/Kconfig. Dont forget to select SERIAL_CORE in this option ; Update drivers/tty/serial/Makefile to make sure your driver gets compiled when your new option is selected Compile your new driver as a module, and after the kernel compilation, run: make INSTALL_MOD_PATH=/path/to/nfsroot modules_install to install the modules (serial_core and your driver) into the root lesystem. c 2004-2012 Free Electrons, CC BY-SA license 27
Free Electrons
Then try to load/unload your module on the target using modprobe. If youre successful, we can now start working on the driver itself.
Free Electrons
symbolic link driver to the atmel_usart driver. If you follow this symbolic link, you should discover that the atmel_usart driver is implemented by the fedrv module. Congratulations!
Free Electrons
->flags should be UPF_BOOT_AUTOCONF so that the config_port() operation gets called to do the conguration ->mapbase should be pdev->resource[0].start, this is the address of the memorymapped registers ->membase should be set to data->regs, where data is the device-specic platform data structure associated to the device. In our case, its a atmel_uart_data structure, available through pdev->dev.platform_data. In the case of the rst serial port data->regs is non-zero and contains the virtual address at which the registers have been remapped. For the other serial ports, we would have to ioremap() them. Then, we need to create stubs for a fairly large number of operations. Even if we dont implement anything inside these operations for the moment, the serial_core layer requires these operations to exist: tx_empty() start_tx() stop_tx() stop_rx() type() startup() shutdown() set_mctrl() get_mctrl() enable_ms() set_termios() release_port() request_port() config_port() First, lets implement whats related to setting and getting the serial port type: In the config_port() operation, if flags & UART_CONFIG_TYPE is true, then set port->type = PORT_ATMEL. There is a global list of serial port types, and we are re-using the existing denition. In the type() operation, if port->type is PORT_ATMEL return a string like ATMEL_ SERIAL, otherwise return NULL. Now, for the transmission itself, we will rst implement tx_empty(). In this function, read the register ATMEL_US_CSR from the hardware (note: the virtual base address of the registers is in port->membase). If the ATMEL_US_TXEMPTY bit is set, it means that the port is ready to transmit, therefore return TIOCSER_TEMT, otherwise return 0. Then, the start_tx() function will do the transmission itself. Iterate until the transmission buffer is empty (use uart_circ_empty()) and do: call an auxiliary function that prints one character update the tail pointer of the transmission buffer 30 c 2004-2012 Free Electrons, CC BY-SA license
The auxiliary function should wait until bit ATMEL_US_TXRDY gets set in the ATMEL_US_CSR register, and then send the character through the ATMEL_US_THR register. Then, compile and load your driver. You should now be able to do: echo "foo" > /dev/ttyS0
Implementing reception
The last part to make our driver usable is to implement reception. We rst need to modify the probe() method to set port->irq to pdev->resource[1] .start so that we fetch the IRQ number from the board-specic platform device denition. Then, in the startup() operation, do the following steps: Disable all interrupts in the serial controller by writing 0UL5 to the ATMEL_US_IDR register. Register the port->irq IRQ channel with a fedrv_interrupt() interrupt handler. Pass port as the dev_id so that we get a pointer to port in the interrupt handler. Make it a shared interrupt. Reset the serial controller by writing ATMEL_US_RSTSTA | ATMEL_US_RSTRX to the ATMEL_US_CR register Enable transmission and reception by writing ATMEL_US_TXEN | ATMEL_US_RXEN to the ATMEL_US_CR register Enable interrupts on reception by writing ATMEL_US_RXRDY to the ATMEL_US_IER register Similarly, in the shutdown() operation, do: Disable all interrupts by writing 0UL to the ATMEL_US_IDR register Free the IRQ channel using free_irq() Then, in the interrupt handler, do the following: Read the ATMEL_US_CSR register to get the controller status and perform the logical and of this value with the enabled interrupts by reading the ATMEL_US_IMR register. If the resulting value is 0, then the interrupt was not for us, return IRQ_NONE. If the result value has bit ATMEL_US_RXRDY set, call an auxiliary function fedrv_rx_ chars() to receive the characters. Finally, we have to implement the fedrv_rx_chars() function. This function should read the ATMEL_US_CSR register, and while ATMEL_US_RXRDY is set in this register, loop to read characters the following way: Read one character from the ATMEL_US_RHR register Increment port->icount.rx Call uart_insert_char() with the value of the status register, overrun to ATMEL_US_ OVRE, and the ag set to TTY_NORMAL (we dont handle break characters, frame or parity errors, etc. for the moment)
5 Thats
31
Free Electrons
Once all characters have been received, we must tell the upper layer to push these characters, using tty_flip_buffer_push(). Now, if you do cat /dev/ttyS0, you should be able to receive characters from the serial port. By default, a ttyS is opened in the so-called canonical mode, so the characters are sent to the reading process only after entering a newline character. You can also try to run the program that will display the login prompt and then a shell: /sbin/getty -L ttyS0 115200 vt100 Go back to picocom, you should be able to login normally, but using your own serial driver!
Improvements
Of course, our serial driver needs several improvements: Real implementation of set_termios() and set_mctrl() Usage of interrupts for transmission Console support for early messages Support of several serial ports Handle parity and frame errors properly Support break characters and SysRq Use of DMA for transmission and reception etc.
32
Free Electrons
Power management
Objective: practice with standard power management interfaces offered by Linux
After this lab, you will be able to: Suspend and resume your Linux system Change the CPU frequency of your system
Setup
Go to the /home/<user>/felabs/powermgt/usage/ directory. Download and extract the latest update to the Linux 3.0 kernel. Suspend/resume support for the Calao board is already included in this kernel. Cpu frequency scaling support for this hardware was developed by Free Electrons and is not yet part of the mainline kernel. Therefore, before compiling the 3.0 kernel, youll have to apply the three patches in the data/ directory of this lab: The rst patch implements the CPU frequency driver itself, which allows to change the frequency on the AT91SAM9263 CPU The second patch adds CPU frequency support to the serial port driver. When the CPU clock is changed, the divisors for the baud rate generator must be modied. This is what this patch does. The third patch adds CPU frequency support to the Ethernet controller driver for similar reasons. Congure your kernel with CPU Frequency scaling support, with the CPU Frequency driver for AT91, and for the different cpufreq governors. Then, compile this kernel, and boot the system over NFS on the root lesystem used in the debugging lab (/home/<user>/felabs/linux/debugging/nfsroot/).
Free Electrons
Using PowerTop
On your development PC. Install the nice PowerTop tool contributed by Intel: sudo apt-get install powertop Run the powertop command, and see it display statistics, and list the top processes that cause you CPU to wake up from a deeper sleep state, causing it to consume more power. You could use this interface to nd power management bugs in the applications running on your system. If youre using a laptop, remove the AC power for a while. This gives you access to live power estimates from ACPI. Also follow the tips that PowerTop gives you to conserve power, and try to make your system consume as little power as possible. Compare your power estimates with other people in the classroom, and try to achieve the best results. Any technique can be used! Thanks to Linaro, PowerTop is also available on ARM now. See https://wiki.linaro. org/WorkingGroups/PowerManagement/Doc/Powertop. So, if your embedded architecture has CPUidle support, you could try this utility on it. If your embedded architecture has CPUidle support, even if you didnt compile powertop, you 34 c 2004-2012 Free Electrons, CC BY-SA license
Free Electrons
can still access idle state statistics by looking at the les in /sys/devices/system/cpu/ cpu<n>/cpuidle.
35
Free Electrons
Git
Objective: Get familiar with git by contributing to the Linux kernel
After this lab, you will be able to: Clone a Git repository Explore the history of a Git repository Create a branch and use it to make improvements to the Linux kernel sources Make your rst contribution to the ofcial Linux kernel sources Rework and reorganize the commits done in your branch Work with a remote tree
Setup
Go to /home/<user>/felabs/linux/git/ This lab assumes that you already installed git software and cloned the Linus Torvalds git tree. See our Kernel source code lab for details (http://free-electrons.com/doc/training/ linux-kernel/).
Conguring Git
Congure your name and email address in git with git config.
Clone a repository
We already cloned Linus Torvalds git tree, but it is useful to remember how to do it again. Go to http://git.kernel.org and make sure you know how to nd the git:// URL of his Linux tree. Cloning downloaded quite a lot of data, but then at the end, we have the full history of the Linux kernel (since the kernel developers started to use Git, around kernel 2.6.12). We can access and explore this history ofine.
Free Electrons
With gitk, look at the full history of the UBIFS lesystem (in fs/ubifs/). On the gitweb interface of Linus Torvalds tree, available at http://git.kernel.org/?p= linux/kernel/git/torvalds/linux.git, search all commits that have been done by Free Electrons (hint: use the search engine by author).
Free Electrons
Dont hesitate to ask your instructor for help. The instructor will also be happy to have a nal look at your changes before you send them for real.
A git fetch will fetch the data for this tree. Of course, Git will optimize the storage, and will no store everything thats common between the two trees. This is the big advantage of having a single local repository to track multiple remote trees, instead of having multiple local repositories. We can then switch to the master branch of the realtime tree: git checkout realtime/master Or look at the difference between the scheduler code in the ofcial tree and in the realtime tree: git diff master..realtime/master kernel/sched/
38
Free Electrons
39