Read-only NFS Root

with Debian 10

CTRL.01.01.013

Sébastien BLANCHET

January 22, 2020

Institut Radio Astronomie Millimétrique

300 rue de la piscine
38406 Saint Martin D’Hères - France
Read-Only NFS Root with Debian 10

2 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 CONTENTS

Contents

1 Introduction 5

2 Build the NFS root 7

2.1 Create the NFS root . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Host computer . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 Setup a NFSROOT variable . . . . . . . . . . . . . . . . . 7
2.1.3 Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.4 Configure /etc/apt/sources/list . . . . . . . . . . . . . . . 8
2.1.5 Configure hostname . . . . . . . . . . . . . . . . . . . . . . 8
2.1.6 Configure /etc/fstab . . . . . . . . . . . . . . . . . . . . . 9
2.1.7 Configure /etc/mtab . . . . . . . . . . . . . . . . . . . . . 10
2.1.8 Configure the root user . . . . . . . . . . . . . . . . . . . . 10
2.1.9 Build a PXE initrd . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Prepare NFS and TFTP server . . . . . . . . . . . . . . . . . . . 11

3 Optimize NFS root 13

3.1 Preempt-RT kernel . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Build a custom kernel . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Graphical environment . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.1 Install wdm and xfce4 . . . . . . . . . . . . . . . . . . . . 15
3.4 Other software . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4.1 Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4.2 NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5.1 iptables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

IRAM – January 22, 2020 3

Read-Only NFS Root with Debian 10 CONTENTS

4 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 1. Introduction

Chapter 1

Introduction

This document explains how to create a read-only network filesystem based on

Debian 10 (Buster). A NFS server will export the filesystem in read-only mode
to be safely shared by several diskless clients. To easily update the filesystem, a
single client will be promoted as the NFS Root Master, with read-write privileges:
any modification on this root master will be automatically applied to all the other
read-only clients.

Prerequesites
• a DHCP server, to provide IP address to the clients when they boot.

• a TFTP server, to allow the clients to download the kernel and initial root
ram disk (initrd).

• a recent PXE file tree1 with menu.c32, pxelinux.0 and the pxelinux.cfg
directory.

• a NFS server, to provide the filesystem to the clients.

Tips
The DHCP and TFTP servers can be easily validated with a basic test: boot
memtest86+ over PXE.

1
these files come from the Syslinux Project: http://syslinux.org

IRAM – January 22, 2020 5

Read-Only NFS Root with Debian 10 1. Introduction

6 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 2. Build the NFS root

Chapter 2

Build the NFS root

2.1 Create the NFS root

2.1.1 Host computer
A Debian host computer is required to create the NFS root filesystem. It is
not the NFS server itself, but only a temporary computer to build the NFS
filesystem. In theory, you can use any Debian computer, whatever the release or
the architecture are. But in practice it is more convenient to create a temporary
virtual machine that runs the same Debian version than the target operating
system to build the new filesystem.
Therefore, create a new virtual machine1 with VirtualBox and install Debian-
10.2.0-amd64 into it.

2.1.2 Setup a NFSROOT variable

Setup a variable with the hosting directory, to avoid typing it everytime.
# export NFSROOT=/home/nfsroot10
# mkdir ${NFSROOT}

2.1.3 Bootstrap
debootstrap creates a basic Debian system.
# apt install -y debootstrap
# debootstrap buster ${NFSROOT} http://ftp2.fr.debian.org/debian
The mirror URL is optional, but it downloads faster if you select a mirror close
to you. See manual debootstrap(8) for details.
Now we customize this basic Debian system in $NFSROOT.
1
If you wish to recompile a custom kernel then prepare a 30 GiB virtual hard disk, otherwise
15 GiB is enough.

IRAM – January 22, 2020 7

Read-Only NFS Root with Debian 10 2. Build the NFS root

2.1.4 Configure /etc/apt/sources/list

$NFSROOT/etc/apt/sources.list
Point to the nearest mirror. If needed add also contrib and non-free reposito-
ries.
Listing 2.1: /etc/apt/sources.list
deb http://deb.debian.org/debian/ buster main non-free contrib

# security patches
deb http://security.debian.org/debian-security buster/updates main non-free contrib

# buster-updates, previously known as ’volatile’

deb http://deb.debian.org/debian/ buster-updates main non-free contrib

2.1.5 Configure hostname

Install required programs in $NFSROOT

# chroot $NFSROOT
# apt update
# apt -y install bind9-host locales

Configure locale

# locale-gen en_US.UTF-8
# dpkg-reconfigure locales

$NFSROOT/bin/whereami
Create a script /bin/whereami to setup client hostname from /etc/hosts
Listing 2.2: $NFSROOT/bin/whereami
#!/bin/bash
#finds node’s hostname based on matching IP in DNS
PATH=/sbin:/usr/sbin:/bin:/usr/bin

# default device for gateway

GWDEV=‘ip route show default |cut -f 5 -d "␣"‘

# get IP address
MYIP=(‘ip address show label ${GWDEV} | tr ’/’ ’␣’‘)
MYIP=${MYIP[1]}

# test if MYIP exist in DNS

8 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 2. Build the NFS root

host $MYIP > /dev/null

status=$?
if [ $status -ne 0 ]; then
MYHOSTNAME="undefined-hostname"
else
MYHOSTNAME=‘host $MYIP | \
sed -e ’s/.*domain␣name␣pointer␣\([a-zA-Z0-9-]*\).*/\1/g’‘
fi
echo $MYHOSTNAME

Add executable bit:

# chmod +x ${NFSROOT}/bin/whereami

$NFSROOT/etc/rc.local
Even if Debian uses systemd, /etc/rc.local is very convenient. So create the
file $NFSROOT/etc/rc.local

# touch ${NFSROOT}/etc/rc.local
# chmod +x ${NFSROOT}/etc/rc.local

Listing 2.3: $NFSROOT/etc/rc.local

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
# set hostname
hostname ‘/bin/whereami‘
exit 0

2.1.6 Configure /etc/fstab

Setup read-write home directories
Install nfs-common and edit $NFSROOT/etc/fstab

# chroot ${NFSROOT}
# apt -y install nfs-common
# exit

Unlike “/”, /home is mounted with read-write privilege from another NFS
share.

IRAM – January 22, 2020 9

Read-Only NFS Root with Debian 10 2. Build the NFS root

Listing 2.4: $NFSROOT/etc/fstab

proc /proc proc defaults 0 0
/dev/nfs / nfs tcp,nolock 0 0
none /tmp tmpfs defaults 0 0
none /var/tmp tmpfs defaults 0 0
none /media tmpfs defaults 0 0
none /var/log tmpfs defaults 0 0
nfsserver:/path/to/home /home nfs tcp,nolock 0 0

The nolock nfs option is very important, otherwise sqlite does not work on
these mount points. See man 5 nfs, for other nfs options.

2.1.7 Configure /etc/mtab

$NFSROOT/etc/mtab
# ln -s /proc/mounts ${NFSROOT}/etc/mtab

2.1.8 Configure the root user

Set the password and move /root to /home/root (to be writeable)

# chroot ${NFSROOT}
# passwd root
# mv /root /home/root
# usermod -d /home/root root
# exit

2.1.9 Build a PXE initrd

Install kernel and firmwares
# chroot $NFSROOT
# apt update
# apt -y install linux-image-amd64 firmware-linux \
firmware-realtek firmware-bnx2
# exit

Modify initramfs.conf to boot on NFS

Edit $NFSROOT/etc/initramfs-tools/initramfs.conf to have a minimal but
functionnal initial root disk.
Listing 2.5: $NFSROOT/etc/initramfs-tools/initramfs.conf
MODULES=netboot

10 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 2. Build the NFS root

BUSYBOX=y
KEYMAP=n
COMPRESS=xz
DEVICE=
NFSROOT=auto
BOOT=nfs

# chroot $NFSROOT
# update-initramfs -u
# exit

Now you have a suitable initrd and kernel in $NFSROOT/boot

$NFSROOT/etc/default
Finaly configure ramdisk for temporary files. See manuals rcS(5) and tmpfs(5)
for details.

# chroot ${NFSROOT}
# echo ASYNCMOUNTNFS=no >> /etc/default/rcS
# echo RAMTMP=yes >> /etc/default/tmpfs
# exit

2.2 Prepare NFS and TFTP server

• Transfer the NFS root to the NFS server.

# cd /home
# tar cvfj nfsroot10.tar.bz2 ${NFSROOT}
# scp nfsroot10.tar.bz2 root@nfsserver:/path/to/exportdir

Then login to NFS server and extract the archive.

• Transfer kernel and initrd to the TFTP server.

# scp /boot/vmlinuz-4.19.0-6-amd64 root@tftpserver:/tftpboot/

# scp /boot/initrd.img-4.19.0-6-amd64 \
root@tftpserver:/tftpboot/initrd.pxe-4.19.0-6-amd64

I usually rename the initrd from .img to .pxe to avoid any confusion, but
it is not mandatory.

IRAM – January 22, 2020 11

Read-Only NFS Root with Debian 10 2. Build the NFS root

Create a pxeconfig file

Listing 2.6: pxeconfig

# boot diskless computer with debian buster
default menu.c32
prompt 0
menu title pc-client

ontimeout linux-4.19.0-6-amd64-buster
timeout 50

label linux-4.19.0-6-amd64-buster
menu label linux-4.19.0-6-amd64 buster
kernel vmlinuz-4.19.0-6-amd64
append root=/dev/nfs initrd=initrd.pxe-4.19.0-6-amd64 \
nfsroot=nfsserver:/path/to/nfsroot ro panic=60 \
ipv6.disable=1 ip=:::::eno1

Note

• the backslash in the append line is only for printing purpose, to show that
the line continue. But indeed, all the arguments must be on the same line.

• If you wish that one diskless PC has read-write acces to easily modifying
the filesystem, replace ro by rw in the append configuration line.

• If you do not use IPv6, disable it on the command line to boot faster.

• Never use the ipappend option, it creates problems on computer with sev-
eral interfaces.

• If you have several Ethernet interfaces, specify which one to use with
ip:::::ethernetname to boot faster.

• If you have DHCP issue2 at the kernel level you must specify a static IP
address on the kernel command line3
ip=<client-ip> :<server-ip> :<gw-ip> :<netmask> :<hostname> :<device> :
<autoconf> :<dns0-ip> :<dns1-ip> :<ntp0-ip>

2
For an unknown reason, the DHCP request for TFTP is always faster and more reliable
than the second DHCP request for the kernel
3
See Linux kernel documentation Documentation/filesystems/nfs/nfsroot.txt for more
details.

12 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 3. Optimize NFS root

Chapter 3

Optimize NFS root

This chapter shows how to optimize the NFS root to match the IRAM needs.

3.1 Preempt-RT kernel

Fortunately there is an official preempt-rt kernel for Debian Buster, so you do
not have to rebuild a custom kernel to get a realtime kernel.

Install Preempt RT kernel

# apt-get install linux-image-4.19.0-6-rt-amd64 linux-headers-4.19.0-6-rt-amd64

# update-initramfs -u

Then transfer /boot/vmlinuz-4.19.0-6-rt-amd64 and initrd.img-4.19.0-6-rt-amd64

to the TFTP server.

3.2 Build a custom kernel

If you need a custom kernel (for example to support VME bridge) you have to
compile it from the sources by following this procedure
https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html

Download the kernel source code

# cd /usr/src
# wget https://kernel.org/pub/linux/kernel/v4.x/linux-4.14.155.tar.xz
# tar xf linux-4.14.155.tar.xz

IRAM – January 22, 2020 13

Read-Only NFS Root with Debian 10 3. Optimize NFS root

Download and apply PREEMPT-RT patch

# wget https://kernel.org/pub/linux/kernel/projects/rt/4.14/\
patch-4.14.155-rt70.patch.xz
# cd linux-4.14.155
# xz -cd ../patch-4.14.155-rt70.patch.xz | patch -p1

Configure the kernel

Install the required compilation tools
# apt-get install build-essential fakeroot bc pkg-config libssl-dev
For a ncurses interface, use the following commands:
# apt-get install libncurses5-dev
# make nconfig
For the Qt5 interface, use the following commands:
# apt-get install qt5-qmake qt5-default
# make xconfig
Configure the kernel with the options you want. For example, enable PREEMPT-
RT and VME bridges.

Build the kernel

Disable module signature, debug info and trusted key, then build the kernel
packages.

# make clean
# scripts/config --disable MODULE_SIG
# scripts/config --disable DEBUG_INFO
# scripts/config --set-str SYSTEM_TRUSTED_KEYS ""
# make deb-pkg

Install the kernel

The packages are created in /usr/src. Install them as regular Debian packages.

3.3 Graphical environment

Even if the computer clients are expected to operate headless, a minimal graphical
environment is very useful during the development process. I have chosen xfce4
because it is a light but comprehesive desktop environment. For the login, I have
chosen wdm because it is easy to setup for a read-only filesystem.

14 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 3. Optimize NFS root

3.3.1 Install wdm and xfce4

On the NFS master

# apt update
# apt install xorg mesa-utils xfce4 wdm

Some small modifications must be done on wdm to support a read-only filesys-

tem:

• Edit /etc/X11/wdm/wdm.options to disable auto-update-wm

Listing 3.1: /etc/X11/wdm/wdm.options

# /etc/X11/wdm/wdm.options
#
# configuration options for wdm
# See wdm.options(5) for an explanation of the available options.

check-local-server
no-ignore-nologin
no-restart-on-upgrade
use-sessreg
#auto-update-wmlist

• Edit /etc/X11/wdm/wdm.wmlist to define only xfce4-session:

Listing 3.2: /etc/X11/wdm/wdm.wmlist

DisplayManager*wdmWm: default:xfce4-session

3.4 Other software

3.4.1 Watchdog
Watchdog restarts computer if it hangs.

# apt -y install watchdog

3.4.2 NTP
The computer clock is controlled by NTP. Install ntp

# apt -y install ntp

IRAM – January 22, 2020 15

Read-Only NFS Root with Debian 10 3. Optimize NFS root

Each computer must save its own ntp drift file on a persistant storage, oth-
erwise it may take some hours to compensate the residual timing errors. But all
computers shares the same ntp.conf, and this file does not support shell variable
expanding. Therefore a small patch is added to /etc/init.d/ntp to preprocess
/etc/ntp.conf. With such a trick we can use $HOSTNAME in filename.

Listing 3.3: /etc/init.d/ntp

...
test -x $DAEMON || exit 0

# patch to expand $HOSTNAME (SBL_2020-01-15)

NTPCONF_SHM=/dev/shm/ntp.conf
sed -e "s/\$HOSTNAME/‘hostname‘/g" /etc/ntp.conf > $NTPCONF_SHM
NTPD_OPTS="$NTPD_OPTS -c $NTPCONF_SHM"
mkdir -p /home/ntp
# patch end

...

The drift will be saved in /home/ntp/$HOSTNAME.drift

Listing 3.4: /etc/ntp.conf
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /home/ntp/$HOSTNAME.drift
...

3.5 Network
3.5.1 iptables
# apt -y install iptables iptables-persistent

iptables-persistent will load /etc/iptables/rules.v4 at the startup.

Listing 3.5: /etc/iptables/rules.v4

*filter

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that
# doesn’t use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

16 IRAM – January 22, 2020

Read-Only NFS Root with Debian 10 3. Optimize NFS root

# Accepts all established inbound connections

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allows all outbound traffic

# You could modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows NFS from NFS server

-A INPUT -p tcp -s 192.168.130.13 --dport 0:1024 -j ACCEPT
-A INPUT -p udp -s 192.168.130.13 --dport 0:1024 -j ACCEPT

# Allows SSH connections

# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
-A INPUT -p tcp -s 192.168.0/16 -m state --state NEW --dport 22 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# Reject all other inbound

# default deny unless explicitly allowed policy:
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

Create a symbolic link to use iptables-apply without arguments

# ln -sf /etc/iptables/rules.v4 /etc/network/iptables.up.rules

IRAM – January 22, 2020 17