Linux as a Real-Time Operating

System
Manas Saksena
TimeSys Corporation
 Trends in Embedded Systems

Increased Functionality in Software

Networked or Connected Devices

Multi-Function Devices

Dynamic Functionality

Time to Market Pressures

Product Differentiation in Application, not Operating Systems

Trend towards increased usage of commercial RTOSes

2
 RTOS Requirements

Hardware Support
• What processors, boards, devices, etc. are supported?

Application Functional Support

• Rich programming API and programming model support
• Third-party Software (today and in future)
• Development Tools
• Deployment & Maintenance Tools

Support for Meeting Real-Time Performance Requirements

Support for Other “Non-functional” Requirements

• High Availability
• Footprint or size constraints
• Power constraints

Increasingly difficult for many RTOS vendors to keep up

3
Embedded OS Trends

4
 Why Linux?

Hardware support
• Wide variety of processors and devices supported
• Portable code-base makes it easy to extend to newer
platforms
• Dedicated open-source community makes it easy to track new
hardware
Application Functional Support
• Unix, POSIX lineage of a full-function OS including rich multi-
tasking, IPC, networking, file systems, memory protection,
multi-user, security and protection, SMP, etc.
• Standard well-known API’s makes it easy for third party
developers to develop middleware, libraries, applications.
• Quickly improving in the state of development tools. A solid
based of GNU development tools, and other command line
tools exists.
• Many open-source developers and interested commercial
vendors keep pushing the boundaries.

5
 Why Linux?

Support for Meeting Real-Time Performance Requirements

• Support for POSIX real-time API’s like scheduling and memory
locking
• Streamlined kernel, continually under improvement for
performance
• Various approaches to make it more suitable for real-time

Support for Other “Non-functional” Requirements

• Robust, reliable kernel
• Modular kernel enabling scaling up and down to support a wide
variety of embedded systems
• Many projects working on features such as high-availability,
footprint, and power-management

6
 Open-Source and Commercial Linux Vendors

Freely Available Source

• Kernel and various enhancements available from the open-
source community
• Commercial Linux vendors must focus on delivering some
value to the customer

Commercial Linux Vendors provide one or more of

• Enhanced real-time capabilities
• Productization
• Support and Subscription Services
• Training
• Integrated Application Stacks targeted to Vertical Markets
• Development Tools
• …….
This talk will focus on enhanced real-time capabilities

7
Understanding Linux
 Linux Tasking & Scheduling Architecture

User Process
Application Level P1 P2 P3 or Thread
Preemptive Scheduling

Kernel Level
Non-Preemptive Scheduling
Kernel Thread P4
Soft IRQ Queue

Task Queue
Task Scheduler

Device Driver
Interrupt handler

9
 Linux Scheduling Architecture

Low Priority

Time-Sharing
Scheduled Tasks Process or
by Task Thread
Scheduler (POSIX) Real-Time Context
Tasks

Soft IRQ Soft-IRQ

Non Processing context
Schedulable
Processing Interrupt Handler Hard-IRQ
Processing context

High Priority

10
 Linux Device Driver Architecture

open close read write ioctl

Virtual File System Layer (vfs) P1

wakeup task
Soft-IRQ Handler

Interrupt handler
Device Driver

11
 Standard Linux Networking Architecture

FTP

BSD Socket

INET Socket
soft
UDP TCP
irq
IP
FIB input forward output

Device Independent Device Driver

IH
Device Dependent Device Driver

12
 Linux Clocks and Timers

Jiffy
• A selectable time quantum (default 10ms)
Jiffy Timer
• A periodic heartbeat timer that interrupts once every jiffy

Software Timers
• Maintained in a list, when a jiffy timer expires, then all expired
software timers are handled
– Implication: Recognition of timer expiry can be delayed by
as much as one jiffy.
– Note:
– Speeding up Jiffy increases overhead

13
 Interrupt Latency

priority inversion
Hardware
HardwareDelay
Delay execution path

Interrupts Interrupts masked by kernel or

InterruptsMasked
Masked driver

Interrupt
InterruptVectoring
Vectoring Linux kernel framework for
handling interrupts.
do_IRQ()
do_IRQ() Interrupts are not necessarily
masked throughout; can get
nested interrupts. Priority
handle_IRQ_Event()
handle_IRQ_Event() inversion?

Interrupt
InterruptHandler
Handler In device driver

14
 Interrupt/Task Response -- Sample

Interrupt
InterruptLatency
Latency priority inversion
execution path
Interrupt
InterruptHandler
Handler
(wake
(wakeup
uptask)
task)

Pending OS defined higher priority

Pending work – may not match
SoftIRQ
SoftIRQProcessing
Processing with application’s needs

Interrupted
Interrupted Non-preemptible kernel
System
SystemCall
CallProcessing
Processing

Scheduler
Scheduler
(context -switch)
(context-switch)

Task
Task
15
 Doing Better Real-Time

Execution Path Latency

• Use faster, better hardware
• Optimize the execution path
• Change the execution path

Priority Inversion
• Lower Priority Work (Classic priority inversion)
– Avoid/Reduce non-preemptible or interrupt-masked regions
– Avoid coarse-grained mutexes for synchronization
– Don’t use semaphores for mutexes
– Use priority inheritance mutexes to protect shared data

• Kernel Defined Higher Priority

– Avoid use of non-schedulable entities and allow application
to prioritize work (as much as possible)

16
Approaches to Real-Time Linux
 Approaches to Real-Time Linux

Compliant Kernel Approach

• LynxOS and Blue Cat Linux;
• QNX;
• Other POSIX API operating systems

Dual Kernel Approach

• Lineo, FSM Labs (RT-Linux or RTAI)
• OnCore

Core Kernel Approach

• TimeSys, MontaVista

18
 Compliant Kernel Approach

Linux Development Tools Linux Development Tools

And Environment And Environment

Linux System Call API Linux System Call API

Linux Kernel Real-Time Kernel

(Embedded Applications) (Real-Time Applications)

19
 Compliant Kernel Approach Philosophy

Linux is broken for real-time; don’t use it.

Real-time requires a real-time OS

It is possible to build a “true” real-time kernel that is

compliant to Linux

20
 Compliant Kernel Approach

Pros
• (Some) Applications developed for Linux can easily port to
real-time kernel
• Standard API familiar to programmers
• Applications can be developed on a Linux box and then easily
ported to real-time kernel

Cons
• Does not benefit from innovations in Linux Kernel
– E.g., devices, network protocols, etc.
• API compliance gives you a false sense of portability
– Many API’s are overloaded to access different features in
the kernel (e.g., drivers, network protocols, file systems,
etc.)

21
 The Dual Kernel Approach

Linux Linux
Process Process
User-Level

Kernel-Level

Real-Time Real-Time Real-Time Linux Kernel

Task Task Task

Real-Time Kernel (RT-Linux or RTAI)

Hardware

22
 Dual Kernel Approach Philosophy

Linux is broken for real-time; don’t use it.

Real-time requires a real-time OS

Real-Time Systems Software should be structured into real-

time and non real-time components and implemented
separately

Linux is good for non real-time components of your system

23
 Dual Kernel Approach

Pros
• Requires very little changes to the Linux kernel
• The Linux side can be preempted at any time by the real-time
side
– Very tight response times possible for real-time tasks
• Significant development can be done on desktop running Linux
and inserting the real-time kernel
• Linux applications can be used in conjunction with real-time
tasks
Cons
• No memory protection in the real-time side
• Duplication of functionality
– Kernel bloat
– Future extensibility is questionable
• Sharing devices between real-time and Linux
• Starvation of non real-time tasks
• Static decomposition of real-time and non real-time

24
 Core Kernel Approach

Linux Linux Linux Linux

Process Process Process Process

Standard Linux API

Linux kernel Modified for Real-Time

Linux Linux
Driver Driver

Hardware

25
 Core Kernel Approach

Linux Linux Linux Linux

Process Process Process Process

Standard Linux API Extended API

Loadable Kernel
Module
Linux kernel Modified
for Real-Time
Linux Linux Loadable Kernel
Driver Driver Module

Hardware

26
 Core Kernel Approach Philosophy

Real-time with Linux

• The benefits of Linux for embedded systems are best obtained by
providing Linux Applications with real-time OS support

Is it even feasible?
• Yes, this is do-able without sacrificing Linux benefits, but does require
careful modifications to the kernel

Some Constraints
• The modifications should be limited and localized so that the benefits
of Linux are maintained
• Existing drivers and kernel modules should work “as-is” (can only be
made source-compatible)
• Existing user-level code (libraries, applications, middleware) should
work “as-is” in binary form, but can be used with better real-time
performance.
• New code can be written to maintain conformance with Linux, but can
benefit from real-time design guidelines.

27
 Summary of Concerns and Solutions

Non-Preemptible Kernel
• Preemption Points via Low-Latency Patches
• Preemptible Kernels
– Two different approaches
Execution Path Latencies
• Real-Time Scheduler
Non-Schedulable Work
• Schedulable Soft-IRQ and IRQ Processing
Coarse Accuracy/Resolution Timers
• High Accuracy/Resolution Timers
User Level (POSIX) Mutexes without Priority Inheritance
• User-Level priority inheritance mutexes

28
 Preemption Points

Basic Idea
• Introduce preemption points on long kernel paths

Approach
• Instrument kernel; identify long kernel paths
• Add Preemption points (call scheduler, if needed) at various
places in the path
– Find safe preemption points, or
– Modify code to introduce safe preemption points

Pros and Cons

• Reduces latencies
• May introduce ugliness in code if preemption points require
code modifications – future maintainability problems

29
 Turning Linux into a Preemptible Kernel

Why is it non-preemptible?
• A non-preemptible kernel does not need to be reentrant
– Note: it has to be interrupt-safe
• Reduces overheads to make the kernel reentrant

But, Linux is already re-entrant

• This was necessary when the kernel was made to run on SMP
– But, only when SMP is turned on
– Uses “spin-locks” to get reentrancy on SMP machines
• Initial approach was to use one big kernel lock to get the
reentrancy
– The locks are significantly more fine-grained now
• SMP spinlocks identify all the critical sections
– A solution based on turning on spin-locks and modifying
spin-lock implementation can make the kernel preemptible

30
 Preemptible Kernel Approaches

Convert Spin Locks to:

• Preempt Locks
– A preempt lock disables preemption from (any) another
task
• Mutex Locks
– A mutex lock does not disable preemption, but achieves
mutual exclusion for critical sections by blocking task when
it is contending for a lock that is held by another task

Maintain “True” Spin Locks where necessary

For SMP machines

• Protect critical sections that previously relied on per-CPU data
structures

31
 Preemptible Kernel Approaches

Preempt Lock Approach

• Lock is non-blocking, and thus can be safely used in system and
interrupt-context
• Preemption disable sections can be long leading to large priority
inversions
• Relatively lower overhead

Mutex Lock Approach

• Lock is blocking and thus cannot be used in system and interrupt-
context
– Solution:
– Run handlers in thread context
– Wakeup handler threads in hard-irq context
• Much better priority inversion control
– Tight latencies for real-time tasks when not contending for long
critical sections with other tasks
– Requires priority inheritance to work well in the worst-case
• Slightly larger overhead

32
 Latency Problems due to Long Critical Sections

Unnecessary Coupling
• A low priority task can cause priority inversion for a high priority task
even if they don’t conflict with each other

Preempt Lock Approach

• All critical sections are coupled
– Priority inversion equals the longest critical section that could be
used by any task in the system
– Design application so that no task can use the problem critical
sections

Mutex Lock Approach

• Coupling only when same mutex lock is used
– Unnecessary coupling is a result of poor lock granularity in the
kernel
– Was a serious problem in 2.2 Linux and only a minor issue in 2.4
Linux
• Design application so that real-time tasks do not contend for locks that
can be held for a long time

33
 Real-Time Scheduler

Problem
• Standard Linux scheduler is inefficient when there are large
number of runnable tasks

Solution
• Efficient real-time schedulers
– constant time dispatching decisions

34
Schedulable Interrupt and Soft-Interrupt Threads

Execute hard and soft-interrupt handlers in threads

• Handlers become schedulable
– Threads can block
– Threads can have priorities that are changeable
• In Linux, kernel threads have user-level visible pid’s
– can change priorities conveniently from user-level
• Allows for better priority control
– A device feeding data to a low-priority task can have its
interrupt handlers run at lower priority
• Can be achieved with small & localized modifications to the
kernel
• Unnecessary overhead when the handlers are very small, or
when the prioritization is not needed

35
 High Accuracy Timers

High Resolution:
• Use a finer-granularity time structure
• POSIX timespec: seconds and nano-seconds

High Accuracy:
• Software Timers
– Maintain logically as a queue of timers sorted by expiry
time
– Include jiffy timer as a software timer
• Hardware Timer
– Use a one-shot hardware timer that expires at the next
software timer’s expiry time
– Reprogram on timer expiry and (if needed) when new
software timers are added

36
Resource Reservations in Linux
 Limitation of Priority based Scheduling

Priority is often a difficult mechanism to work with

• Global effects
– cannot understand the performance of a task without
knowing the priorities of all other things in the system
• Dynamic priorities generally handled by unpredictable heuristic
• Dynamic load means priorities poorly reflect performance
requirements
• Under overload, predictability suffers greatly
• No inherent connection to any performance metric

38
 CPU Reservations

Replenishment Interval: “T”

Execution Budget: “C”
Admission control guarantees maximum
Deadline: “D” of “C”
T units of resource within
T “D” units
Depletion Behavior: in each replenishment interval.
• “Hard” or “Soft”

Soft reserves can compete at

background priority after reserve
is depleted.

• Guaranteed to run for C ms. every T ms.

• Other threads can have all the remaining time
• Priorities handled automatically by OS
• Think of this as a “temporal firewall”

39
 Implications of CPU Reservations

Overload management (Real-Time or not)

• Give reservations to critical tasks so they aren’t affected by
overload
Support for Hard Real-Time
• Predictable Applications can expect hard real-time behavior,
regardless of other applications
Support for Soft Real-Time
• The QoS of an application can be changed dynamically, by
changing the amount of resources they get
• Applications can be written with adaptive behavior
Application Mixes on the same Run-Time System
• Real-time and Non real-time
• Unbounded components mixed with bounded Real-Time
components
Faulty components cannot hog CPU indefinitely

40
 Network Reservation Architecture

FTP

BSD Socket

INET Socket
UDP TCP
IP
FIB input forward output

Device Independent Device Driver

Device Dependent Device Driver

41
 Implications of Network Reservations

Transmit Reservations
• Ensure buffers are available for reserved streams and not
hogged by unreserved streams
• Ensure bandwidth is available for reserved streams
• Ensure reserved streams don’t hog bandwidth and buffers
beyond what they are allocated

Receive Reservations
• Ensures buffer space is available for incoming reservations, but
is not hogged unduly
• A separate execution/data path for reserved streams allows
latencies to be tightly controlled for reserved streams
• CPU reservation needed for thread processing on receive path

42
 Questions?

Manas Saksena
TimeSys Corporation
[email protected]