Jinesh (Real Time Systems in Linux)
Jinesh (Real Time Systems in Linux)
The ‘real-time’ data from the troops can be compared to the now familiar ‘real-
time stock quote’, providing information that was current within the last few seconds
or perhaps minutes. This can be referred to as ‘human real-time’ since short delays
in the tactical data provided from the field are obscured by the much longer human
delays associated with sorting and correlation. The video display observed by the
commander illustrates ‘soft real-time’, where the loss of an occasional frame will not
cause any perceived video degradation, provided that the average case performance
remains acceptable. Although techniques such as interpolation can be used to
compensate for missing frames, the system remains a soft-real time system because
the actual data was missed, and the interpolated frame represents derived, rather
than actual data.
Since an embedded system often performs only a single task, the differences
between soft and hard real-time for these applications are not as critical as one
would think. However, as true multi-tasking operating systems, such as Linux,
are adopted for use in increasingly complex systems, the need for hard real-time
often becomes apparent. To further confuse the real-time issue, the general term
“Real-Time Operating System (RTOS)” is used to refer to one that can provide either
hard or soft real-time capabilities but not necessarily both. Thus all operating
systems labeled as “RTOS” are not created equally.
The real-time Linux scheduler treats the Linux operating system kernel as the
idle task. Linux only executes when there are no real time tasks to run, and the real
time kernel is inactive. The Linux task can never block interrupts or prevent itself
from being preempted. The mechanism that makes this possible is the software
emulation of interrupt control hardware. When any code in Linux tries to disable
interrupts, the real time system intercepts the request, records it, and returns it to
Linux. In fact, Linux is not permitted to ever really disable hardware interrupts, and
hence, regardless of the state of Linux, it cannot add latency to the interrupt
response time of the real time system.
When an interrupt occurs, the real time kernel intercepts the interrupt and
decides what to dispatch. If there is a real time handler for the interrupt, the
appropriate handler is invoked. If there is no real time interrupt handler, or if the
handler indicates that it wants to share the interrupt with Linux, then the interrupt is
marked as pending. If Linux has requested that interrupts be enabled, any pending
interrupts are enabled, and the appropriate Linux interrupt handler invoked - with
hardware interrupts re-enabled. Regardless of the state of Linux: running in kernel
mode; running a user process; disabling or enabling interrupts; the real-time system
is always able to respond to an interrupt. Real-time Linux decouples the mechanisms
of the real time kernel from the mechanisms of the general purpose Linux kernel so
that each can be optimized independently and so that the real-time kernel can be
kept small and simple.
There are two primary variants of hard real-time Linux available: RTLinux and
RTAI.
When an interrupt occurs, the emulation software checks to see whether Linux
has interrupts enabled, if so the interrupt is delivered to the Linux kernel. If not, the
interrupt is held pending the Linux kernel re-enabling interrupts. In this way, the Linux
kernel does not have direct control over interrupts, and cannot delay the processing
of real time interrupts, as these interrupts do not pass through the emulation
software. Instead, they are delivered direct to the real time kernel. This also means
that the scheduling of real time tasks cannot be delayed by Linux.
As mentioned earlier, there are two current ‘flavours’ of Real-Time Linux: RTAI
and RTLinux. We are here to discuss about the RTAI solution for the same.
Ø FPU support
Ø One-shot and periodic schedulers
Ø
RTAI's performance is very competitive with the best commercial Real Time
Operating Systems (such as VxWorks, QNX etc), offering typical context switch times
of 4 uSec, 20 uSec interrupt response, 100 KHz periodic tasks, and 30 KHz one-shot
task rates. The main limitation being imposed by the system hardware, rather than
the real-time software itself.
Like standard Linux, RTAI is open source and thus it can be downloaded from the
internet and manually patched into a Linux system, or, alternatively it can be easily
installed using the Lineo Industrial Solutions Group, Embedix-RealTime installation
CD which allows real-time Linux features, services, and tools to be applied on top of
an existing and already configured
Linux system.
Architecture
The first problem – that of capturing and redirecting the interrupts, was
addressed by creating a small Real-Time Hardware Abstraction Layer (RTHAL)
which intercepts all hardware interrupts and routes them to either standard Linux or
to real-time tasks depending on the requirements of the RTAI schedulers. Interrupts
which are meant for a scheduled real-time task are sent directly to that task, while
those interrupts which are not required by any scheduled real-time task are sent
directly to standard Linux where they are handled according to normal needs. In this
manner, the RTHAL provides a framework onto which RTAI is mounted with the
ability to fully pre-emptible Linux.
A key component of the RTHAL is the Interrupt Descriptor Table (IDT), which
provides a set of pointers, which defines to which processes each of the interrupts
should be routed. The IDT gives the ability to easily implement or disable all RTHAL
services, allowing the developer to easily track low level bugs under the same kernel
configuration, but with and without the HAL in place.
Upon activation of RTAI within the RTHAL, the following actions occur:
Ø Change of the handler functions in the new idt_table to the RTAI dispatchers
so that it takes control of the system hardware and its interrupts.
Ø Provision of a timer and locking services for the real time domain. The timer’s
services provide control of the 8254 and APIC timers. Under these conditions,
Linux is run only as the idle task to the real time domain. RTAI handles the
Linux "real-time clock" properly by passing up the true clock value upon re-
activation of Linux, thus preserving the correct time settings and keeping Linux
oblivious to its presence.
Ø The changes needed to the standard Linux kernel are minimal, a few lines in
eleven source files plus configuration additions to three files in the build
structure, (Makefile, configuration files etc). This lower intrusion on the
standard Linux kernel improves the code maintainability, and makes it easier
to keep the real time modifications up-to-date with the latest release of the
Linux kernel.
Ø The real time extensions can easily be removed by replacing the interrupt
function pointers with the original Linux routines. This is especially useful in
certain debugging situations when it is necessary to remove the extensions,
and when verifying the performance of standard Linux with and without the
real time extensions.
Ø The Linux kernel suffers a slight, but essentially negligible, performance loss
when RTAI is added due to the indirection through pointers to the interrupt
mask, unmask and flag functions.
The third problem is solved, again by using the module loading services of
Linux. Recall that the HAL and real-time modules of RTAI effectively run standard
Linux as the lowest priority task, with the ability (see below) to insert your application-
specific real-time task at a higher priority. In general real-time Linux tasks run with
kernel modules (although extended LXRT is changing this requirement) where they
have direct access to the HAL and RTAI service modules.
Those of you who are familiar with the Linux kernel will note that neither the
memory space occupied by the kernel or its associated modules, is provided any
protection against undesired read/write access. Thus, an improperly implemented
kernel module can over-write critical areas of system memory. This memory over-
write will generally require a system reboot, but it can, in exceptional circumstances
require a complete re-installation of the operating system.
While the issues related to unprotected system memory space are mostly
relevant during the development phase of a kernel module it certainly is a major
inconvenience during development, as once the system has been “damaged”, it is
difficult to debug and may only become operable again by rebooting.
To alleviate this, the developers of RTAI have produced the LXRT (Linux
Real-Time) module that provides the ability to develop real-time applications from
within standard user space using the same RTAI API.
RTAI now also includes the ability to provide hard real-time tasks from user
space using the extended-LXRT feature. Although this provides a protected MMU
context for the real-time task, it still lacks the necessary trap handler to provide fault
recovery during development. This is being address as part of the on-going
development.
Since Linux, like all UNIX systems requires that kernel modules be recompiled
for each version of the kernel that it will be linked to, real-time tasks are generally not
portable across machines running different kernel versions. This requirement means
that kernel module style real-time tasks can only be deployed on machines that run
the same kernel configuration as the development platform.
Ø Provide the application's source code so that the end user can compile and
then install the modules as required, or
Ø In addition to the application, you must provide an associated kernel (of the
correct version) to be compiled and installed by the user.
While either of these solutions is often technically acceptable, the use of LXRT
helps one avoid the kernel dependency issue altogether because it allows a real-time
task to run from standard user space – provided that the deployment platform
includes (and loads) RTAI and the LXRT features. In this case, it is an easy matter to
load the LXRT modules (in addition to those required for standard RTAI services)
and to provide the application as a standard user space task.
RTAI's full feature set can be broken down into a set of basic services – such
as the schedulers, FIFOs, and shared memory, and a set of advanced features such
as POSIX and dynamic memory allocation. Both basic and advanced services are
provided via kernel modules, which can be loaded and unloaded using the standard
Linux insmod and rmmod commands. Although the rtai module is required every
time any real-time service is needed, all other modules are necessary only when
their associated real-time services are desired.
For example, if you want to install only interrupt handlers, you only have to
load the rtai module. If you also want to communicate with standard Linux processes
using FIFOs, then you would then load the rtai_fifos module in addition to the rtai
module. These modules can be dynamically loaded and unloaded – however it is
necessary to pay attention to the order in which they are loaded and unloaded, as
some modules require the services of other. Alternatively, if the modules are installed
in a directory known to modprobe (e.g /lib/modules/<xxx>/misc) and depmod is run,
your real-time module along with all the RTAI modules it depends on may be loaded
by a single 'modprobe' of your application module.
RTAI's basic services are provided by four modules, which allow hard real-
time, fully preemptive scheduling based on a fixed priority scheme. These four key
modules are:
Ø rtai - the basic RTAI framework, plus interrupt dispatching and timer support.
Ø rtai_sched – the real-time, pre-emptive, priority-based scheduler, chosen
according to the hardware configuration.
Ø rtai_fifos – FIFOs and semaphores
Ø rtai_shm - shared memory (note that you can also use the ‘mbuff’ module for
access to shared memory).
The advanced features of RTAI such as LXRT, Pthreads (POSIX 1003.1c) and
Pqueues (POSIX 1003.1b), can be added with these modules:
Ø lxrt - LXRT
Ø rtai_pthread (Pthreads - loaded for POSIX 1003.1c support)
Ø rtai_pqueues (Pqueues – loaded for POSIX 1003.1b message queues
support)
Ø rt_mem_mgr – dynamic memory management for real-time (note that this is
most often simply built-in to the scheduler).
RTAI Schedulers
Although only one type of scheduler can be insmod'ed at any time, RTAI
includes several different types – each uniquely suited to a specific combination of
hardware and tasking requirements. It is generally not necessary for the user to
manually install the proper scheduler because the installation process is usually able
to determine the appropriate scheduler from the hardware configuration of the target
machine. It then copies and links the appropriate scheduler so that it is called by the
generic rtai_sched reference. However, in cases where the developer wants to
investigate other RTAI schedulers, or when he is determining which scheduler should
be installed onto the target platform, an understanding of each option is required.
The RTAI distribution includes three different priority based, pre-emptive real
time schedulers: the Uni-Processor (UP) scheduler; the Multi Uni-Processor
(MUP) scheduler; and the Symmetric Multi-Processor (SMP) scheduler, which
each incorporate standard RTOS scheduling services like resume, yield, suspend,
make periodic, wait until etc.
UP scheduler
SMP scheduler
This scheduler is intended for multi-processor machines but can support either
the 8254 or APIC based timers and supports either single-shot or periodic scheduling
but not both simultaneously. Tasks can run symmetrically on any or a cluster of
CPUs, or be bound to a single CPU.
Depending on the hardware's architecture, either the 8254 or the local APIC
timer schedulers are chosen for SMP operations. The chosen one will be built and
then installed as the ‘generic’ rtai_sched.o module. RTAI supports true Symmetric
Multi Processing (SMP) architectures by providing dynamic task loading and IRQ
management similar to Linux' SMP operations. RTAI contrasts sharply to other real-
time Linux implementations, which do not support standard SMP load balancing
techniques.
Under RTAI, by default all tasks are defined to run on any of the CPUs and
are automatically moved between CPUs as the system's processing and load
requirements change. However, to accommodate situations where manual task
distribution is able to manage the task loading more efficiently than the automatic
load distribution services, the developer also has the ability to assign individual tasks
to any single CPU or to a CPU subset. Additionally, any specific real-time interrupt
service can be assigned to any specific CPU, and because the ability to force an
interrupt to any specific CPU is not related to the SMP scheduler, these two
operations can be performed independently.
To assign individual tasks to any subset of the CPU pool – including just a
single CPU, use:
Ø rt_set_runnable_on_cpus or
Ø rt_set_runnable_on_cpuid
Ø rt_assign_irq_to_cpu, and
Ø rt_reset_irq_to_sym_mode.
MUP scheduler
However, like the SMP schedulers, the MUP can use inter-CPU services
related to semaphores, messages and mailboxes. The main advantage of the MUP
scheduler comes from the ability to be able to use mixed timers simultaneously, i.e.
periodic and one-shot, where periodic timers can be based on different periods.
Be warned that the APIC based scheduler cannot be used for UP, unless you
have the local APIC enabled, i.e. an SMP machine with just one CPU mounted on
the motherboard. It is a fully symmetric scheduler, where at task init all real time
tasks default to using any available cpu. However you can chose either forcing a task
to a single cpu or to let it use any subset of those available by calling the function
"rt_set_runnable_on_cpus". That function set the task structure variable
"runnable_on_cpus" with the bit map of the usable cpus, which is defaulted to
"cpu_present_map", meaning that any available cpu can be used, at task
initialization.
For the APIC timer based scheduler if you want to statically optimize the load
distribution by binding tasks to specific cpus it can be usefull to use
"rt_get_timer_cpu()" just after having started the timer, to know which cpu is using its
local APIC timer to pace the scheduler. Note that for the oneshot case that will be the
main timing cpu but not the only one. In fact which local APIC is shot depends on the
task scheduling out, as that will determine the next shooting.
For the 8254 timer based scheduler a statically optimized load distribution
could bind the 8254 interrupt to a specific cpu by using "rt_assign_irq_to_cpu" and
"rt_reset_irq_to_sym_mode", and then assign tasks in appropriate way to any cpu or
cpu cluster.
Actually there are two schedulers: the pessimistic one keeps the global lock
throughout any scheduling, while the optimistic one releases the lock.
n o t e : In the future it is expected that the MUP will provide the ability to force
critical real-time tasks onto the CPU cache on Pentium IIIs immediately after the task
switch.
The MUP scheduler derives its name from the fact that real time tasks MUST
be bounded to a single CPU at the very task initialization. They can be afterward
moved by using the function "rt_set_runnable_on_cpus". The MUP scheduler can
however use any inter CPUs services related to semaphores, messages and
mailboxes. The advantage of using the MUP scheduler comes mainly from the
possibility of using mixed timers simultaneously, i.e. periodic and one shot, where
periodic timers can be based on different periods, and of possibly forcing critical
tasks on the CPU cache on PIIIs, in the future. With dual SMP one cannot say that
there is a difference in efficiency. MUP has been developed primarily for our not so
fast, a few KHz, PWM actuators, BANG-BANG air jet thrusters, coupled to a periodic
scheduler.
All the functions of UP and SMP schedulers are available in the MUP
scheduler, and MUP specific functions can be used under UP-SMP. Some default
action is implied in scheduler-specific features. The main difference can be seen for
functions whose name ends with "_cpuid". Such functions imply the specification of a
CPU number and came into play with the MUP scheduler whenever a cpuid had to
be declared. Typical examples are: task init and time conversions, when time formats
differ. Please note that there is a difference between "cpuid", i.e. the CPU number,
and "cpu_map", i.e. 1 << cpuid. Thus if you use task init with a cpuid on UP-SMP
schedulers you have it assigned to the only CPU available or mapped to the declared
one, while if you just task init on MUP your task is assigned to the CPU loaded with
less tasks, and so on.
and
void rt_free_apic_timers(void)
The "struct apic_timer_setup_data { int mode, count; };" allows you to define
the mode and count to be used for each timer as:
Conclusion
Are you...
Ø All that on top of a general purpose operating system with penguins inside?
Then RTAI might be the realtime extension of your choice! RTAI is a true
community project. With the support of the OSS community, “Linux + RTAI” has now
become one of the most reliable Real Time Operating System solution with the least
TCO. With the above notions, its clear that Linux/RTAI is the next generation RTOS
solution which is going to be the future.
References
guide.
programmers guide.