2 M ECHANISM : L IMITED D IRECT E XECUTION

OS Program
Create entry for process list
Allocate memory for program
Load program into memory
Set up stack with argc/argv
Clear registers
Execute call main()
Run main()
Execute return from main
Free memory of process
Remove from process list

Figure 6.1: Direct Execution Protocol (Without Limits)

to it, and starts running the user’s code. Figure 6.1 shows this basic di-
rect execution protocol (without any limits, yet), using a normal call and
return to jump to the program’s main() and later back into the kernel.
Sounds simple, no? But this approach gives rise to a few problems
in our quest to virtualize the CPU. The first is simple: if we just run a
program, how can the OS make sure the program doesn’t do anything
that we don’t want it to do, while still running it efficiently? The second:
when we are running a process, how does the operating system stop it
from running and switch to another process, thus implementing the time
sharing we require to virtualize the CPU?
In answering these questions below, we’ll get a much better sense of
what is needed to virtualize the CPU. In developing these techniques,
we’ll also see where the “limited” part of the name arises from; without
limits on running programs, the OS wouldn’t be in control of anything
and thus would be “just a library” — a very sad state of affairs for an
aspiring operating system!

6.2 Problem #1: Restricted Operations

Direct execution has the obvious advantage of being fast; the program
runs natively on the hardware CPU and thus executes as quickly as one
would expect. But running on the CPU introduces a problem: what if
the process wishes to perform some kind of restricted operation, such
as issuing an I/O request to a disk, or gaining access to more system
resources such as CPU or memory?

T HE C RUX : H OW T O P ERFORM R ESTRICTED O PERATIONS

A process must be able to perform I/O and some other restricted oper-
ations, but without giving the process complete control over the system.
How can the OS and hardware work together to do so?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]
 2 M ECHANISM : L IMITED D IRECT E XECUTION

OS Program
Create entry for process list
Allocate memory for program
Load program into memory
Set up stack with argc/argv
Clear registers
Execute call main()
Run main()
Execute return from main
Free memory of process
Remove from process list

Figure 6.1: Direct Execution Protocol (Without Limits)

to it, and starts running the user’s code. Figure 6.1 shows this basic di-
rect execution protocol (without any limits, yet), using a normal call and
return to jump to the program’s main() and later back into the kernel.
Sounds simple, no? But this approach gives rise to a few problems
in our quest to virtualize the CPU. The first is simple: if we just run a
program, how can the OS make sure the program doesn’t do anything
that we don’t want it to do, while still running it efficiently? The second:
when we are running a process, how does the operating system stop it
from running and switch to another process, thus implementing the time
sharing we require to virtualize the CPU?
In answering these questions below, we’ll get a much better sense of
what is needed to virtualize the CPU. In developing these techniques,
we’ll also see where the “limited” part of the name arises from; without
limits on running programs, the OS wouldn’t be in control of anything
and thus would be “just a library” — a very sad state of affairs for an
aspiring operating system!

6.2 Problem #1: Restricted Operations

Direct execution has the obvious advantage of being fast; the program
runs natively on the hardware CPU and thus executes as quickly as one
would expect. But running on the CPU introduces a problem: what if
the process wishes to perform some kind of restricted operation, such
as issuing an I/O request to a disk, or gaining access to more system
resources such as CPU or memory?

T HE C RUX : H OW T O P ERFORM R ESTRICTED O PERATIONS

A process must be able to perform I/O and some other restricted oper-
ations, but without giving the process complete control over the system.
How can the OS and hardware work together to do so?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]
 2 M ECHANISM : L IMITED D IRECT E XECUTION

OS Program
Create entry for process list
Allocate memory for program
Load program into memory
Set up stack with argc/argv
Clear registers
Execute call main()
Run main()
Execute return from main
Free memory of process
Remove from process list

Figure 6.1: Direct Execution Protocol (Without Limits)

to it, and starts running the user’s code. Figure 6.1 shows this basic di-
rect execution protocol (without any limits, yet), using a normal call and
return to jump to the program’s main() and later back into the kernel.
Sounds simple, no? But this approach gives rise to a few problems
in our quest to virtualize the CPU. The first is simple: if we just run a
program, how can the OS make sure the program doesn’t do anything
that we don’t want it to do, while still running it efficiently? The second:
when we are running a process, how does the operating system stop it
from running and switch to another process, thus implementing the time
sharing we require to virtualize the CPU?
In answering these questions below, we’ll get a much better sense of
what is needed to virtualize the CPU. In developing these techniques,
we’ll also see where the “limited” part of the name arises from; without
limits on running programs, the OS wouldn’t be in control of anything
and thus would be “just a library” — a very sad state of affairs for an
aspiring operating system!

6.2 Problem #1: Restricted Operations

Direct execution has the obvious advantage of being fast; the program
runs natively on the hardware CPU and thus executes as quickly as one
would expect. But running on the CPU introduces a problem: what if
the process wishes to perform some kind of restricted operation, such
as issuing an I/O request to a disk, or gaining access to more system
resources such as CPU or memory?

T HE C RUX : H OW T O P ERFORM R ESTRICTED O PERATIONS

A process must be able to perform I/O and some other restricted oper-
ations, but without giving the process complete control over the system.
How can the OS and hardware work together to do so?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]
M ECHANISM : L IMITED D IRECT E XECUTION 5

OS @ boot Hardware
(kernel mode)
initialize trap table
remember address of...
syscall handler

OS @ run Hardware Program

(kernel mode) (user mode)
Create entry for process list
Allocate memory for program
Load program into memory
Setup user stack with argv
Fill kernel stack with reg/PC
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to main
Run main()
...
Call system call
trap into OS
save regs
(to kernel stack)
move to kernel mode
jump to trap handler
Handle trap
Do work of syscall
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to PC after trap
...
return from main
trap (via exit())
Free memory of process
Remove from process list

Figure 6.2: Limited Direct Execution Protocol

locations of these trap handlers, usually with some kind of special in-
struction. Once the hardware is informed, it remembers the location of
these handlers until the machine is next rebooted, and thus the hardware
knows what to do (i.e., what code to jump to) when system calls and other
exceptional events take place.

T HREE
© 2008–23, A RPACI -D USSEAU
E ASY
P IECES
M ECHANISM : L IMITED D IRECT E XECUTION 5

OS @ boot Hardware
(kernel mode)
initialize trap table
remember address of...
syscall handler

OS @ run Hardware Program

(kernel mode) (user mode)
Create entry for process list
Allocate memory for program
Load program into memory
Setup user stack with argv
Fill kernel stack with reg/PC
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to main
Run main()
...
Call system call
trap into OS
save regs
(to kernel stack)
move to kernel mode
jump to trap handler
Handle trap
Do work of syscall
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to PC after trap
...
return from main
trap (via exit())
Free memory of process
Remove from process list

Figure 6.2: Limited Direct Execution Protocol

locations of these trap handlers, usually with some kind of special in-
struction. Once the hardware is informed, it remembers the location of
these handlers until the machine is next rebooted, and thus the hardware
knows what to do (i.e., what code to jump to) when system calls and other
exceptional events take place.

T HREE
© 2008–23, A RPACI -D USSEAU
E ASY
P IECES
M ECHANISM : L IMITED D IRECT E XECUTION 5

OS @ boot Hardware
(kernel mode)
initialize trap table
remember address of...
syscall handler

OS @ run Hardware Program

(kernel mode) (user mode)
Create entry for process list
Allocate memory for program
Load program into memory
Setup user stack with argv
Fill kernel stack with reg/PC
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to main
Run main()
...
Call system call
trap into OS
save regs
(to kernel stack)
move to kernel mode
jump to trap handler
Handle trap
Do work of syscall
return-from-trap
restore regs
(from kernel stack)
move to user mode
jump to PC after trap
...
return from main
trap (via exit())
Free memory of process
Remove from process list

Figure 6.2: Limited Direct Execution Protocol

locations of these trap handlers, usually with some kind of special in-
struction. Once the hardware is informed, it remembers the location of
these handlers until the machine is next rebooted, and thus the hardware
knows what to do (i.e., what code to jump to) when system calls and other
exceptional events take place.

T HREE
© 2008–23, A RPACI -D USSEAU
E ASY
P IECES
8 M ECHANISM : L IMITED D IRECT E XECUTION

memory that it shouldn’t be able to access, it will generate a trap to the

OS. The OS will then have control of the CPU again (and likely terminate
the offending process).
Thus, in a cooperative scheduling system, the OS regains control of
the CPU by waiting for a system call or an illegal operation of some kind
to take place. You might also be thinking: isn’t this passive approach less
than ideal? What happens, for example, if a process (whether malicious,
or just full of bugs) ends up in an infinite loop, and never makes a system
call? What can the OS do then?

A Non-Cooperative Approach: The OS Takes Control

Without some additional help from the hardware, it turns out the OS can’t
do much at all when a process refuses to make system calls (or mistakes)
and thus return control to the OS. In fact, in the cooperative approach,
your only recourse when a process gets stuck in an infinite loop is to
resort to the age-old solution to all problems in computer systems: reboot
the machine. Thus, we again arrive at a subproblem of our general quest
to gain control of the CPU.

T HE C RUX : H OW T O G AIN C ONTROL W ITHOUT C OOPERATION

How can the OS gain control of the CPU even if processes are not being
cooperative? What can the OS do to ensure a rogue process does not take
over the machine?

The answer turns out to be simple and was discovered by a number

of people building computer systems many years ago: a timer interrupt
[M+63]. A timer device can be programmed to raise an interrupt every
so many milliseconds; when the interrupt is raised, the currently running
process is halted, and a pre-configured interrupt handler in the OS runs.
At this point, the OS has regained control of the CPU, and thus can do
what it pleases: stop the current process, and start a different one.
As we discussed before with system calls, the OS must inform the
hardware of which code to run when the timer interrupt occurs; thus,
at boot time, the OS does exactly that. Second, also during the boot

T IP : D EALING W ITH A PPLICATION M ISBEHAVIOR

Operating systems often have to deal with misbehaving processes, those
that either through design (maliciousness) or accident (bugs) attempt to
do something that they shouldn’t. In modern systems, the way the OS
tries to handle such malfeasance is to simply terminate the offender. One
strike and you’re out! Perhaps brutal, but what else should the OS do
when you try to access memory illegally or execute an illegal instruction?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]
8 M ECHANISM : L IMITED D IRECT E XECUTION

memory that it shouldn’t be able to access, it will generate a trap to the

OS. The OS will then have control of the CPU again (and likely terminate
the offending process).
Thus, in a cooperative scheduling system, the OS regains control of
the CPU by waiting for a system call or an illegal operation of some kind
to take place. You might also be thinking: isn’t this passive approach less
than ideal? What happens, for example, if a process (whether malicious,
or just full of bugs) ends up in an infinite loop, and never makes a system
call? What can the OS do then?

A Non-Cooperative Approach: The OS Takes Control

Without some additional help from the hardware, it turns out the OS can’t
do much at all when a process refuses to make system calls (or mistakes)
and thus return control to the OS. In fact, in the cooperative approach,
your only recourse when a process gets stuck in an infinite loop is to
resort to the age-old solution to all problems in computer systems: reboot
the machine. Thus, we again arrive at a subproblem of our general quest
to gain control of the CPU.

T HE C RUX : H OW T O G AIN C ONTROL W ITHOUT C OOPERATION

How can the OS gain control of the CPU even if processes are not being
cooperative? What can the OS do to ensure a rogue process does not take
over the machine?

The answer turns out to be simple and was discovered by a number

of people building computer systems many years ago: a timer interrupt
[M+63]. A timer device can be programmed to raise an interrupt every
so many milliseconds; when the interrupt is raised, the currently running
process is halted, and a pre-configured interrupt handler in the OS runs.
At this point, the OS has regained control of the CPU, and thus can do
what it pleases: stop the current process, and start a different one.
As we discussed before with system calls, the OS must inform the
hardware of which code to run when the timer interrupt occurs; thus,
at boot time, the OS does exactly that. Second, also during the boot

T IP : D EALING W ITH A PPLICATION M ISBEHAVIOR

Operating systems often have to deal with misbehaving processes, those
that either through design (maliciousness) or accident (bugs) attempt to
do something that they shouldn’t. In modern systems, the way the OS
tries to handle such malfeasance is to simply terminate the offender. One
strike and you’re out! Perhaps brutal, but what else should the OS do
when you try to access memory illegally or execute an illegal instruction?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]
8 M ECHANISM : L IMITED D IRECT E XECUTION

memory that it shouldn’t be able to access, it will generate a trap to the

OS. The OS will then have control of the CPU again (and likely terminate
the offending process).
Thus, in a cooperative scheduling system, the OS regains control of
the CPU by waiting for a system call or an illegal operation of some kind
to take place. You might also be thinking: isn’t this passive approach less
than ideal? What happens, for example, if a process (whether malicious,
or just full of bugs) ends up in an infinite loop, and never makes a system
call? What can the OS do then?

A Non-Cooperative Approach: The OS Takes Control

Without some additional help from the hardware, it turns out the OS can’t
do much at all when a process refuses to make system calls (or mistakes)
and thus return control to the OS. In fact, in the cooperative approach,
your only recourse when a process gets stuck in an infinite loop is to
resort to the age-old solution to all problems in computer systems: reboot
the machine. Thus, we again arrive at a subproblem of our general quest
to gain control of the CPU.

T HE C RUX : H OW T O G AIN C ONTROL W ITHOUT C OOPERATION

How can the OS gain control of the CPU even if processes are not being
cooperative? What can the OS do to ensure a rogue process does not take
over the machine?

The answer turns out to be simple and was discovered by a number

of people building computer systems many years ago: a timer interrupt
[M+63]. A timer device can be programmed to raise an interrupt every
so many milliseconds; when the interrupt is raised, the currently running
process is halted, and a pre-configured interrupt handler in the OS runs.
At this point, the OS has regained control of the CPU, and thus can do
what it pleases: stop the current process, and start a different one.
As we discussed before with system calls, the OS must inform the
hardware of which code to run when the timer interrupt occurs; thus,
at boot time, the OS does exactly that. Second, also during the boot

T IP : D EALING W ITH A PPLICATION M ISBEHAVIOR

Operating systems often have to deal with misbehaving processes, those
that either through design (maliciousness) or accident (bugs) attempt to
do something that they shouldn’t. In modern systems, the way the OS
tries to handle such malfeasance is to simply terminate the offender. One
strike and you’re out! Perhaps brutal, but what else should the OS do
when you try to access memory illegally or execute an illegal instruction?

O PERATING
S YSTEMS WWW. OSTEP. ORG
[V ERSION 1.10]