CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 1

80386 Protection Mode

1 Introduction and Initial Discussion

Question 1: Assume that to write to /etc/passwd file, the CPU instructions (Machine codes) that get
executed are c1 , c2 , . . ., cn . And also assume that the instructions related to access control is a1 , . . ., as .
Now let’s construct a new program p′ = c1 , . . ., cn - a1 , ..., as , and let run it directly on CPU, can we
succeed in writing to /etc/passwd file?
Answer: In 8086, you can do this. In 80386, you cannot!

Question 2: Assume that we know the address of the code for system calls write(), which can write
data to disks. There are two ways to call it:

1. Go through the system-call approach, which is subject to access control.

2. Directly jump to that code.

We know the first approach works, but can the second choice succeed? If not, what prevents a program from
jumping to that code?
Answer: the hardware access control disallows it. There is a security policy to prevent the above direct
jumping from happening, and the policy is enforced by hardware. We are interested in how such access
control policy is defined and how hardware enforces the policy.

Discussion: From the above two questions and their answers, it seems certain kind of access control is
protecting the systems. If you get chance to design such a protection scheme, how would you design an
access control like this?

• Three components of a security policy: subject, object, and rules.

• Memory address can be used as object (this is quite straightforward because we are protecting mem-
ory). However, at what granularity, each byte, word, block? What are the disadvantages and advan-
tages of your choices?

• What can be used as subject?

– Can we use user ID or process ID as subjects? No we cannot use things that are defined in
an operating system, because this access control is not part of an OS, it is underneath an OS.
Processes and users are meaningful in an OS, but the underlying hardware does not know what
those are.
– In other words, how to give each instruction an identity?

• How to design the rules (or policies)?

– How to represent the policies?

– Where to store the policies?

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 2

– When to enforce the policies?

– Access matrix: high cost, inflexible, etc.

• Mandatory versus Discretionary Access Control

– If MAC is used, system-wise mandatory access control polices are enforced.

– If DAC is used, the owner of an object can set up security polices.
– 80386 Protection Mode chose MAC.
∗ It is believed that mandatory access control is better for highly secure systems.
∗ The implementation of DAC scheme might be inefficient. Note that such access control
policy is checked for each memory access, so it must be very fast.

• In MAC, objects and subjects are assigned security labels, and access control policies are defined
based on these labels. What can be used as labels and where to put them?

2 The Ring Architecture and Segments

• History

– Late 70’s: 8086, Real Mode and has no protection.

– 1982: 80286, Real Mode and 16b Protected Mode.
– 1985: 80386, Real Mode and 32b Protected Mode.

Protection Rings

Operating System Kernel

Operating System Services

Level 0

Level 1 Applications

Level 2

Level 3

Figure 1: Ring Levels of 80386

• The Ring architecture: the labels used by MAC (see Figure 5).

– 80386 has four rings. Each ring is associated with different privileges. Ring 0 is the most
privileged ring, and the OS kernel usually resides within this layer.
– Each object and subject is associated with a label, called ring. This label is used as the subject
in access control policies.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 3

– Whether a subject can access an object is decided by the mandatory access control policy that
are implemented in the hardware.

• Memory protection across ring boundaries: once we divide the memory into several rings, we can
define security policies based on rings. For example, we can prevent code in ring 3 from accessing
data in ring 0, etc. The question is that, when conducting access control, how CPU learns the ring
labels of a subject and an object.

– When CPU enforces the access control policies, it must know the ring label of both the subject
and object in an efficient way.
– CPL: Current Privilege Level, the label on subjects.
∗ CPL is stored in a register (Bits 0 and 1 of the CS and SS segment registers).
∗ CPL represents the privilege level of the currently executing program or procedure.
∗ Normally, the CPL is equal to the privilege level of the code segment from which instruc-
tions are being fetched (there is one exception, and we will talk about later when we talk
about conforming code segments).
∗ The processor changes the CPL when program control is transferred to a code segment with
a different privilege level.
– DPL: Descriptor Privilege Level, the label on objects.
∗ DPL is the privilege level of an object. When the currently executing code segment attempts
to access an object, the DPL is compared with CPL.
∗ Where should DPL be stored?
· Discussion: stored in each byte? Stored for each block (at the beginning of a block)?
or somewhere else?

• Memory protection within the same ring: Rings can achieve memory protection across ring bound-
aries, but they cannot memory protection within the same ring. For example, when we develop an
operating system for 80386, we would like user processes to run at ring 3, but we do not want one
process to access another process’s memory (all within ring 3). Rings cannot achieve this kind of
protection (memory isolation). We need another access control mechanism for this protection.

– Let us divide memory into segments. Each process can take one or more segments. Whenever a
process tries to access a memory, access control should be enforced to achieve memory isolation.
– Discussion: What access control model do we use? ACL or Capability?
– ACL Approach: we associate each segment with an access control list. Each memory access will
go through this list. This is too time consuming, because the list might be long. The processor
cannot afford to go through a long list for each memory access.
– Capability Approach: each process is assigned a list of capability, each corresponding to one
of its segments. There are two important issues in capability-based access control. First, where
should the capabilities be stored? They cannot be forged by users. Privileged rings are good
places for storing capabilities. Second, there are two common ways to implement capability-
based access control:
∗ Capability List: the code does not need to explicitly show its capabilities when access a
memory; instead, the processor searches the capability list of the process to find the one
that is appropriate, if any. This approach has the same problem as the ACL approach: list
might be too long.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 4

∗ Index of Capabilities: when a code tries to access a memory, it should present a “ticket”,
which is the index of the actual capability stored in a privileged ring. This way, the proces-
sor only needs to check this specific capability. The performance is much better than the
capbility list approach. This is similar to how the file descriptor is implemented.
– 80386 chooses the capability as its access control model to achieve memory isolation; it uses the
index approach.

• Logical and Linear Address (see Figure 2)

– Logical address: consist of segment selector and offset. The processor converts the logical
address to linear address using the segment descriptor indicated by the segment selector.
– Linear address: when the paging is disabled, the linear address is actually the physical address;
when the paging is enabled, the linear address is converted to physical address through paging
mechanisms.
– Segment selectors are provided by segment registers: For example, in the following instruction,
the segment selector is provided by the register DS, and the offset is 80:
MOV %DS:[80], %EAX.

15 0 31 0
Logical
Seg. Selector Offset
Address

Descriptor Table Mandatory Access Control:

(Privilege Check)

Segment
Descriptor +

31 0
Capability−Based Access Control Linear Address
(Selecting Segment)

Figure 2: Logical Address to Linear Address

– Two access control mechanisms are used here:

∗ Capability-based Access Control: the segment selector and segment descriptor are actually
the capability concept. The segment descriptor is the capability, while the segment selector
is the index to the descriptor. Segment selectors are accessible to user programs, but not
segment descriptors.
∗ Mandatory Access Control: even if a process has a capability, its access right is further
restricted by another level of access control that is based on MAC. This level of access
control ensures that ring based access policies are enfored.
∗ Note: one might wonder whether the second-level of access control is redundant; if the
access is not allowed, why bother to create a capability (descriptor) for a process at the first
place? The answer will be partially clear when when talk about Global Descriptor Table
(GDT), which contains capabilities shared by all processes; however, whether a process can
use a capability in GDT depends on its privilege level.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 5

• Segment Selector (see Figure 3)

– TI: Table Indictor. Indicate whether GDT or LDT is used.
– Index: The processor multiplies the index by 8 (the number of bytes in a segment descriptor),
and add the result to the base address of the GDT or LDT based on the TI value (the base
addresses are stored in the GDTR or LDTR register, respectively).
∗ GDT: Global Descriptor Table. Each system must have one GDT defined.
∗ LDT: Local Descriptor Table. One or more LDT can be defined. For example, an LDT can
be defined for each task being run, or some or all tasks can share the same LDT.
– RPL: Request Privilege Level. Specifies the privilege level of the selector. We will explain this
later.

Segment Selector
15 3 2 1 0
T
Index I RPL

TI: Table Indicator

TI=0 TI=1

GDT LDT

GDTR Register LDTR Register

Figure 3: Segment Selector

• Segment Descriptor (see Figure 4)

– Base (32 bits): the base address of the segment.
– Segment Limit (20 bits): the size of the segment. The processor will ensure that the offset of the
address does not go beyond the segment limit.
– Type (4 bits): specify the type of segment. The processor will enforce type rules. For example,
no instruction may write into a data segment if it is not writable, no instruction may read an
executable segment unless the readable flag is set, etc.
∗ Data Type: Read-Only, Read/Write, etc.
∗ Code Type: Execute-Only, Execute/Read
– DPL (2 bits): Descriptor Privilege Level. It specifies the ring level of the segment. DPL is used
in access control.

• Segment Registers
– Due to the address translation step, accessing data or code in memory involves two memory ac-
cess, one for retrieving segment descriptor from the descriptor table, and the other for accessing
the actual memory. To avoid consulting a descriptor table for each memory acess, 80386 caches
information from descriptors in segment registers.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 6

Access Rights
31 24 20 16 8 0
Seg.
Base 31:24 Limit P DPL S Type Base 23:16
19:16
31 16 0

Base Address 15:00 Segment Limit 15:00

Figure 4: Segment Descriptor

– A segment register has a “visible” part and a “hidden” part.

∗ “Visible” part: segment selector.
∗ “Hidden” part: descriptor cache; it caches the descriptor indicated by the segment selector,
including base address, limit, and access information. This cached information allows the
processor to translate addresses without taking extra bus cycles.
– Segment registers in 80386: CS (code segment), DS (data), SS (stack), ES, FS, and GS. By de-
fault, for a code address, the processor uses the segment selector contained in CS, and therefore
fetch the code from the code segment. For an data address, the processor by default uses the seg-
ment selector contained in DS, and for a stack address, the processor uses the segment selector
contained in SS. If one wants to use other segment registers, they can use them as a prefix: e.g.
MOV EAX, FS:[0].
– For a program to access a segment, the segment selector for the segment must have been loaded
in one of the segment registers. The operations that load these registers are normal program
instructions; they are of two classes:
∗ Direct load instructions: e.g. MOV, POP, LDS, LSS, LGS, LFS.
∗ Implied load instructions: e.g. far CALL and JMP. These instructions implicitly reference
the CS register, and load it with a new value.
When segment registers are modified, the processor automatically fetches the base address, limit,
type, and other information from a descriptor table and loads them into the “hidden” part of the
segment register.
– Loading a segment register under 80386 Protected Mode results in special checks and actions,
to make sure the access control policies are satisfied. We will talk about the policies later.

3 The Mandatory Access Control on Data and Code Access

• Privilege Check for Data Access (see Figure)

– We temporarily ignore RPL.

– Policy: CPL ≤ DPL of code segment.
– A subject can only access data objects with the same or lower privilege levels.

• RPL: Request Privilege Level.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 7

– Potential Risk: At ring 0, code can access data at any ring level. This poses a risk when the code
(say A) is invoked by some other code (say B) in a less privileged ring, and B passes a pointer
to A. Normally, the pointer refers to a memory space that belongs to B (and of course A can
also access). However, if B is malicious, B can pass a pointer of a memory that does not belong
to B (B does not have privileges to access the memory). Because A is a privileged code, access
control cannot prevent A from accessing the memory. This way, B can use A to corrupt the
targeted memory in a privileged space.
– Principle of Least Privilege: in the above case, it is really unnecessary to run A with the ring-0
privilege when accessing the pointed memory passed by B. According to the principle of least
privilege, A should drop its privilege to B’s ring level when accessing the memory.
– How does RPL works:
∗ Assume that A is in ring 0 and B is in ring 3, and the memory address’s selector is S.
∗ The last two bit of a selector is used for RPL. It means that when accessing this memory,
the code’s privilege is droped to the RPL level. Therefore, if S’s RPL=3, when A tries to
access the memory in ring 0 (i.e. DPL=0), the access will be denied. If S’s RPL is not
dropped to 3 (instead it is set to 0), the access will succeed because A’s CPL is 0.
∗ In other words, max(RP L, CP L) is actually used for access control.
∗ RPL is usually less than or equal to CPL.
∗ If RPL = 0, RPL will have no effect.
– Policy of access control: max(CPL, RPL) ≤ DPL of data segment.

CS Register CS Register
CPL CPL

Segment Selector Segment Selector

For Data Segment For Code Segment
Privilege Privilege
RPL RPL
Check Check

Data−Segment Descriptor Code−Segment Descriptor

DPL DPL C

(a) Privilege Check for Data Access (b) Privilege Check for Control Transfer Without Using a Gate

Figure 5: Privilege Check for Data Access and Code Transfer (without using Gates)

• Privilege Check for Control Transfer without Using a Gate

– Policy:
∗ For non-comforming segment: transfer is allowed when CPL = DPL.
∗ For comforming segment: trasfer is allowed when CPL ≥ DPL.
∗ RPL does not have much effect here.
– Why can’t we access code with a higher DPL (i.e., lower privilege)?
∗ Possible reason 1: It is easy to jump (lower the CPL) to the code with higher DPL, but it
is difficult to return back, because on returning, we jump from a lower privileged ring to a
higher privileged ring. This violates the mandatory access control policy.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 8

∗ Possible reason 2: Another reason is the data access. If a code A jumps to another code B
at a lower privilege level, B cannot access A’s data because the data are most likely in A’s
ring level.
∗ Possible reason 3: Is there really a need to allow jumping from a higher privilege to a lower
privilege?
– Why can’t we jump to code with a lower DPL (i.e., higher privilege)?
∗ For security reasons, we cannot do this.
∗ Is this type of jump necessary? Yes, we definitely need this. For the device driver code is
usually in a privileged ring. User-level program should be able to jump to the device driver
code somehow.
∗ How can we achieve jumping to lower DPL? Gates are designed for this purpose. We will
talk about gates later.

• The conforming bit

– Permits sharing of procedures, so they can be called from various privilege levels.
– Usually for math library and exception handlers.
– If you have a procedure that you want everybody to be able to call it, where do you put it? in
which ring?
∗ Ring 3: ring 0,1,2 cannot call it.
∗ Ring 0: ring 1,2,3 cannot call it.
∗ Ring 0 and call gate: you need to build a call gate for each library call.
– The conforming segment mechanism permits sharing of procedures that may be called from
various privilege levels but should execute at the privilege level of the calling procedure. When
control is transferred to a conforming segment, the CPL does not change.

4 Call Gates
• Supporting system calls:
In an operating systems, all the privileged operations are carried out in the privileged rings, such as
modifing kernal data structure, interacting with devices, etc. OS does not allow user programs to
invoke them in an arbitrary way, such as jumping to the middle of a privileged operations. Instead, OS
provides a number of interfaces to users programs, which can only invoke those provileged operations
via the interfaces. These interfaces are often called system calls. Invoking system calls is quite
different from invoking a normal function. In the latter case, the call is within the same ring; however
in the former case, the call is from a less privileged ring to a prvileged ring. 80386’s ring protection
does not allow a direct jump like this. Some special mechanism must be provided to allow the control
transfer from a less privileged ring to a privileged ring.

• How to invoke system calls?

– Call Gate: Call gates allow a program to directly call system calls. However, since system calls
are often in a privileged ring, calling them directly is not allowed because of the ring protection.
The 80386 protection mode uses a call-gate concept to allow this kind of transfer. Call gates
enable programs in a lower privileged ring to jump to designated places in a higher privileged
ring.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 9

– Software Interrupt or Trap: In many operating systems, such as Linux and Minix, programs
use int 0x80 to trap to the kernel. Namely, when a program wants to call a system call,
it saves the system call number in the EAX register, and then execute int 0x80 to raise a
software interrupt. Interrupts transfer control to the kernel, so the kernel can execute the intended
system call based on the number stored in EAX. This approach is quite popular in OS designs.
– SYSENTER/SYSEXIT: The Intel Pentium II processor introduces another new facility for
faster system call invocation. The facility uses the instruction SYSENTER to enter the sys-
tem call entry point at ring 0; it uses SYSEXIT to return back. Starting with version 2.5, Linux
kernel introduced started taking advantage of this new system call facility.

• The Call Gate concept.

– The idea of Call Gate: 80386 does allow a program to jump to a more privilege ring, but a
program cannot jump to an arbitrary place, it must go through Call Gates, which basically define
the entry points for the privileged code. Corresponding security checks will be conducted at
those entry points to decide whether the invoking code has sufficient right. These security checks
are enforced by operating systems.
– Like segment descriptors, call-gate entries (call-gate descriptors) are also stored in the GDT (or
LDT) tables. Gates define an entry point of a procedure.
– Call-Gate Descriptor contains the following information:
∗ Code segment to be accessed (segment selector)
∗ Entry point for a procedure in the specified code segment (offset in segment)
∗ Privilege level required for a caller trying to access the procedure (DPL)
∗ Parameter Count: if a stack switch occurs, it specifies the number of optional parameters to
be copied between stacks.
∗ etc.
– How to use call gates?
∗ Call xxxxxx or JMP xxxxxx
∗ xxxxxx specifies the call gate entry in the GDT (or LDT) table
∗ From the table, the entry point of the procedure will be obtained.
∗ DPL of the gate descriptor allows the CPU to decide wither the invocator can enter the gate.

• Access Control Policy for Call Gates

– CPL ≤ DPL of the call gate.

– For CALL: DPL of the code segment ≤ CPL (only calls to the more privileged code segment
are allowed).
– For JMP: DPL of the code segment = CPL.
– Q: why can’t we CALL a less privileged code segment using Gates? Still returning will be a
problem, because returning will be from the less privileged code to the more privileged code,
and it violates the mandatory access control.

• Returning from a Called Procedure

– The RET instruction can be used to perform a near return, a far return at the same privilege level,
and a far return to a different privilege level

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 10

Far Pointer to Call Gate

Seg. Selector Offset
(Required but not used by processor)

Descriptor Table

Offset Call−Gate
Seg Selector Offset Descriptor

Base Base Code−Segment

+ Base Descriptor

Procedure Entry Point

Figure 6: Call-Gate Mechanism

31 24 20 16 8 4 0

Parameter
Offset in Segment 31:16 P DPL Type Count
0 1 1 0 0 0 0 0

31 16 0

Segment Selector Offset in Segment 15:00

Figure 7: Call-Gate Descriptor

– A far return that requires a privilege-level change is only allowed when returning to a less privi-
leged level

5 Miscellaneous
• How to change rings?

– The privilege level can only be changed by POPF, which has to be executed at level 0.

• Protecting descriptor tables.

– GDT, LDT, IDT are very important. They contain the followings.
∗ Call gates
∗ Code Segment Descriptors
∗ Data Segment Descriptors
– These tables should be in a protected memory
– The GDTR, LDTR, and IDTR registers (they store the locations of these tables) can only be set
by privileged code.

April 21, 2008

CIS/CSE 785: Computer Security (Syracuse University) 80386 Protection Mode: 11

• I/O Protection

– Introduction Question: everything can be boiled down to I/O operations. Is Direct I/O from ring
3 possible?
– How to prevent any arbitrary code from conducting I/O operations?
– Instructions: IN, INS, OUT, OUTS
– Policy: CPL ≤ IOPL
– IOPL is stored in EFLAGS, can only be modified by POPF instruction
– POPF can only be executed in ring 0
– This way, the security sensitive I/O operations can only be conducted by the code in the privi-
leged rings (e.g. ring 0).

April 21, 2008