Operating System Security and Access Control
Operating System Security and Access Control
Chapter 5
Operating System Security and Access Control
Copyright
2020-2022
c Paul C. van Oorschot. Under publishing license to Springer.
Mass-produced computers emerged in the 1950s. 1960s time-sharing systems brought se-
curity requirements into focus. 1965-1975 was the golden design age for operating system
(OS) protection mechanisms, hardware protection features and address translation. While
the threat environment was simpler—e.g., computer networks were largely non-existent,
and the number of software authors and programs was far smaller—many challenges were
the same as those we face today: maintaining separation of processes while selectively al-
lowing sharing of resources, protecting programs from others on the same machine, and
restricting access to resources. “Protection” largely meant controlling access to mem-
ory locations. This is more powerful than it first appears. Since both data and programs
are stored in memory, this controls access to running processes; input/output devices and
communications channels are also accessed through memory addresses and files. As files
are simply logical units of data in primary memory and secondary storage, access control
of memory and files provides a general basis for access control of objects and devices.
Initially, protection meant limiting memory addresses accessible to processes, in con-
junction with early virtual memory address translation, and access control lists were devel-
oped to enable resource sharing. These remain protection fundamentals. Learning about
such protection in operating systems provides a solid basis for understanding computer
security. Aside from Unix, we base our discussion in large part on Multics; its segmented
virtual addressing, access control, and protection rings heavily influenced later systems.
Providing security-related details of all major operating systems is not our goal—rather,
considering features of a few specific, real systems allows a coherent coverage highlight-
ing principles and exposing core issues important in any system design. Unix of course has
many flavors and cousins including Linux, making it a good choice. Regarding Multics,
security influenced it from early design (1964-67) through later commercial availability.
It remains among the most carefully engineered commercial systems ever built; an invalu-
able learning resource and distinguishing feature over both early and modern systems is
its rich and detailed technical literature explaining both its motivation and design.
126
5.1. Memory protection, supervisor mode, and accountability 127
2 Section 5.9 defines subject (principal) more precisely, and the relationship to processes and domains.
130 Chapter 5. Operating System Security and Access Control
internally) when acting in a role requiring the privileges of a different domain; abstractly,
distinct UIDs are considered distinct principals. Use of the same username by several
users is generally frowned upon as poor security hygiene, hindering accountability among
other drawbacks. To share resources, a preferred alternative is to use protection groups as
discussed in Section 5.3. Section 5.7 discusses role-based access control (RBAC).
Policy
access not
permi@ed Object1
✗
request access Reference permi@ed Object2
Subject by policy
to object monitor
✗ Object3
Figure 5.3: Reference monitor model. The policy check consults an access control policy.
ACCESS MATRIX . The reference monitor is a subject-object model. A subject (or
Ch.5, Fig.3. Reference monitor model. The policy check may involve, e.g.,
principal, above) is a system entity that may request access to a system object. An object
consul>ng an access control matrix.
is any item that a subject may request to use or alter—e.g., active processes, memory
addresses or segments, code and data (pages in main memory, swapped pages, files in
secondary memory), peripheral devices such as terminals and printers (often involving
input/output, memory or media), and privileged instructions.
In the model, a system first identifies all subjects and objects. For each object, the
types of access (access attributes) are determined, each corresponding to an access per-
mission or privilege. Then for each subject-object pair, the system predefines the autho-
rized access permissions of that subject to that object. Examples of types of access are
read or write for a data item or memory address, execute for code, wakeup or terminate
for a process, search for a directory, and delete for a file. The authorization of privileges
across subjects and objects is modeled as an access matrix with rows i indexed by subjects,
columns j by objects, and entries A(i, j), called access control entries (ACEs), specifying
access permissions subject i has to object j (Fig. 5.4). The ACE will typically contain a
collection of permissions, but we may for ease of discussion refer to the entry as a single
permission with value z, and if A(i, j) = z then say that subject i has z-access to object j.
R EFERENCE MONITOR IMPLEMENTATION . The reference monitor model is im-
5.2. The reference monitor, access matrix, and security kernel 131
column j
Objects
O1 O2 ... Oj ...
Subjects
S1 RW
S2 R
...
row i Si Ai, j
(a)
...
(b)
Figure 5.4: Access control matrix. A(i, j) is an ACE specifying the permissions that
subject i has on object j. (a) Row i can be used to build a capabilities list (C-list) for
subject i. (b) Column j can be used to build an access control list (ACL) for object j. The
access matrix itself is policy-independent; the access control policy is determined by the
Ch.5, Fig.4. Access control matrix.
permission content specified in matrix entries, not the matrix structure.
a guard at the door does an identity check, e.g., using photo ID cards in the physical world,
and any entity whose identity is verified and on an authorized list is allowed access. Ca-
pabilities (tickets) are held by subjects; authorization lists (based on identity) are held by
an object’s guard. Tickets must be unforgeable; identities must be unspoofable.
‡Exercise (Implementing capabilities). To prevent unauthorized copying or tamper-
ing of capabilities by users, capability systems may be implemented using different op-
tions. a) Maintain data structures for capabilities in OS (kernel) memory. b) Maintain
data structures for capabilities in user memory, but with tamper protection by a message
authentication code (MAC) or equivalent. c) Rely on hardware support (one scheme uses
kernel-controlled tag bits designating memory words that hold capabilities). Explain fur-
ther details for each of these mechanism approaches. (Hint: [35]; see also [38].)
‡Exercise (Access control and USB drives). When a USB flash drive is inserted into
a personal computer, which system accounts or processes are given access to the content
on this storage device? When files are copied from the USB drive into the filesystem of an
account on the host, what file permissions result on the copied files (assume a Unix-type
system)? Discuss possible system choices and their implications.
BASIS FOR AUDIT TRAILS . The basic reference monitor idea of mediating every
access provides a natural basis from which to build fine-grained audit trails. Audit logs
support not only debugging and accountability, but intruder detection and forensic inves-
tigations. Whether or not audit records must be tamper-proof depends on intended use.
‡Exercise (Access control through encryption and key release). Access control to
documents can be implemented through web servers and encryption. Give a technical
overview of one such architecture, including how it supports an audit trail indicating ac-
cess times and subjects who access documents. (Hint: [11].)
permissions to the printer are “file permissions”. Thus the study of file permissions gener-
alizes to access control on resources. This explains why file permissions are a main focus
when access control is taught.
ACL ALTERNATIVES . The simple permission mechanisms in early systems pro-
vided basic functionality. Operating systems commonly now support ACLs (Section 5.2)
for system objects including files. ACLs are powerful and offer fine-grained precision—
but also have disadvantages. ACLs can be as long as the list of system principals, con-
suming memory and search time; ACLs may need frequent updates; listing all principals
requiring access to a file can be tedious. A less expressive alternative, the ugo architec-
ture, became popular long ago, and remains widely used; we discuss it after background
on file ownership. Section 5.7 discusses a further alternative: role-based access control.
F ILE OWNER AND GROUP. Unix-based systems assign to each file an owner and
a protection group, respectively identified by two numeric values: a userid (UID) and a
groupid (GID). Initial values are set on file creation—in the simplest case, using the UID
and GID of the creating process. Where are these from? The login username for each
account indexes an entry in world-readable Unix file /etc/passwd, of the form:
username:*:UID:GID:fullname or info:home dir:default shell
Here GID identifies the primary group for this username. The * is where the pass-
word hash, salt, and related information was historically stored; it is now typically in
/etc/shadow, readable only by root. Group memberships are defined in a distinct file,
/etc/group, each line of which lists a groupname (analogous to a username), a GID, and
the usernames of all group members.
S UPERUSER VS . ROOT. Other than for login, the system uses UID for access control,
not username. Superuser means a process running with UID=0; such a process is granted
access to all file resources, independent of protection settings. The username convention-
ally associated with UID=0 is “root”, but technically the string “root” could be assigned
to other UIDs. It is the UID value of 0, not the string name, that determines permissions.
Within this book we will assume root is synonymous with superuser (UID=0).
USER - GROUP - OTHERS MODEL . We can now explain the base architecture for Unix
file permissions.6 The ugo permission model assigns privileges based on three categories
of principals: (user, group, others). The user category refers to the principal that is
the file owner. The group category enables sharing of resources among small to medium-
sized sets of users (say, project groups), with relatively simple permissions management.
The third category, others, is the universal or world group for “everyone else”. It defines
permissions for all users not addressed by the first two categories—as a means to grant
non-empty file permissions to users who are neither the file owner nor in the file’s group.
This provides a compact and efficient way to handle an object for which many (but not all)
users should be given the same privileges. This ugo model allows fixed-size filesystem
meta-data entries, and saves storage and processing time. Whereas ACLs may involve
arbitrary-length lists, here permission checking involves bit-operations on sets of just three
categories of principals; the downside is a significant loss in expressiveness.
6 This may be viewed as supporting principle P4 (COMPLETE - MEDIATION).
5.3. Object permissions and file-based access control 135
Ch.5. Symbolic nota1on for \unix{} file permissions (e.g., from ``ls – l”). % long format
M ETA - DATA AND FILE PERMISSIONS . The above user-group-others mechanism
User is also ``owner”. Other may be called ``all” or ``world”. t-bit is also ``s1cky bit”.
is supported by a per-file filesystem data structure, which also holds other “accounting
Cf. Table 5.2. s turns to S if x-bit is ``-” (this is not a useful combina1on since setuid and
details” related to a file, such as the address of the file contents. A commonly used such
datasetgid have no effect without x).
structure contains the following protection-related fields:7
1) user: indicating the userid (UID) of the file owner.
2) group: indicating the groupid (GID) of the file.
3) 9 bits: 3 protection bits for each of (user, group, others) as shown in Fig. 5.5.
For regular files, their meaning is fairly straightforward, as follows.
R (read): the file content may be read.
W (write): an existing file’s content may be modified.
X (execute): a binary file may be run. To run a shell script requires R to read the
file plus X. For a non-executable file, X is not useful (execution fails).
4) 3 bits: special protection bits setuid, setgid, t-bit (Sections 5.4 and 5.5).
U SE OF PROTECTION BITS . When a user process requests access to a file, the system
checks whether the process has the requested access privilege, based on the permissions
in this data structure. The checks are made in sequence: user, group, others. The first
qualifying category determines privileges. For a process that seeks R access and is the file
owner, if the user category does not grant R, the request fails even if others grants R.
P ERMISSION DISPLAY NOTATION . A common visual display format for file per-
missions is a 10-character string, such as -rwxr-xr-- (Table 5.1). The first character
conveys file type—a leading dash indicates a non-directory file. The next nine characters,
in groups of three, convey permissions for the ugo categories in order. A substring rwx
corresponds to binary 111 indicating read, write and execute, while a dash “-” conveys a
0-bit denoting that the corresponding permission is absent. Additional permissions can be
overlaid onto these ten characters, as shown in Figure 5.5.
P ROTECTION BIT INITIAL VALUES ( NON - DIRECTORY FILES ). A file’s 9 RWX
bits are set at file creation using a 12-bit mode parameter (with the 3 special bits) provided
to the syscall used to create the file. The requested RWX bits are post-modified by the
creator’s 9-bit file creation “unset” mask (umask), typically defined in a user startup file; a
child process inherits its parent’s umask. The mask’s positional 1-bits specify permissions
to turn off or remove (if present). For example for regular files—using octal format, with
7 This is based on the Unix inode (index-node), to illustrate concepts concretely in this section. Other
inode fields unrelated to permissions indicate: file size, last-modified time, number (link count) of directory
entries that link to the file, and fields to signal whether the file is a directory or special Unix I/O device file.
136 Chapter 5. Operating System Security and Access Control
one octal digit for each 3-bit RWX group (Table 5.1)—a common permissions default 666
(RW for all categories), and common mask of 022 (removing W from group and others),
yield a combined initial permission string 644 (RW for user, R only for group and others).
Example (group permissions). Suppose a group identifier accounting is set up to
include userA, userB and userC, and a group executives to include userC, userD and
userE. Then userC will have, e.g., RX access to any file-based resource if the file’s group
is accounting or executives, and the file’s group permissions are also RX.
Exercise (setting/modifying file permissions). The initial value of a Unix mask can
be modified where set in a user startup file, or later by the umask command. (a) Ex-
periment to discover default file permissions on your system by creating a new file with
the command touch, and examining its permissions with ls -l. Change your mask set-
ting (restore it afterwards!) using umask and create a few more files to see the effect.
(b) The command chmod allows the file owner to change a file’s 9-bit protection mode.
Summarize its functionality for a specific flavor of Unix or Linux.
Exercise (modifying file owner and group). Unix commands for modifying file at-
tributes include chown (change owner of file) and chgrp (change a file’s group). Some
systems allow file ownership changes only by superuser; others allow a file owner to
chown their owned files to any other user. Some systems allow a file owner to change the
file’s group to any group that the file owner belongs to; others allow any group whatsoever.
Summarize the functionality and syntax of these commands for a chosen Unix flavor.
Exercise (access control in swapped memory). Paging is common in computer sys-
tems, with data in main memory temporarily stored to secondary memory (“swapped to
disk”). What protection mechanisms apply to swapped memory? Discuss.
F ILE PERMISSIONS AUGMENTED BY ACL S . The ugo permission architecture
above is often augmented by ACLs (Section 5.2). On access requests, the OS then checks
whether the associated UID is in an ACL entry with appropriate permissions.
Exercise (file ACL commands). For a chosen Unix-type system (specify the OS ver-
sion), summarize the design of file ACL protection. In particular, explain what informa-
tion is provided by the command getfacl, and the syntax for the setfacl command.
5.4. Setuid bit and effective userid (eUID) 137
library calls,10 to the specified executable, which continues with the child’s PID and also
inherits its userids (rUID, eUID, sUID) except when the executable is setuid.
Example (userids and login). The Unix command login results in a process running
the root-owned setuid program /bin/login, which prompts for a username-password, does
password verification, sets its process UID and GID to the values specified in the verified
user’s password file entry, and after various other initializations yields control by execut-
ing the user’s declared shell. The shell inherits the UID and GID of this parent process.
D ISPLAYING SETUID AND SETGID BITS . If a file’s setuid or setgid bit is set,
this is displayed in 10-character notation by changing the user or group execute letter as
follows (Figure 5.5): an x changes to s, or a “-” changes to S. The latter conveys status, but
is not useful—setuid, setgid have no effect without execute permission. So -rwsr-xr-x
indicates a file executable by all, with setuid bit set; and -rwSr--r-- indicates a file is
setuid, but not executable (which is not useful; the unusual capital S signals this).
‡Exercise (setuid). Explain how the setuid functionality is of use for user access to
a printer daemon, and what general risks setuid creates. (Chapter 6 discusses privilege
escalation related to setuid programs in greater detail.)
of files therein (they have their own permissions), nor their meta-data.
W: the user may alter (edit) directory content—provided X permission is also held.
W (with X) allows renaming or deleting filename entries, and creating new entries
(by creating a new file, or linking to an existing file as explained in Section 5.6);
the system will modify, remove, or newly add a corresponding dir-entry. Thus
removing a file reference (entry) from a directory requires not W permission on the
target file, but W (with X) on the referencing directory.
X: the user may traverse and “search” the directory. This includes setting it as the
working directory for filesystem references; path-access11 to the directory’s listed
files; and access to their inode meta-data, e.g., by commands like find. Lack of X
on a directory denies path-access to its files, independent of permissions on those
files themselves; their filenames (as directory content) remain listable if R is held.
setuid: this bit typically has no meaning for directory files in Unix and Linux.
setgid: if set, the group value initially assigned to a newly created (non-directory or di-
rectory) file is the GID of its directory (rather than the GID of the creating process);
a newly created sub-directory in addition inherits the directory’s setgid bit. The
aim is to make group sharing of files easier, by setting a directory’s setgid bit.
t-bit: (text or sticky bit) this bit set on a directory prevents deletion or renaming of files
therein owned by other users. The directory owner and root can still delete files.
For non-directory files, the t-bit is now little used (its original use is obsolete).
S TICKY BIT. A primary use of the t-bit is on world-writable directories, e.g., /tmp.
An attack could otherwise remove and replace a file with a malicious one of the same
filename. When set, a t replaces x in position 10 of symbolic display strings (Fig. 5.5).
Example (Directory permissions). In Fig. 5.6, curry and durant are entries in Warriors.
Path-access to Warriors, including X on the inode it references (Warriors inode), is needed
to make this the current directory (via cd) or access any files below it. R on Warriors inode
allows visibility of filenames curry and durant (e.g., via ls). X on Warriors inode allows
access to the meta-data of curry and durant (e.g., via ls -l or find), but to read their
content requires R on these target file inodes (plus X on Warriors inode, in order to get to
them). In summary: access to a file’s name (which is a directory entry), properties (inode
meta-data including permissions), and content are three distinct items. Access to a (dir or
non-dir) file’s meta-data requires path-access (X) on the inode holding the meta-data, and
is distinct from RWX permission on the file content referenced by the inode.
W ORLD - WRITABLE FILES . Some files are writable by all users (world-writable).
This is indicated by w in the second-last character for a display string such as -rwxrwxrwx.
The leading dash indicates a non-directory (regular) file. Some files should not be world-
writable, e.g., a startup file (a script file invoked at startup of a login program, shell,
editor, mail or other executable), lest an attack modify it to invoke a malicious program
11 By Unix path-based permissions, to read a file’s content requires X on all directories on a path to it plus
R on the file; path-access does not require R on directories along the path if the filename is already known.
140 Chapter 5. Operating System Security and Access Control
“/” datablock
filesystem “.”
root node “..”
“/” inode
“NBAeast”
“NBAwest”
...
inode inode
“Spurs”
“Warriors”
Warriors datablock
(dir file)
“.”
pathlink1 Warriors “..” pathlink2
inode “curry”
inodes & datablocks for “durant”
files “Lakers” and “Spurs” “thompson”
Figure 5.6: Directory structure and example (Unix filesystem with inodes). See inline
Ch.5, Fig.5. Directory structure and example.
discussion regarding the roles of pathlink1 and pathlink2. Compare to Fig. 5.7.
PPT: no-shadow, 0.75 solid lines, 1.0 doBed lines,
on each startup. Ashape->freeform->hold down CTRL for straight segments,
world-writable root-owned setuid executable is spectacularly bad—
click to drop a corner, double-click to end
any process could replace the contents with a malicious file, which would run with root
privileges regardless of who invoked it. (Chapter 6 has a further example on this topic.)
‡Exercise (finding world-writable files). The Unix find command can search for files
with specific properties, including permissions. Explore your own system for world-
writable files using: find /users/yourhome -perm -2 -print (replace /users/yourhome
with a directory to explore; start at one with a relatively small subtree, as the output is of-
ten extensive). The search will be denied access to directories that you lack X permission
for; the command will continue with the next directory in a recursive tree search.
‡Example (inode details). The following, and Figure 5.6, explain the internal ac-
tions that occur in creating a new directory node, and provide a detailed example of the
filesystem (directory) structure summarized above. This may help in understanding how
5.5. Directory permissions and inode-based example 141
like ls -l. (Note: on a Mac system, to get a command interpreter shell to the underlying
Unix system, run the Terminal utility, often found in Applications/Utilities.)
Exercise (access control outside of filesystem). Suppose a copy of a filesystem’s data
(as in Figure 5.6) is backed up on secondary storage or copied to a new machine. You
build customized software tools to explore this data. Are your tools constrained by the
permission bits in the relevant inode structures? Explain.
‡Exercise (chroot jails). A chroot jail provides modest isolation protection, support-
ing principle P5 (ISOLATED - COMPARTMENTS) through restricted filesystem visibility. It
does so using the Unix system call chroot(), whose argument becomes the filesystem root
(“/”) of the calling process. Files outside this subsystem are inaccessible to the process.
This restricts resources and limits damage in the case of compromise. Common uses in-
clude for network daemons and guest users. a) Summarize the limitations of chroot jails.
b) Describe why the newer jail() is preferred, and how it works. (Hint: [17].)
a file object whose directory entry references the inode for existing. Since a file’s name
is not part of the file itself, distinct directory entries (here, now two) may name the file
differently. Also, this same command syntax may allow a directory-file to be hardlinked
(i.e., existing may be a directory), although for technical reasons, hardlinking a directory
is usually discouraged or disallowed (due to issues related to directory loops).
Example (Symlink). For a symlink the -s option is used: ln -s existing new2
results in a dir-entry for an item assigned the name new2 but in this case it references a
new inode, of file type symlink, whose datablock provides a symbolic name representing
the object existing, e.g., its ASCII pathname string. When new2 is referenced, the filesys-
file content
Pre-exisAng dir-entry1: “chezpan”
for file pathname /users/chezpan inode
datablock
(a) dir-entry2: “new1” hardlink
Results from (ptr to an inode)
ln /users/chezpan new1
soOlink (symlink)
via dir-entry1
Figure 5.8: Comparison of (a) hardlink, and (b) symbolic link. A symbolic link can be
thought of as a file whose datablock itself points to another file.
Ch.5, Fig.7. Hard link vs. symbolic link.
tem uses this symbolic name to find the inode for existing. If the file is no longer reachable
by pathname existing (e.g., its path or that directory entry itself is changed due to renam-
ing, removed because the file is moved, or deleted from that directory), the symbolic link
will fail, while a hardlink still works. Examining Figure 5.8 may help clarify this.
D ELETING LINKS AND FILES . “Deleting” a file removes the filename from a direc-
tory’s list, but the file itself (inode and datablock) is removed from the filesystem only if
this was the last directory entry referencing the file’s inode.12 Figures 5.8 and 5.9 con-
sider the impact of deletion with different types of links. Deleting a symlink file (e.g.,
new2 directory entry in Fig. 5.8) does not delete the referred-to “file content”. Deleting
a hardlinked file by specifying a particular dir-entry eliminates that directory entry,
but the file content (including inode) is deleted only when its link count (the number of
hardlinks referencing the inode) drops to zero. While at first confusing, this follows di-
rectly from the filesystem design: an inode itself does not “live” in any directory but rather
exists independently; directory entries simply organize inodes into a structure.
permissions
object
subject role
opera@ons
Figure 5.10: Role-based access control (RBAC) model. A user, represented as a subject,
is pre-assigned one or more roles by an administrator. The administrator also assigns
permissions to each role. For each login session, a subset of roles is activated for the user.
Ch.5, Fig.3. RBAC. A user is represented as a subject, and for each login session one
the role GrantManager is assigned read access to files for department member grants. A
or more roles, which are pre-assigned to a subject is ac@vated. RBAC reference
new staff member Alex who is assigned both these roles will then acquire both sets of per-
model: subject, object (opera@on on), permissions, roles. An administrator
missions. When Alex moves to another department and Corey takes over, Corey gets the
associates permissions with each role.
same permissions by being assigned these two roles; if individual file-based permissions
were used, a longer list of individual permissions might have to be reassigned. Roles may
be hierarchically defined, e.g., so that a SeniorManager role is the union of all roles en-
joyed by junior managers, plus some roles specific to the higher position. RBAC system
administrators make design choices as to which tasks (and corresponding permissions)
are associated with different job functions, and define roles accordingly.
M-AC AND SEL INUX . The remainder of this section introduces efforts related to
SELinux: the Flask operating system architecture on which it was built, the Linux Security
Module framework that provides support for it (and other M-AC approaches) within Linux,
and the SEAndroid version of it for the Google Android OS.
‡Exercise (Flask). The Flask security architecture was designed as a prototype during
the 1990s to demonstrate that a commodity OS could support a wide range of (mandatory)
access control policies. a) Summarize the motivation and goals of Flask. b) Describe the
Flask architecture, including a diagram showing how its Client, Object Manager, and
Security Server interact. c) Explain Flask object labeling (include a diagram), and how
the data types security context and security identifier (SID) fit in. (Hint: [34].)
‡Exercise (SELinux). Security-Enhanced Linux (SELinux) is a modified version of
Linux built on the Flask architecture and its use of security labels. SELinux supports
mandatory security policies and enforcement of M-AC policies across all processes. It
was originally integrated into Linux as a kernel patch, and reimplemented as an LSM
(below). a) Summarize details of the SELinux implementation of the Flask architecture,
including the role security labels play and how they are supported. b) Describe the secu-
rity mechanisms provided by SELinux, including mandatory access controls for process
management, filesystem objects, and sockets. c) Describe the SELinux API. d) Give an
overview of the SELinux example security policy configuration that serves as a customiz-
able foundation from which to build other policy specifications. (Hint: [22].)
‡Exercise (LSMs). Linux Security Modules (LSMs) are a general framework by
which the Linux kernel can support, and enforce, diverse advanced access control ap-
proaches including SELinux. This is done by exposing kernel abstractions and operations
146 Chapter 5. Operating System Security and Access Control
a) Concentric rings
n-1 b) Segment Descriptor (basic ring-num)
...
2 physical seg
1 ring-num
ring 0 start addr length
(strongest)
c) Program Counter with ring number
PC procedure word
ring-num segment S number W
Figure 5.11: Protection rings and supporting descriptors. To support protection rings,
ring numbers are added to segment descriptors and to the program counter.
5.8. ‡Protection rings: isolation meets finer-grained sharing 147
having high privileges may be confusing, we also use the terms stronger (inner) rings and
weaker (outer) rings. We add to every segment descriptor (Section 5.1) a new ring-num
field (ring number) per Figure 5.11b, and to the CPU Program Counter (instruction ad-
dress register) a PCring-num for the ring number of the executing process, per Fig. 5.11c.
For access control functionality, we associate with each segment an access bracket (n1 , n2 )
denoting a range of consecutive rings, used as explained below.
P ROCEDURE SEGMENT ACCESS BRACKETS . A user process may desire to transfer
control to programs designed to execute in stronger rings (e.g., for privileged functions,
such as input-output functions, or to change permissions to a segment’s access control list
in supervisor memory); or weaker rings (e.g., for simple shared services). A user process
P1 may wish to allow another user’s process P2 , operating in a weaker ring, access to data
memory in P1 ’s ring, but only under the condition that such access is through a program
(segment) provided by P1 , and verified to have been accessed at a pre-authorized entry
point, specified by a memory address. Rings allow this, but now transfers that change
rings (cross-ring transfers) require extra checks. In the simple “within-bracket” case,
a calling process P1 executing in ring i requests a transfer to procedure segment P2 with
access bracket (n1 , n2 ) and n1 ≤ i ≤ n2 . Transfer is allowed, without any change of control
ring (PCring-num remains i).13 The more complicated “out-of-bracket” case is when i is
outside P2 ’s access bracket. Such transfer requests trigger a fault; control goes to the
supervisor to sort out, as discussed next.
P ROCEDURE SEGMENT GATE EXTENSION . Suppose a process in ring i > n2 at-
tempts transfer to a stronger ring bracketed (n1 , n2 ). Processes are not generally permitted
to call stronger-ring programs, but to allow flexibility, the design includes a parameter,
n3 , designating a gate extension for a triple (n1 , n2 , n3 ). For case n2 < i ≤ n3 , a transfer
request is now allowed, but only to pre-specified entry points. A list (gate list) of such en-
try points is specified by any procedure segment to be reachable by this means. So i > n2
triggers a fault and a software fault handler handles case n2 < i ≤ n3 . The imagery is that
gates are needed to cross rings, mediated by gatekeeper software as summarized next.
R ING GATEKEEPER MEDIATION . When a ring-i process P1 requests transfer to
procedure segment P2 having ring bracket (n1 , n2 , n3 ), a mediation occurs per Table 5.2.
The domain of a process—introduced in Section 5.1 as the permission set associated with
the objects a process can access—can change over time. When a Unix process calls a root-
owned setuid program it retains its process identifier (PID), but temporarily runs with a
different eUID, acquiring different access privileges. Viewing the domain as a room, the
objects in the room are accessible to the subject; when the subject changes rooms, the
accessible objects change. For a more precise definition of protection domain D , the ring
system and segmented virtual addressing of Multics can be used as a basis. We define the
domain of a subject S = (P , D ) associated with process P as
D = (r, T )
Here r is the execution ring of the segment g that P is running in, and T is P ’s descriptor
segment table containing segment descriptors (including for g), each including access
indicators. This yields a more detailed description of a subject as
S = (P , r, T ) = (processID, ring-number, descriptor-seg-ptr)
N OTES . These definitions for subject and domain lead to the following observations.
1) A change of execution ring changes the domain, and thus the privileges associated
with a subject S. At any specific execution point, a process operates in one domain or
context; privileges change with context (mode or ring).
2) A transfer of control to a different segment, but within the same ring (same process,
same descriptor segment), changes neither the domain nor the subject.
3) Access bracket entry points specify allowed, gatekeeper-enforced domain changes.
4) Virtual address translation maps constrain physical memory accessible to a domain.
5) A system with n protection rings defines a strictly ordered set of n protection domains,
D (i) = (i, T ), 0 ≤ i ≤ n − 1. Associating these with a process P and its fixed descriptor
segment defined by T , defines a set of subjects (P , 0, T ), ..., (P , n − 1, T ).
6) Informally, C-lists (Section 5.2) define the environment of a process. Equating C-lists
with domains allows substitution in the definition S = (P , D ).
Exercise (Ring changes vs. switching userid). It is recommended that distinct accounts
be set up on commodity computers to separate regular user and administrative activities.
Tasks requiring superuser privileges employ the administrative account. Discuss how this
compares, from an OS viewpoint, to a process changing domains by changing rings.
H ARDWARE - SUPPORTED CPU MODES UNUSED BY SOFTWARE . Many comput-
ers in use run operating systems supporting only two CPU modes (supervisor, user) de-
spite hardware support for more—thus available hardware support for rings goes unused
(Figure 5.13). Why so? If an OS vendor seeks maximum market share via deployment
on all plausible hardware platforms, the lowest-functionality hardware (here, in terms of
CPU modes) constrains all others. The choices are to abandon deployment on the low-
functioning hardware (ceding market share), incur major costs of redesign and support
for multiple software streams across hardware platforms, or reduce software functional-
ity across all platforms. Operating systems custom-built for specific hardware can offer
richer features, but fewer hardware platforms are then compatible for deployment.
5.10. ‡End notes and further reading 151
rings 1, 2
(typically unused in
commodity systems)
Exercise (Platforms supporting more than two modes). Discuss, with technical de-
tails, how many CPU modes, privilege levels, or rings are supported by the ARMv7 pro-
cessor architecture. Likewise discuss for operating systems OpenVMS and OS/2. Do any
versions of Windows support more than two modes?
Dennis [6] is credited for segmented addressing (although protection features of Multics-
style segmentation were designed out of later commercial systems). Graham [14] gives
an authoritative view of protection rings (including early identification of race condition
issues); see also Graham and Denning [13] (which Section 5.9 follows), and Schroeder
and Saltzer [31] for a Multics-specific discussion of hardware-supported rings and related
software issues. Interest in using hardware-supported rings recurs—for example, Lee [20]
leverages unused x86 rings for portable user-space privilege separation.
The 1970 Ware report [37] explored security controls in resource-sharing computer
systems. The 1972 Anderson report [1, pages 8-14] lays out the central ideas of the
reference monitor concept, access matrix, and security kernel; it expressed early concerns
about requiring trust in the entire computer-manufacturing supply chain, and the ability to
determine that “compiler and linkage editors are either certified free from ‘trap-doors’,
or that their output can be checked and verified independently”—attacks later more fully
explained in Thompson’s Turing-award paper [36] detailing C-code for a Trojan horse
compiler. The 1976 RISOS report [21, p.57] defined a security kernel as “that portion
of an operating system whose operation must be correct in order to ensure the security
of the operating system” (cf. Chapter 1, principle of SMALL - TRUSTED - BASES P8). Their
small-size requirement, originally specified as part of the validation mechanism for the
reference monitor, has made microkernels a focus for security kernels (cf. Jaeger above).
These 1970-era reports indicate longstanding awareness of computer security challenges.
Lampson’s 1973 note on the confinement problem [18] raises the subject of untrusted
programs leaking data over covert channels. Gasser’s lucid 1988 book [12], a practi-
tioner’s integrated treatment on how to build secure computer systems, includes discus-
sion of reference monitors, security kernels, segmented virtual memory, MLS/mandatory
access control, and covert channels.
References (Chapter 5)
[1] J. P. Anderson. Computer Security Technology Planning Study (Vol. I and II, “Anderson report”). James
P. Anderson and Co., Fort Washington, PA, USA, Oct 1972.
[2] S. M. Bellovin. Thinking Security: Stopping Next Year’s Hackers. Addison-Wesley, 2016.
[3] H. Chen, D. Wagner, and D. Dean. Setuid demystified. In USENIX Security, 2002.
[4] D. A. Curry. UNIX System Security: A Guide for Users and System Administrators. Addison-Wesley,
1992.
[5] R. C. Daley and P. G. Neumann. A general-purpose file system for secondary storage. In AFIPS Fall
Joint Computer Conference, pages 213–229, Nov 1965.
[6] J. B. Dennis. Segmentation and the design of multiprogrammed computer systems. Journal of the ACM,
12(4):589–602, 1965.
[7] M. S. Dittmer and M. V. Tripunitara. The UNIX process identity crisis: A standards-driven approach
to setuid. In ACM Comp. & Comm. Security (CCS), pages 1391–1402, 2014.
[8] M. Dowd, J. McDonald, and J. Schuh. The Art of Software Security Assessment: Identifying and
Preventing Software Vulnerabilities. Addison-Wesley, 2006.
[9] D. Ferraiolo and R. Kuhn. Role-based access controls. In National Computer Security Conf. (NCSC),
pages 554–563, Oct. 1992.
[10] D. F. Ferraiolo, R. S. Sandhu, S. I. Gavrila, D. R. Kuhn, and R. Chandramouli. Proposed NIST standard
for role-based access control. ACM Trans. Inf. Systems and Security, 4(3):224–274, 2001.
[11] W. Ford and M. J. Wiener. A key distribution method for object-based protection. In ACM Comp. &
Comm. Security (CCS), pages 193–197, 1994.
[12] M. Gasser. Building a Secure Computer System. Van Nostrand Reinhold, 1988. PDF available online.
[13] G. S. Graham and P. J. Denning. Protection—principles and practice. In AFIPS Spring Joint Computer
Conference, pages 417–429, May 1972.
[14] R. M. Graham. Protection in an information processing utility. Comm. ACM, 11(5):365–369, 1968.
Appeared as the first paper, pp.1-5, first ACM Symposium on Operating System Principles, 1967.
[15] A. Gruenbacher. POSIX access control lists on LINUX. In USENIX Annual Technical Conf., pages
259–272, 2003.
[16] T. Jaeger. Operating System Security. Morgan and Claypool, 2008.
[17] P.-H. Kamp and R. N. M. Watson. Jails: Confining the omnipotent root. In System Admin. and Net-
working Conf. (SANE), 2000. Cf. “Building systems to be shared, securely”, ACM Queue, Aug 2004.
[18] B. W. Lampson. A note on the confinement problem. Comm. ACM, 16(10):613–615, 1973.
[19] B. W. Lampson. Protection. ACM Operating Sys. Review, 8(1):18–24, 1974. Originally published in
Proc. 5th Princeton Conf. on Information Sciences and Systems, 1971.
[20] H. Lee, C. Song, and B. B. Kang. Lord of the x86 rings: A portable user mode privilege separation
architecture on x86. In ACM Comp. & Comm. Security (CCS), pages 1441–1454, 2018.
153
154 References (Chapter 5)
[21] T. Linden. Security Analysis and Enhancements of Computer Operating Systems (“RISOS report”),
Apr 1976. NBSIR 76-1041, The RISOS Project, Lawrence Livermore Laboratory, Livermore, CA.
[22] P. Loscocco and S. Smalley. Integrating flexible support for security policies into the Linux operating
system. In USENIX Annual Technical Conf., pages 29–42, 2001. FREENIX Track. Full technical report,
62 pages, available online.
[23] P. A. Loscocco, S. D. Smalley, P. A. Muckelbauer, R. C. Taylor, S. J. Turner, and J. F. Farrell. The
inevitability of failure: The flawed assumption of security in modern computing environments. In
National Info. Systems Security Conf. (NISSC), pages 303–314, 1998.
[24] B. McCarty. SELinux: NSA’s Open Source Security Enhanced Linux. O’Reilly Media, 2004.
[25] E. I. Organick. The Multics System: An Examination of Its Structure. MIT Press (5th printing, 1985),
1972.
[26] D. M. Ritchie and K. Thompson. The UNIX time-sharing system. Comm. ACM, 17(7):365–375, 1974.
[27] J. H. Saltzer. Protection and the control of information sharing in Multics. Comm. ACM, 17(7):388–402,
1974.
[28] J. H. Saltzer and M. F. Kaashoek. Principles of Computer System Design. Morgan Kaufmann, 2010.
[29] J. H. Saltzer and M. D. Schroeder. The protection of information in computer systems. Proceedings of
the IEEE, 63(9):1278–1308, September 1975.
[30] R. S. Sandhu, E. J. Coyne, H. L. Feinstein, and C. E. Youman. Role-based access control models. IEEE
Computer, 29(2):38–47, 1996.
[31] M. D. Schroeder and J. H. Saltzer. A hardware architecture for implementing protection rings. Comm.
ACM, 15(3):157–170, 1972. Earlier version in ACM SOSP, pages 42–54, 1971.
[32] A. Silberschatz, P. B. Galvin, and G. Gagne. Operating System Concepts (seventh edition). John Wiley
and Sons, 2005.
[33] S. Smalley and R. Craig. Security Enhanced (SE) Android: Bringing Flexible MAC to Android. In
Netw. Dist. Sys. Security (NDSS), 2013.
[34] R. Spencer, S. Smalley, P. Loscocco, M. Hibler, D. Andersen, and J. Lepreau. The Flask security
architecture: System support for diverse security policies. In USENIX Security, 1999.
[35] A. S. Tanenbaum. Modern Operating Systems (3rd edition). Pearson Prentice Hall, 2008.
[36] K. Thompson. Reflections on trusting trust. Comm. ACM, 27(8):761–763, 1984.
[37] W. H. Ware (Chair). Security Controls for Computer Systems: Report of Defense Science Board Task
Force on Computer Security. RAND Report R-609-1 (“Ware report”), 11 Feb 1970. Office of Director
of Defense Research and Engineering, Wash., D.C. Confidential; declassified 10 Oct 1975.
[38] R. N. M. Watson and 14 others. CHERI: A hybrid capability-system architecture for scalable software
compartmentalization. In IEEE Symp. Security and Privacy, pages 20–37, 2015.
[39] R. N. M. Watson, J. Anderson, B. Laurie, and K. Kennaway. Capsicum: Practical capabilities for UNIX.
In USENIX Security, pages 29–46, 2010.
[40] C. Wright, C. Cowan, S. Smalley, J. Morris, and G. Kroah-Hartman. Linux security modules: General
security support for the Linux kernel. In USENIX Security, pages 17–31, 2002.