System Programming (CENG204)

Lecturer: Ahmet ÖZDİL

Course Info
Weekly Topics
1 Overview of systems programming
2 Users, files and manuals
3 Directories, file properties and file systems
4 Terminal control and signals
5 Event driven programming
6 Processes and programs
7 i/o redirection and pipes
8 Mid-term Exam
9 Servers and sockets
10 Threads
11 Memory management system and processes carried out under the operating systems
12 Concurrency control
13 Project/Homework Presentation
14 Project/Homework Presentation
15 Project/Homework Presentation
16 Project/Homework Presentation
2
Users, files and manuals
Types of OSs
Batch
• Single CPU
• User submits large number of tasks at one time
• OS decides what to run and when
• Back-to-back single tasking
• One-single task is running on the CPU at any time
• Need to wait for a task to finished before the next task
can start
• Only runs in sequential order, not parallel
• Not interactive, need to wait for the whole batch to
finish (pressing on a key, need to wait)
3
Users, files and manuals
Types of OSs
Time-Sharing
• Multiple users connected to a single CPU
• Many user terminals
• Multiple tasks run simultaneously using time-sharing,
giving users the feeling of having multiple dedicated
CPUs running in parallel
• More responsive to users (fake feeling to user), like
TDMA, Round-Robin scheduling

4
Users, files and manuals
Types of OSs
Parallel
• N-CPUs in one computer simultaneously
• May be fake Parallel
• Multiple CPUs closely coupled to form one computer
• Higher throughput (multiple tasks running at same
time) and better fault tolerance
• Each CPU can be running batch tasking or time-sharing
multitasking

5
Users, files and manuals
Types of OSs
Distributed
• Integrated multiple complete computer systems into
one chip
• Multiple CPUs loosely coupled via networking
• Not flexible to actual demand
• May be rented to users on demand
• Like: grid computing, could computing

6
Users, files and manuals
Types of OSs
Real-time
• Very strict response time requirements
• Periodical tasks, every job of a task has a strict
deadline
• Used in aviation, military, etc.
• In short time, need to finish the sensing, computing,
and actuating, not miss the DDL (Dead Dead Line)
• To guarantee the the Real-time, very hard, very
expensive.

7
Users, files and manuals
Single Tasking vs. Multitasking
Single
• Tasking System Each task, once started, executes to its
very end without any interruption from other task(s).
• Simple to implement: sequential boundaries between
tasks, and resource accesses.
• Few security risks
• Poor utilization of the CPU and other resources
• Example: MS-DOS

8
Users, files and manuals
Single Tasking vs. Multitasking
Multitasking System
• Very complex
• Serious security issues
o How to protect one task from another, which shares the same block of memory

• Much higher utilization of system resources

• Support interactive user/physical-world interface
• Example: Unix, Windows NT

9
Users, files and manuals
Protection and Security
OS must protect itself from users
• Reserved memory only accessible by
OS OS may protect users from one another
• Use hardware through system calls, constrained by OS
routines

10
Users, files and manuals
The Unix Philosophy
• Modular design
• A collection of simple tools, with limited well-defined
functions
• File system
• Inter-process communication
• The tools can be easily composed (via >, >>, <, |, ``)
by a shell to accomplish complex tasks

11
Users, files and manuals
What makes up Unix

12
Users, files and manuals
What makes up Unix
Kernel is the heart of the OS. It interacts with the
hardware and conducts key functions of the OS, such as
memory management, task scheduling, file management.
Shell is the utility that interacts with the user via
command line. When you type a line command from your
terminal input device, the shell interprets the command
and executes the intended program (or composition of
programs). The shell commands have a standard syntax. C
Shell, Bourne Shell, and Korn Shell are the most famous
shells, which are available in most Unix variations (Linux
commonly uses the Bourne Shell – like bash).

13
Users, files and manuals
Filesystems
Definition of a "filesystem":

14
Users, files and manuals
Filesystems
Definition of a "filesystem":

• A hierarchical arrangement of directories.

15
 Users, files and manuals
Filesystems
In Unix, the root file system starts with "/". However, there are
other subfilesystems, that are part of the root one. To see the
filesystems on your machine, type "df". You'll see something like:
UNIX> df
Filesystem kbytes used avail capacity Mounted on
/dev/sd0a 11167 5952 4099 59% /
/dev/sd0g 140591 84016 42516 66% /usr
/dev/sd0h 171959 141421 13343 91% /blugreen/homes
-memory- 55720 0 55720 0% /tmp
plank:(pid130) 4 4 0 100% /amd
galoob:/export/sun4-sos4_1
335803 313356 0 100% /a/galoob/export/sun4-sos4_1
cs:/var/spool/mail 221351 122607 76609 62% /a/cs/var/spool/mail
alphard:/canary/homes
16
 Users, files and manuals
Filesystems
The last entry on each line in this denotes a different filesystem.
You'll note that all are subdirectories of "/". Each filesystem resides
on a different disk or "disk partition".
• Filename: The name of a file as it appears in a directory.
• Pathname: A sequence of zero or more filenames, separated by
slashes.

17
 Users, files and manuals
Filesystems
When you say "ls -a", it lists all the filenames in the current
directory:
UNIX> cd ~huangj/cs360/notes/Rbtree-1
UNIX> ls -a
. jh.c makefile mysorti.c mysortu2.c
.. lecture mysort.c mysortu0.c mysortu3.c
README lecture.html mysort2.c mysortu1.c randfile
UNIX>
(If you don't say "-a", it will not list filenames beginning with ".")
Note there are two filenames that are in every directory: "." and
".."
"." is the current directory. ".." is the parent of the current
18
 Users, files and manuals
Filesystems
The pwd command tells you the complete pathname of the current directory. The cd
command moves you between directories:
UNIX> cd ~huangj/cs360/notes/Rbtree-1
UNIX> pwd
/home/huangj/cs360/notes/Rbtree-1
UNIX> cd .
UNIX> pwd
/home/huangj/cs360/notes/Rbtree-1
UNIX> cd ..
UNIX> pwd
/home/huangj/cs360/notes
UNIX> cd ../../..
UNIX> pwd
/home
UNIX> cd .././././.
UNIX> pwd
/ 19
 Users, files and manuals
Filesystems
Note that the parent directory of "/" is itself.

• Absolute Pathname: A pathname starting with a slash.

• Relative Pathname: A pathname not starting with a slash.
• Working Directory: The directory to which relative pathnames
are relative.
• Home Directory: The working directory of a user when he or
she first logs in.

20
 Users, files and manuals
Filesystems
Programs and processes:
• Program: A file that can be executed, either directly, or through
the aid of interpreters, compilers, and/or linkers.
• Process: An executing instance of a program.
• Process ID: The number given to a process by the operating
system.

21
 Users, files and manuals
Filesystems
Examples of programs:
• /bin/ls -- this is a program that can be executed directly.
• /usr/ucb/vi -- this is a program that can be executed directly.
• ~huangj/cs360/notes/Rbtree-1/mysort.c -- this is a program
that needs to be compiled in order to execute it.
• /blugreen/homes/plank/bin/calc -- this is a shell script -- it is a
program that needs to be interpreted by /bin/sh.

22
 Users, files and manuals
Filesystems
When you run a program, its executing instance is called a process.
For example, suppose you type "vi ~huangj/cs360/notes/Rbtree-
1/mysort.c" in one window. This executes the vi program, found
in /usr/ucb/vi. The executing instance is called a process. If you go to
another window, and type "ps x", it will list all of the processes that you
are currently executing:
UNIX> ps x

1394 p0 IW 0:00 xclock -geometry 100x100-0-0 -update 60

1416 p0 S 0:01 twm

1436 p1 IW 0:00 -sh (csh)

1427 p2 IW 0:00 -sh (csh)

1428 p3 IW 0:00 -sh (csh)

1443 p5 S 0:01 -sh (csh)

23
 Users, files and manuals
Filesystems
Note that we can run more than one vi process at the same time. Go
to another window and type "vi
/home/huangj/cs360/notes/Rbtree-1/mysort2.c". Now when you
type "ps x", you'll see the second process.
UNIX> ps x

1394 p0 IW 0:00 xclock -geometry 100x100-0-0 -update 60

1416 p0 S 0:01 twm

1436 p1 IW 0:00 -sh (csh)
1427 p2 S 0:00 -sh (csh)
2330 p2 S 0:00 vi /home/huangj/cs360/notes/Rbtree-1/mysort2.c

1428 p3 IW 0:00 -sh (csh)

1443 p5 IW 0:01 -sh (csh)

2328 p5 IW 0:00 vi /home/huangj/cs360/notes/Rbtree-1/mysort.c

24
 Users, files and manuals
Filesystems
• User Identification
• User ID: The number given to each user by the system administrator.
There are two ways that the system deals with identification: User Id's and
group id's. We won't talk much about group id's here. Your user id can be
gotten by the system call "getuid()" -- read the man page.
To print out your UID, try ch1b.c:
#include < stdio.h >
main()
{
printf("%d\n", getuid());
}
25
 Users, files and manuals
Filesystems Signals
• Signal: An interruption of the program.
• Signal Handler: The mechanism by which the program may
gracefully deal with signals.
• Signals are a way for asynchronous events to occur in a
program. For example, compile and run the program ch1c.c:
• #include < stdio.h >
• main()

• {
• int i;

• i = 0;

• while (1) {

• i++;
26
 Users, files and manuals
Filesystems Signals
• This program implements a counter that increments itself every 5
seconds. Let it run for a few seconds, and then type < CNTL-Z >. This
sends the "STOP" signal to the program, which stops it. You'll be back
at the shell now. If you type "ps", you'll see something like:
2483 p5 T 0:00 ch1c
• The "T" means that the process is not running -- it is stopped. To start
it, you can type "fg", which will send it the "START" signal.

27
 Users, files and manuals
Filesystems Signals
• Now, while it's running, type < CNTL-C > to kill the program -- this
sends it the "INT" signal, which teminates it. Segmentation faults are
also signals. You can write your programs to handle signals any way
you please. This is a more advanced programming technique, which we
will deal with later.
• Yes, signal is a part of the ANSI C standard, it should perform the same
(or, very similarly) on all systems claiming to support ANSI'87. The
primary differences stem from which additional signals, beyond the
basic set of signals, are supported on different systems. The basic set
specified by ANSI C standard is rather small.

28
 Users, files and manuals
Filesystems
File paths and their meanings for UNIX and LINUX
- /bin: Contains executable files belonging to the user and the
system administrator.
- /dev: Contains the files required to access hardware.
- /etc: Contains the files required for system configurations.
- /lib: Contains system libraries.
- /sbin: Contains executable files belonging to the system
administrator.
- /home: It is the directory belonging to the users.
-
/mnt: It is the directory that specifies the ports of the hardware
devices
29
that will be connected to the system externally.
 Users, files and manuals
Filesystems
File paths and their meanings for UNIX and LINUX
- /root: It is the directory belonging to the "root" user, who is the
most authorized user in a system.
- /tmp: This is the directory containing temporary files.
- /usr: This is the directory where shared files are located.
Although there are executable files here, there are also files
belonging to documents and programs.
- /var: It is the directory where messages such as log messages
and emails of the system and programs are located.
- /proc: It is the virtual directory containing necessary information
about the system.

30
 Users, files and manuals
Filesystems
UNIX Simple Commands (touch-mv-cp-rm-mkdir)
touch otuceng.txt Generates a text file named #Otuceng.txt. With the same logic, '.c',
'.py', '.rb', '.js' etc. File extensions can be generated as follows.
mv otuceng.txt ceng.txt # It is used to change the name in operations without
changing the directory.

mv otuceng.txt ./lessons # Moves the same work to the lessons directory.

mv otuceng.txt ./dersler/ceng.txt # Moves to the lessons directory by changing the

name.

cp system.txt systemprogramming.txt # The file named system is copied and the

name of the copied file becomes systemprogramming.
rm system.txt # The file named system is deleted.

rm -rv ./lessons # The folder named Lessons and the files under it are deleted.

mkdir lessons # Generates a new subdirectory in the current directory. Its name is
lessons.
31
Users, files and manuals
Files
Files store information on secondary storage. This means
that the information should exist even when the computer
that stored it is powered off.

This is as opposed to primary storage, which only works

while a computer is powered up, and which goes away
forever when a computer is powered down.

32
Users, files and manuals
Files
Directory and Links
• file types: regular, directory, link, device, ... (almost
everything is file)
• directory: a set of related file, support hierarchical structure
• Home directory, root directory, current directory

33
Users, files and manuals
Files
File I/O Model
• stdio library: fopen(), fread(), fwrite(), fclose(), printf(),
scanf(), …
• system call: open(), read(), write(), close(), …
• After open(): file name -> file descriptor

34
Users, files and manuals
Files
When you generate a file in Unix, there are quite a few things
that happen. We are going to focus on three components of
a file in Unix:
• The bytes of the file itself.
• The metadata of the file.
• The links to the file that are relative to a directory.

35
Users, files and manuals
Files
Let's do a simple example:

UNIX> echo "This is f1.txt" > f1.txt

UNIX> ls -lai
total 20
497584377 drwxr-xr-x 2 plank guest 38 Feb 3 13:54 .
430430681 drwxr-xr-x 51 plank guest 4096 Feb 3 2014 ..
497584369 -rw-r--r-- 1 plank loci 15 Feb 3 13:54 f1.txt
497584451 -rw-r--r-- 1 plank guest 9896 Feb 3 13:44
lecture.html
UNIX>

36
Users, files and manuals
Files
We generated a file called "f1.txt", and that places three things
on disk:
A directory entry for "f1.txt" in the current directory. That associates the
name "f1.txt" with an "inode" whose number is 497584369.

An "Inode" with a number of 497584369. This contains metadata, or

information about the file. Typical metadata is the size, owner, links,
protection mode, latest access and modification time, etc. This is
stored on disk in its own location, the operating system knows how to find
it by its inode number.

The actual bytes, "This is f1.txt". These are placed in a disk block, and the
inode has information on how to find the disk block. I won't go into much
detail on this. In this example, the file is contained within one disk block,
and the inode will know how to find it.

37
Users, files and manuals
Files
We can associate a picture with this example:
You can see the inode for f1.txt (497584369), and how it points to
block 0x4ab, which contains the bytes of the file (you don't have access to
this information unless you are the operating system -- I just made up the
number 0x4ab for the sake of the example). I have included the
information in the directory, which is itself a file on disk, and the inode for
that file. See how everything links together?

38
Users, files and manuals
Files
Also, you will note that I haven't put a null character at the end
of the string in the disk block. That's because there is no null
character -- that is only there when you are using a string
inside a C program. When you write it to disk, there is no null
character.
When you give the -i flag to ls, it will tell you the inode number,
as in the example above.

39
Users, files and manuals
Files
To use Unix lingo, the way we name a file is by attaching a
"link" to the inode. Links are stored in "directories" -- each
entry in a directory maps the name of the link to the inode
number of the inode that points to the file. Again, you can
see that in the example above.

40
Users, files and manuals
Files
We can have more than one link point to a file. Suppose we are
in a fresh directory, and we have created the file f1 to
contain the bytes "This is f1\n". Moreover, suppose that file
has an inode number of 34778. And now we do the
following:
UNIX> ln f1.txt f1
UNIX> ln f1 f2
This says to generate another link to the file f1, and call it "f2".
That link is really an entry in the directory that maps "f2" to
inode 34778. What we have now are two pointers to the
same metadata and the same bytes on disk. When we do a
listing:

41
Users, files and manuals
Files
When we do a listing:
UNIX> ls -li f1 f2
34778 -rw-r--r-- 2 plank 11 Sep 16 10:12 f1
34778 -rw-r--r-- 2 plank 11 Sep 16 10:12 f2
UNIX> cat f1
This is f1
UNIX> cat f2
This is f1
UNIX>

42
Users, files and manuals
Files
We see that the files are exactly the same, except that the
links have different names. If we change either of these files --
for example, let's edit f2 using vi, and change the word "This"
to "That", then the change is seen in both f1 and f2, because
they both point to the same bytes on disk:
UNIX> vi f2
...
UNIX> cat f2
That is f1
UNIX> cat f1
That is f1
UNIX> ls -li f1 f2
43
Users, files and manuals
Files
Note that even though we only modified f2, the file
modification time for f1 has changed as well. That is because
file modification time is stored as part of the inode -- thus,
when f2 changes it, the change is seen in f1 as well. Same
with file protection modes. If we change the protection for f1,
then we will see the changes in f2:
UNIX> chmod 0400 f1
UNIX> ls -li f1 f2
34778 -r-------- 2 plank 11 Sep 16 10:14 f1
34778 -r-------- 2 plank 11 Sep 16 10:14 f2
UNIX>

44
Users, files and manuals
Files
Note the third column of the ls command. It is the number of
links to the file. If we make another link to f1, then this column
will be updated:

UNIX> ln f1 f3
UNIX> ls -li f1 f2 f3
34778 -r-------- 3 plank 11 Sep 16 10:14 f1
34778 -r-------- 3 plank 11 Sep 16 10:14 f2
34778 -r-------- 3 plank 11 Sep 16 10:14 f3

45
Users, files and manuals
Files
When we use the "rm" command, we are actually removing
links. E.g.

UNIX> chmod 0644 f1

UNIX> rm f1
UNIX> ls -li f*
34778 -rw-r--r-- 2 plank 11 Sep 16 10:14 f2
34778 -rw-r--r-- 2 plank 11 Sep 16 10:14 f3
UNIX>

46
Users, files and manuals
Files
When the last link to a file is removed, then the file itself,
inode and all, is deleted. As long as there is a link pointing to a
file, however, the file remains. It is interesting to see what
happens when files with links are overwritten. For example,
suppose I do the following:
UNIX> cat > f2
This is now file f2
^D
UNIX> cat f2
This is now file f2
UNIX> cat f3
This
47 is now file f2
Users, files and manuals
Files
By saying you want to redirect output to the file f2, you end
up changing f3. This means that when the shell performs
output redirection, it opens the file and truncates it, instead of
removing the file and creating it anew.
Instead, suppose you do:
UNIX> gcc -o f2 ../Stat/src/ls1.c
UNIX> ls -li f*
34779 -rwxr-xr-x 1 plank 24576 Sep 16 10:16 f2
34778 -rw-r--r-- 1 plank 20 Sep 16 10:16 f3
UNIX>
You'll note that the c compiler gcc did a "rm f2"
before
48 creating f2 as an executable.
Users, files and manuals
Files
All directories have at least 2 links:
UNIX> mkdir test
UNIX> ls -li | grep test
34800 drwxr-xr-x 2 plank 512 Sep 16 10:17 test
UNIX>
This is because every directory contains two subdirectories
"." and ".." The first is a link to itself, and the second is a link to
the parent directory. Thus, there are two links to the directory
file "test": "test" and "test/."

49
Users, files and manuals
Files
Similarly, suppose we make a subdirectory of test:
UNIX> mkdir test/sub
UNIX> ls -li | grep test
34800 drwxr-xr-x 3 plank 512 Sep 16 10:17 test
UNIX>
Now there are three links to "test": "test", "test/.", and
"test/sub/.."

50
 Users, files and manuals
Files
Besides these links which are automatically generated for you,
you cannot manually create links to directories. Instead, there is a
special kind of a link called a "symbolic link" (also called a "soft
link"), which you make using the command "ln -s". For example,
we can create a soft link to the test directory as follows:
UNIX> ln -s test test-soft
UNIX> ls -li | grep test
34800 drwxr-xr-x 3 plank 512 Sep 16 10:17 test
34801 lrwxrwxrwx 1 plank 4 Sep 16 10:18 test-soft -> test
UNIX>
Note that soft links have a different kind of directory listing. Moreover,
note that the generation of a soft link to "test" doesn't update the link
field
51
of test's inode. That only records regular, or "hard" links.
 Users, files and manuals
Files
A soft link is a way of pointing to a file without changing the
file's inode. However, soft links can do pretty much everything that
hard links can do:
UNIX> cat > f1

This is f1

UNIX> ln -s f1 f2
UNIX> cat f2

This is f1
UNIX> cat > f2
This is f2

UNIX> cat f1

This is f2
52
 Users, files and manuals
Files
UNIX> cat > f1
This is f1
UNIX> ln -s f1 f2
UNIX> cat f2
This is f1
UNIX> cat > f2
This is f2
UNIX> cat f1
This is f2
UNIX> ls -l f*
-rw-r--r-- 1 plank 11 Sep 16 10:19 f1
lrwxrwxrwx 1 plank 2 Sep 16 10:18 f2 -> f1
UNIX> chmod 0600 f2
UNIX>
53 ls -l f*
 Users, files and manuals
Files
What is the main difference between hard and soft links then?
Well, if the file to which the soft link points gets deleted or moved,
then the link becomes unusable:
UNIX> rm f1
UNIX> ls -l f*
lrwxrwxrwx 1 plank 2 Sep 16 10:18 f2 -> f1
UNIX> cat f2
cat: f2: No such file or directory
UNIX>

The link is called "unresolved."

54
 Users, files and manuals
Files
In Unix, you cannot make hard links from a file in one filesystem to a
directory in another filesystem. I.e., from your student accounts, you
cannot do a command such as:

UNIX> ln /home/jplank/cs360/notes/Links/lecture.html
~/lecture.html

because your home directory is not on the same filesystem as

administrator. However, you can make a soft link:

UNIX> ln -s /home/jplank/cs360/notes/Links/lecture.html
~/lecture.html

55
 Users, files and manuals
SHELL

56
 Users, files and manuals
Users and Groups
In linux users:
• 3 categories: user, group, others
• Superuser: has special privileges (User ID: 0, login name: root)
• Unix is a multi-user, multi-tasking OS.
• Multiple users may run multiple tasks concurrently.

57
 Users, files and manuals
Users and Groups

58
References

• Dr. James S. Plank

59