Unix File System

Unix file system is a logical method of organizing and storing large amounts of information in a way that
makes it easy to manage. A file is a smallest unit in which the information is stored. Unix file system has
several important features. All data in Unix is organized into files. All files are organized into directories.
These directories are organized into a tree-like structure called the file system.

Files in Unix System are organized into multi-level hierarchy structure known as a directory tree. At the
very top of the file system is a directory called “root” which is represented by a “/”. All other files are
“descendants” of root.

Directories or Files and their description –

 / : The slash / character alone denotes the root of the filesystem tree.

 /bin : Stands for “binaries” and contains certain fundamental utilities, such as ls or cp, which are
generally needed by all users.

 /boot : Contains all the files that are required for successful booting process.

 /dev : Stands for “devices”. Contains file representations of peripheral devices and pseudo-
devices.

 /etc : Contains system-wide configuration files and system databases. Originally also contained
“dangerous maintenance utilities” such as init,but these have typically been moved to /sbin or
elsewhere.
 /home : Contains the home directories for the users.

 /lib : Contains system libraries, and some critical files such as kernel modules or device drivers.

 /media : Default mount point for removable devices, such as USB sticks, media players, etc.

 /mnt : Stands for “mount”. Contains filesystem mount points. These are used, for example, if the
system uses multiple hard disks or hard disk partitions. It is also often used for remote (network)
filesystems, CD-ROM/DVD drives, and so on.

 /proc : procfs virtual filesystem showing information about processes as files.

 /root : The home directory for the superuser “root” – that is, the system administrator. This
account’s home directory is usually on the initial filesystem, and hence not in /home (which may
be a mount point for another filesystem) in case specific maintenance needs to be performed,
during which other filesystems are not available. Such a case could occur, for example, if a hard
disk drive suffers physical failures and cannot be properly mounted.

 /tmp : A place for temporary files. Many systems clear this directory upon startup; it might have
tmpfs mounted atop it, in which case its contents do not survive a reboot, or it might be
explicitly cleared by a startup script at boot time.

 /usr : Originally the directory holding user home directories,its use has changed. It now holds
executables, libraries, and shared resources that are not system critical, like the X Window
System, KDE, Perl, etc. However, on some Unix systems, some user accounts may still have a
home directory that is a direct subdirectory of /usr, such as the default as in Minix. (on modern
systems, these user accounts are often related to server or system use, and not directly used by
a person).

 /usr/bin : This directory stores all binary programs distributed with the operating system not
residing in /bin, /sbin or (rarely) /etc.

 /usr/include : Stores the development headers used throughout the system. Header files are
mostly used by the #include directive in C/C++ programming language.

 /usr/lib : Stores the required libraries and data files for programs stored within /usr or
elsewhere.

 /var : A short for “variable.” A place for files that may change often – especially in size, for
example e-mail sent to users on the system, or process-ID lock files.

 /var/log : Contains system log files.

 /var/mail : The place where all the incoming mails are stored. Users (other than root) can access
their own mail only. Often, this directory is a symbolic link to /var/spool/mail.

 /var/spool : Spool directory. Contains print jobs, mail spools and other queued tasks.
  /var/tmp : A place for temporary files which should be preserved between system reboots.

Types of Unix files – The UNIX files system contains several different types of files

1. Ordinary files – An ordinary file is a file on the system that contains data, text, or program
instructions.

 Used to store your information, such as some text you have written or an image you have
drawn. This is the type of file that you usually work with.

 Always located within/under a directory file.

 Do not contain other files.

 In long-format output of ls -l, this type of file is specified by the “-” symbol.

2. Directories – Directories store both special and ordinary files. For users familiar with Windows or Mac
OS, UNIX directories are equivalent to folders. A directory file contains an entry for every file and
subdirectory that it houses. If you have 10 files in a directory, there will be 10 entries in the directory.
Each entry has two components.
(1) The Filename
(2) A unique identification number for the file or directory (called the inode number)

Branching points in the hierarchical tree.

Used to organize groups of files.

May contain ordinary files, special files or other directories.

Never contain “real” information which you would work with (such as text). Basically, just used for
organizing files.

All files are descendants of the root directory, ( named / ) located at the top of the tree.

In long-format output of ls –l , this type of file is specified by the “d” symbol.

3. Special Files – Used to represent a real physical device such as a printer, tape drive or terminal, used
for Input/Output (I/O) operations. Device or special files are used for device Input/Output(I/O) on UNIX
and Linux systems. They appear in a file system just like an ordinary file or a directory.
On UNIX systems there are two flavors of special files for each device, character special files and block
special files :

 When a character special file is used for device Input/Output(I/O), data is transferred one
character at a time. This type of access is called raw device access.

 When a block special file is used for device Input/Output(I/O), data is transferred in large fixed-
size blocks. This type of access is called block device access.
For terminal devices, it’s one character at a time. For disk devices though, raw access means reading or
writing in whole chunks of data – blocks, which are native to your disk.

 In long-format output of ls -l, character special files are marked by the “c” symbol.

 In long-format output of ls -l, block special files are marked by the “b” symbol.

4. Pipes – UNIX allows you to link commands together using a pipe. The pipe acts a temporary file which
only exists to hold data from one command until it is read by another.A Unix pipe provides a one-way
flow of data.The output or result of the first command sequence is used as the input to the second
command sequence. To make a pipe, put a vertical bar (|) on the command line between two
commands.For example: who | wc -l

In long-format output of ls –l , named pipes are marked by the “p” symbol.

5. Sockets – A Unix socket (or Inter-process communication socket) is a special file which allows for
advanced inter-process communication. A Unix Socket is used in a client-server application framework.
In essence, it is a stream of data, very similar to network stream (and network sockets), but all the
transactions are local to the filesystem.

In long-format output of ls -l, Unix sockets are marked by “s” symbol.

6. Symbolic Link – Symbolic link is used for referencing some other file of the file system.Symbolic link is
also known as Soft link. It contains a text form of the path to the file it references. To an end user,
symbolic link will appear to have its own name, but when you try reading or writing data to this file, it
will instead reference these operations to the file it points to. If we delete the soft link itself , the data
file would still be there.If we delete the source file or move it to a different location, symbolic file will
not function properly.

In long-format output of ls –l , Symbolic link are marked by the “l” symbol (that’s a lower case L).

Linux Shell and Shell Scripting

If you are using any major operating system you are indirectly interacting to shell. If you are running
Ubuntu, Linux Mint or any other Linux distribution, you are interacting to shell every time you use
terminal. In this article I will discuss about linux shells and shell scripting so before understanding shell
scripting we have to get familiar with following terminologies –

 Kernel

 Shell

 Terminal

What is Kernel
The kernel is a computer program that is the core of a computer’s operating system, with complete
control over everything in the system. It manages following resources of the Linux system –

 File management

 Process management

 I/O management

 Memory management

 Device management etc.

It is often mistaken that Linus Torvalds has developed Linux OS, but actually he is only responsible for
development of Linux kernel.
Complete Linux system = Kernel + GNU system utilities and libraries + other management scripts +
installation scripts.

What is Shell

A shell is special user program which provide an interface to user to use operating system services. Shell
accept human readable commands from user and convert them into something which kernel can
understand. It is a command language interpreter that execute commands read from input devices such
as keyboards or from files. The shell gets started when the user logs in or start the terminal.

linux shell

Shell is broadly classified into two categories –

 Command Line Shell

  Graphical shell

Command Line Shell

Shell can be accessed by user using a command line interface. A special program called Terminal in
linux/macOS or Command Prompt in Windows OS is provided to type in the human readable commands
such as “cat”, “ls” etc. and then it is being execute. The result is then displayed on the terminal to the
user. A terminal in Ubuntu 16.4 system looks like this –

linux command line

In above screenshot “ls” command with “-l” option is executed.

It will list all the files in current working directory in long listing format.
Working with command line shell is bit difficult for the beginners because it’s hard to memorize so many
commands. It is very powerful, it allows user to store commands in a file and execute them together.
This way any repetitive task can be easily automated. These files are usually called batch files in
Windows and Shell Scripts in Linux/macOS systems.

Graphical Shells
Graphical shells provide means for manipulating programs based on graphical user interface (GUI), by
allowing for operations such as opening, closing, moving and resizing windows, as well as switching
focus between windows. Window OS or Ubuntu OS can be considered as good example which provide
GUI to user for interacting with program. User do not need to type in command for every actions.A
typical GUI in Ubuntu system –

GUI shell

There are several shells are available for Linux systems like –

 BASH (Bourne Again SHell) – It is most widely used shell in Linux

systems. It is used as default login shell in Linux systems and in
macOS. It can also be installed on Windows OS.

 CSH (C SHell) – The C shell’s syntax and usage are very similar to
the C programming language.

 KSH (Korn SHell) – The Korn Shell also was the base for the
POSIX Shell standard specifications etc.

Each shell does the same job but understand different commands and provide different built in
functions.

Shell Scripting
Usually shells are interactive that mean, they accept command as input from users and execute them.
However some time we want to execute a bunch of commands routinely, so we have type in all
commands each time in terminal.
As shell can also take commands as input from file we can write these commands in a file and can
execute them in shell to avoid this repetitive work. These files are called Shell Scripts or Shell Programs.
Shell scripts are similar to the batch file in MS-DOS. Each shell script is saved with .sh file extension
eg. myscript.sh

A shell script have syntax just like any other programming language. If you have any prior experience
with any programming language like Python, C/C++ etc. it would be very easy to get started with it.
A shell script comprises following elements –

 Shell Keywords – if, else, break etc.

 Shell commands – cd, ls, echo, pwd, touch etc.

 Functions

 Control flow – if..then..else, case and shell loops etc.

Why do we need shell scripts

There are many reasons to write shell scripts –

 To avoid repetitive work and automation

 System admins use shell scripting for routine backups

 System monitoring

 Adding new functionality to the shell etc.

Advantages of shell scripts

 The command and syntax are exactly the same as those directly
entered in command line, so programmer do not need to switch
to entirely different syntax

 Writing shell scripts are much quicker

 Quick start

 Interactive debugging etc.

Disadvantages of shell scripts

 Prone to costly errors, a single mistake can change the

command which might be harmful
  Slow execution speed

 Design flaws within the language syntax or implementation

 Not well suited for large and complex task

 Provide minimal data structure unlike other scripting languages.

etc

Simple demo of shell scripting using Bash Shell

If you work on terminal, something you traverse deep down in directories. Then for coming few
directories up in path we have to execute command like this as shown below to get to the “python”
directory –

It is quite frustrating, so why not we can have a utility where we just have to type the name of directory
and we can directly jump to that without executing “cd ../” command again and again. Save the script
as “jump.sh”

# !/bin/bash
# A simple bash script to move up to desired directory level directly
function jump()
{
# original value of Internal Field Separator
OLDIFS=$IFS

# setting field separator to "/"

 IFS=/

# converting working path into array of directories in path

# eg. /my/path/is/like/this
# into [, my, path, is, like, this]
path_arr=($PWD)

# setting IFS to original value

IFS=$OLDIFS

local pos=-1

# ${path_arr[@]} gives all the values in path_arr

for dir in "${path_arr[@]}"
do
# find the number of directories to move up to
# reach at target directory
pos=$[$pos+1]
if [ "$1" = "$dir" ];then

# length of the path_arr

dir_in_path=${#path_arr[@]}

#current working directory

cwd=$PWD
limit=$[$dir_in_path-$pos-1]
for ((i=0; i<limit; i++))
do
cwd=$cwd/..
done
cd $cwd
break
fi
done
}

For now we cannot execute our shell script because it do not have permissions. We have to make it
executable by typing following command –
$ chmod -x path/to/our/file/jump.sh

Now to make this available on every terminal session, we have to put this in “.bashrc” file.
“.bashrc” is a shell script that Bash shell runs whenever it is started interactively. The purpose of a
.bashrc file is to provide a place where you can set up variables, functions and aliases, define our prompt
and define other settings that we want to use whenever we open a new terminal window.
Now open terminal and type following command –

$ echo “source ~/path/to/our/file/jump.sh”>> ~/.bashrc

Now open you terminal and try out new “jump” functionality by typing following command-

$ jump dir_name

just like below screenshot –

https://nus-unix-workshop.github.io/2021-s1/stdio/

Standard Input/Output

Two of the design decisions of Unix that lead to its simplicity are the decisions to (i) decouple the
physical input/output devices from the programs, allowing programs written for Unix to read
from abstract input and output devices; and (ii) make all programs read and write from these abstract
input and output devices by default. Before Unix, the older operating systems often require
programmers to painstakingly set up connections to the teletype machines and other devices for
reading inputs and printing outputs. With Unix, programmers can now focus on solving the tasks at hand
and let Unix takes care of the input and output.

The two abstract devices that Unix provides are called standard input and standard output. By default,
the standard input refers to the keyboard and the standard output is the terminal.

Examples using cat and wc

Let's look at these concepts closer, by examining some examples.

Remember cat? The cat command takes in a filename and it prints the content of the file to the standard
output.

$ cat test.txt

This is a test file for learning Unix file management commands.

If no filename is given, cat by default try to read from the standard input. Try running:

$ cat

You will see that the command is waiting for you to type in something. Type in anything, as soon as you
press Enter, cat is going to read in the text from the standard input, as if it is the content of a file, and
then prints the content to the standard output. You can keep typing, supplying text to cat, or you can
type Ctrl+D to send the end-of-input command to cat.

Let's look at another command, wc. wc is a utility that counts the number of lines, words, characters. If
we call wc and supply it a file name, it will count the number of lines, words, and characters in that given
file.

$ wc test.txt

1 11 64 test.txt

The output means that there is 1 line, 11 words, and 64 characters in the file test.txt.

But if you do not pass in any file name, wc is going to read in the text from the standard input, as if it is
the content of a file, and prints the three counters to the standard output. Go ahead and try:

$ wc

You will see that the wc command is waiting for you to type in something. Type in a few sentences, you
can hit Enter for a new line. When you are done, type Ctrl+D. wc will count the number of lines, words,
and characters for the text that you just entered.

Output Redirection

By defining two abstract input and output devices (or channels), Unix frees the programmers from
worrying about where to read the input from and write the output to. Most of the time, we can write
the output of the program to the standard output. In instances where we need to write the output to
another location, we can just redirect the output.

The operators > and >> are used to redirect the standard output to a file. The difference is that > will
overwrite the given file, while >> will concatenate into the given file.

For example:
$ wc test.txt > test.count

$ cat test.count

1 11 64 test.txt

The first command redirects the output from wc to a file named test.count, so you do not see anything
printed to the output anymore. We can check by running cat on the new file test.count -- indeed the
original output from wc is now stored in the file test.count.

If we repeat the command wc test.txt > test.count again, you can see that the file has been overwritten
with the output from wc again. But if we replace > with >>, a new line is concatenated into test.count.
So the file now has two lines.

$ wc test.txt > test.count

$ cat test.count

1 11 64 test.txt

$ wc test.txt >> test.count

$ cat test.count

1 11 64 test.txt

1 11 64 test.txt

Input Redirection

The operator < is used to redirect a file into the standard input. So, instead of reading from the
keyboard, we can now read from a file. Commands such as cat and wc already support from a file
directly, so there is no difference in terms of functionality to run the commands by passing in the file
name, or by using the < operator.

$ wc test.txt

1 11 64 test.txt

$ wc < test.txt

1 11 64

$ cat test.txt

This is a test file for learning Unix file management commands.

$ cat < test.txt

This is a test file for learning Unix file management commands.

Note the slight difference in the output format of the second wc above -- it no longer prints the file
name since from wc points of view, it is read from the standard input and not from a file, so it is not
aware of the file named test.txt

In most CS programming assignments, however, to keep things simple, you will be asked to read from
the standard input only, so the < is a great time-saver -- you do not have to repeatedly type in the same
input data over and over from the keyboard. You can just save the input data in a file, then redirect it to
standard input with the < operator.

Unix System Call

In computing, a system call is the programmatic way in which a computer program requests a service
from the kernel of the operating system it is executed on. A system call is a way for programs to interact
with the operating system. A computer program makes a system call when it makes a request to the
operating system’s kernel. System call provides the services of the operating system to the user
programs via Application Program Interface(API). It provides an interface between a process and
operating system to allow user-level processes to request services of the operating system. System calls
are the only entry points into the kernel system. All programs needing resources must use system calls.

Services Provided by System Calls :

1. Process creation and management

2. Main memory management

3. File Access, Directory and File system management

4. Device handling(I/O)

5. Protection

6. Networking, etc.

Types of System Calls : There are 5 different categories of system calls –

1. Process control: end, abort, create, terminate, allocate and free memory.

2. File management: create, open, close, delete, read file etc.

3. Device management

4. Information maintenance

5. Communication
Examples of Windows and Unix System Calls –

Windows Unix

CreateProcess() fork()
ExitProcess() exit()
Process Control WaitForSingleObject() wait()

CreateFile() open()
ReadFile() read()
WriteFile() write()
File Manipulation CloseHandle() close()

SetConsoleMode() ioctl()
ReadConsole() read()
Device Manipulation WriteConsole() write()

GetCurrentProcessID() getpid()
SetTimer() alarm()
Information Maintenance Sleep() sleep()

CreatePipe() pipe()
CreateFileMapping() shmget()
Communication MapViewOfFile() mmap()

SetFileSecurity() chmod()
InitlializeSecurityDescriptor() umask()
Protection SetSecurityDescriptorGroup() chown()