“Unix Essentials”

Course @ ILP, TCS Limited

December 8, 2021 TCS Internal

Course Objective

After the course you should

• Have a fundamental understanding of the Unix architecture

• Be able to use Unix as a platform for application development

December 8, 2021 2
TCS Internal
History of unix

– Ken thompson of AT&T ,BELL laboratories originally designed unix os in late 1960s
which evolved from timesharing OS called multics.
– AT & T licenses source code for low cost
– Originally written in assembly language but with the development of ‘c’ programming
language in 1973,it was then written in ‘c’.
– Two versions of unix emerged are AT&T Unix
and BSD Unix

December 8, 2021 3
TCS Internal
Unix History

– GNU Project started in 1984

– Free Software Foundation
– 1989 AT&T and Sun Micro system joined together and developed system V release 4
(SVR4)
– Two of the main standards mainly in use are POSIX(Portable Operating System
Interface) and X/open standard
– 1988 MIT formed Xconsortium which developed vendor Neutral Xwindow System
– Linux Origins

December 8, 2021 4
TCS Internal
Unix Principles

• Multiuser and Multitasking

• Everything is a file
• Small, single purpose programs
• Configuration data stored in text
• Ability to chain programs together to perform complex tasks

December 8, 2021 5
TCS Internal
 UNIX Architecture
• Shell and kernel together make UNIX system work.

Utilities or Shells
Other Application commands interact
& System Software user with users
user
KERNEL
interacts
user
H/W
with H/W user

Database
Compilers
Packages
user
December 8, 2021 6
TCS Internal
 The UNIX Kernel

• Kernel is a collection of programs mostly written in C, that runs directly on the hardware -
so parts of kernel must be customized to each system’s hardware features.
• Kernel is directly loaded into memory when the system is booted. Low level jobs are its
basic services:
 System Initialization
 Process/Memory/File/I-O Management
 Programming Interface
 Communication Facilities

December 8, 2021 7
TCS Internal
 The UNIX Process

• A process in UNIX is a program in execution with definite life-time and well-defined

hierarchy.
• User Space - Process execution on user data structure that represents an application
and is manipulated by its own functions (user mode execution).
• Kernel Space - Process execution on system’s data structures that represent the
resources (memory, files, devices) and supported by the kernel routines. Depending
upon resource requirement and availability, process’s states are controlled by executing
the kernel routines accessed through system calls (system mode execution).

December 8, 2021 8
TCS Internal
 Environment around the UNIX Process
• When a process is created, UNIX opens 3 streams stdin/stdout/stderr for basic communication with
respect to the process control terminal. In addition, it knows the current working directory for
performing file I/O.
• Each login shell maintains a description of the environment as a table of pointers to strings.
TERM=vt100
extern USER=ashok
char LOGNAME=ashok
HOME=/usr/ashok
**environ PATH=.:/bin:/usr/ahok/bin
A UNIX process
……

• A global shell environment pointer called environ is maintained by UNIX kernel and it can be used
by a process to access its own table
• The shell does not directly use this table, but it creates a child process and calls exec() system call
to execute a command program that uses the table inherited from the shell parent.

December 8, 2021 9
TCS Internal
 Environmental Variables (EVs)

• Children inherit the entire execution environment from the parent. Some examples of the
environment variables are the USER, LOGNAME, HOME, PATH, PS1, PS2, TERM MAIL,
etc.
• The HOME Variable
It specifies an associated directory with every user in a UNIX system. If the HOME
variable for the user Sita contains /usr/sita/stores, every time Sita logs in, she is taken to
the directory stores.
The variable HOME is referenced the same way:
$ echo ${HOME}<Enter>

December 8, 2021 10
TCS Internal
 Environmental Variables (EVs) ...

• The PATH Variable

contains a list of all full path-names (separated by a colon) of directories that are to be
searched for an executable program. For example, the command
$PATH=.:/usr/bin:/bin<Enter> specifies directories to be searched for any executable file
or a command file (current directory, /usr/bin and /bin, in that order).
• The PS1 Variable
The system prompt may be changed by setting the value of this variable to the desired
prompt:
$ PS1=“Hello>”<Enter>
Hello> #can be changed only at the UNIX command line, not within a shell script.
• The PS2 Variable: prompt string for continued command line (default ‘> ‘).

December 8, 2021 11
TCS Internal
 Environmental Variables (EVs) ...

• The LOGNAME Variable

contains user’s login name. Its contents cannot be changed by the user, but can be
displayed:
echo “${LOGNAME}”
• The TERM Variable
Names the kind of terminal you are using; setting it helps to manage your screen more
effectively, say,
$ TERM=vt100<Enter>
• The PWD Variable
The current working directory can be displayed:
echo “${PWD}”
In fact the whole environment table can be displayed.
• $IFS: String of characters which are used as word separators in command line ( space,
tab, newline chars).

December 8, 2021 12
TCS Internal
 Environmental Variables (EVs) ...
• The MAIL Variable
Names the standard file where your mail is kept
• Variables for Abbreviation
It is useful if frequently referred names are abbreviated: d=/very/long/directory/name
• The .profile File
Some of above variables like HOME and LOGNAME are set automatically each time the
user logs in. The others, however, have to be set. The .profile is used for this purpose as
it is executed as soon the user logs in. A sample .profile file would look like:
PATH=.:/bin:/usr/bin
export HOME PATH PS1 MAIL

December 8, 2021 13
TCS Internal
Any Questions ?

December 8, 2021 14
TCS Internal
Utilities and Applications

• UNIX utilities or commands are collection of about 200 programs (ed, sed,
awk, vi, grep, make, link, debug, etc.) that service day-to-day processing
requirements. They are invoked through the shell, which is itself another utility.
• More than a 1000 UNIX-based application programs like DBMS, word
processors, language processors, accounting software, etc. are available from
independent vendors

December 8, 2021 15
TCS Internal
 The UNIX Shell

• It is the interpreter of UNIX to decipher and execute commands - user interface to the
kernel for isolating the user from the knowledge of kernel functions.
• Maintains interactive dialogue with the user and is capable of Input/Output
Redirection. Can do background processing so that time-consuming, non-interactive
tasks can proceed side by side.
• Makes it possible to connect a couple of commands with one command taking input
from another (pipes). The commands that can be connected in this way are called
filters, because they filter or manipulate data in different ways. Pipes and filters form
the central feature of the UNIX philosophy.
• Shell variables can control the behavior of shell as well as other programs by storing
data in them.

December 8, 2021 16
TCS Internal
 The UNIX Shell ...

• Shell includes programming language features which can be used to build shell scripts for
performing complex operations.
• Shell scripts are frequently used sequence of shell commands stored in a file; file-name can be
used later to execute the stored sequence.
• Types of Shells - Bourne Shell - original command interpreter developed at AT&T by Stephen R.
Bourne; fastest official shell distributed with UNIX systems (executable filename sh)
• C Shell - developed by William Joy and others at UCB; gets its name from C due to syntax
resemblance of its programming language (executable file name csh)
• Korn Shell - developed by David Korn, combines best features of both shells, not popular
(executable file name ksh)
• Restricted Shell - restricted version of Bourne shell, typically used for guest logins and in secure
installations (executable file name).

December 8, 2021 17
TCS Internal
Shell variables

• Data in shell scripts and environment settings stored in variables

• Conventionally all upper-case
• Shell variables exist only in current shell instance
• Environment variables passed to subshells
• Shell variables can be exported into environment

December 8, 2021 18
TCS Internal
Shell features

• Command-Line expansion/ substitution

• Command Completion
• Command-Line Editing
• Aliases
• Shell Functions
• Command Recall
• History search
• Job Control

December 8, 2021 19
TCS Internal
Getting Help

• Check the manual pages!

man man - help on the man command

man who - help with the who command
info ls
ls --help

December 8, 2021 20
TCS Internal
 Some Shell Commands

• Command Function
• pwd print working directory
• cd change directory
• mkdir make directory
• rmdir remove directory
• ls list contents of directory
• cat display contents of files
• cp copy one file to another
• rm remove files
• mv change name of a file/directory
or move file from one directory to another
• date display system date
• who display logged-in users’ names
• echo display string; used for prompt
• ln create another link to a file

December 8, 2021 21
TCS Internal
 The UNIX File System (UFS)

• It is a hierarchical collection of 3 types of files: ordinary, directory and special files (device, pipe, fifo,
socket).
• A UNIX file is featureless because it is simply an array of bytes.
 Dominant file type in UNIX is the text file.
 System related files are also stored in text form.
 Separate device can be added by creating a file for it.
• Root is the supremo and is represented by the ‘/’. Every subdirectory must have a parent.
• File names can be up to 14 characters long; can contain both upper and lower case alphabets, digits,
a dot, hyphen (-), underscore (_) anywhere; should not have a blank or tab; are case-sensitive.
• Path names are a sequence of directory names separated by ‘/’. They are used to access files.
• Absolute pathname - file location is determined with respect to the root.
• Relative pathname - file location is determined with respect to the current directory.

December 8, 2021 22
TCS Internal
 The UNIX File System …
Though the UFS looks hierarchical, it is actually a directed
acyclic graph because files can be shared.
root ( / )
/mnt /home …
/bin /lib /proc /tmp /sbin ntpuser1
vi/awk /dev /etc /usr /var to 4
cat/cut audio aliases bin – sh lost+ lost+
chmod hosts etc found found
beep games
date/sort init.d crash ntpusr
cpu dict ftp …
echo fd0 to mail include lib
ed/sed fd7 mknod lib mail cron
grep/tar motd man opt fax
ram sbin news
kill/pwd news tmp
ttyxx share spool mail
ln/ping … passwd tmp mqueue
…
mv/ps/ls uucp … uucp
rm/rmdir … …
December 8, 2021 23
… TCS Internal
Important Directories in Linux File System

/home – It holds user’s home directories. In other UNIX systems, this can be /usr directory.
/bin – It holds many of the basic Linux programs; bin stands for binaries, files that are executable.
/usr – It holds many user-oriented directories:
bin – It holds user-oriented Linux programs.
sbin – It holds system administration files.
spool – It has several subdirectories:
. mail holds mail files
. spool holds files to be printed
. uucp holds files copied between Linux machines.
docs – various documents including useful Linux info
man – man pages accessed by typing the man <command>
games – the fun stuff!
/sbin – It holds system files that are usually run automatically.
/etc – It and its subdirectories hold many of Linux config files.
/dev – It holds device files. All info sent to /dev/null is thrown into trash. Your terminal is one of the /dev/tty
files.

December 8, 2021 24
TCS Internal
 File Access Permissions (FAP)

• FAP refer to permissions associated with a file with respect to the following:
• File Owner, Group Owner, Other Users.
Read r Allows displaying /copying
Write w Allows editing/deleting
Execute x Allows execution of a file
• The protection on a file is referred to as its file mode on UNIX systems. The long version of
ls command also displays FAP in addition to user and group ownership. The first character
indicates the file type, a hyphen for a plain file, and a ‘d’ for a directory; rest of the bytes
may be rwxrwxrwx or a ‘-’ for r/w/x if that particular permission does not apply.

December 8, 2021 25
TCS Internal
 File Access Permissions (FAP) ...

• Meaning of 0123456789 FAP bytes of a directory:

• No read does not allow list <directory> or remove directory.
• No write does not allow copy files to directory, remove files from directory, rename files in
directory, make a subdirectory, remove a subdirectory, move files to/from directory.
• No execute does not allow display<directory> from within directory, change to directory,
display a file in dir, copy a file to/from dir.
• Byte 0 is ‘b’ (block), ‘c’ (character) or ‘p’ (device) for special files.

December 8, 2021 26
TCS Internal
Any Questions ?

December 8, 2021 27
TCS Internal
 Filters and Pipes

• Filter is a program that takes its input from standard input file, processes it, and sends its output to
standard output file (simple filters: head, tail, cut, paste, sort, uniq, etc; advanced filters: grep, pg, wc, tr,
awk).
• The grep filter (global search for regular expressions) searches files for a particular pattern. The format
of grep command is:
grep [options] reg-expr filenames
Options:
-n prints each line matching the pattern with line no.
-c prints only the count of lines that match the pattern.
-v prints out all those lines that do not match the pattern.
Specifying grep regular expressions
\c Backslash turns off any special meaning of character c.
^ Pattern following it must occur at the beginning of line.
$ Pattern preceding it must occur at the end of line.
. Matches any single character
[…] Matches any one of chars in …; ranges like a-z are legal
[^…] Matches any single char not in …; ranges are legal.
r1r2 Concatenation of regular expressions: r1 followed by r2

December 8, 2021 28
TCS Internal
 Filters and Pipes ...

• pg displays a large file, a screen-full at a time.

• wc counts number of lines, words, characters
• tr is used to translate one set of characters to another; does not accept a filename as an
argument but only the redirected file input.
• cut (vertical file slit) - extracts only the desired columns from the output of certain
commands.
• paste (vertical file paste) - what you cut with previous cmd can be pasted back with the
paste cmd. But while cat pastes more than one file horizontally, paste does it vertically.
• pipes (|)combine filters and commands such that standard output of one can be sent as
standard input to another.
Advantage of pipes: UNIX tools (commands) can be combined rather than be rewritten.

December 8, 2021 29
TCS Internal
Filters and Pipes - awk

Pattern Scanning and Processing Language

$ awk ‘program’ filenames …where program is:
pattern { action }
pattern { action }
… # By Aho, Weinberger, Kernighan
awk reads the input in the filenames one line at a time. Each line is compared with each
pattern in order; for each pattern that matches the line, the corresponding action is
performed. Awk does not alter its input files.
Patterns can be regular expressions exactly as in egrep, or C-like complicated conditions.

December 8, 2021 30
TCS Internal
awk ...
Example 1
$ awk ‘/regular expression/ { print }’ filenames …
$ awk ‘/regular expression/’ filenames …
Both print every line that matches the regular expression
Example 2
$ awk ‘{ print }’ filenames …
does exactly what cat does, albeit more slowly.
Example 3
$ who | awk ‘{ print $1, $5 }’
prints names of logged-in users and time of login. Awk splits each input line automatically
into fields strings) separated by blanks or tab.

December 8, 2021 31
TCS Internal
sed

• Stream editor
• Reads a file or stream of data, writes out the data, performing search and replace
instructions
• Uses regular expressions in search string

December 8, 2021 32
TCS Internal
Using sed

Sed addresses
• sed ‘s/TCS/Tata Consultancy Services Ltd/g’ company.txt
• Sed ‘1,50s/TCS/Tata Consultancy Services Ltd/g’ company.txt
• Sed ‘s/[Tt]cs/TCS/g’ company.txt

Multiple sed instructions

December 8, 2021 33
TCS Internal
Any Questions ?

December 8, 2021 34
TCS Internal
 The vi Editor

• It is a visual editor used to enter and edit text files

• Invoking vi with/without filename puts it in command mode:
$ vi [<file name>]
• vi command mode syntax:
[<repeat-count>] <command>
• vi works in three different modes:
 Input - where any key is entered as text
 Command - where keys are used as commands to act on text
 ex - where ex commands can be entered in last line to act on text

December 8, 2021 35
TCS Internal
Relationship between 3 Modes

Command
ss s SMode

:<E
yP oO

<E
:sh

nte
Ke r R

nte
re

^d

r>
<E
ey
aA

cK

r>
or

nte
iI

Es

exit

r>
Input ex
Mode Mode

December 8, 2021 36
TCS Internal
 Input Mode Commands

Command Action
a Appends text to right of cursor
A Appends text at end of line
i Inserts text to left of cursor
o Inserts blank line below + inserts text
O Inserts blank line above + inserts text
rx Replace current character with char x
Rtext<Esc> Replaces character with text
s Replaces character under cursor with
many characters
S Replaces entire line
20i- followed by <Enter> or <Esc> enters
20 hyphens (-) in one line

December 8, 2021 37
TCS Internal
Save and Exit from vi
Command Action
ZZ saves file only if vi started with a filename
Save and Exit Commands of ex Mode
:w save file and remain in edit mode
:wq save file and quit edit mode
:w <filename> write buffer to filename
:q quit editing mode when no changes
are made
:q! quit editing mode but after
abandoning changes
:x save file and quit editing mode

December 8, 2021 38
TCS Internal
Cursor Movement Commands

Command Action
h or backspace Moves Cursor Left
l or spacebar Moves Cursor Right
j Moves Cursor Down
k Moves Cursor Up
nG Go to line number n
$ Moves cursor to end of line
w Moves cursor to next word
b Moves cursor back to previous word
e Moves cursor forward to end of word
Note: W, B, E perform functions similar to w, b, e except that punctuation is
skipped)

December 8, 2021 39
TCS Internal
Paging and Scrolling Commands
Command Action
^d Scroll down half screen
^u Scroll up half screen
^f or <PfDn> Full Page forward (12 lines by
^b or <PgUp> Full Page backward default)
^l Redraw Screen
[[ Scroll to beginning of document
]] Scroll to end of document
{ Scroll to previous paragraph
} Scroll to next paragraph
( or ) Scroll to previous or next sentence

December 8, 2021 40
TCS Internal
Search and Repeat-Search Commands

Command Action
/pat Search forward for pattern pat
?pat Search backward for pattern pat
n Repeat search in previous search
direction (no repeat factor)
N same as n but in opposite direction
fx or tx Move cursor forward to or before the
first occurrence of character x in current line
Fx or Tx Same as above but move backward
(repeat factor works with all f/F and t/T)
; or , Repeat last character search in same
or opposite direction (by f/F/t/T) only
in current line (repeat factor works)

December 8, 2021 41
TCS Internal
Deletion and Modification Commands
Command Action
dw Delete Current Word
dd Delete Current Line
d$ or D Delete from cursor to end of line
cw Change Current Word
cc Change Current Line
c$ or C Change from cursor to end of line
x Delete character under cursor
X Delete character before cursor
J Join current, next lines (also nJ)
.(dot) Repeat last editing instruction
u Undo single last change
U Restore all changes to line since
cursor moved to it

December 8, 2021 42
TCS Internal
Commands to Move or Copy Lines
[“<named-buffer>][n]dd
Cut (delete) n lines starting from current line
[“<named-buffer>][n]yy
Copy n lines starting from current line
[“<named-buffer>]p
Put yanked text after current cursor position
[“<named-buffer>]P
Put yanked text before current cursor position
• Note: Named-buffer is useful for copying an area from one file into another. Open
some files simultaneously by vi. Mark an area in one file by dd or yy; move to
another file (by :next) to paste (by p or P) the named area; then say :rewind to
return to the parent file.

December 8, 2021 43
TCS Internal
 Customizing vi

• vi can be customized as per users’ requirements using ex-mode commands.

ex Commands Action
:set all Display all set options; those pre-fixed with no are not
operative
:set autoindent (ai for short)
Extremely useful to programmers for
indentation of lines
:set number (nu for short)
Displays all lines with numbers which are not preserved on
saving file
:set nonumber (nonu for short) - Reverses number setting
:set tabstop=6 (ts for short)
Changes default tab setting (8 spaces)
:set ignorecase (ic for short)
ignores case while pattern matching

December 8, 2021 44
TCS Internal
 Customizing vi ...

:set showmatch (sm for short)

Helps locate matching brackets when ) or } is entered in input mode; beeps when no match found to
alert for correction
:set autowrite (aw for short)
Writes current file automatically whenever switching files with :n and escaping to shell with :sh
:set showmode - Displays a message when vi goes to input mode
:next (n for short) Moves to next file opened in vi
:rewind Comes back to parent file
• All sets can be stored in .exrc file used by vi for its startup instructions.
• Equivalently an environment variable, EXINIT can be assigned to store the settings:
$EXINIT=“set nu tabstop=6 ignorecase”
• It is better to make this assignment in the .profile so that it is available for all sessions.

December 8, 2021 45
TCS Internal