Linux Unit 4.1 Notes
Linux Unit 4.1 Notes
Introduction
• The shell sits between user and the operating system, acting as a command
interpreter.
• It reads user terminal input and translates the commands into actions taken
by the system.
• The shell is analogous to command.com in DOS.
• When you log into the system you are given a default shell.
• When the shell starts up it reads its startup files and may set environment
variables, command search paths.
• The original shell was the Bourne shell, sh. Every Unix platform will either
have the Bourne shell, or a Bourne compatible shell available.
• It has very good features for controlling input and output, but is not well
suited for the interactive user. To meet the latter need the C shell, csh, was
written and is now found on most, but not all, Unix systems.
• It uses C type syntax, the language Unix is written in, but has a more
awkward input/output implementation.
• The default prompt for the Bourne shell is $ (or #, for the root user). The
default prompt for the C shell is %.
• Numerous other shells are available from the network. Almost all of them
are based on either sh or csh with extensions to provide job control to sh
• Some of the more well known of these may be on your favorite Unix
system: the Korn shell, ksh, by David Korn and the Bourne Again Shell,
bash, from the Free Software Foundations GNU project, both based on sh,
the T-C shell, tcsh, and the extended C shell, cshe, both based on csh.
• The C shell, the Korn shell and some other more advanced shells, retain
information about the former commands you’ve executed in the shell.
Shell’s Functions :
• Program execution
• Name substitution and gobbling [Globbing is mainly used to match
filenames or searching for content in a file. Globbing uses wildcard
characters to create the pattern.]
• Input/output redirection and pipes
• Environment control.
• Programming language
The sh command :
• When you log in, you get $ prompt. As long as you do not enter something,
the prompt remains idle. But, the UNIX command sh is running.
• The shell sh starts running the moment you log in. sh command can be
located in /bin. sh is waiting for you to enter command and the system is
idling.
• The shell swings into action the moment you enter something through the
keyboard. The shell, technically the UNIX command itself, it accepts other
UNIX commands as input.
• The shell first sees whether the command line is in the form the kernel can
understand. If it is not, it processes the request to recreate a simplified
command line.
• It then leaves the job of command execution to the kernel.
• The kernel directly communicates with all hardware and takes care of all
simple or complex processes. Since the kernel interacts directly with the
shell, the user remains transparent to the complex internal processes that
take place between the kernel and hardware.
• The major time spent by the shell is in waiting for the input from the user.
With this input, it performs series of processing tasks. It interacts with the
kernel if required. After the job is complete, it returns to its waiting role, to
start the next cycle.
• The following activities are typically performed by the shell in each cycle-
1. It issues $ prompt and waits for user to enter a command
2. After the command has been entered, the shell scans the command line for
some special characters and then rebuilds the command line after processing
is complete.
3. The command is then passed to kernel for execution and the shell waits
for its completion.
4. The $ prompt appears and the shell waits for new command.
• When there is no input from the user, the shell is said to be sleeping. It
indicates this by $ prompt; which indicates that it is ready to accept new
command. It wakes up whenever user enters some input and presses enter
key. Sleeping, waiting and waking are the UNIX terms used for this.
1) The ? and * :
• The command rm* is used to delete all the files in the current directory.
• The *, known as a metacharacter , is one of the characters of the shell’s special
set.
• It matches any number of characters (including none).
• When this character is appended to the string chap, the pattern chap*
expands to all files, in which the first four characters constitute the string
chap.
• It thus matches all the files specified in the command line
Example:-
$ ls –l chap chap01 chap02 chap03 chap04
• It also matches the string chap.
• This previous command can now be shortened with this sequence.
$ ls -x chap*
chap.txt chap01.txt chap02.txt chap03.txt chap04.txt
• Now, when the shell encounters the above command line, it looks for
metacharacters, and identifies the * immediately.
• It then matches all the files in the current directory with the pattern chap*,
and replaces the pattern with the list that matches the pattern.
• It then reconstructs the command line as below and passes it on to the kernel
for execution.
$ ls –x chap chap01 chap02 chap03 chap04
• may occur anywhere in a filename, and not merely at the end.
• Thus *chap* matches all the following filenames.
$ ls –x *chap*
chap newchap chap01 chap02 chap03.txt
What does the command wc* do ?
• It makes a word count of each and every file in the current directory (plus
a total at the end!)
Wildcard Significance
5) Matching a dot-
• The * does not match all files beginning with a ‘.’ or / of a pathname. Such
files must be matched explicitly.
• If you want to list all the hidden files in your directory having atleast 3
characters after the dot, then the dot must be matched explicitly.
$ls –x .???*
.exrc .news_time .profile
• However, if a filename contains a dot anywhere except at the beginning, it
need not be matched explicitly.
• Example :-
The expression emp*lst matches a dot embedded in the filename.
$ls –x emp*lst
emp.lst emp1.lst emp22.lst emp2.lst
6) Using rm with * :
• Be cautious when using rm with *
– rm chap*
• Removes all the chapters
• You inadvertently introduce a space
– rm chap * // space between chap and *
– Even if there is no file named chap in the current directory and you
get an error message to indicate that, the command will still remove
all files in this directory because of the singular existence of the *!.
• This happens because the shell rightly treats the * as a separate argument.
• When using a wild-card expression with the rm command, you should
pause and check the command line before you finally press the <Enter>key
• The shell has to treat these metacharacters in a special manner because they
represent a feature of the shell and not of the command using them.
• It is necessary for the shell to do the interpretation first because chap*
means nothing to rm
• Nor does it mean anything to any command using filename as an argument.
• The design of the UNIX system prevents the execution of a command till the
shell has expanded all wild-card expressions.
Quoting :
• Escaping is not a convenient solution when you need to despecialize the
meaning of group of characters.
• When an argument to a command is enclosed in quotes, the meaning of all
special characters is turned off.
What happens when you Enter shell metacharacter with the Echo command?
Quoting includes :
1) Quoting preserves spaces
2) Escaping in echo
• The above arguments to echo could have been preserved by escaping the
space character wherever it occurs at least twice :
2) Escaping in echo
• Apart from the shell, there are some commands that use the \ as part of its
syntax.
• Rather than remove the special meaning, the \ is also used to emphasize a
character so that a command treat it as a special character.
$ echo ‘Enter your name : \c’
Enter your name :$ _
• Observe that the prompt has been returned, not in the next line, but at the end of
the echoed string.
• \c used here represents an escape sequence, which results in in the positioning
of the cursor immediately after the argument, instead of the newline.
• echo here treats the character c as special.
• Echo also accepts certain escape sequences such as \t, \f, \n
$ echo -e “line 1 \n line 2\n line 3\n …”
line 1
line 2
line 3
…
• echo accepts certain escape sequences that manipulate the cursor motion in a
number of ways; \c is just one of them.
• There are other sequences ;
– \t produces a tab,
– \f creates a form feed (page skip),
– while \n produces a newline:
$echo ‘\t This message is broken here\n \n into three lines’
This message is broken here
--------------------------------
into three lines
Redirection :
• The family of commands take input from a file and send output to the terminal.
These commands can also operate on character streams(sequence of bytes).
Without really knowing the source of the ‘stream’.
• Similar , they also write the output in1 the form: of a character stream,
without knowing it its destination.
• These streams are nothing but a sequence of bytes that many commands
see as input and output.
• A command can be ignorant about source and destination of these streams. The
shell sees that these streams are handled properly.
• When the commands cat and wc are used without arguments in the following
manner:
cat # No arguments
wc
• It is an indication to a shell that the input to each of these commands no
longer comes from a file, but from a stream.
• The default source of this stream is the user’s keyboard and has to be keyed
in.
• It is the responsibility of the shell to set up the connection between ‘the
command and the keyboard without the command knowing anything about
it.
• This stream is called the standard input to the command.
• cat and wc also send the output as a character stream.
• The default destination of this stream is the terminal,
• Here, the shell has to setup another connection. between the command and
the terminal so that all output is displayed on the screen.
• This stream is called the standard output of the command.
• Here again, the command is ignorant of the destination of the stream that it
has generated.
• When you enter wrong commands, or when the shell or the command
encounters an error, like the non-existence of a file that you are trying to
open, certain diagnostic messages are echoed by the system.
• This constitutes a third stream, and, like the standard output, its default
destination is also the terminal.
• This file is called the standard error. Even though the standard output and
standard error have the same default destinations, they are in fact two
separate streams, and the shell possesses a mechanism of capturing them
individually.
• Trying to “cat” a non-existent file produces the third stream:
Redirection includes :
1) Standard Input
2) Standard Output
3) Combining standard input and standard output
4) Standard Error
1) Standard Input :
• The command considers its own input as a stream. This stream can come from-
1) the keyboard (default)
2) a file (using redirection)
3) another program ( using pipeline)
• We will see how the shell manipulates the input source depending on the
instructions it sees in the command line.
eg cat and wc commands need filename(s) as an argument.
• Both commands ‘have a built-in mechanism of taking standard input too.
• It is a general feature of many commands using filenames to look to the shell
for input when the filename is omitted.
• wc is one of them, and when you use it without a file name, the input has to be
keyed in –
• This situation is similar to the one encountered when creating a file with the
cat command.
• Enter the three lines of text, signify the end of input with <Ctrl-d>, and then
press <Enter>.
• wc, which takes the stream from the standard input, immediately counts 3
lines, 20 words and 103 characters.
• reassignment feature is available in the shell so that you can choose to take the
standard input from a file rather than the keyboard.
• This is the second source of the stream, and the connection is set up by the shell
using the metacharacter < (left chevron):
$ wc < infile
--- --- ----
• In both the above cases, the input was from the standard input.
• In the former case, the shell assigned the default file, i.e., the keyboard, as the
source of the stream.
• In the second case, the shell has redirected the stream, or reassigned the
standard input file to come from a disk file instead.
• The important thing here is that wc has no idea where the stream came from; it
is not even aware that the shell had to open the file infile and take input from
there!
• This isn't the case when we is used with a filename as an argument.
The output is similar, but not identical:
• Everything remains the same, except for the presence of the fourth column that
represents the filename.
• It was possible for the command to print the filename this time since the file
was opened by the command, and not the shell.
• Note: When the standard input is redirected to come from a file, it is the shell
who opens the file, and not the command. In fact, the command is totally
ignorant of what the shell is doing.
• When the input is taken from a multiple source, a file, as well as the standard
input, the – symbol must be used to indicate the sequence of taking the input.
• The meaning of the following sequences should be quite obvious:
2) Standard Output:
• Like the standard input, the standard output stream also has three similar
destinations-
1. Output can be directed to terminal ( default)
2. Output can be directed to file( using redirection)
3. Output can serve as an input to another program.
• Any command displaying output on the screen is normally using standard
output. The mechanism of redirection is also available with standard output.
• A similar feature of redirection is available with standard output also.
• You can replace the default destination of any file using > (right chevron)
operator, followed by a file name.
$ wc infile > newfile
$ cat newfile
3 20 103 infile
• This sends output of wc to a file newfile.
• Nothing appears on the terminal screen except the return of the prompt.
• If the output file doesn't exist, the shell will create it before executing the
command.
• If the newfile does not exists, it is created. If it exists, it is overwritten.
• A useful application of the concept of standard output can be the
concatenation of all the chapters of the book to a file textbook.
• Wild-cards help set up an abbreviated command line:
• We can also append to a file, instead of overwriting, by using the >> (the right
chevron used twice) symbol.
• So, if you want to append the output of the who command to the file newfile,
which already contains a single line of text, you can use
• Two or more commands can be combined, and the aggregate output also sent to
a file.
• A pair of parentheses groups the commands, and a single > symbol can be used
to redirect both of them:
• Note: When the output of a command is redirected to a file, the output file is
created by the shell before the command is executed.
• Redirection is one feature that doesn't really care about the type of file that it
encounters.
• You can use a device name to echo a message on someone's terminal screen.
• Any user working on the terminal / dev/tty02 will see this echoed message on her
terminal (provided the terminal is set up accordingly).
4) Standard Error:
• The third stream, standard error includes all error messages written to the
terminal. This output may be generated by command or by the shell, but in
either case the default destination is terminal. This output can also be reassigned
to a file.
• If you try to "cat" a file that doesn't exist, and redirect the output to a file with
the > symbol, you get an unusual result:
$ cat bar > errorfile
cat : can not open bar: no such file or directory
• The system still displays output on the terminal, the error massage is not
directed to the errorfile.
• The standard error can't be redirected in the same way the standard output
can (with the > or >> symbol).
• File descriptors:
• Each of these three standard files has a number associated with them, called
File descriptor, which is used for identification.
0: represents standard input
1: represents standard output
2: represents standard error
• The File descriptors are implicitly prefixed to the redirection symbol.
• For instance, > an 1> mean the same thing to the shell ,
while < and 0< also are identical.
• We don't need to use the numbers 0 and 1 to prefix the redirect symbols
because they are the default values.
• However, we need to use the descriptor 2> for the standard error:
$ cat bar 2> errorfile
$ cat errorfile
cat : can not open bar: no such file or directory
• You thus have a method for holding such error 'messages in a separate file.
This enables to run long programs and save error output to be viewed at the
end of the day.
• The standard output and standard error symbols can also be used in the
same command line.
• If you try to "cat" two files newfile and nofile, where newfile exists and nofile
doesn't, you can use separate redirection symbols for both the streams:
• Sometimes, you will need to direct both the standard output and standard
error streams to the same file.
• You will have to use some more special symbols.
Pipes :
• The standard input and standard output constitute two separate streams, which
can be manipulated separately by the shell. The shell can also connect these
streams together, so that one command takes input from the other. This is
important features of UNIX.
• eg who : displays list of users, one user per line
• If you save this output in a file and apply the wc -l command to count the
number of lines in this file, then you would have effectively counted the
number of users:
• To count the number of users-
$who > user.lst
$cat user.lst
abc tty0 April 18 10:45
pqr tty1 April 18 10:49
$wc-l user.lst
2 user.lst
• Disadvantages of this method-
– You need intermediate files that has to be removed after the wc
command has completed its run.
– Process is slow. The second command cant act unless the first has.
completed its job.
– When handling complex tasks involving a series of commands to be
used in sequence, this approach can be at best be termed pedestrian.
– When handling large files, temporary files can build up easily and
eat up disk space in no time.
• The shell enables the standard output of one command to be connected as
(std) input to another.
• The | pipe symbol is used as a connector of commands.
• Since who sends out a character stream, and wc accepts it, the above
sequence of commands can be combined into a single instruction as-
$who | wc –l <enter>
2
Here the output of who has been passed directly to the input of wc and who
is said o be piped to wc. No intermediate files are created when they are
used.
When a sequence of commands are combined together, a pipeline is said to
be formed.
• It is the shell that sets up this interconnection, and the commands have no
knowledge of it.
$ls | wc –l : counts number of files in a current directory
15
• Because wc uses standard output, you can redirect this output to a file:
• Note: In a pipeline, the command on the left of the | must use standard
output, and the one on the right must use standard input.
tee:
• UNIX provides a feature by which you can save the standard output in a file
as well as display it on terminal( or pipe it to a another process). This is
made possible by the tee command, which is available in /bin.
• Tee uses standard input and standard output, so that it can be placed
anywhere in the pipeline.
• Tee breaks up the input into two components -
• one component is saved in a file and
• the other is connected to the standard output.
• Tee performs no filtering action.
• You can use tee to save the output of who command in a file, as well as
display it.
$who | tee user.lst
abc tty0 April 18 10:45
pqr tty1 April 18 10:49
One channel is saved in user.lst, while the other is displayed on the standard
output.
• You can cross-check the display with the contents of the file user.lst:
Since the tee uses standard output, you can pipe its output to another
command, say, wc
$who | tee user.lst | wc –l
3
• How do you use tee to display, both the list of users, as well as its count
on the terminal?
• Since the terminal is also a file, you can use the device name /dev/tty as an
argument to tee:
• Here the terminal is treated in the same way as any disk file
• The -a (append) option appends the output to the file specified as
argument.
• The following sequence appends one stream to calfile and overwrites
calfile2 with the other stream.
Command Substitution:
• UNIX allows to connect two commands in different ways. Pipe is one
method.
• The shell allows the argument of a command to be obtained from the output
of another command. This feature is called Command Substitution.
• When a command is enclosed in a pair of back quotes (``), the shell executes
the command first, and the enclosed command text is replaced by the output
of the command. It is like running a command within command.
• To consider a simple example, suppose you wish to echo today's date
with the echo statement.
Using the feature of command grouping, you need to issue sequentially
the echo and date commands:
$echo The todays date is ; date
The todays date is Wed April 18 1:25:15 IST 2020
$echo The todays date is `date`
The todays date is Wed April 18 1:25:15 IST 2020
• Command substitution offers the facility of doing this in a single
statement.
• Use the expression `date` as an argument to echo:
• When scanning a command, if the shell comes across with the pair of `
metacharacters, the shell executes the enclosed command and puts the output
in its place.
• Like pipes, no intermediate files are used, nor are any variables required
to store them.
• For the Command Substitution to work, enclosed command must use
standard output.
• This feature can be used to generate useful messages.
• For example, you can use two commands in a pipeline, and use the output
as the argument to a third:
$echo There are `ls |wc –l` files in current directory
There are 58 files in current directory
• The ` is one of the few characters interpreted by the shell when placed
within double quotes, though it loses its meaning in single quotes
• The above argument would have worked just as well even if the
arguments had been quoted:
Shell Variables:
The shell provides the facility to define and use variables in the command line.
These variables are called shell variables. No type declaration and initialization
are required before its use.
Shell variables are assigned with = operator and evaluated by prefixing the
variable name with $
$ x=37
$echo $x
37
• A variable name can consist of letters of the alphabet, numerals and the
underscore character.
• The first character necessarily has to be a letter.
• Moreover, the shell is sensitive to case; the variable x is different from X.
• The shell uses another notation when evaluating variables-a pair of curly
braces to enclose a variable name.
• Here is an alternative form of evaluating the variable fname:
• This form has certain advantages; you can tag a string to it without needing
to quote it. This way you can generate a second set of filenames by affixing
the single character x to each one:
• Note how the variable z was defined as the concatenated output of two other
variables.
Examples :
$ MyVar=999
$ echo $MyVar
999
$ echo "$MyVar"
999
$ echo '$MyVar'
$MyVar
$ city=Delhi
$ echo "We are in $city today."
We are in Delhi today.
$ echo 'We are in $city today.'
We are in $city today.