Richard Brittain, Dartmouth College Computing Services. © 2003,2004,2010 Dartmouth College.

Comments
and questions, contact Richard.Brittain @ dartmouth.edu

Unix shell scripting with

ksh/bash
Course Handout

These notes may be found at http://www.dartmouth.edu/~rc/classes/ksh. The online version has

many links to additional information and may be more up to date than the printed notes
The goals of this class are to enable you to:

 Learn what kinds of problems are suited to shell scripts

 Review the most commonly used Unix commands that are useful in shell scripts.
 Write simple shell scripts using the Bourne, Korn or Bash shells

These notes are intended for use in a 2-part class, total duration 3 hours.

Assumptions:
It is assumed that you already know how to:

 log in and get a command line window (any shell)

 run basic commands, navigate directories
 use simple I/O redirection and pipes
 use a text editor (any one)
 look up details of command usage in man pages

Example commands are shown like this. Many commands are shown with links to their
full man pages (sh)
Output from commands is shown like this; optional items are [ in brackets ].

Some descriptions in these notes have more detail available, and are denoted like this:

More details of this item would appear here. The printed notes include all of the additional information

Permission is granted to download and use these notes and example scripts, as long as all copyright notices are
kept intact. Some of the examples are taken from texts or online resources which have granted permission to
redistribute.
These notes are updated from time to time. The "development" set of notes are http://northstar-www.dartmouth.edu/~richard/classes/ksh (Dartmouth
only)

Richard Brittain, Dartmouth College Computing Services.

© 2003,2004,2010 Dartmouth College.
Comments and questions, contact Richard.Brittain @ dartmouth.edu

1
Table of Contents
The goals of this class are to enable you to: ................................................................................ 1
1. What is a Shell Script ........................................................................................................................... 5
2. Why use Shell Scripts........................................................................................................................... 5
2.1. Typical uses................................................................................................................................... 6
3. History of Shells ................................................................................................................................... 6
4. Comparison of shell features .............................................................................................................. 7
4.1. Core Similarities (and recap of basic command line usage)......................................................... 7
4.2. Principal Differences..................................................................................................................... 8
5. Other Scripting Languages................................................................................................................... 8
6. ksh/bash vs sh ..................................................................................................................................... 9
7. Basic sh script syntax ......................................................................................................................... 10
7.1. Exit status ................................................................................................................................... 10
8. Filename Wildcards ........................................................................................................................... 11
9. Shell Variables ................................................................................................................................... 11
9.1. Setting and exporting variables.................................................................................................. 11
9.2. Using variables............................................................................................................................ 12
9.3. Conditional modifiers ................................................................................................................. 12
9.4. Variable assignment command prefix........................................................................................ 12
10. Preset Shell Variables ...................................................................................................................... 13
10.1. Login environment ................................................................................................................... 13
10.2. Shell internal settings ............................................................................................................... 13
10.3. Process ID variables.................................................................................................................. 14
10.4. ksh/bash additional features.................................................................................................... 14
11. Command Line (positional) arguments ........................................................................................... 14
11.1. Setting new positional arguments............................................................................................ 15
12. Shell options .................................................................................................................................... 16
13. Command Substitution.................................................................................................................... 16
13.1. sh syntax................................................................................................................................... 16
13.2. ksh/bash syntax........................................................................................................................ 17
14. I/O redirection and pipelines........................................................................................................... 17
14.1. Output redirection.................................................................................................................... 17

2
 14.2. Input redirection ...................................................................................................................... 17
14.3. Command pipelines.................................................................................................................. 18
15. Input and Output............................................................................................................................. 18
15.1. Script output............................................................................................................................. 18
15.2. Script input ............................................................................................................................... 18
16. Conditional tests for [...] and [[...]] commands ............................................................................... 19
16.1. File tests.................................................................................................................................... 19
16.2. Character string tests ............................................................................................................... 19
17. More conditional tests for [...] and [[...]] commands...................................................................... 20
17.1. Arithmetic tests ........................................................................................................................ 20
17.2. Additional tests for [[...]] (ksh and bash).................................................................................. 20
17.3. Negating and Combining tests ................................................................................................. 20
18. Flow Control and Compound Commands ....................................................................................... 21
18.1. Conditional execution: if/else .................................................................................................. 21
18.2. Looping: 'while' and 'for' loops................................................................................................. 21
19. Flow Control and Compound Commands (contd.).......................................................................... 23
19.1. Case statement: pattern matching........................................................................................... 23
19.2. Miscellaneous flow control and subshells................................................................................ 23
20. Conditional Test Examples .............................................................................................................. 24
21. Miscellaneous internal commands.................................................................................................. 24
22. Manipulating Variables (ksh/bash only).......................................................................................... 25
22.1. Text variables............................................................................................................................ 25
22.2. Numeric variables..................................................................................................................... 26
23. Shell Functions................................................................................................................................. 26
23.1. Defining functions .................................................................................................................... 26
23.2. Calling functions. ...................................................................................................................... 27
23.3. Reuseable functions ................................................................................................................. 28
24. Advanced I/O................................................................................................................................... 28
24.1. Redirecting for the whole script............................................................................................... 28
24.2. Explicitly opening or duplicating file descriptors ..................................................................... 28
24.3. Input and output to open file descriptors (ksh) ....................................................................... 29
24.4. Closing file handles................................................................................................................... 29
24.5. "Here" documents.................................................................................................................... 29
25. Wizard Level I/O .............................................................................................................................. 30

3
 25.1. Pass stderr of a command into a pipeline for further processing............................................ 31
25.2. Capture the exit status of a command in the middle of a pipeline.......................................... 31
26. Coprocesses and Background jobs .................................................................................................. 33
26.1. ksh coprocesses........................................................................................................................ 33
27. Delivering and Trapping Signals ...................................................................................................... 35
28. Security issues in shell scripts.......................................................................................................... 36
29. Style ................................................................................................................................................. 38
29.1. When not to use shell scripts ................................................................................................... 38
30. Some longer examples .................................................................................................................... 38
31. A Toolkit of commonly used external commands........................................................................... 39
31.1. Listing, copying and moving files and directories .................................................................... 39
31.2. Displaying text, files or parts of files ........................................................................................ 40
31.3. Compression and archiving ...................................................................................................... 40
31.4. Sorting and searching for patterns........................................................................................... 40
31.5. System information (users, processes, time) ........................................................................... 41
31.6. Conditional tests....................................................................................................................... 41
31.7. Stream Editing .......................................................................................................................... 41
31.8. Finding and comparing files ..................................................................................................... 42
31.9. Arithmetic and String Manipulation......................................................................................... 42
31.10. Merging files........................................................................................................................... 42
32. References, Resources, Man pages etc. .......................................................................................... 43
32.1. Books ........................................................................................................................................ 43
32.2. Online Resources...................................................................................................................... 43
32.3. Unix-like shells and utilities for Microsoft Windows................................................................ 44

4
 (1)

1. What is a Shell Script

 A text file containing commands which could have been typed directly into the shell.

There is no difference in syntax between interactive command line use and placing the commands in a
file. Some commands are only useful when used interactively (e.g. command line history recall) and
other commands are too complex to use interactively.

 The shell itself has limited capabilities -- the power comes from using it as a "glue"
language to combine the standard Unix utilities, and custom software, to produce a
tool more useful than the component parts alone.
 Any shell can be used for writing a shell script. To allow for this, the first line of every
script is:
#!/path/to/shell (e.g. #!/bin/ksh).

The #! characters tell the system to locate the following pathname, start it up and feed it the rest of the
file as input. Any program which can read commands from a file can be started up this way, as long as it
recognizes the # comment convention. The program is started, and then the script file is given to it as an
argument. Because of this, the script must be readable as well as executable. Examples are perl, awk, tcl
and python.

 Any file can be used as input to a shell by using the syntax:

ksh myscript
 If the file is made executable using chmod, it becomes a new command and available
for use (subject to the usual $PATH search).
chmod +x myscript

A shell script can be as simple as a sequence of commands that you type regularly. By putting
them into a script, you reduce them to a single command.

Example: ex0 display, text

1: #!/bin/sh
2: date
3: pwd
4: du -k

(2)

2. Why use Shell Scripts

 Combine lengthy and repetitive sequences of commands into a single, simple
command.
 Generalize a sequence of operations on one set of data, into a procedure that can be
applied to any similar set of data.

(e.g. apply the same analysis to every data file on a CD, without needing to repeat the commands)

5
  Create new commands using combinations of utilities in ways the original authors
never thought of.
 Simple shell scripts might be written as shell aliases, but the script can be made
available to all users and all processes. Shell aliases apply only to the current shell.
 Wrap programs over which you have no control inside an environment that you can
control.

e.g. set environment variables, switch to a special directory, create or select a configuration file, redirect
output, log usage, and then run the program.

 Create customized datasets on the fly, and call applications (e.g. matlab, sas, idl,
gnuplot) to work on them, or create customized application commands/procedures.
 Rapid prototyping (but avoid letting prototypes become production)

2.1. Typical uses

 System boot scripts (/etc/init.d)
 System administrators, for automating many aspects of computer maintenance, user
account creation etc.
 Application package installation tools

Other tools may create fancier installers (e.g. tcl/tk), but can not be assumed to be installed already.
Shell scripts are used because they are very portable. Some software comes with a complete installation
of the tool it wants to use (tcl/tk/python) in order to be self contained, but this leads to software bloat.

 Application startup scripts, especially unattended applications (e.g. started from cron
or at)
 Any user needing to automate the process of setting up and running commercial
applications, or their own code.

AUTOMATE, AUTOMATE, AUTOMATE

(3)

3. History of Shells
sh
aka "Bourne" shell, written by Steve Bourne at AT&T Bell Labs for Unix V7 (1979).
Small, simple, and (originally) very few internal commands, so it called external
programs for even the simplest of tasks. It is always available on everything that looks
vaguely like Unix.
csh
The "C" shell. (Bill Joy, at Berkeley). Many things in common with the Bourne shell,
but many enhancements to improve interactive use. The internal commands used only
in scripts are very different from "sh", and similar (by design) to the "C" language
syntax.
tcsh
The "TC" shell. Freely available and based on "csh". It has many additional features to
make interactive use more convenient.

6
 We use it as the default interactive shell for new accounts on all of our public systems.
Not many people write scripts in [t]csh. See Csh Programming Considered Harmful by Tom
Christiansen for a discussion of problems with programming csh scripts.
ksh
The "Korn" shell, written by David Korn of AT&T Bell Labs (now AT&T Research).
Written as a major upgrade to "sh" and backwards compatible with it, but has many
internal commands for the most frequently used functions. It also incorporates many
of the features from tcsh which enhance interactive use (command line history recall
etc.).
It was slow to gain acceptance because earlier versions were encumbered by AT&T licensing. This
shell is now freely available on all systems, but sometimes not installed by default on "free" Unix.
There are two major versions. ksh88 was the version incorporated into AT&T SVR4 Unix, and may
still be installed by some of the commercial Unix vendors. ksh93 added more features, primarily for
programming, and better POSIX compliance.
POSIX 1003.2 Shell Standard.
Standards committees worked over the Bourne shell and added many features of the
Korn shell (ksh88) and C shell to define a standard set of features which all compliant
shells must have.
On most systems, /bin/sh is now a POSIX compliant shell. Korn shell and Bash are POSIX compliant,
but have many features which go beyond the standard. On Solaris, the POSIX/XPG4 commands which
differ slightly in behaviour from traditional SunOS commands are located in /usr/xpg4/bin
bash
The "Bourne again" shell. Written as part of the GNU/Linux Open Source effort, and
the default shell for Linux and Mac OS-X. It is a functional clone of sh, with
additional features to enhance interactive use, add POSIX compliance, and partial ksh
compatability.
zsh
A freeware functional clone of sh, with parts of ksh, bash and full POSIX compliance,
and many new interactive command-line editing features. It was installed as the
default shell on early MacOSX systems.

(4)

4. Comparison of shell features

All the shells just listed share some common features, and the major differences in syntax
generally only affect script writers. It is not unusual to use one shell (e.g. tcsh) for interactive
use, but another (sh or ksh) for writing scripts.

4.1. Core Similarities (and recap of basic command line usage)

Each of these items is discussed in more detail later.

 Parse lines by whitespace, search for external commands using $PATH.

 Can run a shell script using shellname scriptfile, or run a single command using
shellname -c "command"
 Pass expanded command line arguments to programs; get exit status back.
 Pass environment variables to programs.
 Expand filename wildcards using []*?. Each shell has some additional wildcard
metacharacters, but these are common to all shells.
 Standard I/O redirection and piping with <,>,>>,|
 A few internal functions (cd)

7
  Backgrounding commands with &
 Quoting rules: "double quotes" protect most things, but allow $var interpretation;
'single quotes' protect all metacharacters from interpretation.
 Home directory expansion using ~user (except for sh)
 # comments
 Command substitution using `command` (backtics)
 Expand variables using $varname syntax
 Conditional execution using && and ||
 Line continuation with "\"

4.2. Principal Differences

between sh (+derivitives), and csh (+derivitives).

 Syntax of all the flow control constructs and conditional tests.

 Syntax for string manipulation inside of scripts
 Syntax for arithmetic manipulation inside of scripts
 Syntax for setting local variables (used only in the script) and environment variables
(which are passed to child processes). setenv vs export
 Syntax for redirecting I/O streams other than stdin/stdout
 Login startup files (.cshrc and .login, vs .profile) and default options
 Reading other shell scripts into the current shell (source filename, vs . filename)
 Handling of signals (interrupts)

(5)

5. Other Scripting Languages

There are many other programs which read a file of commands and carry out a sequence of
actions. The "#!/path/to/program" convention allows any of them to be used as a scripting
language to create new commands. Some are highly specialized, and some are much more
efficient than the equivalent shell scripts at certain tasks. There is never only one way to
perform a function, and often the choice comes down to factors like:

 what is installed already - many other scripting languages are not available by default
 what similar code already exists
 what you are most familiar with and can use most efficiently. Your time is always
more expensive than computer cycles.

Some major players (all of these are freely available) in the general purpose scripting
languages are:

 awk

A pattern matching and data (text and numeric) manipulation tool. Predates perl. Installed on all Unix
systems. Often used in combination with shell scripts.

 perl

8
 The most used scripting language for Web CGI applications and system administration tasks. Perl is
harder to learn, and is uusually installed by default now. It is more efficient and has an enormous library
of functions available. You could use Perl for almost all scripting tasks, but the syntax is very different
to the shell command line

 python

An object-oriented scripting language. Commonly installed by default on modern systems.

 tcl/tk

Tool Command Language. Another general purpose scripting language. The "tk" component is a
scripted interface to standard X-windows graphical components, so the combination is often used to
create graphical user interfaces.

Ksh93 can be extended by linking to shared libraries providing additional internal commands. One
example of an extended shell is tksh which incorporates Tcl/Tk with ksh and allows generation of
scripts using both languages. It can be used for prototyping GUI applications.

(6)

6. ksh/bash vs sh
Ksh and bash are both supersets of sh. For maximum portability, even to very old computers,
you should stick to the commands found in sh. Where possible, ksh or bash-specific features
will be noted in the following pages. In general, the newer shells run a little faster and scripts
are often more readable because logic can be expressed more cleanly user the newer syntax.
Many commands and conditional tests are now internal.
The philosophy of separate Unix tools each performing a single operation was followed closely by the designers
of the original shell, so it had very few internal commands and used external tools for very trivial operations
(like echo and [). Ksh and bash internally performs many of the basic string and numeric manipulations and
conditional tests. Occasional problems arise because the internal versions of some commands like echo are not
fully compatible with the external utility they replaced.

The action taken every time a shell needs to run an external program is to locate the program (via $PATH),
fork(), which creates a second copy of the shell, adjust the standard input/output for the external program, and
exec(), which replaces the second shell with the external program. This process is computationally expensive
(relatively), so when the script does something trivial many times over in a loop, it saves a lot of time if the
function is handled internally.

If you follow textbooks on Bourne shell programming, all of the advice should apply no
matter which of the Bourne-derived shells you use. Unfortunately, many vendors have added
features over the years and achieving complete portability can be a challenge. Explicitly
writing for ksh (or bash) and insisting on that shell being installed, can often be simpler.

The sh and ksh man pages use the term special command for the internal commands -
handled by the shell itself.

(7)

9
7. Basic sh script syntax
The most basic shell script is a list of commands exactly as could be typed interactively,
prefaced by the #! magic header. All the parsing rules, filename wildcards, $PATH searches
etc., which were summarized above, apply.
In addition:
# as the first non-whitespace character on a line
flags the line as a comment, and the rest of the line is completely ignored. Use
comments liberally in your scripts, as in all other forms of programming.
\ as the last character on a line
causes the following line to be logically joined before interpretation. This allows
single very long commands to be entered in the script in a more readable fashion. You
can continue the line as many times as needed.
This is actually just a particular instance of \ being to escape, or remove the special meaning from, the
following character.
;as a separator between words on a line
is interpreted as a newline. It allows you to put multiple commands on a single line.
There are few occasions when you must do this, but often it is used to improve the
layout of compound commands.
Example: ex1 display, text
1: #!/bin/ksh
2: # For the purposes of display, parts of the script have
3: # been rendered in glorious technicolor.
4: ## Some comments are bold to flag special sections
5:
6: # Line numbers on the left are not part of the script.
7: # They are just added to the HTML for reference.
8:
9: # Built-in commands and keywords (e.g. print) are in blue
10: # Command substitutions are purple. Variables are black
11: print "Disk usage summary for $USER on `date`"
12:
13: # Everything else is red - mostly that is external
14: # commands, and the arguments to all of the commands.
15: print These are my files # end of line comment for print
16: # List the files in columns
17: ls -C
18: # Summarize the disk usage
19: print
20: print Disk space usage
21: du -k
22: exit 0

7.1. Exit status

Every command (program) has a value or exit status which it returns to the calling program.
This is separate from any output generated. The exit status of a shell script can be explicitly
set using exit N, or it defaults to the value of the last command run.
The exit status is an integer 0-255. Conventionally 0=success and any other value indicates a problem. Think of
it as only one way for everything to work, but many possible ways to fail. If the command was terminated by a
signal, the value is 128 plus the signal value.

(8)

10
8. Filename Wildcards
The following characters are interpreted by the shell as filename wildcards, and any word
containing them is replaced by a sorted list of all the matching files.
Wildcards may be used in the directory parts of a pathname as well as the filename part. If no files match the
wildcard, it is left unchanged. Wildcards are not full regular expressions. Sed, grep, awk etc. work with more
flexible (and more complex) string matching operators.
*
Match zero or more characters.
?
Match any single character
[...]
Match any single character from the bracketed set. A range of characters can be
specified with [ - ]
[!...]
Match any single character NOT in the bracketed set.

 An initial "." in a filename does not match a wildcard unless explicitly given in the
pattern. In this sense filenames starting with "." are hidden. A "." elsewhere in the
filename is not special.
 Pattern operators can be combined

Example:
chapter[1-5].* could match chapter1.tex, chapter4.tex, chapter5.tex.old. It would
not match chapter10.tex or chapter1

(9)

9. Shell Variables
Scripts are not very useful if all the commands and options and filenames are explicitly coded.
By using variables, you can make a script generic and apply it to different situations. Variable
names consist of letters, numbers and underscores ([a-zA-Z0-9_], cannot start with a number,
and are case sensitive. Several special variables (always uppercase names) are used by the
system -- resetting these may cause unexpected behaviour. Some special variables may be
read-only. Using lowercase names for your own variables is safest.

9.1. Setting and exporting variables

srcfile=dataset1
Creates (if it didn't exist) a variable named "srcfile" and sets it to the value "dataset1".
If the variable already existed, it is overwritten. Variables are treated as text strings,
unless the context implies a numeric interpretation. You can make a variable always
be treated as a number. Note there must be no spaces around the "=".
set
Display all the variables currently set in the shell
unset srcfile
Remove the variable "srcfile"
srcfile=
Give the variable a null value, (not the same as removing it).
export srcfile

11
 Added srcfile to the list of variables which will be made available to external program
through the environment. If you don't do this, the variable is local to this shell instance.
export
List all the variables currently being exported - this is the environment which will be
passed to external programs.

9.2. Using variables

$srcfile
Prefacing the variable name with $ causes the value of the variable to be substituted in
place of the name.
${srcfile}
If the variable is not surrounded by whitespace (or other characters that can't be in a
name), the name must be surrounded by "{}" braces so that the shell knows what
characters you intend to be part of the name.

Example:

datafile=census2000
# Tries to find $datafile_part1, which doesn't exist
echo $datafile_part1.sas
# This is what we intended
echo ${datafile}_part1.sas

9.3. Conditional modifiers

There are various ways to conditionally use a variable in a command.
${datafile-default}
Substitute the value of $datafile, if it has been defined, otherwise use the string
"default". This is an easy way to allow for optional variables, and have sensible
defaults if they haven't been set. If datafile was undefined, it remains so.
${datafile=default}
Similar to the above, except if datafile has not been defined, set it to the string
"default".
${datafile+default}
If variable datafile has been defined, use the string "default", otherwise use null. In
this case the actual value $datafile is not used.
${datafile?"error message"}
Substitute the value of $datafile, if it has been defined, otherwise display datafile:
error message. This is used for diagnostics when a variable should have been set and
there is no sensible default value to use.
Placing a colon (:) before the operator character in these constructs has the effect of counting a null
value the same as an undefined variable. Variables may be given a null value by setting them to an
empty string, e.g. datafile= .
Example: echo ${datafile:-mydata.dat}
Echo the value of variable datafile if it has been set and is non-null, otherwise echo "mydata.dat".

9.4. Variable assignment command prefix

It is possible to export a variable just for the duration of a single command using the syntax:
var=value command args

12
 (10)

10. Preset Shell Variables

Several special variables are used by the system -- you can use these, but may not be able to
change them. The special variables use uppercase names, or punctuation characters. Some
variables are set by the login process and inherited by the shell (e.g. $USER), while others are
used only by the shell.
Try running set or env
These are some of the more commonly used ones:

10.1. Login environment

$USER, $LOGNAME
Preset to the currently logged-in username.
$PATH
The list of directories that will be searched for external commands. You can change
this in a script to make sure you get the programs you intend, and don't accidentally
get other versions which might have been installed.
$TERM
The terminal type in which the shell session is currently executing. Usually "xterm" or
"vt100". Many programs need to know this to figure out what special character
sequences to send to achieve special effects.
$PAGER
If set, this contains the name of the program which the user prefers to use for text file
viewing. Usually set to "more" or "less" or something similar. Many programs which
need to present multipage information to the user will respect this setting (e.g. man).
This isn't actually used by the shell itself, but shell scripts should honour it if they
need to page output to the user.
$EDITOR
If set, this contains the name of the program which the user prefers to use for text file
editing. A program which needs to have the user manually edit a file might choose to
start up this program instead of some built-in default (e.g. "crontab -e". This also
determines the default command-line-editing behaviour in interactive shells.

10.2. Shell internal settings

$PWD
Always set the current working directory (readonly)
$OLDPWD
The previous directory (before the most recent cd command). However, changing
directories in a script is often dangerous.
$? (readonly)
Set to the exit status of the last command run, so you can test success or failure. Every
command resets this so it must be saved immediately if you want to use it later.
$-
Set to the currently set options flags.
$IFS

13
 Internal Field Separators: the set of characters (normally space and tab) which are used
to parse a command line into separate arguments. This may be set by the user for
special purposes, but things get very confusing if it isn't changed back.

10.3. Process ID variables

$$ (readonly)
Set to the process ID of the current shell - useful in making unique temporary files, e.g.
/tmp/$0.$$
$PPID (readonly)
Set to the process ID of the parent process of this shell - useful for discovering how
the script was called.
$! (readonly)
Set to the process ID of the last command started in background - useful for checking
on background processes.

10.4. ksh/bash additional features

$SECONDS (readonly)
Integer number of seconds since this shell was started. Can be used for timing
commands.
$RANDOM
Every time it is valuated, $RANDOM returns a random integer in the range 0-32k.
RANDOM may be set to "seed" the random number generator.
$LINENO (readonly)
Always evaluates to the current line number of the script being executed - useful for
debugging.

(11)

11. Command Line (positional) arguments

To customize the behaviour of a script at run time, you can give it any number of arguments
on the command line.
These are often filenames, but can be interpreted by the script in any way. Options are often specified using the
"-flag" convention used by most Unix programs, and a ksh command getopts is available to help parse them.
The shell expands wildcards and makes variable and command substitutions as normal, then
parses the resulting words by whitespace (actually special variable $IFS), and places the
resulting text strings into the positional variables as follows:
$0, $1, $2, ... $9
The first 9 arguments are made available directly as $1-$9. To access more than 9, use
shift, or $*, $@. The variable $0 contains the name of the script itself.
${10}, ${11}, ...
Positional arguments greater than 9 are set by ksh and bash. Remember to use braces
to refer to them.
shift
discard $1 and renumber all the other variables. "shift N" will shift N arguments at
once.
$#
contains the number of arguments that were set (not including $0).

14
$*
contains all of the arguments in a single string, with one space separating them.
$@
similar to $*, but if used in quotes, it effectively quotes each argument and keeps them
separate. If any argument contains whitespace, the distinction is important.
e.g. if the argument list is: a1 a2 "a3 which contains spaces" a4
then: $1=a1, $2=a2, $3=a3 which contains spaces, $4=a4
and: $*=a1 a2 a3 which contains spaces a4
and: "$@"="a1" "a2" "a3 which contains spaces" "a4"

Only using the form "$@" preserves quoted arguments. If the arguments are being passed
from the script directly to some other program, it may make a big difference to the meaning.

Example: ex7 display, text

1: #!/bin/sh
2: #
3: # Check positional argument handling
4: echo "Number of arguments: $#"
5: echo "\$0 = $0"
6:
7: echo "Loop over \$*"
8: for a in $*; do
9: echo \"$a\"
10: done
11:
12: echo "Loop over \"\$@\""
13: for a in "$@"; do
14: echo \"$a\"
15: done

11.1. Setting new positional arguments

The set command, followed by a set of arguments, creates a new set of positional arguments.
This is often used, assuming the original arguments are no longer needed, to parse a set of
words (possibly using different field separators). Arguments may be reset any number of
times.

Example: ex2 display, text

1: #!/bin/sh
2: # Find an entry in the password file
3: pwent=`grep '^root:' /etc/passwd`
4: # Turn off globbing - passwd lines often contain '*'
5: set -o noglob
6: # The "full name" and other comments are in
7: # field 5, colon delimited. Get this field using shell word
splitting
8: OIFS=$IFS; IFS=: ; set $pwent; IFS=$OIFS
9: echo $5

Example: pickrandom display, text

Selects a random file from a directory. Uses the ksh RANDOM feature.

15
 1: #!/bin/ksh
2:
3: # Select a random image from the background logo collection
4: # This could be used to configure a screen saver, for example.
5: #
6: # This works even if the filenames contain spaces.
7:
8: # switch to the logos directory to avoid long paths
9: logos=/afs/northstar/common/usr/lib/X11/logos/backgrounds
10: cd $logos
11:
12: # '*' is a filename wildcard to match all files in the current
directory
13: set *
14:
15: # Use the syntax for arithmetic expressions. "%" is the modulo
operator
16: # Shift arguments by a random number between 0 and the number of
files
17: shift $(($RANDOM % $#))
18:
19: # Output the resulting first argument
20: echo "$logos/$1"

(12)

12. Shell options

Startup options. ksh -options scriptname
-x
echo line to stderr before executing it
-n
read commands and check for syntax errors, but do not execute.
-a
all variables are automatically exported
-f
disable wildcard filename expansion (globbing)
set -x
Set an option within a shell script
$-
contains the currently set option letters
There are many other options, not often needed. Options in ksh and bash can also be set using
long names (e.g. -o noglob instead of -f). Many options are unique to ksh or bash.

(13)

13. Command Substitution

13.1. sh syntax
`command`
A command (plus optional arguments) enclosed in backticks is executed and the
standard output of that command is substituted. If the command produces multiline

16
 output, the newlines are retained. If the resultant string is displayed, unquoted, using
echo, newlines and multiple spaces will be removed.

13.2. ksh/bash syntax

$(command)
This syntax is functionally the same as backticks, but commands can be more easily
nested.
$(<file)
This is equivalent to `cat file`, but implemented internally for efficiency.

Example: ex3 display, text

1: #!/bin/ksh
2:
3: echo Today is `date`
4:
5: file=/etc/hosts
6: echo The file $file has $(wc -l < $file) lines
7:
8: hostname -s > myhostname
9: echo This system has host name $(<myhostname)

(14)

14. I/O redirection and pipelines

Any simple command (or shell function, or compound command) may have its input and
output redirected using the following operators. This is performed by the shell before the
command is run.

14.1. Output redirection

> filename
Standard ouput (file descriptor 1) is redirected to the named file. The file is
overwritten unless the noclobber option is set. The file is created if it does not exist.
The special device file /dev/null can be used to explicitly discard unwanted output. Reading from
/dev/null results in an End of File status.
>> filename
Standard ouput is appended to the named file. The file is created if it does not exist.
>| filename
Output redirect, and override the noclobber option, if set.

14.2. Input redirection

< filename
Standard input (file descriptor 0) is redirected to the named file. The file must already
exist.

17
14.3. Command pipelines
command | command [ | command ...]
Pipe multiple commands together. The standard output of the first command becomes
the standard input of the second command. All commands run simultaneously, and
data transfer happens via memory buffers. This is one of the most powerful constructs
in Unix. Compound commands may also be used with pipes. Pipes play very nicely
with multiprocessor systems.
No more than one command in a pipeline should be interactive (attempt to read from the terminal). This
construct is much more efficient than using temporary files, and most standard Unix utilities are
designed such that they work well in pipelines.

The exit status of a pipeline is the exit status of the last command. In compound commands, a pipeline
can be used anywhere a simple command could be used.

(15)

15. Input and Output

Shell scripts can generate output directly or read input into variables using the following
commands:

15.1. Script output

echo
Print arguments, separated by spaces, and terminated by a newline, to stdout. Use
quotes to preserve spacing. Echo also understands C-like escape conventions.
Beware that the shell may process backslashes before echo sees them (may need to double backslash).
Internal in most shells, but was originally external.
\b backspace \c print line without new-line (some versions)
\f form-feed \n new-line
\r carriage return \t tab
\v vertical tab \\ backslash
\0n where n is the 8-bit character whose ASCII code is the 1-, 2- or 3-digit octal number representing
that character.
-n
suppress newline
print (ksh internal)
Print arguments, separated by spaces, and terminated by a newline, to stdout. Print
observes the same escape conventions as echo.
-n
suppress newline
-r
raw mode - ignore \-escape conventions
-R
raw mode - ignore \-escape conventions and -options except -n.

15.2. Script input

read var1 var2 rest

18
 read a line from stdin, parsing by $IFS, and placing the words into the named variables.
Any left over words all go into the last variable. A '\' as the last character on a line
removes significance of the newline, and input continues with the following line.
-r
raw mode - ignore \-escape conventions

Example: ex4a display, text

1: #!/bin/sh
2: echo "Testing interactive user input: enter some keystrokes and press
return"
3: read x more
4: echo "First word was \"$x\""
5: echo "Rest of the line (if any) was \"$more\""

(16)

16. Conditional tests for [...] and [[...]] commands

Most of the useful flow-control operators involve making some conditional test and branching
on the result (true/false). The test can be either the test command, or its alias, [, or the
ksh/bash built-in [[ ... ]] command, which has slightly different options, or it can be any
command which returns a suitable exit status. Zero is taken to be "True", while any non-zero
value is "False". Note that this is backwards from the C language convention.

16.1. File tests

-e file
True if file exists (can be of any type).
-f file
True if file exists and is an ordinary file.
-d file
True if file exists and is a directory.
-r file
True if file exists and is readable
Similarly, -w = writable, -x = executable, -L = is a symlink.
-s file
True if file exists and has size greater than zero
-t filedescriptor
True if the open filedescriptor is associated with a terminal device. E.g. this is used to
determine if standard output has been redirected to a file.

16.2. Character string tests

-n "string"
true if string has non-zero length
-z "string"
true if string has zero length
With [, the argument must be quoted, because if it is a variable that has a null value, the resulting
expansion ( [ -z ] ) is a syntax error. An expansion resulting in "" counts as a null string.
For [ only, a quoted string alone is equivalent to the -n test, e.g. [ "$var" ]. In older shells for which [ is

19
 an external program, the only way to test for a null string is:
if [ "X$var" = "X" ]
This is rarely needed now, but is still often found.
$variable = text
True if $variable matches text.
$variable < text
True if $variable comes before (lexically) text
Similarly, > = comes after

(17)

17. More conditional tests for [...] and [[...]]

commands
17.1. Arithmetic tests
$variable -eq number
True if $variable, interpreted as a number, is equal to number.
$variable -ne number
True if $variable, interpreted as a number, is not equal to number.
Similarly, -lt = less than, -le = less than or equal, -gt = greater than, -ge = greater
than or equal

17.2. Additional tests for [[...]] (ksh and bash)

$variable = pattern
True if $variable matches pattern. If pattern contains no wildcards, then this is just an
exact text match. The same wildcards as used for filename matching are used.
The pattern must not be quoted. Since [[...]] is internal to the shell, the pattern in this case is treated
differently and not filename-expanded as an external command would require.
file1 -nt file2
True if file1 is newer than file2.
Similarly -ot = older than
file1 -ef file2
true if file1 is effectively the same as file2, after following symlinks and hard links.

17.3. Negating and Combining tests

Tests may be negated by prepending the ! operator, and combined with boolean AND and
OR operators using the syntax:
conditional -a conditional, conditional -o conditional
AND and OR syntax for test and [
conditional && conditional, conditional || conditional
AND and OR syntax for [[ ... ]]
Parentheses may be inserted to resolve ambiguities or override the default operator
precedence rules.

Examples:

if [[ -x /usr/local/bin/lserve && \
-w /var/logs/lserve.log ]]; then
/usr/local/bin/lserve >> /var/logs/lserve.log &

20
fi

pwent=`grep '^richard:' /etc/passwd`

if [ -z "$pwent" ]; then
echo richard not found
fi

(18)

18. Flow Control and Compound Commands

A list in these descriptions is a simple command, or a pipeline. The value of the list is the
value of the last simple command run in it.
A list can also be a set of simple commands or pipelines separated by ";,&,&&,||,|&". For the compound
commands which branch on the success or failure of some list, it is usually [ or [[, but can be anything.

18.1. Conditional execution: if/else

list && list
Execute the first list. If true (success), execute the second one.
list || list
Execute the first list. If false (failure), execute the second one.

Example:

mkdir tempdir && cp workfile tempdir

sshd || echo "sshd failed to start"

You can use both forms together (with care) - they are processed left to right, and && must
come first.
Example:
mkdir tempdir && cp workfile tempdir || \
echo "Failed to create tempdir"
if list; then list ; elif list; then list; else list; fi
Execute the first list, and if true (success), execute the "then" list, otherwise execute
the "else" list. The "elif" and "else" lists are optional.

Example:

if [ -r $myfile ]
then
cat $myfile
else
echo $myfile not readable
fi

18.2. Looping: 'while' and 'for' loops

while list; do list; done
until list; do list; done
Execute the first list and if true (success), execute the second list. Repeat as long as the
first list is true. The until form just negates the test.

21
 Example: ex4 display, text

1: #!/bin/ksh
2: count=0
3: max=10
4: while [[ $count -lt $max ]]
5: do
6: echo $count
7: count=$((count + 1))
8: done
9: echo "Value of count after loop is: $count"
for identifier [ in words ]; do; list; done
Set identifier in turn to each word in words and execute the list. Omitting the "in
words" clause implies using $@, i.e. the identifier is set in turn to each positional
argument.

Example:

for file in *.dat

do
echo Processing $file
done
As with most programming languages, there are often several ways to express the same action.
Running a command and then explicitly examining $? can be used instead of some of the
above.

Compound commands can be thought of as running in an implicit subshell. They can have I/O
redirection independant of the rest of the script. Setting of variables in a real subshell does not
leave them set in the parent script. Setting variables in implicit subshells varies in behaviour
among shells. Older sh could not set variables in an implicit subshell and then use them later,
but current ksh can do this (mostly).

Example: ex11 display, text

Reading a file line by line. The book by Randal Michael contains 12 example ways to read a
file line by line, which vary tremendously in efficiency. This example shows the simplest and
fastest way.

1: #!/bin/sh
2:
3: # Demonstrate reading a file line-by-line, using I/O
4: # redirection in a compound command
5: # Also test variable setting inside an implicit subshell.
6: # Test this under sh and ksh and compare the output.
7:
8: line="TEST"
9: save=
10:
11: if [ -z "$1" ]; then
12: echo "Usage: $0 filename"
13: else
14: if [ -r $1 ]; then
15: while read line; do
16: echo "$line"
17: save=$line

22
 18: done < $1
19: fi
20: fi
21: echo "End value of \$line is $line"
22: echo "End value of \$save is $save"

(19)

19. Flow Control and Compound Commands

(contd.)
19.1. Case statement: pattern matching
case word in pattern) list;; esac
Compare word with each pattern) in turn, and executes the first list for which the word
matches. The patterns follow the same rules as for filename wildcards.
(ksh and bash only) A pattern-list is a list of one or more patterns separated from each other with a |.
Composite patterns can be formed with one or more of the following:
?(pattern-list)
Optionally matches any one of the given patterns.
*(pattern-list)
Matches zero or more occurrences of the given patterns.
+(pattern-list)
Matches one or more occurrences of the given patterns.
@(pattern-list)
Matches exactly one of the given patterns.
!(pattern-list)
Matches anything, except one of the given patterns.

Example:

case $filename in
*.dat)
echo Processing a .dat file
;;
*.sas)
echo Processing a .sas file
;;
*)
# catch anything else that doesn't match patterns
echo "Don't know how to deal with $filename"
;;
esac

19.2. Miscellaneous flow control and subshells

break [n]
Break out of the current (or n'th) enclosing loop. Control jumps to the next statement
after the loop
continue [n];
Resume iteration of the current (or n'th) enclosing loop. Control jumps to the top of
the loop, which generally causes re-evaluation of a while or processing the next
element of a for.
. filename

23
 Read the contents of the named file into the current shell and execute as if in line.
Uses $PATH to locate the file, and can be passed positional parameters. This is often
used to read in shell functions that are common to multiple scripts. There are security
implications if the pathname is not fully specified.
( ... ) Command grouping
Commands grouped in "( )" are executed in a subshell, with a separate environment
(can not affect the variables in the rest of the script).

(20)

20. Conditional Test Examples

As with most aspects of shell scripting, there are usually several possible ways to
accomplish a task. Certain idioms show up commonly. These are five ways to examine
and branch on the initial character of a string.
Use case with a pattern:
case $var in
/*) echo "starts with /" ;;
Works in all shells, and uses no extra processes
Use `cut`:
if [ "`echo $var | cut -c1`" = "/" ] ; then .
Works in all shells, but inefficiently uses a pipe and external process for a trivial task.
Use POSIX variable truncation:
if [ "${var%${var#?}}" = "/" ]; then
Works with ksh, bash and other POSIX-compliant shells. Not obvious if you have not
seen this one before. Fails on old Bourne shells. Dave Taylor in "Wicked Cool Shell
Scripts" likes this one.
Use POSIX pattern match inside of [[...]]:
if [[ $var = /* ]]; then
Works with ksh, bash and other POSIX-compliant shells. Note that you must use [[...]]
and no quotes around the pattern.
The [[...]] syntax is handled internally by the shell and can therefore interpret "wildcard" patterns
differently than an external command. An unquoted wildcard is interpreted as a pattern to be matched,
while a quoted wildcard is taken literally. The [...] syntax, even if handled internally, is treated as
though it were external for backward compatability. This requires that wildcard patterns be expanded to
matching filenames.
Use ksh (93 and later) and bash variable substrings:
if [ "${var:0:1}" = "/" ]; then
ksh93 and later versions, and bash, have a syntax for directly extracting substrings by
character position. ${varname:start:length}

Example: ex17 display, text

(21)

21. Miscellaneous internal commands

The shells (ksh in particular) have many more internal commands. Some are used
more in interactive shells. The commands listed here are used in scripts, but don't
conveniently fit elsewhere in the class.
eval args

24
 The args are read as input to the shell and the resulting command executed. Allows
"double" expansion of some constructs. For example, constructing a variable name out
of pieces, and then obtaining the value of that variable.
netdev=NETDEV_
NETDEV_1=hme0 # As part of an initialization step defining
multiple devices

devnum=1 # As part of a loop over those devices

ifname=$netdev$devnum # construct a variable name NETDEV_1
eval device=\$$ifname # evaluate it - device is set to hme0
exec command args
The command is executed in place of the current shell. There is no return from an exec.
I/O redirection may be used. This is also used to change the I/O for the current shell.
:
The line is variable-expanded, but otherwise treated as a comment. Sometimes used as
a synonym for "true" in a loop.
while :; do
# this loop will go forever until broken by
# a conditional test inside, or a signal
done
unset var ...
Remove the named variables. This is not the same as setting their values to null.
typeset [+/- options] [ name[=value] ] ... (ksh only, bash uses declare for
similar functions)
Set attributes and values for shell variables and functions. When used inside a function,
a local variable is created. Some of the options are:
-L[n]
Left justify and remove leading blanks. The variable always has length n if specified.
-R[n]
Right justify and fill with leading blanks. The variable always has length n if specified.
-l
The named variable is always treated as an integer. This makes arithmetic faster. The
reserved word integer is an alias for typeset -i.
-Z[n]
As for -R, but fill with zeroes if the value is a number
-i
Lower-case convert the named variables
-u
Upper-case convert the named variables
-r
Mark the variables as readonly
-x
Export the named variables to the enviroment
-ft
The variables are taken as function names. Turn on execution tracing.

(22)

22. Manipulating Variables (ksh/bash only)

22.1. Text variables
The pattern in the following uses the same wildcards as for filename matching.
${#var}
returns the length of $var in characters
${var%pattern}

25
 removes the shortest suffix of $var patching pattern
${var%%pattern}
removes the longest suffix of $var patching pattern
${var#pattern}
removes the shortest prefix of $var patching pattern
${var##pattern}
removes the longest prefix of $var patching pattern

22.2. Numeric variables

$(( integer expression ))
The $(( ... )) construction interprets the contents as an arithmetic expression (integer
only). Variables are referenced by name without the "$". Most of the arithmetic syntax
of the 'C' language is supported, including bit manipulations (*,/,+,-,|,&,<<,>>. Use
parentheses for changing precedence).
Examples
datapath=/data/public/project/trials/set1/datafile.dat
filename=${datapath##*/}
filename is set to "datafile.dat" since the longest prefix pattern matching "*/" is the
leading directory path (compare basename)
path=${datapath%/*}
path is set to "/data/public/project/trials/set1" since the shortest suffix pattern
matching "/*" is the filename in the last directory (compare dirname)
i=$((i+1))
often used in while loops

(23)

23. Shell Functions

All but the earliest versions of sh allow you define shell functions, which are visible
only to the shell script and can be used like any other command. Shell functions take
precedence over external commands if the same name is used. Functions execute in
the same process as the caller, and must be defined before use (appear earlier in the
file). They allow a script to be broken into maintainable chunks, and encourage code
reuse between scripts.

23.1. Defining functions

identifier() { list; }
POSIX syntax for shell functions. Such functions do not restrict scope of variables or
signal traps. The identifier follows the rules for variable names, but uses a separate
namespace.
function identifier { list; }
Ksh and bash optional syntax for defining a function. These functions may define
local variables and local signal traps and so can more easily avoid side effects and be
reused by multiple scripts.

A function may read or modify any shell variable that exists in the calling script. Such
variables are global.

26
 (ksh and bash only) Functions may also declare local variables in the function using
typeset or declare. Local variables are visible to the current function and any
functions called by it.

return [n], exit [n]

Return from a function with the given value, or exit the whole script with the given
value.
Without a return, the function returns when it reaches the end, and the value is the
exit status of the last command it ran.

Example:

die()
{
# Print an error message and exit with given status
# call as: die status "message" ["message" ...]
exitstat=$1; shift
for i in "$@"; do
print -R "$i"
done
exit $exitstat
}

23.2. Calling functions.

Functions are called like any other command. The output may be redirected
independantly of the script, and arguments passed to the function. Shell option flags
like -x are unset in a function - you must explicitly set them in each function to trace
the execution. Shell functions may even be backgrounded and run asynchronously, or
run as coprocesses (ksh).

Example:

[ -w $filename ] || \
die 1 "$file not writeable" "check permissions"

Example: Backgrounded function call. ex12 display, text

1: #!/bin/sh
2:
3: background()
4: {
5: sleep 10
6: echo "Background"
7: sleep 10
8: # Function will return here - if backgrounded, the
subprocess will exit.
9: }
10:
11: echo "ps before background function"
12: ps
13: background &
14: echo "My PID=$$"
15: echo "Background function PID=$!"

27
 16: echo "ps after background function"
17: ps
18: exit 0

Example:

vprint()
{
# Print or not depending on global "$verbosity"
# Change the verbosity with a single variable.
# Arg. 1 is the level for this message.
level=$1; shift
if [[ $level -le $verbosity ]]; then
print -R $*
fi
}

verbosity=2
vprint 1 This message will appear
vprint 3 This only appears if verbosity is 3 or higher

23.3. Reuseable functions

By using only command line arguments, not global variables, and taking care to
minimise the side effects of functions, they can be made reusable by multiple scripts.
Typically they would be placed in a separate file and read with the "." operator.

Functions may generate output to stdout, stderr, or any other file or filehandle.
Messages to stdout may be captured by command substitution (`myfunction`, which
provides another way for a function to return information to the calling script. Beware
of side-effects (and reducing reusability) in functions which perform I/O.

(24)

24. Advanced I/O

Unix I/O is performed by assigning file descriptors to files or devices, and then using
those descriptors for reading and writing. Descriptors 0, 1, and 2 are always used for
stdin, stdout and stderr respectively. Stdin defaults to the keyboard, while stdout and
stderr both default to the current terminal window.

24.1. Redirecting for the whole script

Redirecting stdout, stderr and other file descriptors for the whole script can be done
with the exec command.
exec > outfile < infile
with no command, the exec just reassigns the I/O of the current shell.
exec n>outfile
The form n<, n> opens file descriptor n instead of the default stdin/stdout. This can
then be used with read -u or print -u.

24.2. Explicitly opening or duplicating file descriptors

28
 One reason to do this is to save the current state of stdin/stdout, temporarily reassign
them, then restore them.
>&n
standard output is moved to whatever file descriptor n is currently pointing to
<&n
standard input is moved to whatever file descriptor n is currently pointing to
n>file
file descriptor n is opened for writing on the named file.
n>&1
file descriptor n is set to whatever file descriptor 1 is currently pointing to.
Example Sending messages to stderr (2) instead of stdout (1)
echo "Error: program failed" >&2
Echo always writes to stdout, but stdout can be temporarily reassigned to duplicate
stderr (or other file descriptors). Conventionally Unix programs send error messages
to stderr to keep them separated from stdout.

24.3. Input and output to open file descriptors (ksh)

Printing to file descriptors (usually more efficient than open/append/close):
print -u n args
print to file descriptor n.
-p
write to the pipe to a coprocess (opened by |&)

Reading from file descriptors other than stdin:

read -u n var1 var2 rest

read a line from file descriptor n, parsing by $IFS, and placing the words into the
named variables. Any left over words all go into the last variable.
-p
read from the pipe to a coprocess (opened by |&)

24.4. Closing file handles

<&-
standard input is explicitly closed
>&-
standard output is explicitly closed
For example, to indicate to another program downstream in a pipeline that no more
data will be coming. All file descriptors are closed when a script exits.

I/O redirection operators are evaluated left-to-right. This makes a difference in a

statement like: ">filename 2>&1". (Many books with example scripts get this wrong)

24.5. "Here" documents

<< [-]string
redirect input to the temporary file formed by everything up the matching string at the
start of a line. Allows for placing file content inline in a script.
Example: ex5 display, text

29
 1: #!/bin/sh
2: echo "Example of unquoted here document, with variable and
command substitution"
3:
4: cat <<EOF
5: This text will be fed to the "cat" program as
6: standard input. It will also have variable
7: and command substitutions performed.
8: I am logged in as $USER and today is `date`
9: EOF
10: echo
11: echo "Example of quoted here document, with no variable or
command substitution"
12: # The terminating string must be at the start of a line.
13: cat <<"EndOfInput"
14: This text will be fed to the "cat" program as standard
15: input. Since the text string marking the end was quoted, it
does not get
16: variable and command subsitutions.
17: I am logged in as $USER and today is `date`
18: EndOfInput

Example: duplex display, text

1: #!/bin/sh
2: # Add in the magic postscript preface to perform
3: # duplex printer control for Xerox docuprint.
4:
5: # To have this script send the files directly to the
printer, use
6: # a subshell to collect the output of the two 'cat'
commands.
7:
8: ## (
9: cat << EOP
10: %!PS
11: %%BeginFeature: *Duplex DuplexTumble
12: <</Duplex true /Tumble false>> setpagedevice
13: %%EndFeature
14: EOP
15: cat "$@"
16: ## ) | lpr

(25)

25. Wizard Level I/O

More complicated manipulations of file descriptors can be arranged. Two such
examples are shown here:

This short test script can be used to generate suitable output.

ex13: display, text

echo "This goes to stdout"

echo "This goes to stdout and has foo in the line"
echo "This goes to stderr" >&2
exit 99

30
25.1. Pass stderr of a command into a pipeline for further
processing
Example: ex14 display, text

exec 3>&1
./ex13.sh 2>&1 1>&3 3>&- | sed 's/stderr/STDERR/' 1>&2

We duplicate stdout to another file descriptor (3), then run the first command with
stderr redirected to stdout and stdout redirected to the saved descriptor (3). The result
is piped into other commands as needed. The output of the pipeline is redirected back
to stderr, so that stdout and stderr of the script as a whole are what we expect.

1: #!/bin/sh
2: # Example 14
3: # Take stderr from a command and pass it into a pipe
4: # for further processing.
5:
6: # Uses ex13.sh to generate some output to stderr
7: # stdout of ex13 is processed normally
8:
9: # Save a copy of original stdout
10: exec 3>&1
11:
12: # stdout from ex13.sh is directed to the original stdout
(3)
13: # stderr is passed into the pipe for further processing.
14: # stdout from the pipe is redirected back to stderr
15: ./ex13.sh 2>&1 1>&3 3>&- | sed 's/stderr/STDERR/' 1>&2
16:
17: # 3 is closed before running the command, just in case it
cares
18: # about inheriting open file descriptors.

25.2. Capture the exit status of a command in the middle of a

pipeline
Example: ex15 display, text

exec 3>&1
ex13stat=`((./ex13.sh; echo $? >&4) | grep 'foo' 1>&3) 4>&1`

This script uses nested subshells captured in backtics. Again we first duplicate stdout
to another file descriptor (3). The inner subshell runs the first command, then writes
the exit status to fd 4. The outer subshell redirects 4 to stdout so that it is captured by
the backtics. Standard output from the first command (inner subshell) is passed into
the pipeline as normal, but the final output of the pipeline is redirected to 3 so that it
appears on the original stdout and is not captured by the backtics.

If any of the commands really care about inheriting open file descriptors that they
don't need then a more correct command line closes the descriptors before running the
commands.

31
 1: #!/bin/sh
2: # Example 15
3:
4: # Uses ex13.sh to generate some output and give us an
5: # exit status to capture.
6:
7: # Get the exit status of ex13 into $ex13stat.
8: # stdout of ex13 is processed normally
9:
10: # Save a copy of stdout
11: exec 3>&1
12: # Run a subshell, with 4 duplicated to 1 so we get it in
stdout.
13: # Capture the output in ``
14: # ex13stat=`( ... ) 4>&1`
15: # Inside the subshell, run another subshell to execute
ex13,
16: # and echo the status code to 4
17: # (./ex13.sh; echo $? >&4)
18: # stdout from the inner subshell is processed normally,
but the
19: # subsequent output must be directed to 3 so it goes to
the
20: # original stdout and not be captured by the ``
21: ex13stat=`((./ex13.sh; echo $? >&4) | grep 'foo' 1>&3)
4>&1`
22:
23: echo Last command status=$?
24: echo ex13stat=$ex13stat
25:
26: # If any of the commands really care about inheriting
open file
27: # descriptors that they don't need then a more correct
command line
28: # closes the descriptors before running the commands
29: exec 3>&1
30: ex13stat=`((./ex13.sh 3>&- 4>&- ; echo $? >&4) | \
31: grep 'foo' 1>&3 3>&- 4>&- ) 4>&1`
32: echo Last command status=$?
33: echo ex13stat=$ex13stat

Combine the above two techniques:

Example: ex16 display, text

exec 3>&1
ex13stat=`((./ex13.sh 2>&1 1>&3 3>&- 4>&- ; echo $? >&4) | \
sed s/err/ERR/ 1>&2 3>&- 4>&- ) 4>&1`

1: #!/bin/sh
2: # Example 16
3:
4: # Uses ex13.sh to generate some output and give us an
5: # exit status to capture.
6:
7: # Get the exit status of ex13 into ex13stat.
8: # stderr of ex13 is processed by the pipe, stdout
9: # is left alone.

32
 10:
11: # Save a copy of stdout
12: exec 3>&1
13:
14: # Run a subshell, with 4 copied to 1 so we get it in
stdout.
15: # Capture the output in backtics`
16: # ex13stat=`( ) 4>&1`
17:
18: # In the subshell, run another subshell to execute ex13,
and
19: # echo the status code to 4
20: # (./ex13.sh; echo $? >&4)
21:
22: # stdout from the inner subshell is directed to the
original stdout (3)
23: # stderr is passed into the pipe for further processing.
24: # stdout from the pipe is redirected back to stderr
25:
26: # Close the extra descriptors before running the commands
27: exec 3>&1
28: ex13stat=`((./ex13.sh 2>&1 1>&3 3>&- 4>&- ; echo $? >&4)
| \
29: sed s/err/ERR/ 1>&2 3>&- 4>&- ) 4>&1`
30:
31: echo Last command status=$?
32: echo ex13stat=$ex13stat
33:
A practical application of this would be running a utility such as dd where the exit
status is important to capture, but the error output is overly chatty and may need to be
filtered before delivering to other parts of a script.

(26)

26. Coprocesses and Background jobs

Scripts can start any number of background jobs (any external command), which run
in parallel with the parent script, and asynchronously. Processes which require no
further interaction or synchronization (fire and forget) are easy. Interaction with
background jobs is tricky. You can use signals, pipes, named pipes, or disk files for
communication.
command &
Start command as a background process. Control returns immediately to the shell.
bgpid=$!
The special variable $! contains the process ID of the last background job that was
started. You can save that and examine the process later (ps -p $bgpid) or send it a
signal (kill -HUP $bgpid).

26.1. ksh coprocesses

Coprocesses are a way of starting a separate process which runs asychronously, but
has stdin/stdout connected to the parent script via pipes.
command |&
Start a coprocess with a 2-way pipe to it
read -p var

33
Read from the pipe to the coprocess, instead of standard input
print -p args
Write to the pipe connected to the coprocess, instead of standard output
Multiple coprocesses can be handled by moving the special file descriptors connected
to the pipes onto standard input and output, and or to explicitly specified file
descriptors.
exec <&p
The input from the coprocess is moved to standard input
exec >&p
The output from the coprocess is moved to standard output

Example: ex9 display, text

A script wants to save a copy of all output in a file, but also wants a copy to the screen.
This is equivalent to always running the script as
script | tee outfile

1: #!/bin/ksh
2:
3: # If we have not redirected standard output, save a copy
of
4: # the output of this script into a file, but still send a
5: # copy to the screen.
6:
7: if [[ -t 1 ]] ; then
8: # Only do this if fd 1 (stdout) is still connected
9: # to a terminal
10:
11: # We want the standard output of the "tee" process
12: # to go explicitly to the screen (/dev/tty)
13: # and the second copy goes into a logfile named $0.out
14:
15: tee $0.out >/dev/tty |&
16:
17: # Our stdout all goes into this coprocess
18: exec 1>&p
19: fi
20:
21: # Now generate some output
22: print "User activity snapshot on $(hostname) at $(date)"
23: print
24: who

Example: ex10 display, text

Start a coprocess to look up usernames in some database. It is faster to run a single
process than to run a separate lookup for each user.

1: #!/bin/ksh
2: # This example uses a locally written tool for Dartmouth
Name Directory lookups
3:
4: # Start the dndlookup program as a coprocess
5: # Tell it to output only the canonical full name, and to
not print multiple matches
6: dndlookup -fname -u |&
7:
8: # move the input/output streams so we

34
 9: # can use other coprocesses too
10: exec 4>&p
11: exec 5<&p
12:
13: echo "Name file contents:"
14: cat namefile
15: echo
16:
17: # read the names from a file "namefile"
18: while read uname; do
19: print -u4 $uname
20: read -u5 dndname
21: case $dndname in
22: *many\ matches*)
23: # handle case where the name wasn't unique
24: print "Multiple matches to \"$uname\" in DND"
25: ;;
26: *no\ match*)
27: # handle case where the name wasn't found
28: print "No matches to \"$uname\" in DND"
29: ;;
30: *)
31: # we seem to have a hit - process the
32: # canonical named retrieved from dndlookup
33: print "Unique DND match: full name for \"$uname\" is
\"$dndname\""
34: ;;
35: esac
36: sleep 2
37: done < namefile
38:
39: # We've read all the names, but the coprocess
40: # is still running. Close the pipe to tell it
41: # we have finished.
42: exec 4>&-

(27)

27. Delivering and Trapping Signals

Unix signals (software interrupts) can be sent as asynchronous events to shell scripts,
just as they can to any other program. The default behaviour is to ignore some signals
and immediately exit on others. Scripts may detect signals and divert control to a
handler function or external program. This is often used to perform clean-up actions
before exiting, or restart certain procedures. Execution resumes where it left off, if the
signal handler returns. Signal traps must be set separately inside of shell functions.
Signals can be sent to a process with kill.
trap handler sig ...
handler is a command to be read (evaluated first) and executed on receipt of the
specified sigs. Signals can be specified by name or number (see kill(1)) e.g. HUP,
INT, QUIT, TERM. A Ctrl-C at the terminal generates a INT.

 A handler of - resets the signals to their default values

 A handler of '' (null) ignores the signals
 Special signal values are as follows:

35
 EXIT
the handler is called when the function exits, or when the whole script exits. The exit
signal has value 0.
ERR (ksh)
the handler is called when any command has a non-zero exit status
DEBUG (ksh)
the handler is called after each command.

Example: ex8 display, text

1: #!/bin/bash
2: # Try this under bash, ksh and sh
3:
4: trap huphandler HUP
5: trap '' QUIT
6: trap exithandler TERM INT
7:
8: huphandler()
9: {
10: echo 'Received SIGHUP'
11: echo "continuing"
12: }
13:
14: exithandler()
15: {
16: echo 'Received SIGTERM or SIGINT'
17: exit 1
18: }
19: ## Execution starts here - infinite loop until
interrupted
20: # Use ":" or "true" for infinite loop
21: # SECONDS is built-in to bash and ksh. It is number of
seconds since script started
22: : is like a comment, but it is evaluated for side effects
and evaluates to true
23: seconds=0
24: while : ; do
25: # while true; do
26: sleep 5
27: seconds=$((seconds + 5))
28: echo -n "$SECONDS $seconds - "
29: done
Exit handlers can be defined to clean up temporary files or reset the state of devices.
This can be useful if the script has multiple possible exit points.

(28)

28. Security issues in shell scripts

Shell scripts are often used by system administrators and are run as a priviledged user.

 Don't use set-UID scripts.

36
 Most systems don't even allow a script to be made set-UID. It is impossible (due to inherent
race conditions) to ensure that a set-uid script cannot be compromised. Use wrapper programs
like sudo instead.

 Always explicitly set $PATH at the start of a script, so that you know exactly
which external programs will be used.
 If possible, don't use temporary files. If they cannot be avoided, use $TMPDIR,
and create files safely (e.g. mktemp).

Often scripts will write to a fixed, or trivially generated temporary filename in /tmp. If the file
already exists and you don't have permission to overwrite it, the script will fail. If you do have
permission to overwrite it, you will delete the previous contents. Since /tmp is public write,
another user may create files in it, or possibly fill it completely.
Example:

1. A link is created by an unprivileged user in /tmp: /tmp/scratch -> /vmunix

2. A root user runs a script that blindly writes a scratch file to /tmp/scratch, and
overwrites the operating system.

Environment variable $TMPDIR is often used to indicate a preferred location for temporary
files (e.g., a per-user directory). Some systems may use $TMP or $TEMP. Safe scratch files can
be made by creating a new directory, owned and writeable only by you, then creating files in
there.
Example:

(umask 077 && mkdir /tmp/tempdir.$$) || exit 1

or (deluxe version)

tmp=${TMPDIR:-/tmp}
tmp=$tmp/tempdir.$RANDOM.$RANDOM.$RANDOM.$$
(umask 077 && mkdir $tmp) || {
echo "Could not create temporary directory" 1>&2
exit 1
}

Alternatively, many systems have mktemp to safely create a temporary file and return the
filename, which can be used by the script and then deleted.

 Check exit status of everything you do.

 Don't trust user input
o contents of files
o data piped from other programs
o file names. Output of filename generation with wildcards, or directly
from ls or find

Example:
Consider the effects of a file named "myfile;cd /;rm *" if processed, unquoted, by
your script.

One possible way to protect against weirdo characters in file names:

# A function to massage a list of filenames
# to protect weirdo characters
# e.g. find ... | protect_filenames | xargs command
#

37
 # We are backslash-protecting the characters \'" ?*;
protect_filenames()
{
sed -es/\\\\/\\\\\\\\/g \
-es/\\\'/\\\\\'/g \
-es/\\\"/\\\\\"/g \
-es/\\\;/\\\\\;/g \
-es/\\\?/\\\\\?/g \
-es/\\\*/\\\\\*/g \
-es/\\\ /\\\\\ /g
}
If using GNU find and xargs, there is a much cleaner option to null-terminate generated
pathnames.

(29)

29. Style
Shell scripts are very frequently written quickly for a single purpose, used once and
discarded. They are also as frequently kept and used many times, and migrate into
other uses, but often do not receive the same level of testing and debugging that other
software would be given in the same situation. It is possible to apply general principles
of good software engineering to shell scripts.

 Preface scripts with a statement of purpose, author, date and revision notes
 Use a revision control system for complex scripts with a long lifetime
 Assume your script will have a long lifetime unless you are certain it won't
 Document any non-standard external utilities which your script needs
 Document your scripts with inline comments - you'll need them in a few
months when you edit it.
 Treat standard input and output in the normal way, so that your script can be
used in combination with other programs (the Unix toolkit philosophy)
 Be consistent in the format of your output, so that other programs can rely on it
 Use options to control behaviour such as verbosity of output. Overly chatty
programs are very hard to combine with other utilities
 Use interactive features (prompting the user for information) very sparingly.
Doing so renders the script unuseable in pipeline combinations with other
programs, or in unattended operations.
 Test (a lot)

29.1. When not to use shell scripts

 If an existing tool already does what you need - use it.
 If the script is more than a few hundred lines, you are probably using the
wrong tool.
 If performance is horrible, and the script is used a lot, you might want to
consider another language.

(30)

30. Some longer examples

38
 The class accounts have directories with all of the examples from the books by Blinn,
Michael, Rosenblatt, and Taylor. These can also be downloaded (see the References
page). Some of these are linked below (but not included in the printed notes), with
additional comments.

Download a compressed tar file of all example scripts used in these notes.

 postprint display, text

A wrapper script for printing a mix of text and postscript files
 checkpath display, text
Check all the directories in the $PATH for possibly conflicting programs.
 run-with-timeout display, text
Run a command with a timeout. Kill the command if it hasn't finished when
the timeout expires.
 MailPkg display, text
Tar, compress, split and uuendcode a set of files for mailing. (Blinn)
 Ptree (original) display, text
 Ptree (ksh version) display, text
Runs "ps" to get a process listing and then reformats to show the process
family hierarchies. The original example is pure Bourne shell and inefficient.
The ksh version is a fairly simple translation to use ksh internal commands
where possible, and avoid writing scratch files, and runs very much faster.
(Blinn).

This entire tutorial was created from individual HTML pages using a content
management system written as ksh scripts (heavily using sed to edit the pages),
coordinated by make.

You can even write an entire web server as a shell script. This one is part of the LEAF
(Linux Embedded Appliance Firewall) project. This wouldn't be suitable for much
load, but handles occasional queries on static HTML and CGI scripts.
(www.nisi.ab.ca/lrp/Packages/weblet.htm)

 sh-httpd display, text

(31)

31. A Toolkit of commonly used external commands

The following commands are very frequently used in shell scripts. Many of them are
used in the examples in these notes. This is just a brief recap -- see the man pages for
details on usage. The most useful are flagged with *.

Most of these commands will operate on a one or more named files, or will operate on
a stream of data from standard input if no files are named.

31.1. Listing, copying and moving files and directories

ls *

39
 list contents of a directory, or list details of files and directories.
mkdir; rmdir *
Make and Remove directories.
rm; cp; mv *
Remove (delete), Copy and Move (rename) files and directories
touch *
Update the last modifed timestamp on a file, to make it appear to have just been
written.
If the file does not exist, a new zero-byte file is created, which is often useful to signify that an event
has occurred.
tee
Make a duplicate copy of a data stream - used in pipelines to send one copy to a log
file and a second copy on to another program. (Think plumbing).

31.2. Displaying text, files or parts of files

echo *
Echo the arguments to standard output -- used for messages from scripts. Some
versions of "sh", and all csh/ksh/bash shells internalized "echo".
Conflicts sometimes arise over the syntax for echoing a line with no trailing CR/LF. Some use "\c" and
some use option "-n". To avoid these problems, ksh also provides the "print" command for output.
cat *
Copy and concatenate files; display contents of a file
head, tail *
Display the beginning of a file, or the end of it.
cut
Extract selected fields from each line of a file. Often awk is easier to use, even though
it is a more complex program.
wc
Count lines, words and characters in the input.

31.3. Compression and archiving

compress; gzip, zip; tar *
Various utilities to compress/uncompress individual files, combine multiple files into a
single archive, or do both.

31.4. Sorting and searching for patterns

sort *
Sort data alphabetically or numerically.
grep *
Search a file for lines containing character patterns. The patterns can be simple fixed
text, or very complex regular expressions.
The name comes from "Global Regular Expression and Print" -- a function from the Unix editors which
was used frequently enough to warrant getting its own program.
uniq *
Remove duplicate lines, and generate a count of repeated lines.
wc *
Count lines, words and characters in a file.

40
31.5. System information (users, processes, time)
date *
Display the current date and time (flexible format). Useful for conditional execution
based on time, and for timestamping output.
ps *
List the to a running processes.
kill *
Send a signal (interrupt) to a running process.
id
Print the user name and UID and group of the current user (e.g. to distinguish
priviledged users before attempting to run programs which may fail with permission
errors)
who
Display who is logged on the system, and from where they logged in.
uname *
Display information about the system, OS version, hardware architecture etc.
mail *
Send mail, from a file or standard input, to named recipients. Since scripts are often
used to automate long-running background jobs, sending notification of completion by
mail is a common trick.
logger
Place a message in the central system logging facility. Scripts can submit messages
with all the facilities available to compiled programs.
hostname
Display the hostname of the current host - usful to keep track of where your programs
are running

31.6. Conditional tests

test; [ *
The conditional test, used extensively in scripts, is also an external program which
evaluates the expression given as an argument and returns true (0) or false (1) exit
status. The name "[" is a link to the "test" program, so a line like:
if [ -w logfile ]
actually runs a program "[", with arguments "-w logfile ]", and returns a true/false
value to the "if" command.
In ksh and most newer versions of sh, "[" is replaced with a compatible internal command, but the
argument parsing is performed as if it were an external command. Ksh also provides the internal "[["
operator, with simplified syntax.

31.7. Stream Editing

awk *
A pattern matching and data manipulation utility, which has its own scripting language.
It also duplicates much functionality from 'sed','grep','cut','wc', etc.
Complex scripts can be written entirely using awk, but it is frequently used just to extract fields from
lines of a file (similar to 'cut').
sed *
Stream Editor. A flexible editor which operates by applying editing rules to every line
in a data stream in turn.

41
 Since it makes a single pass through the file, keeping only a few lines in memory at once, it can be used
with infinitely large data sets. It is mostly used for global search and replace operations. It is a superset
of 'tr', 'grep', and 'cut', but is more complicated to use.
tr
Transliterate - perform very simple single-character edits on a file.

31.8. Finding and comparing files

find *
Search the filesystem and find files matching certain criteria (name pattern, age, owner,
size, last modified etc.)
xargs *
Apply multiple filename arguments to a named command and run it.
Xargs is often used in combination with "find" to apply some command to all the files matching certain
criteria. Since "find" may result in a very large list of pathnames, using the results directly may
overflow command line buffers. Xargs avoids this problem, and is much more efficient than running a
command on every pathname individually.
diff *
Compare two files and list the differences between them.
basename pathname
Returns the base filename portion of the named pathname, stripping off all the
directories
dirname pathname
Returns the directory portion of the named pathname, stripping off the filename

31.9. Arithmetic and String Manipulation

expr *
The "expr" command takes an numeric or text pattern expression as an argument,
evaluates it, and returns a result to stdout. The original Bourne shell had no built-in
arithmetic operators. E.g.
expr 2 + 1
expr 2 '*' '(' 21 + 3 ')'
Used with text strings, "expr" can match regular expressions and extract sub
expressions. Similar functionality can be achived with sed. e.g.
expr SP99302L.Z00 : '[A-Z0-9]\{4\}\([0-9]\{3\}\)L\.*'
dc
Desk Calculator - an RPN calculator, using arbitrary precision arithmetic and user-
specified bases. Useful for more complex arithmetic expressions than can be
performed internally or using expr
bc
A preprocessor for dc which provides infix notation and a C-like syntax for
expressions and functions.

31.10. Merging files

paste
Merge lines from multiple files into tab-delimited columns.
join
Perform a join (in the relational database sense) of lines in two sorted input files.

(32)

42
32. References, Resources, Man pages etc.
The standard man pages for sh and ksh are quite complete, but not easy to learn from.
The following is a sampling of the many available books on the subject. The Bolsky
and Korn book might be viewed as the standard "reference". The Blinn book is Bourne
shell, but everything in it should work for either shell.
The links are to publisher's web sites, or Amazon.com. Some links are also given to
the example scripts provided with the books.

32.1. Books
 The New KornShell Command And Programming Language, by Morris I.
Bolsky, David G. Korn (Contributor). More info

 Learning the Korn Shell, 2nd Edn. by Bill Rosenblatt and Arnold Robbins.
More info

 Korn Shell Programming by Example, by Dennis O'Brien, David Pitts

(Contributor). More info

 The Korn Shell Linux and Unix Programming Manual (2nd Edn) by Anatole
Olczak. More info

 Portable Shell Programming: An Extensive Collection of Bourne Shell

Examples by Bruce Blinn. More info
Examples from this book can be downloaded for study.

 Linux Shell Scripting with Bash by Ken O. Burtch. More info

 Unix Shell Programming by Stephen Kochan and Patrick Wood (third Edition).
More info

 Teach yourself Shell Programming in 24 Hours, by S. Veeraraghavan. SAMS

2nd Edn. (2002) More info

 Mastering Unix Shell Scripting by Randal K. Michael, Wiley (2003) More info
Light on basics, but develops scripting through examples. Ksh only. Examples can be
downloaded from the Wiley site (www.wiley.com/legacy/compbooks/michael/).

 Wicked Cool Shell Scripts by Dave Taylor, No Starch Press (2004) More info
Develops scripting entirely through examples, drawn from Linux and OSX in addition to
traditional Unix. Recommended, but not for beginners. Examples can be downloaded from the
Intuitive site (www.intuitive.com/wicked/wicked-cool-shell-script-library.shtml).

 Unix Power Tools, by S. Powers, J. Peek, T. O'Reilly, M. Loudikes et al. More

info

32.2. Online Resources

 Shelldorado (http://www.shelldorado.com)
Lots of links to scripting resources

43
  Kornshell (http://www.kornshell.com)
The official Korn shell home page, with download links.
 Mac OSX Unix tutorial (http://www.osxfaq.com/Tutorials/LearningCenter/)
Good resource on advanced use of OSX and Unix shell scripting in general

32.3. Unix-like shells and utilities for Microsoft Windows

 U/Win (http://www.research.att.com/sw/tools/uwin/)
A free port of ksh and Unix command line utilities, plus Windows DLL for Unix compatability.
Developed by AT&T Research.
 Cygwin (http://www.cygwin.com/)
A free Linux-like environment for Windows. Provides bash, command line utilities and DLLs.
Developed by RedHat. An X server is also available.
 MKS Toolkit (http://www.mkssoftware.com/)
A commercial ksh clone and command line utilities, plus DLL for Unix compatability. An X
server is also available.
 Microsoft Services for UNIX (http://www.microsoft.com/windows/sfu/)
A POSIX environment for Windows, with ksh, csh, command line tools, libraries and software
development tools. Developed by Interix and bought by Microsoft. Free download.

Unix shell scripting with ksh/bash: Course Handout

(last update Friday, 26-Mar-2010 14:16:02 EDT) ©Dartmouth College http://www.dartmouth.edu/~rc/classes/ksh

44