How To Write A Shell Script
How To Write A Shell Script
How To Write A Shell Script
Introduction
A shell is a command line interpretor. It takes commands and executes them. As such, it
implements a programming language. The Bourne shell is used to create shell scripts -ie. programs that are interpreted/executed by the shell. You can write shell scripts with
the C-shell; however, this is not covered here.
Creating a Script
Suppose you often type the command
find . -name file -print
Observations
This quick example is far from adequate but some observations:
1. Shell scripts are simple text files created with an editor.
2. Shell scripts are marked as executeable
3.
4. Should be located in your search path and ~/bin should be in your search path.
5. You likely need to rehash if you're a Csh (tcsh) user (but not again when you
login).
6. Arguments are passed from the command line and referenced. For example, as $1.
#!/bin/sh
All Bourne Shell scripts should begin with the sequence
#!/bin/sh
You can get away without this, but you shouldn't. All good scripts state the interpretor
explicitly. Long ago there was just one (the Bourne Shell) but these days there are many
interpretors -- Csh, Ksh, Bash, and others.
Comments
Comments are any text beginning with the pound (#) sign. A comment can start anywhere
on a line and continue until the end of the line.
Search Path
All shell scripts should include a search path specifica- tion:
PATH=/usr/ucb:/usr/bin:/bin; export PATH
A PATH specification is recommended -- often times a script will fail for some people
because they have a different or incomplete search path.
The Bourne Shell does not export environment variables to children unless explicitly
instructed to do so by using the export command.
Argument Checking
A good shell script should verify that the arguments sup- plied (if any) are correct.
if [ $# -ne 3 ]; then
echo 1>&2 Usage: $0 19 Oct 91
exit 127
fi
Exit status
All Unix utilities should return an exit status.
# is the year out of range for me?
if [ $year -lt 1901 -o $year -gt 2099 ]; then
echo 1>&2 Year \"$year\" out of range
exit 127
fi
etc...
# All done, exit ok
exit 0
A non-zero exit status indicates an error condition of some sort while a zero exit status
indicates things worked as expected.
On BSD systems there's been an attempt to categorize some of the more common exit
status codes. See /usr/include/sysexits.h.
For example,
Your code should be written with the expectation that others will use it. Making sure you
return a meaningful exit status will help.
Error messages should appear on stderr not on stdout! Output should appear on stdout. As
for input/output dialogue:
# give the fellow a chance to quit
if tty -s
echo
echo
case
; then
This will remove all files in $* since ...
$n Ok to procede? $c;
read ans
"$ans" in
n*|N*)
Note: this code behaves differently if there's a user to communicate with (ie. if the
standard input is a tty rather than a pipe, or file, or etc. See tty(1)).
Language Constructs
For loop iteration
Substitute values for variable and perform task:
for variable in word ...
do
command
done
For example:
Case
Switch to statements depending on pattern match
case word in
[ pattern [ | pattern ... ] )
command ;; ] ...
esac
For example:
case "$year" in
[0-9][0-9])
year=19${year}
years=`expr $year - 1901`
;;
[0-9][0-9][0-9][0-9])
years=`expr $year - 1901`
;;
*)
echo 1>&2 Year \"$year\" out of range ...
exit 127
;;
esac
Conditional Execution
Test exit status of command and branch
if command
then
command
[ else
command ]
fi
For example:
if [ $# -ne 3 ]; then
echo 1>&2 Usage: $0 19 Oct 91
exit 127
fi
While/Until Iteration
Repeat task while command returns good exit status.
{while | until} command
do
command
done
For example:
# for each argument mentioned, purge that directory
while [ $# -ge 1 ]; do
_purge $1
shift
done
Variables
Variables are sequences of letters, digits, or underscores beginning with a letter or
underscore. To get the contents of a variable you must prepend the name with a $.
Numeric variables (eg. like $1, etc.) are positional vari- ables for argument
communication.
o
Variable Assignment
Assign a value to a variable by variable=value. For example:
PATH=/usr/ucb:/usr/bin:/bin; export PATH
or
TODAY=`(set \`date\`; echo $1)`
o
Exporting Variables
Variables are not exported to children unless explicitly marked.
# We MUST have a DISPLAY environment variable
if [ "$DISPLAY" = "" ]; then
if tty -s ; then
echo "DISPLAY (`hostname`:0.0)? \c";
read DISPLAY
fi
if [ "$DISPLAY" = "" ]; then
DISPLAY=`hostname`:0.0
fi
export DISPLAY
fi
Likewise, for variables like the PRINTER which you want hon- ored by
lpr(1). From a user's .profile:
PRINTER=PostScript; export PRINTER
Referencing Variables
Use $variable (or, if necessary, ${variable}) to reference the value.
# Most user's have a /bin of their own
if [ "$USER" != "root" ]; then
PATH=$HOME/bin:$PATH
else
PATH=/etc:/usr/etc:$PATH
fi
$p_01
The value of the variable "p" with "_01" pasted onto the end.
o
o
Conditional Reference
${variable-word}
If the variable has been set, use it's value, else use word.
POSTSCRIPT=${POSTSCRIPT-PostScript};
export POSTSCRIPT
${variable:-word}
If the variable has been set and is not null, use it's value, else use word.
These are useful constructions for honoring the user envi- ronment. Ie. the
user of the script can override variable assignments. Cf. programs like
lpr(1) honor the PRINTER environment variable, you can do the same
trick with your shell scripts.
${variable:?word}
If variable is set use it's value, else print out word and exit. Useful for
bailing out.
o
Arguments
Command line arguments to shell scripts are positional vari- ables:
$0, $1, ...
The command and arguments. With $0 the command and the rest the
arguments.
$#
All the arguments as a blank separated string. Watch out for "$*" vs.
"$@".
And, some commands:
shift
Special Variables
$$
Current process id. This is very useful for constructing temporary files.
tmp=/tmp/cal0$$
trap "rm -f $tmp /tmp/cal1$$ /tmp/cal2$$"
trap exit 1 2 13 15
/usr/lib/calprog >$tmp
$?
etc...
then
fi
Quotes/Special Characters
Special characters to terminate words:
; & ( ) | ^ < > new-line space tab
These are for command sequences, background jobs, etc. To quote any of these
use a backslash (\) or bracket with quote marks ("" or '').
Single Quotes
Within single quotes all characters are quoted -- including the backslash. The
result is one word.
grep :${gid}: /etc/group | awk -F: '{print $1}'
Double Quotes
Within double quotes you have variable subsitution (ie. the dollar sign is
interpreted) but no file name generation (ie. * and ? are quoted). The result is one
word.
if [ ! "${parent}" ]; then
parent=${people}/${group}/${user}
fi
Back Quotes
Back quotes mean run the command and substitute the output.
if [ "`echo -n`" = "-n" ]; then
n=""
c="\c"
else
n="-n"
c=""
fi
and
TODAY=`(set \`date\`; echo $1)`
Functions
Functions are a powerful feature that aren't used often enough. Syntax is
name ()
{
commands
}
For example:
# Purge a directory
_purge()
{
if [ ! -d $1 ]; then
echo $1: No such directory 1>&2
return
fi
}
etc...
Within a function the positional parmeters $0, $1, etc. are the arguments to the
function (not the arguments to the script).
Within a function use return instead of exit.
Functions are good for encapsulations. You can pipe, redi- rect input, etc. to
functions. For example:
# deal with a file, add people one at a time
do_file()
{
while parse_one
}
etc...
etc...
# take standard input (or a specified file) and do it.
if [ "$1" != "" ]; then
cat $1 | do_file
else
do_file
fi
Sourcing commands
You can execute shell scripts from within shell scripts. A couple of choices:
sh
command
This runs the shell script as a separate shell. For example, on Sun machines in
/etc/rc:
sh /etc/rc.local
. command
This runs the shell script from within the current shell script. For example:
# Read in configuration information
. /etc/hostconfig
What are the virtues of each? What's the difference? The second form is useful for
configuration files where environment variable are set for the script. For example:
for HOST in $HOSTS; do
# is there a config file for this host?
.
if [ -r ${BACKUPHOME}/${HOST} ]; then
${BACKUPHOME}/${HOST}
fi
etc...
Using configuration files in this manner makes it possible to write scripts that are
automatically tailored for differ- ent situations.
Some Tricks
Test
The most powerful command is test(1).
if test expression; then
etc...
On System V machines this is a builtin (check out the com- mand /bin/test).
On BSD systems (like the Suns) compare the command /usr/bin/test with /usr/bin/
[.
Useful expressions are:
String matching
The test command provides limited string matching tests. A more powerful trick is
to match strings with the case switch.
# parse argument list
while [ $# -ge 1 ]; do
case $1 in
-c*)
rate=`echo $1 | cut -c3-`;;
-c)
shift; rate=$1 ;;
-p*)
prefix=`echo $1 | cut -c3-`;;
-p)
shift; prefix=$1 ;;
-*)
echo $Usage; exit 1 ;;
*)
disks=$*;
break
;;
esac
shift
done
read ans
Is there a person?
The Unix tradition is that programs should execute as qui- etly as possible.
Especially for pipelines, cron jobs, etc.
User prompts aren't required if there's no user.
# If there's a person out there, prod him a bit.
if tty -s; then
echo Enter text end with \^D
fi
Beware: just because stdin is a tty that doesn't mean that stdout is too. User
prompts should be directed to the user terminal.
# If there's a person out there, prod him a bit.
if tty -s; then
echo Enter text end with \^D >&0
fi
Have you ever had a program stop waiting for keyboard input when the output is
directed elsewhere?
Creating Input
We're familiar with redirecting input. For example:
# take standard input (or a specified file) and do it.
if [ "$1" != "" ]; then
cat $1 | do_file
else
do_file
fi
String Manipulations
One of the more common things you'll need to do is parse strings. Some tricks
TIME=`date | cut -c12-19`
TIME=`date | sed 's/.* .* .* \(.*\) .* .*/\1/'`
TIME=`date | awk '{print $4}'`
With some care, redefining the input field separators can help.
#!/bin/sh
# convert IP number to in-addr.arpa name
name()
{
set `IFS=".";echo $1`
echo $4.$3.$2.$1.in-addr.arpa
}
if [ $# -ne 1 ]; then
echo 1>&2 Usage: bynum IP-address
exit 127
fi
add=`name $1`
nslookup < < EOF | grep "$add" | sed 's/.*= //'
set type=any
$add
EOF
Debugging
The shell has a number of flags that make debugging easier:
sh -n
command
Read the shell script but don't execute the commands. IE. check syntax.
sh -x
command
For example, I recently wrote a script to make a backup of one of the subdirectories
where I was developing a project. I quickly wrote a shell script that uses /bin/tar to create
an archive of the entire subdirectory and then copy it to one of our backup systems at my
computer center and store it under a subdirectory named according to today's date.
As another example, I have some software that runs on UNIX that I distribute and people
were having trouble unpacking the software and getting it running. I designed and wrote
a shell script that automated the process of unpacking the software and configuring it.
Now people can get and install the software without having to contact me for help, which
is good for them and good for me, too!
For shell script experts one of the things to consider is whether to use the Bourne shell (or
ksh or bash), the C shell, or a richer scripting language like perl or python. I like all these
tools and am not especially biased toward any one of them. The best thing is to use the
right tool for each job. If all you need to do is run some UNIX commands over and over
again, use a Bourne or C shell script. If you need a script that does a lot of arithmetic or
string manipulation, then you will be better off with perl or python. If you have a Bourne
shell script that runs too slowly then you might want to rewrite it in perl or python
because they can be much faster.
Historically, people have been biased toward the Bourne shell over the C shell because in
the early days the C shell was buggy. These problems are fixed in many C shell
implementations these days, especially the excellent 'T' C shell (tcsh), but many still
prefer the Bourne shell.
There are other good shells available. I don't mean to neglect them but rather to talk about
the tools I am familiar with.
If you are interested also in learning about programming in the C shell I also have a
comparison between features of the C shell and Bourne shell.
Table of Contents:
1. Review of a few Basic UNIX Topics (Page 1)
2. Storing Frequently Used Commands in Files: Shell Scripts
(Page 6)
3. More on Using UNIX Utilities (Page 9)
4. Performing Search and Replace in Several Files (Page 11)
5. Using Command-line Arguments for Flexibility (Page 14)
6. Using Functions (Page 30)
7. Miscellaneous (Page 38)
8. Trapping Signals (Page 43)
9. Understanding Command Translation (Page 50)
10. Writing Advanced Loops (Page 59)
11. Creating Remote Shells (Page 67)
12. More Miscellaneous (Page 73)
13. Using Quotes (Page 75)
Variables
The quotes are required in the example above because the string contains a special
character (the space)
A variable may store a number
num=137
Try defining num as '7m8' and try the expr command again
What happens when num is not a valid number?
Now you may exit the Bourne shell with
exit
Page 1
I/O Redirection
sort /etc/passwd
sort < /etc/passwd
wc /etc/passwd
wc -l /etc/passwd
You can save the output of wc (or any other command) with output redirection
wc /etc/passwd > wc.file
Many UNIX commands allow you to specify the input file by name or by input
redirection
You can also append lines to the end of an existing file with output redirection
wc -l /etc/passwd >> wc.file
Page 2
Backquotes
Notice how echo prints the output of 'date', and gives the time when you defined
the save_date variable
Store the following in a file named backquotes.sh and execute it (right click and
save in a file)
#!/bin/sh
# Illustrates using backquotes
# Output of 'date' stored in a variable
Today="`date`"
echo Today is $Today
The example above shows you how you can write commands into a file and
execute the file with a Bourne shell
Backquotes are very useful, but be aware that they slow down a script if you use
them hundreds of times
You can save the output of any command with backquotes, but be aware that the
results will be reformated into one line. Try this:
LS=`ls -l`
echo $LS
Page 3
Pipes
head -5 /etc/passwd
head -5 < /etc/passwd
ls -al | sort -n -r +4
You could accomplish the same thing more efficiently with either of the two
commands:
For example, this command displays all the files in the current directory sorted by
file size
The command ls -al writes the file size in the fifth column, which is why we skip
the first four columns using +4.
The options -n and -r request a numeric sort (which is different than the normal
alphabetic sort) in reverse order
Page 4
awk
Cut and paste this line into a Bourne shell and you should see a column of file
sizes, one per file in your current directory.
A more complicated example shows how to sum the file sizes and print the result
at the end of the awk run
ls -al | awk '{sum = sum + $5} END {print sum}'
In this example you should see printed just one number, which is the sum of the
file sizes in the current directory.
Page 5
Shell Scripts
What is the difference between -v and -x? Notice that with -v you see '$USER' but
with -x you see your login name
Run the command 'echo $USER' at your terminal prompt and see that the variable
$USER stores your login name
With -v or -x (or both) you can easily relate any error message that may appear to
the command that generated it
When an error occurs in a script, the script continues executing at the next
command
Verify this by changing 'cal' to 'caal' to cause an error, and then run the script
again
Run the 'caal' script with 'sh -v simple.sh' and with 'sh -x simple.sh' and verify the
error message comes from cal
Other standard variable names include: $HOME, $PATH, $PRINTER. Use echo
to examine the values of these variables
Page 6
Topics covered: variables store strings such as file names, more on creating and
using variables
Utilities covered: echo, ls, wc
A variable is a name that stores a string
It's often convenient to store a filename in a variable
Store the following in a file named variables.sh and execute it
#!/bin/sh
# An example with variables
filename="/etc/passwd"
echo "Check the permissions on $filename"
ls -l $filename
echo "Find out how many accounts there are on this system"
wc -l $filename
Topics covered: global search and replace, input and output redirection
Utilities covered: sed
Here's how you can use sed to modify the contents of a variable:
echo "Hello Jim" | sed -e 's/Hello/Bye/'
Copy the file nlanr.txt to your home directory and notice how the word 'vBNS'
appears in it several times
Change 'vBNS' to 'NETWORK' with
sed -e 's/vBNS/NETWORK/g' < nlanr.txt
You can save the modified text in a file with output redirection
sed -e 's/vBNS/NETWORK/g' < nlanr.txt > nlanr.new
Sed can be used for many complex editing tasks, we have only scratched the
surface here
Page 8
Performing Arithmetic
Topics covered: integer arithmetic, preceding '*' with backslash to avoid file
name wildcard expansion
Utilities covered: expr
Arithmetic is done with expr
expr 5 + 7
expr 5 \* 7
Page 9
Translating Characters
Topics covered: converting one character to another, translating and saving string
stored in a variable
Utilities covered: tr
Copy the file sdsc.txt to your home directory
The utility tr translates characters
tr 'a' 'Z' < sdsc.txt
This example shows how to translate the contents of a variable and display the
result on the screen with tr
Store the following in a file named tr1.sh and execute it
#!/bin/sh
# Translate the contents of a variable
Cat_name="Piewacket"
echo $Cat_name | tr 'a' 'i'
Now you can change the value of the variable and your script has access to the
new value
Page 10
This executes the three commands echo, ls and wc for each of the three file names
You should see three lines of output for each file name
filename is a variable, set by "for" statement and referenced as $filename
Now we know how to execute a series of commands on each of several files
Page 11
You should see three lines of output for each file name ending in '.sh'
The file name wildcard pattern *.sh gets replaced by the list of filenames that
exist in the current directory
For another example with filename wildcards try this command
echo *.sh
Page 12
Topics covered: combining for loops with utilities for global search and replace
in several files
Utilities covered: mv
Sed performs global search and replace on a single file
sed -e 's/application/APPLICATION/g' sdsc.txt > sdsc.txt.new
What if you want to run the script but with different file names?
To execute for loops on different files, the user has to know how to edit the script
Not simple enough for general use by the masses
Wouldn't it be useful if we could easily specify different file names for each
execution of a script?
Page 14
How many command-line arguments were given to wc? It depends on how many
files in the current directory match the pattern *.sh
Use 'echo *.sh' to see them
Most UNIX commands take command-line arguments. Your scripts may also have
arguments
Page 15
This script runs properly with any number of arguments, including zero
The shorter form of the for statement shown below does exactly the same thing
for filename
do
...
Don't use
for filename in $*
If Blocks
Page 19
echo "The next command should fail and return a status greater
than zero"
ls /nosuchdirectory
echo "Status is $? from command: ls /nosuchdirectory"
echo "The next command should succeed and return a status equal to
zero"
ls /tmp
echo "Status is $? from command: ls /tmp"
Regular Expressions
Page 21
Eagerness: a regular expression will find the first match if several are present in
the line
Execute this command and see whether 'big' or 'bag' is matched by the regular
expression
echo 'big bag' | sed -e 's/b.g/___/'
Hint: a* matches zero or more a's, and there are many places where zero a's
appear
Try the example above with the extra 'g'
echo 'black dog' | sed -e 's/a*/_/g'
Page 22
regexp
wildcard
meaning
.*
.
[aCg]
*
?
[aCg]
Page 23
Page 24
;;
cal)
echo "Running cal..."
cal
;;
*)
echo "Bad command, your choices are: who, list, or cal"
;;
esac
exit 0
The last case above is the default, which corresponds to an unrecognized entry
The next example uses the first command-line arg instead of asking the user to
type a command
Store the following in a file named case2.sh and execute it
#!/bin/sh
# An example with the case statement
# Reads a command from the user and processes it
# Execute with one of
# sh case2.sh who
# sh case2.sh ls
# sh case2.sh cal
echo "Took command from the argument list: '$1'"
case "$1" in
who)
echo "Running who..."
who
;;
list)
echo "Running ls..."
ls
;;
cal)
echo "Running cal..."
cal
;;
*)
echo "Bad command, your choices are: who, list, or cal"
;;
esac
The patterns in the case statement may use file name wildcards
Page 25
Page 26
The entire while loop reads its stdin from the pipe
Each read command reads another line from the file coming from cat
The entire while loop runs in a subshell because of the pipe
Variable values set inside while loop not available after while loop
Page 27
"h")
?)
;;
opt_h="1"
;;
echo "getopts2.sh: Bad option specified...quitting"
exit 1
;;
esac
done
shift `expr $OPTIND - 1`
if [ "$opt_P" != "" ]
then
echo "Option P used with argument '$opt_P'"
fi
if [ "$opt_h" != "" ]
then
echo "Option h used"
fi
if [ "$*" != "" ]
then
echo "Remaining command-line:"
for arg in "$@"
do
echo " $arg"
done
fi
Functions
Define a Function
Define a function
echo_it () {
echo "In function echo_it"
}
Function Arguments
echo_it Barney
Page 33
Functions in Pipes
Inherited Variables
func_y () {
echo "A is $A"
return 7
}
A='bub'
func_y
if [ $? -eq 7 ] ; then ...
Try it: is a variable defined inside a function available to the main program?
Page 35
Operate in pipes
echo "test string" | ls_sorter
Page 36
Libraries of Functions
Section 7: Miscellaneous
Here Files
set +v
set -xv
Common Signals
Page 44
Send a Signal
Page 45
Trap Signals
Handling Signals
trap "echo Interrupted; exit 2" 2
Ignoring Signals
trap "" 2 3
Page 46
See file
/usr/include/sys/signal.h
Page 47
User Signals
Page 49
Command Translation
Order of Translations
Echos command if -v
Interprets quotes
Performs variable substitution
Page 52
Exceptional Case
Page 55
Examples (continued)
x=*
echo $x
Examples (continued)
Wildcards expanded after redirection (assuming file* matches exactly one file):
cat < file*
file*: No such file or directory
Page 57
Eval Command
While loops
Page 59
Until loops
Redirection of Loops
Continue Command
Break Command
do
if [ ! -r $name ] ; then
echo "Cannot read $name, quitting loop"
break
fi
echo "Found file or directory $name"
done
Example loops over files and directories, quits if one is not readable
Page 63
Case Command
Page 64
done
echo "All done"
When you run it, the script waits for you to type one of:
list
freespace
quit
Quit
Try it: modify the example so any command beginning with characters "free" runs
df
Page 65
Infinite Loops
Remote Shells
Rsh command
rsh hostname "commands"
Page 67
Page 70
Return Status
Temporary Files
Wait Command
Quotes
Page 75
Metacharacters
Page 76
Page 79
Last modified: Thursday, May 11, 2006 06:11:35