04_ShellProgramming
04_ShellProgramming
2
Outline
● Variable pre-operations
● args, argc in Shell Scripts
● Arithmetic and Logics
○ Test commands
● Control Structures: if-else, switch-case, for/while loops
● Input/output: Read from keyboard
● Defining Functions & Parsing Arguments
● Error Handling and Debug tool (sh -x)
● Regular Expression
● Advanced scripting: sed and awk
● A Shell Script Sample: Failure Detection on Servers 3
Bourne Shell
● We use Bourne Shell in this slide
● Check your login shell
$ echo $SHELL
/bin/tcsh
4
Sample script
● Print "Hello World" 3 times
#!/bin/sh
# ^ shebang: tell the system which interpreter to use
for i in `seq 1 3` ; do
echo "Hello world $i" # the body of the script
done
● Output
$ chmod +x test.sh # grant execution permission
$ ./test.sh # execute the script. Must specify the directory(./)
5
Executable script
● Shebang (#!), or called Shabang
○ Sharp (#) + Bang (!)
■ or Hash Bang
○ Specify which interpreter is going to execute this script
○ Many interpreted language uses # as comment indicators
○ The first widely known appearance of this feature was on BSD
6
Executable script
● Shebang examples
○ #!/bin/sh
○ #!/bin/sh -x
○ #!/bin/bash
○ #!/usr/local/bin/bash
○ #!/usr/bin/env bash
○ #!/usr/bin/env python
● Execution
○ $ sh test.sh
■ Can execute without shebang
○ $ chmod a+x test.sh
○ $ ./test.sh 7
Shell variables (1)
● Assignment
Syntax Scope
Variable my=test Process
Local variable local my=test Function
Environment variable export my Process and sub-process
● Example
$ export PAGER=/usr/bin/less
$ current_month=`date +%m`
$ myFun() { local arg1="$1" }
8
Shell variables (2)
● There are two ways to call variable
○ $ echo "$PAGER"
○ $ echo "${PAGER}" <= Why?
■ Use {} to avoid ambiguity
● Example
$ temp_name="haha" && temp="hehe" # No Space Beside "="
$ echo $temp
hehe
$ echo $temp_name
haha
$ echo ${temp}_name
hehe_name
$ echo ${temp_name}
haha 9
Quotation marks
Quotes Description Example
Single quote, Preserves the
10
Shell variable operator (1)
● sh(1): Parameter Expansion
Operator Description
${var:=value} If "Bad", use the given value and assign to var.
${var:+value} If "Good", use the given value. Otherwise, null is used but not assign to var.
=> Replace if "Good", not assign to var.
${var:-value} If "Good", use the value of var. Otherwise, use the given value but not assign to var
=> Replace if "Bad", not assign to var.
${var:?value} If "Bad", print given value (stderr) and shell exits (The command stops
immediately).
● Script ● Result
#!/bin/sh 32
var="Nothing happened end closing end" happened end closing end
echo ${#var} end
echo ${var#*ing} Nothing happened end closing
echo ${var##*ing} Nothing happened
echo ${var%end*}
echo ${var%%end*} 13
Predefined shell variables
● Environment Variables
● Other useful variables
○ Similar to C program's "int main(argc, argv)" – arguments of
program
○ e.g. ls –a ~
14
Predefined shell variables
● Example:
○ ls –a ~ sh Description
$# Number of positional arguments (start from 0)
15
Usage of $* and $@
● The difference between $* and $@
○ $* : all arguments are formed into a long string
○ $@ : all arguments are formed into separated strings
● Examples: test.sh
for i in "$*" ; do for i in "$@" ; do
echo "In loop: $i" echo "In loop: $i"
done done
% test.sh 1 2 3
% test.sh 1 2 3 In loop: 1
In loop: 1 2 3 In loop: 2
In loop: 3
16
The "test" command
● Checking file status, string, numbers, etc
● test(1)
○ test expression
○ [ expression ]
● Test and return 0 (true) or 1 (false) in $?
○ test -e News ; echo $?
■ If there exist the file named "News"
○ test "haha" = "hehe" ; echo $?
■ Whether "haha" equal "hehe"
○ test 10 -eq 11 ; echo $?
■ Whether 10 equal 11
17
Test command – File test
● -e file
○ True if file exists (regardless of type)
● -s file
○ True if file exists and has size greater than zero
● -d file
○ True if file exists and is a directory
● -f file
○ True if file exists and is a regular file
18
Test command – File test
● -L file
○ True if file exists and is a symbolic link
● -r file
○ True if file exists and is readable
● -w file
○ True if file exists and is writable
● -x file
○ True if file exists and is executable
19
Test command – File test
● file1 -nt file2
○ True if file1 exists and is newer than file2
● file1 -ot file2
○ True if file1 exists and is older than file2
● file1 -ef file2
○ True if file1 and file2 exist and refer to the same file
20
Test command – String test
● -z string
○ True if the length of string is zero
● -n string
○ True if the length of string is nonzero
● string
○ True if string is not the null string
● s1 = s2 (though some implementation recognize ==)
○ True if the strings s1 and s2 are identical
● s1 != s2
○ True if the strings s1 and s2 are not identical
● s1 < s2
○ True if string s1 comes before s2 based on the binary value of their characters
(lexicographical order)
● s1 > s2
○ True if string s1 comes after s2 based on the binary value of their characters 21
Test command – Number comparison
● Number comparison with “>”
22
Test command – Number test
● n1 -eq n2 ==, !=, >, <, >=, <= fashion does not apply here
○ True if the integers n1 and n2 are algebraically equal
● n1 -ne n2
○ True if the integers n1 and n2 are not algebraically equal
● n1 -gt n2
○ True if the integer n1 is algebraically greater than the integer n2
● n1 -ge n2
○ True if the integer n1 is algebraically greater than or equal to the integer n2
● n1 -lt n2
○ True if the integer n1 is algebraically less than the integer n2
● n1 -le n2
○ True if the integer n1 is algebraically less than or equal to the integer n2
23
Test Command – Combination
● ! expression
○ True if expression is false.
○ $ [ ! A == B ] => Test expression, invert the internal result
○ $ ! [ A == B ] => Invert the whole test command result
● expression1 -a expression2
○ True if both expression1 and expression2 are true.
○ $ [ A == B –a C == D ]
● expression1 -o expression2
○ True if either expression1 or expression2 are true.
○ The -a operator has higher precedence than the -o operator.
○ $ [ A == B –o C == D ]
24
Test Command – Combination Example
● ! [ "A" = "A" -o 1 -eq 1 ]
○ false
● [ ! "A" = "A" -o 1 -eq 1 ]
○ true
25
Test Command – In Script
● Add space beside = <= != [ ]…
○ $ [A=B] # error
○ $ [ A=B ] # error
○ $ [A = B] # error
● If the var may be null or may not be set, add ""
○ $ [ $var = "A" ] may be parsed to [ = "A" ] and cause syntax error!!
○ $ [ "$var" = "A" ] become [ "" = "A" ]
27
expr command (2)
● $ expr1 && expr2
○ if expr1 is false then expr2 won't be evaluate
● $ expr1 || expr2
○ if expr1 is true then expr2 won't be evaluate
● Ex:
○ $ [ -e SomeFile ] && rm SomeFile
○ $ checkSomething || exit 1
28
Arithmetic Expansion
echo $(( 1 + 2 )) 3
a=8 // a=8
a=$(( $a + 9 )) // a=17
a=$(( $a + 17 )) // a=34
a=$(( $a + 9453 )) // a=9487
echo $a 9487
29
if-then-else structure
if [ test conditions ] ; then
command-list
elif [ test conditions ] ; then
command-list
else
command-list
fi
# Or in one line
if [ a = a ]; then echo "Yes"; else echo "No"; fi
30
switch-case structure (1)
case $var in case $sshd_enable in
value1) [Yy][Ee][Ss])
action1
;; action1
value2) ;;
action2 [Nn][Oo])
;; action2
value3|value4)
;;
action3
;; *)
*) ???
default-action ;;
;; esac
esac
31
For loop
for var in var1 var2 …; do a=""
action for var in `ls`; do
done a="$a $var"
done
echo $a
for i in A B C D E F G; do
mkdir $i;
done
32
While loop
while [ expression ] ; do
action
done
break
continue
33
Read from stdin
#!/bin/sh
echo -n "Do you want to 'rm -rf /' (yes/no)? "
read answer # read from stdin and assign to variable
case $answer in
[Yy][Ee][Ss])
echo "Hahaha"
;;
[Nn][Oo])
echo "No~~~"
;;
*)
echo "removing..."
;;
esac
34
Create tmp file/dir
● TMPDIR=`mktemp –d tmp.XXXXXX`
● TMPFILE=`mktemp ${TMPDIR}/tmp.XXXXXX`
● echo "program output" >> ${TMPFILE}
35
functions (1)
● Define function
function_name ( ) {
command_list
}
● Removing function definition
unset function_name
● Function execution
function_name
● Function definition is local to the current shell
● Define the function before first use
36
functions (2) - scoping
func() { func() {
# global variable # local variable
echo $a local a="bar"
a="bar" echo $a
} }
a="foo" a="foo"
func func
echo $a echo $a
foo bar
bar foo
37
functions (3) - arguments check
func() {
if [ $# -eq 2 ] ; then
echo $1 $2
else
echo "Wrong"
fi
}
func
func hi
func hello world
Wrong
Wrong
hello world
38
functions (4) - return value
func() {
if [ $# -eq 2 ] ; then
return 2
else
return 0
fi
}
func
echo $?
func hello world
echo $?
0
2 39
Scope
● Local var can only be read and written inside the function.
● Subprocess can only read the environment variable, the
modification of the variable will NOT be effective to the current
process. (Subprocess may include some PIPE execution)
● If something wrong, try to print every variable.
#!/bin/sh 10 20 #!/bin/sh
a=10 10 21 a=10
export b=20 10 22 export b=20
cat test.sh | while read line; do 10 23 cat test.sh | while read line; do
echo "$a $b $line" 10 24 echo "$a $b $line"
b=$((b+1)) 10 25 b=$((b+1))
done 10 26 done
echo b is $b 10 27 echo b is $b
b is 20 40
test.sh
Parsing arguments
● Use getopts
#!/bin/sh $ ./test.sh -a -b -c -f hi
echo "Initial OPTIND: $OPTIND" Initial OPTIND: 1
while getopts abcf: op ; do 2-th arg
echo "${OPTIND}-th arg" one of ABC
3-th arg
case $op in one of ABC
a|b|c) 4-th arg
echo "one of ABC" ;; one of ABC
f) 6-th arg
echo $OPTARG ;; hi
*)
echo "Default" ;; ● ":" means additional arg.
esac ● $OPTARG: content of additional arguments
done
● $OPTIND: index of the next argument
○ Need manually reset for the second call
41
Handling Error Conditions
● Internal error
○ Program crash
○ Failing to perform sub commands
○ Invalid input
○ Syntax error
● External error
○ Signal from OS
■ The system telling you that some system-level event has occurred
○ Ctrl+C
■ SIGINT
42
Handling Error Conditions –
Internal Error cut: you must specify a list of bytes, characters, or fields
Try 'cut --help' for more information.
● Example:
○ Handling the errors by yourself
program name
#!/bin/sh
UsageString="Usage: $0 -man=val1 -woman=val2"
if [ $# != 2 ] ; then
echo "$UsageString"
else How about c but not -c?
echo "ok!"
man=`echo $1 | cut -c 6-`
woman=`echo $2 | cut -c 8-`
echo "Man is ${man}"
echo "Woman is ${woman}"
fi
43
Handling Error Conditions –
External Error (1)
● Using trap in Bourne shell
○ To handle events like Ctrl+C (SIGINT, signal number is 2)
○ trap [command-list] [signal-list]
■ Perform command-list when receiving any signal in signal-list
44
Handling Error Conditions –
External Error (2) Catch: perform something when trapped
Block: prevent system actions
# Name Description Default Catch Block Dump Core
1 SIGHUP Hangup Terminate ✔ ✔ ❌
47
Regular Expression
Pattern Matching
國立陽明交通⼤學資⼯系資訊中⼼
Information Technology Center, Department of Computer Science, NYCU
Regular Expression (1)
● Informal definition
○ Basis:
■ A single character "a" is a R.E.
○ Hypothesis
■ If r and s are R.E.
○ Inductive
■ Union: r + s is R.E
● Ex: a + b
■ Concatenation: rs is R.E.
● Ex: ab
■ Kleene closure: r* is R.E.
● Ex: a*
49
Regular Expression (2)
● Pattern-matching operator Description
○ Special operators . Any single character
[] Any character in []
[^] Any character not in []
^ start of a line
$ end of a line
* zero or more
? zero or one
+ one or more
{m,n} At least m times and at most n times
{m,} At least m times.
{m} Exactly m times.
\ Escape character
50
Regular Expression (3)
● Examples
○ r.n
■ Any 3-character string that start with r and end with n
● r1n, rxn, r&n will match
● r1xn, axn will not match
○ ..Z..
■ Any 5-character strings that have Z as 3rd character
● aeZoo, 12Zos will match
● aeooZ, aeZoom will not match
○ r[a-z]n
■ Any 3-character string that start with r and end with n and the 2nd character is
an alphabet
■ rxn will match
■ r1n, r&n will not match 51
Regular Expression (4)
● Examples
○ ^John
■ Any string starts with John
● John Snow -> will match
● Hi John -> will not match
○ [Ee][Nn][Dd]$
■ Any string ends with any combination of "end"
○ [A-Za-z0-9]+
■ String of characters
52
Regular Expression (5)
● Utilities using RE
○ grep
○ awk
○ sed
○ find
● Different tools, different RE
○ BRE (Basic)
○ ERE (Extended)
○ PCRE (Perl Compatible)
○ https://en.wikipedia.org/wiki/Regular_expression#Standards
53
Advanced scripting
- sed and awk
Details on using sed and awk…
國立陽明交通⼤學資⼯系資訊中⼼
Information Technology Center, Department of Computer Science, NYCU
sed – Stream EDitor (1)
● sed(1)
○ sed -e "command" -e "command"… file
○ sed -f script-file file
■ Sed will (1) read the file line by line and (2) do the commands,
then (3) output to stdout
■ e.g. sed -e '1,10d' -e 's/yellow/black/g' yel.dat
● Command format
○ [address1[,address2]]function[argument]
■ From address 1 to address 2
■ Do what action
● Address format
○ n → line number
○ /R.E./ → the line that matches R.E
55
sed – Stream EDitor (2)
● Address format
○ Example of address format
■ sed -e 10d
■ sed -e /man/d
■ sed -e 10,100d
■ sed -e 10,/man/d
● Delete line from line 10 to the line contain "man"
56
sed – Stream Editor
Function: print (1)
● print
○ Syntax:
■ [addr1, addr2]p
● Ex:
○ sed -n -e ‘/^tsaimh/p' # Print out the lines that begins with tsaimh
-n: By default, each line of input is echoed to the standard output after
all of the commands have been applied to it. The -n option suppresses
this behavior.
57
sed – Stream Editor
Function: print (2)
hello
tsaimh
world
input.txt
58
sed – Stream Editor
Function: substitution (1)
● substitution
○ Syntax
■ s/pattern/replace/flags
○ Flags
■ N: Make the substitution only for the N'th occurrence
■ g: replace all matches
■ p: print the matched and replaced line
■ w: write the matched and replaced line to a file
59
sed – Stream Editor
Function: substitution (2)
● Example:
file.txt
○ sed -e ‘s/tsaimh/TSAIMH/2' file.txt I am jon
I am jon I am john
I am john I am tsaimh
I am tsaimh I am tsaimh
I am tsaimh I am nothing
I am nothing
○ sed -e ‘s/tsaimh/TSAIMH/g' file.txt
I am jon
I am john
I am TSAIMH
I am TSAIMH
I am nothing
60
sed – Stream Editor
Function: substitution (3)
● Example:
file.txt
○ sed -e ‘s/tsaimh/TSAIMH/p' file.txt I am jon
I am jon I am john
I am john I am tsaimh
I am TSAIMH I am tsaimh
I am TSAIMH I am nothing
I am TSAIMH
I am TSAIMH
I am nothing
○ sed -n -e ‘s/tsaimh/TSAIMH/p' file.txt
I am TSAIMH
I am TSAIMH
○ sed -e ‘s/tsaimh/TSAIMH/w wfile' file.txt
○ cat wfile
I am TSAIMH
I am TSAIMH
61
sed – Stream Editor
Function: delete
● delete
○ Syntax:
■ [address]d
● Ex:
○ sed -e 10d
○ sed -e /man/d
○ sed -e 10,100d
○ sed -e 10,/man/d
62
sed – Stream EDitor
Function: append, insert, change
● Function
○ append
■ append after the line
○ insert
■ insert before the line
○ change
■ replace whole line sed.src
/tsaimh/i \
● Example: Meet tsaimh, Hello I am jon
○ sed -f sed.src file.txt I am john
I am jon Meet tsaimh, Hello
I am john I am tsaimh
I am tsaimh Meet tsaimh, Hello
I am tsaimh I am tsaimh
I am nothing I am nothing
file.txt Result
63
sed – Stream EDitor
Example: split lines into multiple files
● cat host.txt | sed -e '/cs/w cs_host.txt' -e '/google/w google_host.txt'
bsd1.cs.nctu.edu.tw
www.google.com linux3.cs.nctu.edu.tw
bsd1.cs.nctu.edu.tw cs.nctu.edu.tw
linux3.cs.nctu.edu.tw cs_host.txt
store.google.com
cs.nctu.edu.tw
host.txt
www.google.com
store.google.com
google_host.txt
64
awk
● awk(1)
○ awk [-F fs] [ 'awk_program' | -f program_file] [data_file ……]
■ awk will read the file line by line and evaluate the pattern, then do the
action if the test is true
■ Ex:
● awk '{print "Hello World"}' file
● awk '{print $1}' file
● Program structure Amy 32 0800995995 nctu.csie
○ pattern { action } $1 $2 $3 $4
○ missing pattern means always matches
○ missing { action } means print the line
65
awk – Pattern formats
● pattern formats
○ Regular expression
■ awk '/[0-9]+/ {print "This is an integer" }'
■ awk '/[A-Za-z]+/ {print "This is a string" }'
■ awk '/^$/ {print "this is a blank line."}'
○ BEGIN
■ before reading any data
● awk ' BEGIN {print "Nice to meet you"}'
○ END
■ after the last line is read
● awk ' END {print "Bye Bye"}'
66
awk – action format
● Actions
○ Print
■ Assignment
■ if( expression ) statement [; else statement2]
● awk ' { if( $2 ~ /am/ ) print $1}' file
■ while( expression ) statement variable usage: no need for "$"
tsaimh:*:1002:20:Meng-Hsun Tsai:/home/tsaimh:/bin/tcsh
69
Shell Script Examples
國立陽明交通⼤學資⼯系資訊中⼼
Information Technology Center, Department of Computer Science, NYCU
check alive(1)
● ping
$ /sbin/ping -c 3 bsd1.cs.nctu.edu.tw
71
check alive(2)
#!/bin/sh
# [Usage] isAlive.sh bsd1.cs.nctu.edu.tw
Usage="[Usage] $0 host"
temp="$1.ping"
Admin=”tsaimh fs"
count="3"
default 10 times
if [ $# != 1 ] ; then Grep "tran…"
echo $Usage write to the temp file
else
/sbin/ping -c ${count:=10} $1 | /usr/bin/grep 'transmitted' > $temp
Lost=`awk -F" " '{print $7}' $temp | awk -F"." '{print $1}' `
73