The command line for scientific
computing
Katerina Michalickova, November 11, 12 and 13, 2019
Graduate School
slides – http://tinyurl.com/ic-gs-nov11 and http://tinyurl.com/ic-gs-nov13
Outline
• Finding things
http://swcarpentry.github.io/shell-novice/07-find/index.html
• Nelle’s story continued
• Variables
• Tests
• Loops
• (Functions)
Bash script
A shell script is a series of commands written in a plain text
file. When a script is executed, all commands are
performed in a sequence allowing a user to accomplish
multiple or repetitive tasks without much typing. Shell
scripts utilise variables, control statements, loops and
functions. These building block can turn a simple script
into a program in its own right.
Bash script
Shell scripting can be used for:
• Managing big data
• Setting up workflows
• System management
• Deploying programs on High Performance Computing
systems
A minimal example
#!/bin/bash
# My first script
echo "This is my first script!"
first.sh
Executing a script
bash-4.1$ cat first.sh
#!/bin/bash
# My first script
echo "This is my first script!"
bash-4.1$
bash-4.1$
bash-4.1$ bash first.sh
This is my first script!
bash-4.1$
bash-4.1$
bash-4.1$ chmod a+x first.sh
bash-4.1$
bash-4.1$ ./first.sh
This is my first script!
bash-4.1$
Scripting in Bash shell
Shell scripts utilise the same basic building blocks as other
scripting languages:
• Variables (represent values and make scripts more
versatile)
• Instruction flow control statements
• Conditionals (if this is true, do this; otherwise do this)
• Loops (do this repeatedly)
• Functions
The above can be combined with shell commands to
create a powerful tool that can help us do more in less
time and with less pain.
Variables - user-defined
#!/bin/bash
# My first script with a variable
MESSAGE=”Hello world!”
echo $MESSAGE
varfirst.sh
Variables - user-defined
#!/bin/bash
# My first script with a variable
GREETING=Hello
NAME=Bob
MESSAGE=”$GREETING $NAME”
echo $MESSAGE
varfirst.sh
Variables – assign an output of a command
#!/bin/bash
# date is a command
VAR=$(date)
echo "today is $VAR"
varfirst.sh
Variable declaration
Rules for declaring variables:
• no spaces are allowed around the the equal sign in the
variable declaration line
• variable names have to start with a letter
• no spaces are allowed in the variable name
• no punctuation marks are allowed in the name
(underscore is ok)
Shell variables are not typed, essentially, they are
strings. Arithmetic operations are permitted on variables
containing only digits. No floating point operations by default.
Variables - dollar sign and curly brackets
• Variable appears without “$” at assignment
• Otherwise preceded by $ (with some exceptions*)
• $MESSAGE is the same as ${MESSAGE}
• Curly brackets are often skipped
*unset and arithmetic with (( ))
Variables - system-defined
• Environment variables
• e.g. $PATH, $SHELL, $USER, $HOME
• Positional input parameters
• $@, $1, $2, ${10}, $#
• Exit value
• $?
Variable type
• All shell variables are considered text (not numbers).
• Text containing white space have to be quoted inside the
scripts.
MESSAGE="This is my first script!"
• Some text characters have special meaning
• $ denotes a variable
• ` evokes a command execution
• " closes initial double quote
• \ means ignore special meaning of the next character
Single and double quotes
• The choice of single or double quotes determines if
special character will be interpreted.
• Single quotes do not allow interpretation of special
characters.
• Special characters enclosed in double quotes will be
interpreted (unless they are preceded with \).
• It is possible to mix single and double quotes in one
statement.
Quoting and escaping
VAR=foo
echo $VAR
foo
• variable recognised in double quotes (variable expansion)
echo "$VAR"
foo
• $ variable not expanded in single quotes
echo '$VAR'
$VAR
• if escaped with \ in double quotes, $ is interpreted as $
echo "\$VAR"
$VAR
Variable scope
• Variables declared inside the script exist only during
execution of the script.
• If there is a need to keep a variable set after the script
finished, the script can be "sourced”.
• Correspondingly, a variable that is instantiated in the
shell is not propagated to the script (as this runs in its
own shell).
• Variables can be “exported” in the shell to be inherited
by the script.
Sourcing a script
bash-4.1$ cat varfirst.sh bash-4.1$./varfirst.sh
#!/bin/bash This is my first script!
# My first script bash-4.1$
MESSAGE="This is my first bash-4.1$ echo $MESSAGE
script!"
echo $MESSAGE bash-4.1$
bash-4.1$ bash-4.1$ source first.sh
This is my first script!
bash-4.1$
bash-4.1$
bash-4.1$ echo $MESSAGE
This is my first script!
bash-4.1$
Math with Bash
bash-4.1$ a=1
bash-4.1$ b=2
bash-4.1$ ((c=a+b))
bash-4.1$ echo $c
3
bash-4.1$ echo $((b-a))
1
bash-4.1$ echo $((b*a))
2
bash-4.1$ echo $((a/b))
0
bash-4.1$ echo $((b**3))
8
bash-4.1$
Control flow statements
Control flow determines the order of statements during
the execution.
• if
• test
• loops
• case
If statement
if [ expression ]; then
echo "it is true"
else
echo "it is false"
fi
ifelse.sh
If + elif statements
if [ expression1 ]; then
# if code
elif [ expression2 ]; then
# elif code
else
# else code
fi
Logical expression
• [ $var1 = $var2 ] evaluates to true or false
• it is equivalent to test $var1 = $var2
• “test” is a program, $var1 = $var2 are arguments to the
program
• $var1 = $var2 without spaces (inside and around)
would be impossible for the test program to read
Logical expressions - comparing text
$var1 = $var2 var1 is equal to var2
$var1 != $var2 var1 is not equal to var2
Logical expressions – comparing numbers
-eq is equal to
-ne is not equal to
-lt is less than
-le is less than or equal to
-gt is grater than
-ge is greater than or equal to
Logical expressions – logical comparison
! expression logical NOT
expression1 -a expression2 logical AND
expression1 -o expression2 logical OR
Logical expressions – file tests
-e file exists [ -e file.txt ]
-f is a regular file (not a directory)
-s has non-zero size
-d is a directory
-w has write permissions
-r has read permissions
-x is executable
Loops
Repeat a block of code
The number of repetitions is controlled by a condition
For loop
for i in {1..4}
do
echo "Iteration $i."
done
bash-4.1$ ./forloop.sh
Iteration 1.
Iteration 2.
Iteration 3.
Iteration 4.
forloop.sh
For loop with a list
for planet in Mercury Venus Earth Mars Jupiter
do
echo $planet
done
bash-4.1$ ./forloop.sh
Mercury
Venus
Earth
Mars
Jupiter
forloop.sh
For loop with a string – quotes matter
for planet in "Mercury Venus Earth Mars Jupiter"
do
echo $planet
done
bash-4.1$ ./forloop.sh
Mercury Venus Earth Mars Jupiter
forloop.sh
For loop – iterations controlled by a variable
number=4
for num in $(seq 1 $number); do
echo $num;
done
bash-4.1$ ./forseq.sh
1
2
3
4
forseq.sh
While loop – iterates until a condition is false
while [ condition ]
do
echo "condition still true"
done
echo "condition is false"
While loop - read a file
while read line
do
echo $line
done <$filename
while.sh
Case
Note unmatched parentheses and double semicolons.
$var="Monday"
case $var in
"Monday") echo "New week, new possibilities!";;
"Friday") echo "TGIF!";;
*) echo "Hang in there!";;
esac
caseweek.sh
Functions
A code unit with defined input and output. A function
can be called from multiple places in a script.
Functions - syntax
myfunction () {
echo "This is a function!"
}
echo "Script execution starts here."
# function call
myfunction
echo "End of execution"
func.sh
Functions – variable scope
# by default, variables in functions are global
ftest () {
x=3
echo "x in function: $x"
}
x=2
echo "x in main: $x"
ftest
echo "x in main: $x"
bash fnctn.sh
x in main: 2
x in function: 3
x in main: 3 fnct.sh
Functions – local variables
# declare function vars as local to limit their scope
ftest () {
local x=3
echo ”x in function: $x”
}
x=2
echo ”main x: $x"
ftest
echo ”main x: $x”
bash-4.1$ ./funcvar.sh
main x: 2
x in function: 3
main x: 2 funcvar.sh
Functions – passing parameters
hello ()
{
echo "Function params: $@"
}
#function call
hello 1 2 3
bash-4.1$ ./funcposind.sh
Function params: 1 2 3
funcpos.sh
Functions – return values
A bash function can return an integer between 0 and 255 using the
“return” command that is essentially designed to use the Unix exit
status codes.
fhello (){
return 0;
}
fhello
echo "return is $?"
funcsimple.sh
Follow your script step-by-step
multiply () $ bash -x test.sh 3 3
{ + a=3
(( resval = a * b )) + b=3
echo $resval ++ multiply 3 3
} ++ (( resval = a * b ))
a=$1 ++ echo 9
b=$2 + prod=9
prod=`multiply $a $b` + echo 'return is 9'
echo "return is $prod” return is 9
Sources (in order of complexity)
• Research Computing Service tutorial:
https://wiki.imperial.ac.uk/display/HPC/Shell+scripting
• Steve Parker’s Bash tutorial:
http://steve-parker.org/sh/sh.shtml
• The Linux documentation project: http://tldp.org/
• The Bash hackers wiki: http://wiki.bash-hackers.org/
