The command line for scientific

computing

Katerina Michalickova, November 11, 12 and 13, 2019

Graduate School

slides – http://tinyurl.com/ic-gs-nov11 and http://tinyurl.com/ic-gs-nov13

Outline

• Finding things
http://swcarpentry.github.io/shell-novice/07-find/index.html

• Nelle’s story continued

• Variables
• Tests
• Loops
• (Functions)
 Bash script

A shell script is a series of commands written in a plain text

file. When a script is executed, all commands are
performed in a sequence allowing a user to accomplish
multiple or repetitive tasks without much typing. Shell
scripts utilise variables, control statements, loops and
functions. These building block can turn a simple script
into a program in its own right.
 Bash script

Shell scripting can be used for:

• Managing big data

• Setting up workflows
• System management
• Deploying programs on High Performance Computing
systems
A minimal example

#!/bin/bash

# My first script

echo "This is my first script!"

first.sh
Executing a script

bash-4.1$ cat first.sh

#!/bin/bash
# My first script
echo "This is my first script!"

bash-4.1$
bash-4.1$
bash-4.1$ bash first.sh
This is my first script!
bash-4.1$
bash-4.1$
bash-4.1$ chmod a+x first.sh
bash-4.1$
bash-4.1$ ./first.sh
This is my first script!
bash-4.1$
 Scripting in Bash shell

Shell scripts utilise the same basic building blocks as other

scripting languages:

• Variables (represent values and make scripts more

versatile)
• Instruction flow control statements
• Conditionals (if this is true, do this; otherwise do this)
• Loops (do this repeatedly)
• Functions

The above can be combined with shell commands to

create a powerful tool that can help us do more in less
time and with less pain.
Variables - user-defined

#!/bin/bash

# My first script with a variable

MESSAGE=”Hello world!”

echo $MESSAGE

varfirst.sh
Variables - user-defined

#!/bin/bash

# My first script with a variable

GREETING=Hello
NAME=Bob
MESSAGE=”$GREETING $NAME”

echo $MESSAGE
varfirst.sh
Variables – assign an output of a command

#!/bin/bash

# date is a command

VAR=$(date)
echo "today is $VAR"

varfirst.sh
 Variable declaration

Rules for declaring variables:

• no spaces are allowed around the the equal sign in the

variable declaration line
• variable names have to start with a letter
• no spaces are allowed in the variable name
• no punctuation marks are allowed in the name
(underscore is ok)

Shell variables are not typed, essentially, they are

strings. Arithmetic operations are permitted on variables
containing only digits. No floating point operations by default.
Variables - dollar sign and curly brackets

• Variable appears without “$” at assignment

• Otherwise preceded by $ (with some exceptions*)

• $MESSAGE is the same as ${MESSAGE}

• Curly brackets are often skipped

*unset and arithmetic with (( ))

Variables - system-defined

• Environment variables
• e.g. $PATH, $SHELL, $USER, $HOME

• Positional input parameters

• $@, $1, $2, ${10}, $#

• Exit value
• $?
Variable type

• All shell variables are considered text (not numbers).

• Text containing white space have to be quoted inside the

scripts.
MESSAGE="This is my first script!"

• Some text characters have special meaning

• $ denotes a variable
• ` evokes a command execution
• " closes initial double quote
• \ means ignore special meaning of the next character
Single and double quotes

• The choice of single or double quotes determines if

special character will be interpreted.

• Single quotes do not allow interpretation of special

characters.

• Special characters enclosed in double quotes will be

interpreted (unless they are preceded with \).

• It is possible to mix single and double quotes in one

statement.
 Quoting and escaping

VAR=foo
echo $VAR
foo

• variable recognised in double quotes (variable expansion)

echo "$VAR"
foo

• $ variable not expanded in single quotes

echo '$VAR'
$VAR

• if escaped with \ in double quotes, $ is interpreted as $

echo "\$VAR"
$VAR
Variable scope

• Variables declared inside the script exist only during

execution of the script.
• If there is a need to keep a variable set after the script
finished, the script can be "sourced”.

• Correspondingly, a variable that is instantiated in the

shell is not propagated to the script (as this runs in its
own shell).
• Variables can be “exported” in the shell to be inherited
by the script.
 Sourcing a script

bash-4.1$ cat varfirst.sh bash-4.1$./varfirst.sh

#!/bin/bash This is my first script!
# My first script bash-4.1$
MESSAGE="This is my first bash-4.1$ echo $MESSAGE
script!"
echo $MESSAGE bash-4.1$
bash-4.1$ bash-4.1$ source first.sh
This is my first script!
bash-4.1$
bash-4.1$
bash-4.1$ echo $MESSAGE
This is my first script!
bash-4.1$
Math with Bash

bash-4.1$ a=1
bash-4.1$ b=2
bash-4.1$ ((c=a+b))
bash-4.1$ echo $c
3
bash-4.1$ echo $((b-a))
1
bash-4.1$ echo $((b*a))
2
bash-4.1$ echo $((a/b))
0
bash-4.1$ echo $((b**3))
8
bash-4.1$
Control flow statements

Control flow determines the order of statements during

the execution.

• if
• test
• loops
• case
If statement

if [ expression ]; then
echo "it is true"
else
echo "it is false"
fi

ifelse.sh
If + elif statements

if [ expression1 ]; then
# if code
elif [ expression2 ]; then
# elif code
else
# else code
fi
Logical expression

• [ $var1 = $var2 ] evaluates to true or false

• it is equivalent to test $var1 = $var2

• “test” is a program, $var1 = $var2 are arguments to the

program

• $var1 = $var2 without spaces (inside and around)

would be impossible for the test program to read
Logical expressions - comparing text

$var1 = $var2 var1 is equal to var2

$var1 != $var2 var1 is not equal to var2

Logical expressions – comparing numbers

-eq is equal to
-ne is not equal to
-lt is less than
-le is less than or equal to
-gt is grater than
-ge is greater than or equal to
Logical expressions – logical comparison

! expression logical NOT

expression1 -a expression2 logical AND

expression1 -o expression2 logical OR

Logical expressions – file tests

-e file exists [ -e file.txt ]

-f is a regular file (not a directory)
-s has non-zero size
-d is a directory
-w has write permissions
-r has read permissions
-x is executable
Loops

Repeat a block of code

The number of repetitions is controlled by a condition
For loop

for i in {1..4}
do
echo "Iteration $i."
done

bash-4.1$ ./forloop.sh
Iteration 1.
Iteration 2.
Iteration 3.
Iteration 4.
forloop.sh
For loop with a list

for planet in Mercury Venus Earth Mars Jupiter

do
echo $planet
done

bash-4.1$ ./forloop.sh
Mercury
Venus
Earth
Mars
Jupiter

forloop.sh
For loop with a string – quotes matter

for planet in "Mercury Venus Earth Mars Jupiter"

do
echo $planet
done

bash-4.1$ ./forloop.sh
Mercury Venus Earth Mars Jupiter

forloop.sh
For loop – iterations controlled by a variable

number=4
for num in $(seq 1 $number); do
echo $num;
done

bash-4.1$ ./forseq.sh
1
2
3
4
forseq.sh
While loop – iterates until a condition is false

while [ condition ]
do
echo "condition still true"
done
echo "condition is false"
While loop - read a file

while read line

do
echo $line
done <$filename

while.sh
Case

Note unmatched parentheses and double semicolons.

$var="Monday"
case $var in
"Monday") echo "New week, new possibilities!";;
"Friday") echo "TGIF!";;
*) echo "Hang in there!";;
esac

caseweek.sh
Functions

A code unit with defined input and output. A function

can be called from multiple places in a script.
Functions - syntax

myfunction () {
echo "This is a function!"
}

echo "Script execution starts here."

# function call
myfunction
echo "End of execution"

func.sh
Functions – variable scope

# by default, variables in functions are global

ftest () {
x=3
echo "x in function: $x"
}

x=2
echo "x in main: $x"
ftest
echo "x in main: $x"

bash fnctn.sh
x in main: 2
x in function: 3
x in main: 3 fnct.sh
Functions – local variables
# declare function vars as local to limit their scope

ftest () {
local x=3
echo ”x in function: $x”
}

x=2
echo ”main x: $x"
ftest
echo ”main x: $x”

bash-4.1$ ./funcvar.sh
main x: 2
x in function: 3
main x: 2 funcvar.sh
Functions – passing parameters

hello ()
{
echo "Function params: $@"
}

#function call
hello 1 2 3

bash-4.1$ ./funcposind.sh
Function params: 1 2 3

funcpos.sh
Functions – return values

A bash function can return an integer between 0 and 255 using the
“return” command that is essentially designed to use the Unix exit
status codes.

fhello (){
return 0;
}

fhello
echo "return is $?"

funcsimple.sh
 Follow your script step-by-step

multiply () $ bash -x test.sh 3 3

{ + a=3
(( resval = a * b )) + b=3
echo $resval ++ multiply 3 3
} ++ (( resval = a * b ))
a=$1 ++ echo 9
b=$2 + prod=9
prod=`multiply $a $b` + echo 'return is 9'
echo "return is $prod” return is 9
 Sources (in order of complexity)

• Research Computing Service tutorial:

https://wiki.imperial.ac.uk/display/HPC/Shell+scripting
• Steve Parker’s Bash tutorial:
http://steve-parker.org/sh/sh.shtml
• The Linux documentation project: http://tldp.org/
• The Bash hackers wiki: http://wiki.bash-hackers.org/