The Linux command line for scientific computing

November 11,12 and 13, 2019

Katerina Michalickova
[email protected]
Senior Teaching Fellow, The Graduate School

1
Setup
Wi-fi
If using a laptop, make sure you are on the Imperial-WPA wi-fi.

Command line on Mac and Linux

Choose either A or B.

A. Using the Terminal application on your laptop

Open the Terminal application and you are ready to go.

B. Using a remote system

Open the Terminal and type (replace “username” with your Imperial login name):
ssh [email protected]

You will be prompted for your password. When you type your password, nothing
will appear on the screen. That is normal, just type carefully and press enter.

Command line on Windows laptops and classroom PCs

Choose either A, B or C.

A. Using Git for Windows in Software Hub

1. Download Git (https://git-scm.com/downloads) or open the Software Hub and
launch Git; this will open the command line interface
2. Use the following commands to navigate to your desktop (replace “username”
with your Imperial username)

cd /c/Users/username/Desktop

B. Using a remote system through your browser

1. Go to http://selfservice.rcs.imperial.ac.uk/
2. Select the leftmost tab “Interactive Services” -> “Jupyter”
3. Sign in with your College credentials
4. Select “1 core, 8 GB, 8 hours” and “Spawn”
5. Select “Terminal”

C. Using a remote system via Putty ssh client

2
 1. Download Putty from
https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html (choose the
64-bit putty.exe binary) or start Putty through the Software Hub.
2. In Putty, put the following text into the Host Name box (replace “username”
with your Imperial username):
[email protected]

You will be prompted for your password. When you type your password,
nothing will appear on the screen. That is normal, just type carefully and press
enter.

Exercise files (any platform)

• If you are not using login.hpc.ic.ac.uk, download the file “data-shell.zip” from
http://swcarpentry.github.io/shell-novice/setup.html and place it on your Desktop.
Unzip the file, and navigate to the data-shell directory using the command line.

cd data-shell

• If you are using login.hpc.ic.ac.uk, type the following on the command line (press
Enter at the end of each line). This will download the files needed for today and put
you in the right directory.

cd
module load gs_setup
gs_setup
cd data-shell

3
Part 1 – Oct 1, 2019
Why the Linux command line

Computing has become an integral part of science and the great majority of powerful
computational tools require the command line environment.
Often, the command line is the only way to interact with a remote system such as a central
computing resource.

What is the Linux command line

The command line is a text-based interface that stands between you and the Linux (or Mac)
operating system. The command line itself is a program that is used to interpret other
programs.
The command line is used by typing a command followed by Enter (or Return). The
command line executes the command, prints the output and waits for another command.

The flavour of the command line in the figure below is called “bash”. The $ sign is called the
prompt.

Teaching materials

Topics 1-3 from the Software Carpentry Unix Shell lesson -

http://swcarpentry.github.io/shell-novice/

1. Introducing the shell

2. Navigating files and directories
3. Working with files and directories

4
Part 2 – Oct 2, 2019

Review exercise
1. Change directory to data-shell/creatures

2. You’ve been working on the basilisk project and you managed to sequence a bit
more DNA from the creature. You are now going to enter this data.

3. Make a directory called backup in data-shell/creatures

4. Copy the original file basilisk.dat into backup under the name basilisk_original.dat

5. Edit basilisk.dat in data-shell/creatures:

a. add the following sequence to the end “TGAACCCGAC”
b. change the date of the last update to today’s date
c. save the file and exit

Teaching materials

Topics 4-7 from the Software Carpentry Unix Shell lesson -

http://swcarpentry.github.io/shell-novice/

4. Pipes and filters

5. Loops
6. Shell scripts
7. Finding things

5
Part 3 – Oct 4, 2019

Shell scripts

A shell script is a series of commands written in a plain text file. When a script is executed,
all commands are performed in a sequence allowing a user to accomplish multiple or
repetitive tasks without much typing. Shell scripts utilise variables, control statements,
loops and functions. These building block can turn a simple script into a program in its own
right.

Shell scripts can be used for managing big data, setting up workflows, system management,
deploying programs on High Performance Computing systems etc.

Shell scripts utilise the same basic building blocks as other scripting languages:
• Variables (represent values and make scripts more versatile)
• Instruction flow control statements
• Conditionals (if this is true, do this; otherwise do this)
• Loops (do this repeatedly)
• Functions (reusable blocks of code)

The above can be combined with shell commands to create a powerful tool that can help us
do more in less time and with less pain.

Syllabus:
• Little Women (http://swcarpentry.github.io/shell-novice/07-find/index.html)
• Nelle’s story
• Variables
• Tests
• Loops
• Functions

Slides can be downloaded from: http://tinyurl.com/ic-gs-commline-slides

The script examples can be downloaded by typing:

git clone https://github.com/kmichali/gs_shell_scripts.git

6
Commands and concepts
Connecting to a remote Linux computer:
To connect from your laptop or a workstation, use “ssh username@hostname”. This gives
you a simple window with the command line interface to the remote server.

File and directory management:

• pwd - print path to the current directory
• cd - change directory
• mkdir – make a new directory
• ls - list directory content
• cp - copy file/directory
• mv - move file/directory
• mkdir - remove empty directory
• rm - remove file/directory
• rmdir - remove an empty directory

Commands for viewing and editing files:

• cat - output contents of a file
• less - page though contents of a file
• wc - count lines, characters and bytes in a file
• head - show lines from the beginning of a file (tail shows the end of the file)
• nano - simple text editor
• touch - make an empty file

Commands for file operations:

• sort - sort content of a file
• uniq - remove adjacent duplicated entries
• grep - find search string in a file
• cut - extract a column from a file

Path
The path specifies a position in the directory tree, it is a sequence of directory names
separated by a forward slash.

• Absolute path always starts at at the top of the tree (at the root directory), for
example “/home/username”. The absolute path always starts with a slash “/” (root)
that is the head of the filesystem.
• The relative path is the path to the destination from your current position and it
does not start with a “/”, e.g. "../home/otheruser" (the double dot means one
directory level up).

7
Directory and file names:
Do not use spaces, only letters, numbers, underscore, dash and period. Never start with a
dash. The names are case sensitive.

Flags:
Most of shell commands can be used with flags (or arguments) that alter their behaviour.
The flags are preceded by a dash, e.g. ls –aF.

Help:
To get a complete list of flags for a command, use “command --help” or “man command”.

Output forwarding:
Unless specified otherwise, the shell commands show output onto the screen. To redirect
the output to a file, use the following: command > file_name

Pipe:
Pipe is used to forward output of one command directly to the next: command1 |
command2

Shortcuts:
^C – abort process on the command line
q – get out of “man” or “less” mode
^D – log out of the shell
^A – go to the start of the line
^E – go to the end of the line

8
 Command line reference
Description Command Examples Choice Flags
Find your current location- pwd pwd
print working directory
List files/directories ls ls -l -l long form
ls /home/user -t sort by time of
ls ../ modification
ls *.txt (list all files ending -S sort by size
with “.txt”)
ll (ls –l)
Make a new directory mkdir mkdir projectx -p make all parent directories
mkdir /home/user/tutorial if needed
Change directory cd cd or cd ~(change home)
cd – (change back to last)
cd projectx or cd ./projectx
cd /home/user/project
cd ../work (change one level
up and then to work dir)
Copy file cp cp afile bfile -a recursive, keeps structure
cp afile projectx/ and attributes
cp –a projectx/ ../work
Move file/directory mv mv myfile yourfile -n do not overwrite
mv projectx
/home/user/work
Remove file/directory rm rm afile -i always ask before deleting
rm ../projectb/bfile -f force deletion
rm –r mydir -r recursive delete
Print file content onto a cat cat afile
screen cat afile bfile
Page through a file more more myfile
Show top of a file head head ./work/myfile -n number of lines to show
head –n 10 afile
Show end of a file tail tail –n 20 bfile -n number of lines
Find a pattern in a file grep grep “my pattern” file.txt -i case insensitive
-c count occurences
-l show only file names
-B10 show 10 lines before
-A5 show 5 lines after
--colour colour the pattern
Show help page man man ls
Edit a file nano nano myfile ^O save
^W search
^X exit

9
Resources
RCS Bash: https://wiki.imperial.ac.uk/display/HPC/Command+line
RCS Shell scripting: https://wiki.imperial.ac.uk/display/HPC/Shell+scripting
Unix tutorial: http://www.ee.surrey.ac.uk/Teaching/Unix/index.html
Advanced materials: http://tldp.org/guides.html
Have fun while you learn with this exercise: https://github.com/veltman/clmystery
The murder mystery (created by Noah Veltman) presents you with a few clues and a lot of
data that hide snippets of information needed to solve a crime. You will need a few basic
commands to sift through the data to unravel the mystery. Hint - grep and pipe will come
handy.

10