The

Shell

Mats Tpel
Department of Marine Sciences
[email protected]
www.matstopel.se, @matstopel
 UNIX?

Terminal, shell, GNU/Linux, Mac OSX?

Ken Tompson & Dennis Rithcie

EvoluJon of UNIX-like systems
The UNIX system
The UNIX system

Kernel
The UNIX system

Shell

Kernel
The UNIX system
Webbrowser

Shell Perl
Mail

Kernel

Text-
Media editor
player

Terminal
(virtual) Terminal vs. shell vs. bash

Terminal window (Graphical

User Interface [GUI] that runs
the shell)
e.g. iTerm, Terminal, Konsole,
gnome-terminal
(virtual) Terminal vs. shell vs. bash

Terminal window (Graphical

User Interface [GUI] that runs
the shell)
e.g. iTerm, Terminal, Konsole,
gnome-terminal

The shell (a.k.a Comand Line

Interface [CLI]. Program that
starts other programs)
e.g. bash, sh, tcsh
 (virtual) Terminal vs. shell vs. bash

Terminal window (Graphical

Prompt User Interface [GUI] that runs
the shell)
e.g. iTerm, Terminal, Konsole,
gnome-terminal

The shell (a.k.a Comand Line

Interface [CLI]. Program that
starts other programs)
e.g. bash, sh, tcsh
 UNIX.
Why so successful?
 UNIX design strategy

Do one thing and do it good

Dont be chaZy

 UNIX design strategy

Do one thing and do it good

ls [List les and directories]
rm [Remove les]
mkdir [Create directories]
wc [Count words, lines and characters]
 UNIX design strategy

Do one thing and do it good

Dont be chaZy
ls [Only outputs the names of les and dirs]
rm [Will remove a le without any warning]

 UNIX design strategy

Do one thing and do it good

Dont be chaZy
ls [Only outputs the names of les and dirs]
rm [Will remove a le without any warning]

This behavior can be modied using opJons a.k.a. ags
ls
ls -l
 ls -l

File permissions (more on this later)

 ls -l

User and group that owns the le

 ls -l

File size in bytes

 ls -l

ModicaJon date
 ls -l

Filenames
 UNIX design strategy

Do one thing and do it good

Dont be chaZy
Redirect >, append >> and pipe |
 UNIX design strategy

Do one thing and do it good

Dont be chaZy
Redirect >, append >> and pipe |
> Output from a program can be saved in a new
le
UNIX design strategy

ls > my_le.txt
 UNIX design strategy

Do one thing and do it good

Dont be chaZy
Redirect >, append >> and pipe |
> Output from a program can be saved in a new
le
>> or appended to an exisJng le
 UNIX design strategy

ls -la >> my_le.txt

 UNIX design strategy

ls -la >> my_le.txt

(append to le)

(include all les, also those that are hidden)

 UNIX design strategy

cat my_le.txt
(this command will show the content of my_le.txt)
 UNIX design strategy

Do one thing and do it good

Dont be chaZy
Redirect >, append >> and pipe |
> Output from a program can be saved in a new
le
>> or appended to an exisJng le
| or sent to another program.
 UNIX design strategy

How many les?

UNIX design strategy

ls -a | wc -l
 UNIX design strategy

How many sequences

in a le?
 UNIX design strategy

The fasta le format

>otToc34
MSDEEYSEDEYVSSEGEGAEGRDVEPQAQQAAPVLGPKEWVGLSSMPAATQQALLETLGK
KANAAGEKVVNERPWLTDMVGTMVEMACAKESYEYDHKKAGKKLDPNNKYKIWMLPLFLF
QAFVLRPLMIGQIRRDIRKSDEDKKKKSARPAPKK
>cmToc34
MDSETPTKNEAEQRASPDRTGQVTQRPVSLFRKNGTFAYPLEDSDVYEEQVLALEDVPEP
TAQAILRLALAAERQGVPRLRFLVVGAPGVGKSSLINTLLNENLCSVSAWERGTKNAQVC
ARQVDSVVIEFIDTPGIAPCRRSGLEASRRQVQRLRKLLDARGADEHPYLRSFHAILYVM
RLDDTRPDLVDYHNWKVLMEFF
 UNIX design strategy

The fasta le format

>otToc34
MSDEEYSEDEYVSSEGEGAEGRDVEPQAQQAAPVLGPKEWVGLSSMPAATQQALLETLGK
KANAAGEKVVNERPWLTDMVGTMVEMACAKESYEYDHKKAGKKLDPNNKYKIWMLPLFLF
QAFVLRPLMIGQIRRDIRKSDEDKKKKSARPAPKK
>cmToc34
MDSETPTKNEAEQRASPDRTGQVTQRPVSLFRKNGTFAYPLEDSDVYEEQVLALEDVPEP
TAQAILRLALAAERQGVPRLRFLVVGAPGVGKSSLINTLLNENLCSVSAWERGTKNAQVC
ARQVDSVVIEFIDTPGIAPCRRSGLEASRRQVQRLRKLLDARGADEHPYLRSFHAILYVM
RLDDTRPDLVDYHNWKVLMEFF
 UNIX design strategy

grep > my_seq-le.fst | wc -l

 UNIX design strategy

grep > my_seq-le.fst | wc -l

grep -c > my_seq-le.fst
 File permissions

File permissions (more on this now)

File permissions

-rwxrw-r--
 File permissions

-rwxrw-r--
r = read
w = write
x = execute
 File permissions

User Everyone else

-rwxrw-r--
Group
r = read
w = write
x = execute
 File permissions

-rwxrw-r--
Type (le, link, directory etc.)
r = read
w = write
x = execute
 File permissions
File permissions can be changed using the
command chmod
The command

chmod +x my_le.txt

will make my_le.txt executable

 UNIX le system

usr bin lib home

local bin share

bin lib include

 UNIX le system

usr bin lib home

local bin share

bin lib include

/usr/local/bin
 UNIX le system

usr bin lib home

local bin share

bin lib include

/usr/local/lib
 UNIX le system

cd /usr/local/bin

cd Change Directory

 UNIX le system

usr bin lib home

local bin share mats alice

bin lib include

/home/mats = ~
 UNIX le system

usr bin lib home

local bin share mats alice

bin lib include tests

RelaJve path = tests/make clear make

 UNIX le system

cd tests/make

cd Change Directory

Punng it all together!

 WriJng a program
A program is a set of instrucJons collected in a
le
This le is executable
The le usually contains informaJon about
which programing language the instrucJons
are wriZen in (the shebang)
 how_many_les.sh
1. #!/bin/bash
2. ls -a | wc -l
 covgc_plot.sh
(by Alvar Almstedt)
#!/bin/bash

SMRT_JOB_NUM=$1
OUT_PREFIX=$2

echo "CreaJng symlink and copying assembly..."
ln -s /data01/smrtanalysis/userdata/jobs/016/$SMRT_JOB_NUM/data/aligned_reads.bam
cp /data01/smrtanalysis/userdata/jobs/016/$SMRT_JOB_NUM/data/polished_assembly.fasta.gz ./
echo "Symlink created and assembly copied..."

gunzip polished_assembly.fasta.gz
echo "Assembly gunzipped..."

echo "Commencing generaJon of coverage le..."
genomeCoverageBed -ibam aligned_reads.bam -d > $OUT_PREFIX.coverage
echo "Coverage le generated..."

echo "Cunng..."
cut -f 1,3 $OUT_PREFIX.coverage > cut_$OUT_PREFIX.coverage
echo "Cut made..."
echo "Launching conJg_average_coverage.py..."
conJg_average_coverage.py cut_$OUT_PREFIX.coverage > cut_$OUT_PREFIX.avg.coverage
echo "Average calculated..."

echo "DeleJng..."
rm $OUT_PREFIX.coverage
rm cut_$OUT_PREFIX.coverage
echo "Intermediary les removed..."

echo "Formanng fasta le headers..."
sed 's/|quiver//g' polished_assembly.fasta > temp && mv temp polished_assembly.fasta
echo "Headers formaZed..."

echo "Launching fasta_analyzer.py"
fasta_analyzer.py -cg polished_assembly.fasta cut_$OUT_PREFIX.avg.coverage
Thank you for your attention!
 Albiorix

Address: albiorix.bioenv.gu.se

Exercises:
hZps://github.com/The-BioinformaJcs-Group/Learning_Unix/
wiki/Learn-Unix-with-applicaJons-to-NGS-data
 Bash on Windows computers
I suggest you use the mobaxterm program. Go to

hZp://mobaxterm.mobatek.net/download-
home-ediJon.html

and download the portable ediJon.
 Suggested addiJonal tutorials
To start with:
Do tutorial One, Two, Three, Four, Five [up to 5.2] (and
opJonally Six) at
hZp://www.ee.surrey.ac.uk/Teaching/Unix/

Codecademy
hZps://www.codecademy.com/en/courses/learn-the-
command-line

For experienced users:
hZps://wikis.utexas.edu/display/bioiteam/Shell+Script