Introduction to Linux in HPC

By
Karuna Prasad
SSDG, CDAC-Bangalore
Agenda
● Background
● Linux Directory Structure
● Files
● Text Display and Search
● Users and Permissions
● Processes
● Shell scripting
● Environment variables
● Remote file transfer
Historical Background
- 1969: Unix (Bell Laboratories)
- Written in C
- Already a successor to Multics
- Over time, many variations
- 1990: POSIX (Portable Operating System Interface) Standard
- Interface that all unix systems implement
- Adopted by Unix-like systems (including Linux)
- Linux is an operating system that evolved from a kernel created by Linus
Torvalds when he was a student at the University of Helsinki.
- Open Source, GNU General Public License
- Linux is the most popular OS used in a Supercomputer
- Security and stability
- Flexible and scalable
OS used in top 500 Supercomputers
Linux is the most

popular OS used in a

Supercomputer
○ Security and stability
○ Flexible and scalable

Ref :https://en.wikipedia.org/wiki/Supercomputer_operating_system
Distribution of OS in Supercomputers

Ref- https://www.top500.org/statistics/list/
Architecture of Linux
● User space
○ The space in memory where user processes
run
○ This space is protected
○ The system prevents one process from
interfacing with another process
○ Only kernel processes can access a user
process
● Kernel space
○ The space in memory where kernel processes run
○ The user call has access to it only through the
system call
● System call
○ User space and kernel space are different spaces
○ When a system call is executed the arguments to
the call are passed from user space to kernel
space
○ The user process becomes a kernel process when
it executes a system call
Architecture of Linux
● Shell
○ It is an interface among the kernel
and user.
○ It can afford the services of kernel.
○ It can take commands through the
user and runs the functions of the
kernel.
○ The shell is available in distinct
types of OSes.
○ These operating systems are
categorized into two different
types, which are the graphical
shells and command-line shells.
Command Line Interface
● A line where you type commands
● Other terms:
○ CLI (command line interface)
○ Console / Terminal
○ Shell
● Advantages
○ Simple (nearly always works)
○ Easy to program (in comparison to GUI)
○ Fast and efficient to use (if you know the commands)
● Disadvantage: lots of memorizing (vs. GUI buttons)
Command Line Interface
Command Line Interface
Navigating the Command Line
● <Enter> run command
● <Up-Arrow> navigate command history back in time
● <Down-Arrow> navigate command history forward in time
● <Tab> auto-completion (a.k.a. “tab completion”)
○ If enough letters to identify command: completes command
○ More than one possibility: nothing shown
○ Second Tab to list possible completions
● <Ctrl-C> abort current command.
● Always case-sensitive
○ Popular source for errors
● Command line options:
○ Usually start with dash
○ Common convention: single dash for “short options”, double dash for “long options”
Navigating the Command Line
● Manual Page
○ Man <command Name>
● Built-in help
○ -h or –help option
○ Similar to man pages
● CLI no “UNDO” button
● No confirmation Dialog box
● Careful, if root can have destroy entire system
● Make sure you are in right directory
● Make sure you are not “root”, if not required
Directory Structure
● Directory tree structure different from Windows
○ No drive letters ( C:\ )
○ Top level identical on every Linux system
○ “Mounting points”: location of hard drive in tree structure
● “Path”: location inside file system
○ Example /home/karuna/SamplePrograms
○ Absolute path: the location of a file or directory from the root
directory(/) and provides the full path (starts with / )
○ Relative path: relative to current working directory
● Print working directory : pwd
Directory Structure
Directory Structure
Directory
● Linux - everything is a file. Directory is also a file and stores file names and
related information
○ /dev : Device files
○ /proc : System information files
● Every command is a program or script somewhere
○ which <commandname>
● Directories abbreviations
○ Current Directory: . (dot)
○ Parent directory: .. (2 dots)
○ Your home directory: ~ (tilde)
Navigating Directories
● Linux - everything is a file
● cd Command (change directory)
● Use : cd <path>
○ Path can be relative or absolute path
○ Absolute path
○ Relative Path
● Common mistake: cd.. (no space in between)
● ls Command
○ List directory contents
○ ls -l : ll(shortcut)
○ ls -a : show hidden files
○ ls -t : to sort by time modified
Files
● Linux: extensions do not matter
○ But: conventions to help humans
○ Some programs also look at extensions
● Most important: text file or not?
○ Configuration files
○ Scripts
○ System information files
● Binary file: generally not searchable
● Use file <filename> to identify file type
File Manipulation Commands
● Rename file/directory: mv <oldname> <newname> (move)
● Copy file/directory: cp <filename> <newname> (copy)
○ Needs -r for directories
●
● Create directory: mkdir <dirname>
●
● Create empty file: touch <filename>
●
● Remove fire/directory: rm <filename>
○ Check access permissions
○ Recursively delete sub-directories: rm -r
○ Common option: -f (force) - Never prompt for confirmation
● Select multiple files according to patterns
○ * zero or more characters
○ ? exactly one character
○ [ ] range of characters
Console specific commands
● <Ctrl-C> stop current command
● <Ctrl-Z> suspend current command
● <Ctrl-D> send “End-of-File” to application
○ Will usually quit console when on an empty command line
○ exit to Quit console
○ clear command to clear screen
● Paste selected text
■ NOT Ctrl-C / Ctrl-V, see below
Searching Files
● find command
● Syntax: find <targetdir> <options>
●
● Eg: find . -name "ex1.txt" -type f
●
● Allows very complex searches
○ Wildcards
○ Only files modified after X
● Allows executing command for every found file: -exec
● Ex: find command -name option
● $ find . -type f -name “*test*”
Console Input and Output
● Console has three main ways of
communicating with process
(streams)
○ Standard input ( stdin)
○ Standard output ( stdout )
○ Standard error ( stderr )
● stdin : What you type into
console
● stdout + stderr : what you
see in console
○ Separate stream so that error
messages are separated from
normal output
Console Input and Output
● Input / output streams can be redirected to
○ Other commands
○ Files
● Redirect stdout

command > filename

● Redirect stderr

command 2> filename

● Redirect stdin

command < filename

● Use output of one command as input to another : pipe symbol

command1 | command2
Redirection
● Append to file without overwriting
command >> filename
● Streams are numbered
○ 0 : stdin
○ 1 : stdout
○ 2 : stderr
● Examples
command > out.log 2> err.log
command 2>&1 > out_err.log
Text Display
● Different ways to display and edit text
● cat command
○ Outputs contents to the console
● less command
○ Allows going back and forth
○ Also used by man pages
● Others
○ head : display first lines
○ tail : display last lines
Searching File Contents
● Search using grep command
● Syntax: grep <options> <string> <filename>
○ Ex: grep -i -r “test” example*.txt
● Can use wildcards
● Options
○ -r Recursive ( include subdirectories)
○ -i Ignore upper / lower case
○ -I Ignore binary files (uppercase i)
● Situation: command with lot of text output and need to search something
inside output
● Solution: pipe output into grep
○ $ ll | grep - i test
○ ps -aef | grep mysql
Users and Permissions
● Linux is a multi-user system
○ Everyone should only be able to access own files
○ Others only see / changes what you want them to
○ Some files / directories should only be accessible to admins
● Everyone is logged in as specific user (account)
○ and every user has certain permissions
● Only admins can set permissions for others
File / Directory Permissions
● Each file and directory has certain permissions
○ Determines what you can do
● root user (superuser) can do everything
● Users can have temporary root permissions
● sudo <command>
● Users belong to groups, each user has a primary group
● Read : who can read the contents of file /directory
● Write : who can change contents of file / directory
● Execute :
○ File : who can execute file ( or program)
○ Directory : who can traverse directory
■ who can execute files inside but not see them
Example output of ls -l
Changing Permissions
● Modify owner / group (needs root ) :
● chown <newowner> <filename>
● chown <newowner> : <newgroup> <filename>
●
● Modify permissions
● chmod u+x <filename>
● u = User, g = Group, o = Other, a = All
● + or -
● r = Read , w = Write, x = Execute
Processes
● Process : running instance of a program
○ System
○ User
○ User (manually launched)
● Like windows
○ Equivalent to Task Manager: top
○ Short Overview: pstree
● Each process has an owner
○ Process can/can't do what owner can/can't do
● Each process has an ID number ( PID)
Top command output and Navigation
Top command output and Navigation
● Single-letter commands to navigate top
● u : filter processes from a specific user
● K : kill a specific process
● h : show help
● f : toggle displayed columns
● x : highlight current sort column
● <> : select column to sort
● R : Reverse sorting
● q : quit top
Process- Background and Foreground
● If you enter command, it runs in the shell
● Enter <command> & to start it in background
● Send foreground command to background by Ctrl-Z (pauses it) and typing
bg
● Bring to foreground with fg <jobID> - Jobid is different from process ID
● Can be displayed with jobs
VIM Text Editor
● Default Linux text editor: vi
● Usually: vim (vi improved)
● includes syntax highlighting
● Completely inside console
● Advantages:
○ Always available
○ Very fast once you know commands
● When in doubt which Mode you are in , use Esc key and u for undo
Test Editor
Common Vi Commands
● $ vi filename
● :w write(save) file
● :w filename -write as new filename
● :wq or :x or zz write (save) file and quit
● :q! Close file without saving

Cursor movement commands

● <arrow keys> move cursor in arrow direction

● h , j , k , l move cursor left, down, up, right
● $ Move to end of line
● gg Move cursor to first line
● G Move cursor to last line
● w Jump forward to next word
● b Jump backward to previous word
● % Jump to matching character (default pairs: () , {} , [] )
Common Vi Commands
Common Editing Commands

● u Undo last change

● <ctrl r> Redo last change
● . Repeat last command
● x Delete character
● dd Delete (cut) entire line
● yy or Y Yank (copy) entire line
● p Paste after cursor

Common commands Searching

● /pattern Forward search for regular expression

● ?pattern Backward search for regular expression
● n Repeat last search
● N Repeat last search in opposite direction
● %s/old/new/ Replace old pattern with new pattern on current line
● %s/old/new/g Replace old pattern with new pattern in entire file
Shell Scripts
● Interaction with Linux: just a series of commands
○ Commands can be put into a text file
○ Text file is fed to console
○ console run commands one after another
● Advantage: very easy automation
● Shell script : execute like a program
○ provide the "execute" permissions
● Command to run script
○ Full script name (including location)
○ eg ./scriptname.sh
● Why only scriptname?
○ Linux looks commands in specific folders
○ safety feature (not everyone can run everything)
● File needs execute permissions
○ another safety feature
○ Use chmod command ( chmod u+ x filename )
Example Shell Script
Variables
● Store output of commands
● Assignment via = (equal sign)
● var="value"
● note - no spaces around =
● Always text
● Quotes necessary when whitespace, special characters in value
●
● Retrieve with $ sign
● $ var
● Eg - echo $var prints value to screen
Variables
● Brackets and quotes in variables
○ single quotes: exact text
○ double quotes: variables will be expanded
○ parentheses ( round brackets) : command inside will be evaluated
● var="bla" will save text bla to var
● var='$bla' will save the text $bla to var
● var="$bla" will look for a variable named bla
● var=$(bla) will execute command bla and save its output to var
●
● Use command line arguments: $0 - $9, ${10}
● eg: script.sh - f 5.0
● here, $0-script.sh, $1=-f , $2=5.0
Environment Variables
● Environment- which variables are defined and available
○ To a process
○ Within a shell
● Avoids hard coding the data/ path/ information
● Eg: current home directory, HOME=/home/karuna
● Helpful to provide configuration scripts
● Change information in a single location
● Keep business logic apart from config
Environment Variables
● Many environment variables already defined
● By system (eg: "$USER")
● By installed software
●
● Command env to show all currently defined variables
● Usually in Capital letters
●
● Set the environment variables
● $ export MY_VAR="value"
● Available in child processes
Environment Modules
● Cluster:different environments for different people
○ Admins cannot predict who needs what
○ Different version of same software: collision of environment variables!
● Solution:make it easy to switch environments
○ Environment modules: sets of environment settings
○ Not limited to clusters
● Example: OpenMPI module(compiled with GCC)
● $ module load openmpi/gcc/64/1.10.3
● Contains at least three things
○ Description what module does
○ Prepend to path and other variables
○ Add new variables
● Usually prepends rather than appending
PATH variable
● Environment variable PATH
○ List of directories (seperated by : )
○ Console will look for command names
■ Command may be in multiple directories: first hit is used
○ Own commands: add directory to path
○ export PATH=$PATH:/path/to/file
● Core concept of operating system
● Also used by other software
○ Eg PYTHONPATH
File Transfer
● Copying files between cluster and local
● Use scp command (secure cp)
● $ scp [options] sourcehost:sourcefile targethost:targetfile
● Option -r for copying directories
● $scp -p portno filename username@hostname:/home/username
● $scp -p 4422 filename [email protected]:/home/tpw45
SSH Connection
● Clusters typically accessed via Secure Shell (SSH) protocol
● Most commonly OpenSSH software
● Available for all operating systems
○ Linux: original
○ Mac OS: basically identical
○ Windows 10 (since 2019): integrated in cmd/Powershell
● Additional tools, especially on Windows: Putty, MobaXTerm
● Connect with ssh command: ssh [options] <username>@<hostname>
Other Commands
● du - Shows disk usage
● Common options: -h (human-readable) -s (Show total), -c (Show
● individual files)
● Example: du -sch
● df - Disk free
● history - Lists previous commands (same as Up-Arrow/Down-Arrow)
● Text file in your home directory: ~/.bash_history
● Advantage: searchable
● Example: history | grep <commandname>
● When you forget what options you used