HPC Linux Spring 2024
HPC Linux Spring 2024
HPC Linux Spring 2024
Feng Chen
HPC User Services
LSU HPC & LONI
[email protected]
• File editing
Introduction to Linux 2
Why Linux for HPC
OS Family System Share
Introduction to Linux 3
History of Linux (1)
▪ Unix was initially designed and implemented at AT&T Bell Labs
1969 by Ken Thompson, Dennis Ritchie, Douglas McIlroy, and
Joe Ossanna
Introduction to Linux 4
History of Linux (2)
Introduction to Linux 5
History of Linux (3)
Introduction to Linux 6
What is Linux
▪ Essential components: Linux kernel + GNU system utilities +
installation scripts + management utilities etc.
Introduction to Linux 7
Redhat Desktop
Introduction to Linux 8
Ubuntu Desktop
Introduction to Linux 9
CentOS GNOME Desktop
Introduction to Linux 10
Debian MATE Desktop
Introduction to Linux 11
Linux System Architecture
Introduction to Linux 12
Linux Kernel
What is a Kernel
Introduction to Linux 13
Linux Shell
What is a Shell
Introduction to Linux 14
Shell Comparison
*: not by default
http://www.cis.rit.edu/class/simg211/unixintro/Shell.html
Introduction to Linux 15
Shell Comparison
*: not by default
http://www.cis.rit.edu/class/simg211/unixintro/Shell.html
Introduction to Linux 16
Linux Applications
▪ GNU compilers, e.g., gcc, gfortran
▪ OpenOffice
▪ parallel
▪ wget
▪ cat, ls, cp
….
https://directory.fsf.org/wiki/GNU
Introduction to Linux 17
What can you do with a shell?
▪ Check the current shell
▪ echo $SHELL
▪ List available shells on the system
▪ cat /etc/shells
▪ Change to another shell
▪ exec sh
▪ Date and time
▪ date
▪ wget: get online files
▪ wget https://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.gz
▪ Compile and run applications
▪ gcc hello.c –o hello
▪ ./hello
Introduction to Linux 18
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 19
Files
▪ One of the defining features of Linux and other UNIX-like
operating systems is that “everything is a file.”
Introduction to Linux 20
File Directory Structure
❖ All files are arranged in directories.
❖ These directores are organized into the file system
bin boot dev etc home lib mnt tmp usr var work
user1 user2 ··
user1 user2 ·· ·
Desktop ·
bin lib local include share
Documents
Downloads
compilers packages ··
Public ·
·· Intel amber
· GNU python
·· · ··
·
Introduction to Linux 21
Important Directories
contains files that are essential for system operation, available for use by all
/bin
users.
contains libraries that are essential for system operation, available for use by
/lib,/lib64
all users.
/var used to store files which change frequently (system level not user level)
/etc contains various system configurations
/dev contains various devices such as hard disk, CD-ROM drive etc
/sbin same as bin but only accessible by root
/tmp temporary file storage
/boot contains bootable kernel and bootloader
/usr contains user documentations, binaries, libraries etc
contains home directories of all users. This is the directory where you are at
/home
when you login to a Linux/UNIX system.
Introduction to Linux 22
File Path
Definition: position/address in the directory tree
▪ Absolute path
▪ Uniquely defined and does NOT depend on the current path
▪ Start with "/"
▪ E.g., /tmp is unique
▪ Relative path
▪ Depend on the current location in the directory tree
▪ Does not start with "/"
▪ . is the current working directory
▪ .. is one directory up
▪ E.g., ../tmp is not unique
Introduction to Linux 20
Linux is Case Sensitive
▪ All names are case sensitive
▪ Commands, variables, files etc.
Introduction to Linux 24
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 25
Basic Commands
▪ Command format: command_name [options] arguments
ls –l /home/$USER
▪ Command prompt: a sequence of characters used in a command
line interface to indicate the readiness to accept commands
▪ Prompt user to take action
▪ A prompt usually ends with one of the characters $,%#,:,>
and often includes information such as user name and the
current working directory
▪ The format can be changed via $PS1
Introduction to Linux 26
Get More Information
▪ Man: show the manual for a command or program
▪ The manual shows how to use the command and list the
different options and arguments
▪ Usage: man <command name>
▪ Example: man ls
▪ Apropos: show all of the man pages that may be relevant to a
certain command or topic
▪ Usage: apropos <string>
▪ Example: apropos editor
▪ Info: Information of documents
▪ info ls
Introduction to Linux 27
Commands: pwd and cd
▪ pwd:print the current working directory
▪ Usage: pwd
▪ Example: pwd
▪ cd :allow one to change the current working directory
▪ Usage: cd [destination]
▪ Example: cd /tmp
▪ The default destination is the home directory if [destination]
is omitted
▪ ~ stands for home directory (bash)
Introduction to Linux 28
Commands: ls
▪ ls command list the contents of a directory
▪ Usage: ls <options> <path>
▪ Example: ls
▪ The default path will be current working directory if path is
omitted.
▪ Options
▪ -l: show long listing format
▪ -a: (--all) show hidden files(name starts with an “.” is hidden)
▪ -r: reverse order when sorting
▪ -t: show modification times
▪ -h: (--human-readable) use file sized in SI units (bytes, kbytes,
megabytes etc.)
▪ -d: (--directory) list directory entries instead of contents, and do not
dereference symbolic links
Introduction to Linux 29
Commands: cat, more/less, head/tail
▪ Display the content of a file to screen
▪ cat: show content of a file
▪ more: display contents one page at a time
▪ less: display contents one page at a time, and allow
forward/backward scrolling
▪ Usage: cat/more/less <options> <filename>
Introduction to Linux 30
Auto-completion
▪ Allows automatic completion of typing file, directory or command
name via the TAB key
▪ Convenient, also error-proof
▪ If there is no unique name, all matching names will show
Introduction to Linux 31
Wildcards
Linux allows the use of wildcards for strings
Introduction to Linux 32
Commands: mkdir
▪ mkdir:create a directory
▪ Usage: mkdir <options> <path>
▪ Example: mkdir ~/testdir
▪ By default, the directory is created in the current directory
▪ Options
-p: create the target directory as well as any directories that
appear in the path but does not exist
Introduction to Linux 33
Commands: cp
▪ cp:copy a file or directory
▪ Usage: cp <options> <sources> <destination>
▪ Example: cp $HOME/.bashrc ~/testdir
▪ Options
▪ -r: copy recursively, required when copying directories.
▪ -i: prompt if file exists on destination and can be copied over.
▪ -p: preserve file access times, ownership etc.
Introduction to Linux 34
Commands: rm
▪ rm: removes files and directories
▪ Usage: rm <options> <list of files/dirs>
▪ Examples: rm testdir/.bashrc ~/testfile
▪ Options
▪ -r: remove recursively, required when deleting directories
▪ -i: prompt if the file really needs to be deleted
▪ -f: force remove (override the -i option)
Introduction to Linux 40
Commands: mv
▪ mv:moves or renames a file or directory
▪ Usage: mv <options> <sources> <dest>
▪ Example: mv test test1
▪ Use the –i option to prompt if a file or directory will be overwritten.
▪ If there are more than one source files, the destination must be a
directory
Introduction to Linux 36
Commands: alias
▪ alias: create a shortcut to another command or name to execute a
long string
▪ Usage
▪ bash/sh/ksh: alias <name>=“<actual command>”
▪ csh/tcsh: alias <name> “<actual command>”
▪ Example
▪ bash/sh/ksh: alias lla=“ls –altr”
▪ csh/tcsh: alias lls “ls –altr”
Introduction to Linux 37
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 38
File Permission
Introduction to Linux 39
Linux File Permission
▪ Designed as the multiuser environment, the access restriction
of files to other users on the system is embedded.
▪ Three types of file permission
▪ Read (r)
▪ Write (w)
▪ Execute (x)
▪ Three types of user
▪ User (u) (owner)
▪ Group (g) (group members)
▪ World (o) (everyone else on the system)
Introduction to Linux 40
Linux File Permission
• Owner permissions: determine what actions the owner of the file can
perform on a file
• Group permissions: determine what actions a user, who is a member
of the group that a file belongs to, can perform on a file
• Other (world) permissions: indicate what action all other users can
perform on a file
Introduction to Linux 41
File Permission
Introduction to Linux 42
File Permission
Introduction to Linux 43
Changing File Permission
▪ chmod is a *NIX command to change permissions on a file
▪ Usage: chmod <option> <permissions> <file or directory name>
▪ –R: change permission recursively in a directory
▪ chmod in Symbolic Mode:
Chmod operator Description
Introduction to Linux 44
Changing File Permission
▪ chmod in Absolute Mode:
0 No permission ---
Introduction to Linux 45
Permission Effect on File vs Directory
Introduction to Linux 46
User Groups at HPC/LONI
▪ Users are organized into groups
▪ groups command to find your group membership
Introduction to Linux 47
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 48
Variables
▪ Linux allows the use of variables
▪ Similar to programming languages
▪ Number, character or string
▪ Rules for variable names
▪ Must start with a letter or underscore
▪ Number can be used anywhere else
▪ Do not use special characters such as @,#,%,$
▪ Case sensitive
▪ Allowed: VARIABLE, VAR1234able, var_name, _VAR
▪ Not allowed: 1var, %name, $myvar, var@NAME
▪ Two types of variables:
▪ Global variables (ENVIRONMENT variables)
▪ Local variables (user defined variables)
Introduction to Linux 49
Global Variables
Introduction to Linux 50
List of Some Environment Variables
PATH is an environmental variable in Linux and other Unix-like operating
systems that tells the shell which directories to search for executable files (i.e.,
PATH
ready-to-run programs) in response to commands issued by a user. It is widely
considered to be the single most important environmental variable.
LD_LIBRARY_PATH colon-separated set of directories where libraries should be searched for first
HOME indicate where a user's home directory is located in the file system.
TERM specifies the type of computer terminal or terminal emulator being used
Introduction to Linux 51
Editing Variables
▪ Assign values to variables
Introduction to Linux 52
Editing Variables at the curent login
Introduction to Linux 53
Persistent variables for each login
▪ Not to change at each login
Introduction to Linux 54
Input & Output Commands
The basis I/O statements:
▪ echo: display info to screen
▪ The echo arguments command will print arguments
to screen or standard output, where arguments can
be a single or multiple variables, string or numbers
▪ read: reading input from screen/keyboard/prompt
▪ The read statement takes all characters typed until
the Enter key is pressed
▪ Usage: read <variable name>
▪ Example: read name
Introduction to Linux 55
Input & Output Commands
▪ Examples
▪ echo “hello !”
▪ hello
▪ By default, echo eliminates redundant whitespaces (multiple
spaces and tabs) and replaces it with a single whitespace
between arguments.
▪ To include redundant whitespace, enclose the arguments
within single/double quotes
▪ To expand variables, enclose the arguments with double
quotes
Introduction to Linux 56
Other Useful Commands
passwd Change password (does not work on LSU HPC and LONI systems)
chsh Change default shell (does not work on LSU HPC and LONI systems)
df Report disk space usage by filesystem
Estimate file space usage - space used under a particular directory or files
du
on a file system.
sudo Run command as root (only if you have access)
mount Mount file system (root only)
umount Unmount file system (root only)
shutdown Reboot or turn off machine (root only)
top Produces an ordered list of running processes
free Display amount of free and used memory in the system
find Find a file
alias enables replacement of a word by another string
Introduction to Linux 57
Other Useful Commands
vi Edit a file using VI/VIM
emacs Edit a file using Emacs
file Determine file type
wc Count words, lines and characters in a file wc -l file
grep Find patterns in a file grep alias file
awk File processing and report generating awk ’{print $1}’ file
sed Stream Editor sed ’s/home/HOME/g’ file
set manipulate environment variables set -o emacs
touch change file timestamps or create file if not present
date display or set date and time
which Location of a command
Introduction to Linux 58
Other Useful Commands
ln Link a file to another file ln -s file1 file2
wait Wait for each specified process and return its termination status.
which Shows the full path of (shell) commands
who Show who is logged on
whoami Print effective userid
finger User information lookup program
whatis Display manual page descriptions
history Display the command history list with line numbers. An argument
of n lists only the last n lines.
Introduction to Linux 59
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 60
Login Remote Systems
▪ Most Linux systems allocate secure shell connections from
other systems
▪ Log in using the ssh command to the LSU HPC and LONI clusters
▪ Usage: ssh <username>@<remote host name>
▪ Example: ssh [email protected]
▪ –X option: forward the display of an application
▪ The default port is 22 for ssh
▪ ssh –p <port number> <username>@<hostname>
Introduction to Linux 61
File Transfer between Two Systems
▪ scp : copy files between two hosts over the ssh protocol
▪ Usage:
▪ scp <options> <user>@<host>:/path/to/source
<user>@<host>:/path/to/destination
Introduction to Linux 62
File Transfer between Two Systems
▪ rsync is another utility for file transferring
▪ Usage: rsync <options> <source> <destination>
▪ Delta-transfer algorithm
▪ Only transfer the bits that are different between source and
destination
▪ Widely used for backups and mirroring as an improved copy
command for everyday use
▪ Command options
▪ -a: archive mode
▪ -r: recursive mode
▪ -v: increase verbosity
▪ -z: compress files during transfer
▪ -u: skip files that are newer on the receiver
▪ -t: preserve modification times
Introduction to Linux 63
Compressing and Archiving a single file
Reduce storage usage or bandwidth while transferring files.
Introduction to Linux 64
Compressing and Archiving Files
▪ tar: create and manipulate streaming archived files.
▪ Usage: tar <options> <file> <path>
▪ <file> tar archived file, usually with extension .tar
▪ <path> files/directories being archived
▪ Common options
▪ -c: create/compress an archive file
▪ -x: extract/decompress an archive file
▪ -t: list contents of archive (for testing)
▪ -z: filter the archive through gzip
▪ -j: filter the archive through bzip2
▪ -f: archive
▪ -v: verbosely list files processed
Introduction to Linux 65
Compressing and Archiving Files
tar: create and manipulate streaming archived files.
Examples:
▪ File compressing
▪ tar czvf file.tgz ${HOME}/*
▪ tar cjvf file.tgz2 ${HOME}/*
▪ File decompressing
▪ tar xzvf file.tgz –C [dest]
▪ tar xjvf file.tgz2 –C [dest]
• File listing
• tar tvf file.tgz
Introduction to Linux 66
Pipes
▪ Pipe commands: connect two or more commands together
using “|”
▪ grep: searches certain patterns from a file(s)
cat file | grep [option] pattern
Option Description
-v Print all lines not match pattern
-n Print the matched line and line number
-l Print only the names of files with matching pattern
-i Match either upper- or lowercase.
-c Print the count of matching lines
Introduction to Linux 67
Pipes: sort, wc, more, less
▪ ls |wc
▪ cat file | more
▪ Cat file | less
Introduction to Linux 68
I/O Redirection
▪ Three file descriptors for I/O streams (everything is a file in Linux)
Introduction to Linux 69
I/O Redirection Examples
▪ Write STDOUT to file:
▪ ls –l xxx > ls.out
▪ Write STDERR to file:
▪ ls xxx 2 > ls.err
▪ Write STDERR and STDOUT to file:
▪ ls –l xxx >output 2>&1
▪ Discard STDOUT and STDERR:
command > /dev/null 2>&1
Introduction to Linux 70
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 71
Processes
Introduction to Linux 72
Processes and Jobs
Introduction to Linux 73
Managing Processes and Jobs
Introduction to Linux 74
Roadmap
• What is Linux
• Linux file system
• Basic commands
• File permissions
• Variables
• Use HPC clusters
• Processes and jobs
• File editing
Introduction to Linux 75
File Editing
▪ Most commonly used editors on Linux/Unix systems
▪ nano (if you only have slight edit requirements)
▪ vi or vim (vi improved)
▪ Emacs
▪ vi/vim is installed by default on Linux/Unix systems and has
only a command line interface (CLI).
▪ Emacs has both a command line interface (CLI) and a
graphical user interface (GUI).
▪ use emacs –nw to open file in console
▪ Other editors: nano, pico, kate, gedit, gvim, kwrite, nedit
Introduction to Linux 76
File Editing (vi)
vi works in two modes:
– Command mode
• This is the mode when entering vi
• Commands can be issued at the bottom of the screen,
e.g. copy, paste, search, replace etc.
• Press “i” to enter editing mode
– Editing mode
• Text can be entered in this mode
• Press “esc” key to go back to the command mode
Introduction to Linux 77
Most used commands (vi)
Description Command
Insert at cursor i
Insert at the beginning of line I
Delete a line dd
Copy a line yy
Paste p
Search forward /pattern
Search backward ?pattern
Search again n
Go to line #n n
Replace text %s/new/old/g
Save and exit wq
Introduction to Linux 78
Editor cheatsheet (1)
0
Introduction to Linux 79
Editor cheatsheet (2)
Introduction to Linux 80
Editor cheatsheet (3)
Introduction to Linux 81
Shell Scripts
▪ Script: a program written for a software environment to automate
execution of tasks
▪ A series of shell commands put together in a file
▪ When the script is executed, those commands will be executed
one line at a time automatically
▪ The majority of script programs are “quick and dirty”, where the main
goal is to get the program written quickly
▪ May not be as efficient as programs written in C and Fortran
Introduction to Linux 82
Script Example (~/.bashrc)
# .bashrc
Introduction to Linux 83
Getting Help
▪ User Guides
▪ LSU HPC: http://www.hpc.lsu.edu/docs/guides.php#hpc
▪ LONI: http://www.hpc.lsu.edu/docs/guides.php#loni
▪ Documentation: http://www.hpc.lsu.edu/docs
▪ Archived HPC
training:http://www.hpc.lsu.edu/training/archive/tutorials.php
▪ Contact us
▪ Email ticket system: [email protected]
▪ Telephone Help Desk: 225-578-0900
Introduction to Linux 84
Upcoming Trainings
Introduction to Linux 85
Exercise (1)
Introduction to Linux 86
Exercise (1)
$ echo hello world
$ pwd
$ whoami
$ cd /tmp
$ cd ~ (cd /home/uid)
$ mkdir -p test/testagain
$ cd test/testagain
$ touch test.txt
❖ Go back to your home directory
❖ Figure out which shell you are using
Introduction to Linux 87
Exercise (2)
Introduction to Linux 88
Exercise (4)
If you have never used vim or emacs, go through the vim
tutorial: vimtutor
===============================================================================
= W e l c o m e t o t h e V I M T u t o r - Version 1.7 =
===============================================================================
Vim is a very powerful editor that has many commands, too many to
explain in a tutor such as this. This tutor is designed to describe
enough of the commands that you will be able to easily use Vim as
an all-purpose editor.
Introduction to Linux 89