Advanced UNIX Tutorial N OTE : N OT THE FIN AL VER SIO N ! Otheus !

Shelling,

For University of Innsbruck, March 2007

Topics Covered
Why Unix? Brief History Architecture Inter-process Communication Filesystem Regular Expressions Program Execution
PATH to enlightenment Dynamic Linking
root:/ $

The Shells
BASH Programming Shell Startup Short-cuts

Cool Commands

Why UNIX? And Why Not?

High Performance Fine-grain control over hardware Fine-grain control over OS WYSIWYG .. Not WYSIAYG
Windows has caught up in performance Windows is easier to administrate Windows systems dont require fine-grain control

Flavors, Versions, Distributions

Flavors UNIX comes in various Flavors
System V, BSD, Linux

Distributions A particular Flavor will be packaged and marketed through various Distributions
FreeBSD, RedHat, Gentoo

Versions A particular Distribution will be released with a specific Version number

RedHat EL4 US, Solaris 10 (2.10)

Other version designations The kernel and various software tools and applications each have their own version numbering
X11R6, Linux 2.6, Perl 5.6

UNIX Architecture System Level

Devices managed by drivers

Drivers interact through an API with the Kernel

Kernel manages core resources Processes Memory (physical and virtual) Interprocess communication (IPC) presents to users: system calls device files

UNIX Architecture User Level

User:
interacts with an application through a device

Application:
commands ls, qsub, run-time environment bash, perl, python, Java, PHP, mozilla subsystem X Windows, SGE

Shared libraries:
glibc, gtk
Each application is in its own process and has its own private data. The code can be shared across instances. Each process runs under the permissions of the user that started it effective user id The application and its libraries make system calls and operates on files.

Inter-Process Communication

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes Signals
*otheus* ~/eg$ ls -1 example example2 exists myexample ozzie *otheus* ~/eg$ ls -1 >files *otheus* ~/eg$ sort -k 1.2,1.4 files files example example2 exists myexample ozzie *otheus* ~/eg$

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes
Unnamed Named / FIFOs Network Sockets
*otheus* ~/eg$ ls -1 example example2 exists files myexample ozzie *otheus* ~/eg$ ls -1 | sort -k 1.2,1.4 files example example2 exists myexample ozzie *otheus* ~/eg$

RPC Signals

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes
Unnamed Named / FIFOs Network Sockets
*otheus* ~/eg$ mkfifo newfifo *otheus* ~/eg$ *otheus* ~/eg$ ls -l newfifo prw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo *otheus* ~/eg$ { while true; > do echo $RANDOM >>/home/otheus/eg/newfifo ; > sleep 5; done } & [1] 15533 *otheus* ~/eg$ ls -l newfifo prw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo *otheus* ~/eg$ date ; cat newfifo ; date Wed Mar 28 19:52:49 CEST 2007 32102 Wed Mar 28 19:52:49 CEST 2007 *otheus* ~/eg$ date ; cat newfifo ; date Wed Mar 28 19:52:51 CEST 2007 26331 Wed Mar 28 19:52:54 CEST 2007 *otheus* ~/eg$

RPC Signals

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes
Unnamed Named / FIFOs Network Sockets
2-way pipes over a network *otheus* ~/eg$ sudo netstat -anp | > grep -E "(181):(3306|80|22) " | > head -3 tcp 0 0.0.0.0:* 0 138.232.66.181:22 LISTEN 4176/sshd

tcp 0 48 138.232.66.181:22 192.168.64.52:2131 ESTABLISHED 9343/5 tcp 0 0 138.232.66.181:3306 138.232.66.164:49325 TIME_WAIT -

RPC Signals

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes RPC
RPC process Portmapper process YP/NIS & NFS
*otheus* ~/eg$ ps -e -o uid,pid,cmd | > grep rpc 29 3338 0 3599 0 4093 0 4274 0 4312 0 4323 201 17112 rpc.statd rpc.idmapd rpc.yppasswdd -e chsh -e chfn rpc.rquotad [rpciod] rpc.mountd grep rpc

*otheus* ~/eg$ rpcinfo -p | uniq -f 4 program vers proto 100000 2 tcp 100024 1 udp 100007 2 udp 100004 2 udp 100009 1 udp 100011 1 udp 100003 2 udp 100021 1 udp 100005 1 udp port 111 32768 856 845 878 640 2049 32774 686

Signals

portmapper status ypbind ypserv yppasswdd rquotad nfs nlockmgr mountd

Inter-Process communication (IPC)

Semaphores Shared Memory Filesystem Pipes RPC Signals
Only within a process family Processes 0, -1 *otheus* ~/eg$ kill -l | head -8 1) SIGHUP 2) SIGINT 3) SIGQUIT 5) SIGTRAP 6) SIGABRT 7) SIGBUS 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 13) SIGPIPE 14) SIGALRM 15) SIGTERM 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 22) SIGTTOU 23) SIGURG 24) SIGXCPU 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 30) SIGPWR 31) SIGSYS 34) SIGRTMIN *otheus* ~/eg$ echo $$ 2432 *otheus* ~/eg$ /bin/ash *\u* \w$ echo $$ 18201 *\u* \w$ kill -HUP 18201 Hangup *otheus* ~/eg$ echo $$ 2432 *otheus* ~/eg$ trapped & [1] 18664 *otheus* ~/eg$ kill -1 18664 ; sleep 1 Caught signal! *otheus* ~/eg$ kill 18664 ; sleep 1 [1]+ Terminated trapped *otheus* ~/eg$ 4) 8) 12) 17) 21) 25) 29) 35) SIGILL SIGFPE SIGUSR2 SIGCHLD SIGTTIN SIGXFSZ SIGIO SIGRTMI

SIGHUP SIGTERM SIGKILL SIGSEGV

UNIX Filesystem

UNIX Filesystem Tree Structure

Resembles a tree
File-types Mounting drives Breaking structure

Promiscuous Permissions

UNIX Filesystem Tree Structure

Resembles a tree
File-types Mounting drives Breaking structure

Promiscuous Permissions

UNIX Filesystem Tree Structure

Resembles a tree
File-types Mounting drives Breaking structure
Soft-links Loopback mounts Bind mounts

Promiscuous Permissions

UNIX Filesystem Tree Structure

Resembles a tree
File-types Mounting drives Breaking structure
Soft-links Loopback mounts Bind mounts

Promiscuous Permissions

UNIX Filesystem Tree Structure

Resembles a tree
File-types Mounting drives Breaking structure
Soft-links Loopback mounts Bind mounts

Promiscuous Permissions

UNIX Filesystem Very Promiscuous

Resembles a tree Promiscuous:
Nearly everything in UNIX can be accessed through the filesystem
Devices Processes (Linux & Solaris) via /proc filesystem Kernel stats and configuration (Linux: /proc) Exception: Network interfaces

Permissions

UNIX Filesystem Very Promiscuous

Resembles a tree Promiscuous
Devices Processes Kernel stats and configuration
MAKEDEV XOR arpd cdrom cdrom1 console core cpu cpu0 cpu1 cpu2 cpu3 dnrtmsg dvd fd floppy full fwmonitor hda hw_random initctl input ip6_fw js0 kmsg log loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7 lp0 lp1 lp2 lp3 mapper mcelog md0 mem mice mosal net nflog null parport0 parport1 parport2 parport3 port ppp ptmx pts ram ram0 ram1 ram10 ram11 ram12 ram13 ram14 ram15 ram2 ram3 ram4 ram5 ram6 ram7 ram8 ram9 ramdisk random rawctl root route route6 rtc scd0 sda sdb sdc sdd sde sde1 sde2 sde3 sdf sdf1 sdf2 sdf3 shm skip stderr stdin stdout systty tap0 tap1 tap10 tap11 tap12 tap13 tap14 tap15 tap2 tap3 tap4 tap5 tap6 tap7 tap8 tap9 tcpdiag ts_ua0 ts_ua1 ts_ua2 ts_ua3 ts_ua4 ts_ua5 ts_ua6 tty0 tty1 tty2 tty3 tty4 tty5 tty6 tty7 tty8 tty9 urandom usersock vcs vcs1 vcs2 vcs3 vcs4 vcs5 vcs6 vcsa vcsa1 vcsa2 vcsa3 vcsa4 vcsa5 vcsa6 vipkl xfrm zero

Permissions

UNIX Filesystem Very Promiscuous

Resembles a tree Promiscuous
Devices Processes Kernel stats and configuration
*otheus* /proc$ ls -Cd [1-2]* 1 14642 17888 21007 21942 258 10 14644 17889 21008 21943 2619 10377 14657 17890 21009 2251 2620 10395 15 17892 21010 2269 2647 11 155 17894 21017 2270 2648 11639 17473 19570 21019 2312 2649 12 17476 2 21020 2313 2650 13 17478 20998 21021 240 2658 13956 17511 20999 21022 241 27265 13964 17871 21000 2121 2431 286 14 17884 21002 21646 2432 1433 17887 21003 21941 257 *otheus* /proc$ cat 1/cmdline; echo init [4] *otheus* /proc$ ls -F 1 attr/ environ maps stat wchan auxv exe@ mem statm cmdline fd/ mounts status cwd@ loginuid root@ task/ *otheus* /proc$

Permissions

UNIX Filesystem Very Promiscuous

Resembles a tree Promiscuous
Devices Processes Kernel stats and configuration
*otheus* /proc$ ls -CdF [a-z]* acpi/ iomem net/ buddyinfo ioports partitions bus/ irq/ pci cmdline kallsyms scsi/ cpuinfo kcore self@ crypto key-users slabinfo devices keys stat diskstats kmsg swaps dma loadavg sys/ driver/ locks sysrq-trigger execdomains mdstat sysvipc/ fb meminfo topspin@ filesystems misc tty/ fs/ modules uptime ide/ mounts@ version infiniband/ mpt/ vmstat interrupts mtrr *otheus* /proc$ cat uptime 704398.43 593929.58 *otheus* /proc$ uptime 20:56:58 up 8 days, 3:39, 7 users, load average: 1.72, 1.75, 1.67 *otheus* /proc$

Permissions

UNIX Filesystem Security

Resembles a tree Promiscuous Permissions
3 Classes:
User Group Other (World)

4 bits:
Read Write Execute Special

UNIX Filesystem Security

Resembles a tree Promiscuous Permissions
3 Classes:
User Group Other (World)

4 bits:
Read Write Execute Special

UNIX Filesystem Security

Resembles a tree Promiscuous Permissions
4 bits:
Read Write Execute Special
Directories Set-uid Set-gid

UNIX Filesystem Security

Resembles a tree Promiscuous Permissions
4 bits:
Read Write Execute Special
Directories Set-uid Set-gid

Regular Expressions

Regular Expressions Quick Overview

Regular Expressions Examples

10

Regular Expressions Examples

Regular Expressions Examples

Program Execution

11

Program Execution - Overview

Program Execution Example

Program Execution Example

12

The Shells

The Shells
Run applications and commands
Interactively Batch mode In parallel

Interact with Files Manage environment used by applications

The Shells
Run applications and commands Interact with Files
Send files to commands. Send commands output to files.

Manage environment used by applications

13

The Shells
Run applications and commands Interact with Files Manage environment used by applications
Environment variables File creation mask

The Shells
Flavors
Bourne (sh)
Korn (ksh) Bourne-Again (bash) Almquist (ash) Z shell (zsh)

C Shell (csh)
tcsh

Exotic
fish zoidberg

GUI
Konqueror

Text-GUI
Midnight Commander (mc)

The Shells
Mans Best Friend help (BASH only) man
more or less man k man (Solaris) man (GNU/BSD)

info

14

The Shells
Mans Best Friend help (BASH only) man info
Hyperlink viewer CTRL-H --vi-keys
File: info.info, Node: Top, arted, Up: (dir) Info: An Introduction ********************* The GNU Project distributes most of its on-lin\ e manuals in the "Info format", which you read using an "Info reader"\ . You are probably using an Info reader to read this now. If you are new to the Info reader and want \ to learn how to use it, type the command `h' now. It brings you to a \ programmed instruction --zz-Info: (info.info.gz)Top, 35 lines --Top--Basic Commands in Info Windows ****************************** C-x 0 C-x C-c h Quit this help. Quit Info altogether. Invoke the Info tutorial. Next: Getting St\

Selecting other nodes: ---------------------n Move to the "next" node of this no\ de. p Move to the "previous" node of thi\ s node. u Move "up" from this node. m Pick menu item specified by name. -----Info: *Info Help*, 763 lines --Top-------No cross references in this node.

The Shells Using Aliases

What is it? Creating an Alias Automatic aliasing (bash)
type / which alias unalias hash

The Shells Using Aliases

What is it? Creating an Alias Automatic aliasing (bash)
type / which alias unalias hash

15

Advanced Shell Programming

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Manipulation File Specification Negative Logic Execution Control Functions
pwd cd set / echo export eval fg / bg exec

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Manipulation File Specification Negative Logic Execution Control Functions
pwd cd set / echo export eval fg / bg exec

16

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Manipulation File Specification Negative Logic Execution Control Functions
export eval fg / bg exec jobs kill ls

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection and Manipulation
Listing Files Viewing Files
exec ls cat more / less head / tail grep rm /cp / mv

File Specification Negative Logic Execution Control Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Manipulation
standard streams Copying, moving, renaming, new dir
head / tail grep rm /cp /mv mkdir / rmdir chmod / chown id / groups newgrp

File Specification Negative Logic Execution Control Functions

17

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Manipulation
Changing owner Changing permissions
mkdir / rmdir chmod / chown id / groups newgrp umask

File Specification Negative Logic Execution Control Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification
File globs
id / groups newgrp umask find xargs fg / bg exec

Negative Logic Execution Control Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification
find and xargs
id / groups newgrp umask find xargs fg / bg exec

Negative Logic Execution Control Functions

18

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control Functions
eval test true / false if .. fi case W in .. esac for VAR in .. do .. done

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
if/elif/else/fi
true / false if .. fi case W in .. esac for VAR in .. do .. done while.. do.. done

Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
case WORD in ... esac
true / false if .. fi case W in .. esac for VAR in .. do .. done while.. do.. done

Functions

19

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
case WORD in ... esac
true / false if .. fi case W in .. esac for VAR in .. do .. done while.. do.. done

Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
for VAR in ... done
true / false if .. fi case W in .. esac for VAR in .. do .. done while.. do.. done

Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
while ... do ... done
true / false if .. fi case W in .. esac for VAR in .. do .. done while.. do.. done

Functions

20

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
break, next, exit ||, && type, which, whence
while.. do.. done break next exit || and &&

Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control
break, next, exit ||, && type, which, whence
type which whence

Functions

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control Functions
Declaring, returning
return declare command

21

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control Functions
Using Overriding builtins & commands
declare command builtin

Advanced Shell Programming with BASH

Working Directory Variable Creation & Expansion Job Control File Redirection File Specification Negative Logic Execution Control Functions
Using Overriding builtins & commands
type command builtin

Shell Startup

22

Shell Startup (bash)

Shell Startup (tcsh)

Cool Commands

23

Cool Commands

Cool Commands

Cool Commands

24

Complex Examples

Complex Examples

Complex Examples

25

Complex Examples

Complex Examples

Complex Examples

26