Scripting
COSC 301 Lecture 4
Zhiyi Huang
Computer Science
University of Otago
Outline
Purpose, History
Unix scripting
Unix failings
Other solutions
Scripting is...
‘Easier’
Glue
Weakly typed
Interpreted
Who scripts?
Users
Power users
Administrators
Developers
Testers
Developments
Job Control Language
1960s Unix pipe
1993 Applescript
2005 Automator
2006 Windows PowerShell
Available shells in Linux
bash, sh, tcsh, csh; use cat /etc/shells
Unix Shell Scripting
Unix Philosophy
Write programs that do one thing and
do it well. Write programs to work
together. Write programs to handle
text streams, because that is a
universal interface.
Doug McIlroy
Inventor of the | construct
./hello No extension
#!/bin/sh
echo “Hello, World!”
$ chmod +x ./hello
$ ./hello
Hello, World!
The correct UNIX way
$ sh ./hello
Hello, World!
echo -e “no newline\c”
Another example
#!/bin/bash
clear
echo "This is information provided by [Link]. Program starts
now."
echo "Hello, $USER"
echo
echo "Today's date is `date`, this is week `date +"%V"`."
echo
echo "These users are currently connected:"
w | cut -d " " -f 1 | grep -v USER | sort -u
echo
echo "This is `uname -s` running on a `uname -m` processor."
echo
echo "This is the uptime information:"
uptime
echo
echo "That's all folks!"
#! “Sh-Bang”
First line
#!/bin/sh
#!/usr/bin/perl -wnl
#!/usr/bin/env python
Default is /bin/sh
SetUID not honoured
# is also used for comments
Design Patterns
Source: ls
read from file and write to stdout
Filter: sort
read from stdin and write to stdout
Sink: less
read from stdin and write to file
“Cantrip”: rm
do something but return nothing
Compiler: tar
read from file and write to another file
Good scripts
A sensible name
don’t clash with existing commands and
programs
No errors
Perform the intended task
Have a clear logic
Efficient, no unnecessary work
Informative, notifying users about what it is
doing
Reusable
BASH basics
Files read by bash
/etc/profile, .bash_profile, .bashrc
depending on login, interactive, non-interactive, or use
sh directly
Built-in commands like cd and eval, exit, exec, export, …
Three types of commands
built-in, function, executable programs
debugging a script: bash -xv script_file
Some self-study required. Read Bash Beginners Guide
Environment Variables
Environment variable
A variable with name and value used by shells and processes
Use printenv or env to find them
They can be set by
Globally, /etc/profile, /etc/[Link]
Per user, ~/.bash_profile,~/.bashrc, ~/.profile
Non login shell, non interactive shell (shell scripts)
/etc/profile, ~/.bash_profile, ~/.bash_logout
Used by login shells
/etc/[Link], ~/.bashrc
used by interactive, non-login shells
For details: [Link]
environment_variables
I/O Channels
stdin prev pipe or terminal
^D to ‘end-of-file’
stdout next pipe or terminal
stderr not piped
FD 0,1,2 respectively
Redirection
command > file-overwriting
command >> file-appending
command 2> file
redirect stderr to file
echo “Warning to stderr” >&2
redirect stdout to stderr
echo “To black hole” 2> /dev/null >&2
command < f1_in > f2_out 2> f3_err
| “Pipe”
5 biggest dirs in the current dir
du -xkd 1 \
|grep -v "^[0-9]*[[:space:]]*\.$"\
|sort -rn | head -5
All I/O via kernel, slow
Variables
varname=value Assignment
no spaces around ‘=’
$varname Deference
Global and local variables
Seen by subshell/child processes if
export PATH=$HOME/bin:$PATH
Beware white-space!
varname=”foo bar”
Interpolation
‘non-interpolated string’
`command`
“interp. string $varname `command`”
foo=`command \`command\``
foo=$(command $(command)) (Bash specific)
html=”$1”; txt=”${html:%.html}.txt”
links -dump “$html” > “$txt”
Conditions—if
see test(1)
if␣[␣$# -lt 2 ]; then
if-less-than-two-arguments
elif␣[␣\(␣“$1”␣=␣‘foo’␣\)␣-a␣\
\(␣-r␣/etc/foorc␣\)␣]; then
if-arg1-is-foo-and-foorc-is-readable
else
if-otherwise
fi
if␣!␣grep -q ...; then
if-grep-did-not-find
fi
Conditionals—case
case “$fo_proc” in
‘fop’)
command;;
‘xep’)
command1; commandN;;
*)
default-command >&2
exit 1;;
esac
Loops—for
for i in foo bar baz
do
echo $i
done
(( ... ; ... ; ... )) is a Bash-ism
for ((i=128; i<160; i++)); do
printf “ip%03d\tA\t192.168.1.%d\n” $i $i
done
Loops—while
ls | while read filename
do
do stuff with “$filename”
done
while true
do
infinite loop body
done
Subshells
[Link] - - [26/Mar/[Link] +1200] ↩
"GET /path/to/[Link] HTTP/1.0" 304 -
(echo "IP Freq";
(cat access_log;
gzcat access_log.*.gz)
| cut -d’ ‘ -f1 | sort | uniq -c
| sort -rn | awk '{print $2,$1}'
) | column -t
getting ugly, start making functions
Arithmetic
expr 2 \* 8
16
echo $((2 * 8)) Bash-ism
16
echo 'scale=2; 1/3' | bc
.33
echo 'ibase=10; obase=2; 192' | bc
11000000
Sed and Awk
Read a book!
Regular expressions!
Takes a while to learn
A few recipes are useful
sed—Stream Editor
Delete header on first line
sed -e 1d
Disable FTP service in inetd
sed -e ‘s/^ftp/#&/’ < [Link] > \
[Link]
mv [Link]{,~}; mv [Link]{.new,}
What requests got a 404?
gzcat access_log.*.gz | sed -ne '/ 404
[0-9]*/s/^.*"[A-Z]* \(.*\) HTTP\/
[0-9.]*".*$/\1/p'
awk
Re-order fields
echo 'a b c a c b' | tr ' ' '\n' | sort \
| uniq -c | awk '{print $2,$1}' \
| sort -r -k2
Collation
echo -e '1\n2\n3\n4' | awk '
BEGIN{sum=0;max="?"}
max=="?"{max=$1}
{sum+=$1}
$1>max{max=$1}
END{print "Avg:" sum/NR "\nMax:" max}'
A command a day...
List descriptions of system
commands
find /bin /usr/bin /sbin /usr/sbin \
-type f -perm /111 | \
xargs -L1 basename | \
xargs -L1 whatis | grep '([18])'
Unix failings
“Prayerful parsing”
I/O is expensive
Interface inconsistency
Lack of re-use
Security: TOCTTOU
rm /tmp/*/* (find /tmp -not-accessed-
recently | xargs rm)
Other Systems
Applescript example
Is 10% of disk available?
[Link]
applescriptlangguide/conceptual/ASLR_lexical_conventions.html#//apple_ref/doc/uid/
TP40000983-CH214-SW1
tell application "Finder"
set the percent_free to ¬
(((the free space of the startup disk) / ¬
(the capacity of the startup disk)) * 100) div 1
end tell
if the percent_free is less than 10 then
tell application (path to frontmost application as text)
display dialog "The startup disk has only " & ¬
the percent_free & ¬
" percent of its capacity available." & return & return & ¬
"Should this script continue?" with icon 1
end tell
end if
PowerShell examples
These examples from Monad Manifesto
What is filling up my application logs?
Verb
Noun
Get-EventLog application|Group source|Select –
first 5|Format-Table
counter Property
====== ===========
1,269 crypt32
1,234 MsiInstaller
Not text, but objects
1,062 Ci
280 Userenv
are passed around
278 SceCli
Why is MsiInstaller filling my log?
Object Property
Get-EventLog application |Where {$_.source -eq
“MsiInstaller”}|Group Message |Select –first 5
|Format-Table
counter Message
====== =============================================================…
344 Detection of product '{90600409-6E45-45CA-BFCF-C1E1BEF5B3F7}…
344 Detection of product '{90600409-6E45-45CA-BFCF-C1E1BEF5B3F7}…
336 Product: Visual [Link] 7.0 Enterprise - English – Inter…
145 Failed to connect to server. Error: 0x800401F0
8 Product: Microsoft Office XP Professional with FrontPage --…
Change Format-Table to output XML, CSV, LIST,
HTML, Excel...
Is my eventlog usage regular across the week?
Get-EventLog application | Group
{$_.[Link]}
counter DayofWeek
All these commands run in
====== ========
the same run-time
1,333 Tuesday
environment (.NET) so I/O
1,251 Wednesday
is cheap.
744 Thursday
680 Monday
The shell can validate
651 Friday
properties etc. using
556 Sunday
reflection, meaning it can
426 Saturday look at what methods etc.
are available.
Least Privilege Principle
No process or file should be given more
privileges than it needs to do its job.
Setuid programs: don’t set unless necessary
Run programs under special user id such as
www and nobody if possible
Some applications such as httpd can change
its user id from root to nobody after opening
the privileged port number 80.
Temporary files shouldn’t be in /tmp
Suggested Reading
The Art of Unix Programming Eric S. Raymond
The Unix Hater’s Handbook Simson Garfinkel,
Daniel Weise, and Steven Strassmann
Monad Manifesto Jeffrey P. Snover
Scripting: Higher Level Programming for the 21st
Century John K. Ousterhout (father of Tcl)
Bash Guide for Beginners Machtelt
Garrels
[Reference] bash(1)
