Peepcode - Command Line
Peepcode - Command Line
Using
Unix
Get started with the command line
by Casimir Saternos
Unix Basics
2
contents 30 Tow the (Command) Line 52 Somebody’s Watching Me…
56 Monitoring File Space
8 About UNIX 33 Organization
56 Monitoring a Process
8 History 33 Moving in Stereo
57 Scripting
9 Philosophy 33 It’s in the Files
57 The Job That Ate My Brain
9 Do One Thing Well 36 Find Your Way Back
58 Debugging
9 Output is Input
39 Concatenation
10 Everything is a File 61 Appendix: Online Resources
39 Here, There and Everywhere
10 Help! 62 Appendix: Info Script
39 Sequencing
13 Navigation 39 Redirection
13 Starting Here, Starting Now 40 Appending
41 Pipes
14 Where Am I?
41 Filters
15 Path Descriptions
46 I Know the Difference
17 Where Am I Going?
47 Time After Time
18 Who Am I? 48 World on a String
20 Power Tip: SSH Keys
50 Administration
23 Modification 50 Installation
23 Editing Files 50 Unix/Linux Installation
25 Permissions 51 Starting Up / Shutting Down
26 Aliases and Inodes 52 Users/Group Administration
52 System Monitoring
30 Customization 52 Backup/Recovery
about this book
Using UNIX can be intimidating! If you are familiar with Windows or
the graphical elements of Mac OS X, you may be overwhelmed when
presented with a blank terminal screen, waiting for a command.
But don’t panic! You only need a dozen short commands for most of
what you’ll need to do as a web developer. As you become comfort-
able, you’ll learn to customize your environment for greater efficiency.
This book is designed to provide the information to quickly and effec-
tively address your daily challenges and is extensive enough to give
experienced developers insights that will increase their effectiveness
and productivity.
The purpose of this book is to give you insights into how to develop,
install, monitor, and maintain web applications on UNIX servers. We’ll
assume that either:
Even if you are not using UNIX on a daily basis, it is worth noting that
UNIX has exerted a considerable influence on the design of other
5
technologies. Mac OS X, Tivo, routers, and hand held devices run
Linux under the hood. In addition, many of the utilities have corol-
laries in classes in the Ruby programming language. See the chart
below for some examples of features of Ruby that are based upon
UNIX utilities.
6
(called distros) of UNIX operating systems. Each of these have spe-
cific features and unique implementations that differentiate them
from their siblings. However, there is a rather large body of common
functionality which allows most users to use a consistent set of com-
mands on each system. This book will focus on functionality that is
available to the majority of those in popular use.
The commands that are introduced in this book will be described, but
not in a comprehensive manner. The book can be read from begin-
ning to end to provide a relatively comprehensive overview of UNIX
that focuses on the interests of web developers. You can also read
individual sections to get an introduction to the resources available
in a particular problem domain and specific examples of how to use
UNIX effectively for a particular task.
7
About UNIX
chapter 1
History
In the mid 1960’s, a group of companies set out to build a comput-
ing utility service that could be used by many people at once like the
phone or electric utilities. MULTICS (http://en.wikipedia.org/wiki/Multics) never
became as popular as originally hoped, but some of the engineers
who worked on it went on to develop a more focused operating sys-
tem called UNIX.
8
Philosophy
The UNIX philosophy was never formalized, but here is an attempt to
summarize the fundamental design principles that guided the devel-
opers of the operating system.
note Most UNIX commands are only a few characters long. You
may want to shorten commonly used commands even further
with aliases, which we’ll talk about later.
Output is Input
Very few UNIX utilities require interactive input, and their output is in
a form that is suitable for use by another program: plain text! This
makes it easy to use several programs together, even if they aren’t
aware of the other program. Programs can be chained together in a
sequence, and some programs have a primary usage as filters for
other programs. Each filter in a series of commands serves one small
purpose in modifying a stream of text before it is transformed into its
final output.
Help!
Since this book is not comprehensive, you should be aware of how
you can expand your knowledge on your own in a particular area of
interest. There are a huge number of utilities, scripts and programs
available on most standard UNIX installations. You will most likely find
that there are several thousand programs available in yours—some of
which are so complex that entire books are dedicated to them.
The man and info commands are the standard utilities to use to learn
more about each command you encounter. The man command can
be called with a program as the argument, and the program’s manual
page will be displayed. If you are not sure of what program you need,
the -k option will be of assistance. For instance, type the following to
get information on programs related to the Apache web server:
% man -k apache
ab(8) - Apache HTTP server benchmarking tool
apachectl(8) - Apache HTTP Server Control Interface
apxs(8) - APache eXtenSion tool
10
httpd(8) - Apache Hypertext Transfer Protocol Server
Once a given utility has been identified, you can simply type man
followed by the utility name to see the manual page. Press space to
scroll through the page and q to quit. And in typical recursive fashion,
you can man man to find what other man options are available.
% man ab
NAME
ab - Apache HTTP server benchmarking tool
SYNOPSIS
ab [ -A auth-username:password ] ...
Command Description
man Display the manual page.
Press space bar to scroll
through and q to quit
man -k Search the whatis data-
base for commands that
match
man -f Equivalent to whatis, which
provides a short description
of a system command
info Display an info page (more
comprehensive than man)
<command> -h Display the help screen
type -t Identify a command as an
alias, keyword, function,
builtin, or file
11
12
Navigation
chapter 2
You’ll learn best if you try to follow along with these examples or the
accompanying screencast.
There are a number of ways that you might access a server. Although
there are a variety of graphical desktop options available when log-
ging at the console or through VNC (remote screen sharing), the focus
of this book is going to be command line access. Much of the power
of the UNIX operating system can only be realized when accessing
the system from the command line.
The telnet and rsh (remote shell) applications are old, standard,
13
insecure means for accessing systems. It is more common (and
secure) to use ssh today.
% ssh [email protected]
You will be prompted for your password and can now issue com-
mands on the remote server.
Where Am I?
Logging in locally or via SSH will drop you into your home directory.
This is the directory where you can store files and create directories.
The pwd command (print working directory) can be used to print your
current location in the filesystem. The cd command can then be run
to navigate to a new directory. Within a directory or set of directories,
14
you can view the contents using the list (ls) command.
Path Descriptions
Many commands take a filename as an argument. The argument
can be an absolute path, a relative path, or a special character that
designates a directory. Here are a few:
Path Description
~/ Your home directory
/Users/topfunky An absolute path. Starts
with a slash and specifies
all the intervening direc-
tories needed to get to a
location
./bin A relative path. The two
periods and a slash signify
one directory up from your
current location. These can
be added an several times
to navigate up several
directories.
- The last directory you were
in. This might be in a com-
pletely different location
and is useful for returning
there (with cd) or copying
files.
Wildcard Description
? Represents any single
character to be matched
* Represents any number of
characters to be matched
[] Specifies a range matched
{} Contains a list of comma-
separated values, any of
which will be matched
[!] Excludes a range matched
Used as an escape char-
acter to treat a special
character as a literal
The tool time is noteworthy when trying to track down what file in a
group has changed recently. To sort the results of time, use the -t
option. If you are more concerned about the size of the files, include
the -S option. Piping the results to more or less will allow you to page
through a large list of results. Grab the first line in a file piping to head
-1, or the last line using tail -1.
If you want to use the the listing of files as string literal input to a sub-
sequent program, you can use the -Q option to enclose each name in
16
quotes.
If you wanted to remove a set of files that you have listed, you could
use xargs command (rather than directly piping to the rm com-
mand):
% ls \*/\*diffs.txt | xargs rm
Command Description
cd Change directories
ls List files
xargs Build and execute com-
mand lines from argu-
ments in standard input
Where Am I Going?
Once you have logged on to a server, you will want to get familiar
with what resources are available and begin making customiza-
tions to meet your particular needs. Typically, you will immediately
check your that your environmental variables (e.g. PATH) are set as
intended, and perhaps validate the system specifications.
In the bash shell, a file named .bash_profile within the user’s home
directory contains environmental settings specific to the user. You
might want to adjust the settings in this file. In addition, there are
times that you might source this file (execute the script in such a
way that the variable settings become present in the current session).
Scripts scheduled through cron often have this requirement. The way
to source your bash profile is as follows:
% . .bash_profile
17
tip See dotfiles.org (http://dotfiles.org) for profile samples from other
developers.
You can use the env command to see the settings of variables within
your environment. You can use export to make new variable settings
available to the environment.
Command Description
env Outputs the variables that
are set in the environment
export Make variable settings
available to the current
environment
alias Display or set substitutions
for commands or groups of
commands
who Display information about
users that are logged in
uname Output information about
the system (kernel release,
hardware info, processor,
etc)
hostname Output the name that
identifies the server
Who Am I?
As in the rest of life, it’s important to know who you are.
18
Because UNIX supports multiple users and each user has specific
privileges, it is important to understand who you are logged in as.
Problems can occur when a file is created by one user and a later
attempt is made to access the file by another user.
Most installations of UNIX start out with several user accounts and
groups. Here are a few that may be present on your system:
User Description
root The administrative super-
user account. The root user
can perform any com-
mand on the system. If an
attacker gained access to
the root account, he could
take over the rest of the
system.
nobody An unprivileged account
used by applications that
want to restrict access as
much as possible.
daemon A faceless account used to
run background tasks.
In this example, I’ll try to find the file sizes of all the directories on the
current machine. Some are protected, so I’ll have to rerun the com-
mand with sudo in order to have access to all the files.
% du -hs /*
du: `/usr/share/ssl/CA’: Permission denied
19
% sudo du -hs /*
Password:
3.6M /bin
2.7M /boot
568K /dev
4.5M /etc
772M /home
1.3G /usr
Command Description
whoami Identify the user you are
currently logged in as
passwd Change your password
su Switch user
sudo Execute a command as
another user. Frequently
used to allow super user
(root) access
visudo Editor for the sudoers con-
figurations file which limits
access available through
sudo
The following steps can be used to generate a key pair, transfer the
public key to a server, and log in (or copy files via scp) without a
password. In this example, the destination server name is destina-
tionserver and the OS username is webuser.
3. Login and transfer are now available from the client to the server
without a password.
% ssh webuser@destinationserver
% scp source_file.txt webuser@destinationserver:destination_file.txt
21
Command Description
ssh Secure shell remote login
ssh-keygen SSH key generation and
management
22
Modification
chapter 3
Editing Files
Because everything in UNIX is a file, it is crucial to be conversant with
the options available for viewing, editing, and manipulating files. The
vi editor is a common, powerful interactive text editor. You will need to
invest a bit of time to be somewhat conversant with it, but is well worth
the time. It also serves as the editor for a number of special editors
that are designed to safely modify certain system configuration files.
For example:
% tail -f production.log
The sed stream editor can be used to do search and replace opera-
tions in files. When more complex stream-oriented modification of
files is needed you will likely want to switch over to ruby or perl. Finally
redirection to a file using the > (create new) or >> (append) operators
is a way to write the results of a command to a new or existing file.
Command Description
cat When passed a single file
as an argument, displays
the contents to standard
out
tac Concatenate/print a file in
reverse
more Filter for viewing text one
screen at a time
less Functions like more but
allows backwards scrolling
and other features
head Output the first part of a
file
24
Command Description
tail Output the last part of a
file
vi An editor standard to most
UNIX installations
emacs An extensible customizable
text editor that includes a
Lisp interpreter
Permissions
Many problems encountered by new UNIX users are related to setting
file ownership and file permissions. Permissions can apply to users,
groups or others.
The read, write and execute bits are the standard permissions and
there are also some special modes that can be set. There is an octal
notation that is used to specify file permissions summarized the chart
below.
25
You can change both the owner and the group of a file at one time:
Command Description
chown Change Owner of files/
directories
chmod Change Modification of
files/directories
chgrp Change Group of files/
directories
umask Set the default file system
mode for newly created
files and directories
alias Display or set an alternate
name for a command
aliases
The @alias@ command with no arguments displays the aliases in use
in the shell. Aliases can be used to help users who are familiar with
commands on other operating systems.
They can also be used to correctly interpret typos (e.g. aliasing ls-l
which is missing the space between the command and option to
ls -l). Commands such as cp, mv and rm can be aliased to prompt
before overwriting or removing files.
inodes
You can use ls -i to print the index number, or inode of each file.
The stat command will display the inode number as well as its
status and attributes. An inode contains all of the file’s properties
(permissions, ownership, type of file) as well as the addresses for the
data blocks containing the file’s content. Several file names can link
to the same inode. The rm command removes the link that points to
its inode, but does not remove the file contents (so the inode has no
name linking to it at all). If another process still has the file open, you
can even restore a deleted file.
Now let’s force a process to hold on to the file while we deleted it. We
open it with more but only read a line at a time
% more -1 file_to_remove.txt
Type Ctrl-Z while more is still running. This will force the process into
the background. The following line is initially displayed and also will
be seen if you type jobs.
% rm file_to_remove.txt
28
Based upon the output displayed above, we could run a command to
restore the file and view the contents:
% cp /proc/26133/fd/3 restored.txt
% cat restored.txt
Command Description
alias View or create aliases
unalias Remove aliases
lsof List open files
shopt View or modify shell
options
29
Customization
chapter 4
Once you become familiar with the basics, you may want to look at
the unique features of other shells:
• zsh (http://www.zsh.org)
• fish (http://fishshell.org)
• tcsh (http://www.tcsh.org)
Shortcuts are a key factor in improving your accuracy and produc-
tivity in any environment. The table below lists some of the popular
ones. Others are available online at the Nuby on Rails blog (http://
nubyonrails.com/articles/useful-shell-shortcuts).
30
Shortcut Key Description
Ctrl-R Search your bash history.
Type the first few letters
after hitting Ctrl-R and
press enter when the com-
mand is selected
Ctrl-T Swap the last two charac-
ters you typed
Ctrl-C Cancel the currently run-
ning command
Ctrl-D Log out of the current ter-
minal
Ctrl-Z Stop a running Job (see bg
and fg for more informa-
tion )
!! Repeat the last command
!$ Recall the previous argu-
ment
<arrow-up>/<arrow-down> Scroll through Command
History
<TAB> Complete a command or
name based upon entries
available in a given context
Command Description
history Print the command history
31
Command Description
whatis Short description of a com-
mand
clear Clear the screen
stty Change or display terminal
settings
32
Organization
chapter 5
Moving in Stereo
Files can be moved around the file system using mv and copied using
cp. Moving files between computers involves the use of other com-
mands or protocols (ftp, scp).
33
Links are analogous to Windows shortcuts. Symbolic links allow a
given symlink or soft link with one name to refer to a file with another.
Some operations (like an rm) operate on the link itself but most
operations affect the targeted file. Symbolic links are created using
the -s option:
% ln -s /<target> <my_link_name>
% mkdir testing
% cd testing
% ls -lth
total 0
% ln -s /work
% ls -lth
total 0
lrwxrwxrwx 1 csaternos users 5 2008-03-20 09:34 work -> /work
34
% cd work
Now let’s look at the parent directory—Oh wow—it’s the parent of the
destination!
% ls ..
bin boot dev etc home lib mnt opt proc root sbin sys tmp usr var work
% cd ..
% ls
work
There are numerous interactive editors available for UNIX. The classic
vi editor is available on nearly all platforms and emacs has become
a favorite among rails developers. One early editor is named ed, and
a stream oriented variant of this editor is named sed. The sed pro-
gram can be used to do search and replace operations in files.
If the tasks become more sophisticated, perl and ruby can be used to
your advantage. In particular, one-liners may be created that require
a simple invocation of the perl or ruby interpreter along with argu-
ments that include a few commands and files to process.
35
Find Your Way Back
The ls command is sufficient when viewing the contents of a single
directory or a few directories withing a given hierarchy. However, it is
not useful for identifying files that are in a number of different loca-
tions. The find utility can be used to search for files in a more sophis-
ticated manner. One simple example to get you started:
The dot indicates to search within the current working directory. The
*.rb indicates that any file name ending with the .rb extension is an
eligible target. Other file attributes (e.g. size, permissions, type) can
be used; for instance, this command shows all files within the current
working directory that have been modified in the past 5 minutes:
% find . -mmin -5
The find command is not limited to simply listing file names. find can
execute commands directly, and you can also pipe results to xargs:
#!/bin/env ruby
require ‘find’
Find.find(‘.’)do |x|
# Process the file here...
end
36
The find command is very flexible and comprehensive in its ability to
search for files, but it can be slow because of its design. The locate
command relies upon the existence of a database of files. Typically
a system is set up to run a process on a regular basis to keep the
database up to date. The locate command is quicker than find, but
is only as accurate as the database it is accessing. In the following
example, the locate database needs to be updated before it can be
used to find a file.
% locate test.txt
warning: locate: warning: database
/var/lib/slocate/slocate.db’ is more than 8 days old
% locate -u .
% locate test.txt
/root/Desktop/download/Python-2.5.2/Lib/test/test_doctest.txt
/root/test.txt
/usr/local/lib/python2.5/test/test_doctest.txt
The find and locate commands are great general purpose utilities,
but a rather common need is to determine the location of a program
or determine the version of the program in use. The whereis and
which commands are targeted for this usage.
Command Description
find Search for files within a
given directory hierarchy
locate Faster than find—but uses
cached information in a
database
37
Command Description
whereis Display the location of the
binary, source, and man
page
which Show the full path to
commands—useful for
determining the version of
the command in use
38
Concatenation
chapter 6
Sequencing
In simplest form, you can execute one command per line. However,
you can execute several commands in sequence by separating them
by semicolons. Note that the input and output of each command is
independent of the others in the series.
Redirection
Before UNIX, programs generally had to explicitly connect to the
appropriate input and output data. Standard Input (STDIN) is data
that serves as input to a program. It is expected to come from the
text terminal that started the program. However, you can redirect
input instead. Standard Output (STDOUT) is the output from a pro-
gram. The output by default is directed to the text terminal that
started the program, but this can be redirected as well.
39
% read x # Reading from STDIN
HelloWorld
Appending
To append to the end of a file (when using STDERR or STDOUT),
40
change the > to >>. For example
Pipes
The output of one program can serve as the input to another one.
This is one area where UNIX really shines. Many UNIX users never
make the connection that the many small utilities can be combined
to very powerful effect. Pipes serve as the connectors between utilities
that are combined to perform tasks.
% cat result.txt
It’s tee for two!
The pipe redirects STDOUT to STDIN of the next process. The tee
command reads from STDIN and redirects to STDOUT and the file
result.txt at the same time
Filters
41
sorting/ordering
Many commands include options that allow you to sort their output
(such as ls discussed earlier). If you need to order the results in a
command in a way that is not possible for the command in question,
you can use the sort command. The sort command allows sorting
that varies with space used, type of element (numeric, alphabetic etc),
and location of the element that serves as a key for sorting.
The sort command is often used to filter input that is passed to the
uniq command. Uniq removes duplicates from the list of lines passed
to it. If the lines have not been previously ordered by sort, the out-
put of the uniq command will only eliminate duplicate entries that
appear sequentially in the list of lines.
The sort command includes an option that allows you reverse the
output of an other command. To sort on the 6th field first use -t,
switch to tell sort that the delimiter is a comma and -k to tell sort
which field to sort on:
But there are also several other commands that can be used to
reverse output in various ways. To illustrate, lets start by producing a
list of zero-padded numbers between one and eleven.
% seq -w 11
We can reverse the list by piping to tac (which works like cat in
42
reverse):
% seq -w 11 | tac
We can reverse each element in the list using the rev command:
% seq -w 11 | rev
limiting
There are a number of different options available to limit the output
of files. The grep utility and awk programming language (that is com-
monly used to execute one liners) have a broad range of functionality
that are well beyond the scope of this book. A few minimal examples
will suffice to get you started.
The name of the file in question will be displayed before each line if
there are multiple entries. If you wanted to list every file that did not
contain the word split, you would use the -v option.
The following example uses a regular expression that lists lines that
include 2 or 9.
43
% seq -w 1000 1010 | nl | grep -E ‘2|9’
If you want to use a different delimiter, use the -F option and specify
the delimiter name.
enhancing
Enhancing might not be the best way to describe this type of filtering,
but adding line numbers to output is a common requirement.
% cat environment.rb | nl
44
grouping
It is common to want to redirect the output of a number of a group of
commands. Rather than individually specifying the redirection:
% ls >> output.txt
% df >> output.txt
% who >> output.txt
% (ls
df
who) >> output.txt
The uniq utility with the -c option provides a count of unique entries.
This is somewhat analogous to the use of a GROUP BY in SQL. The
following example displays each file owner within the directory fol-
lowed by the number of files that they own:
Command Description
grep Print lines that match a
specified pattern
awk Special purpose pattern
matching and data format-
ting language
sort Order lines based on
specified criteria
uniq Remove any duplicates
from input
45
Command Description
nl Number lines
tac A version of cat that reads
through a file from the end
to the beginning
rev Reverses inputted lines
If you simply want to see the side-by-side output of two files to make
comparisons for yourself, the paste utility can be used. The join
command can be used to display the lines with a common key field
between two files. Although this is useful for simple comparisons,
comparisons using key fields are better handled in an RDBMS.
46
Command Description
Diff Display the differences
between two files
patch Use the output of diff and
apply it to a file
cmp Compare two files
join Join lines in two files using
a key field
paste Combine two files side by
side
comm Compare two sorted files
side by side
47
% time <my long running command>
Command Description
date Return the system date or
manipulate date strings
time Measure the resource
usage and elapsed execu-
tion time of a command
cal Display a calendar
World on a String
The bash shell provides for string manipulation—but in a rather incon-
sistent manner. Some commands are based on parameter substitu-
tion while others require the expr command.
% function slash_date {
d=$1
echo ${d:0:2}/${d:2:2}/${d:4:4}
}
48
The length of a string can be outputted as follows:
% echo ${username-`whoami`}
Command Description
expr Evaluate an expression
tr Translate or delete charac-
ters in a string
49
Administration
chapter 7
As mentioned earlier, the focus of this books in not on system admin-
istration, and so this section is intentionally brief. It is intended to
serve as a point of reference for communication with system adminis-
trators or as a stepping stone to further investigation.
Installation
Tools such as apt-get, yum, and port can compile and install soft-
ware easily. See the accompanying screencast for an example.
Unix/Linux Installation
If you are installing an operating system for the first time, you will
probably perform the installation in an interactive manner. If you are
going to have to repeat the installation, you should probably consider
automating the installation process.
50
Besides the base operating system, you will have to install other
software (web servers, databases, other language environments,
and packages). You will need to become familiar with the installation
utilities and package manager (rpm) associated with the particular
system in use. You may need to build from source using a series of
commands that will become all too familiar:
% ./configure
% make
% make install
51
Users/Group Administration
Administrators are responsible for creating, updating, and deleting
users (using the commands useradd, usermod, userdel). Users are
categorized into groups, and administrators are also responsible for
maintaining these (groupadd, groupmod, groupdel).
System Monitoring
System administrators typically monitor servers to determine the
general availability of the system (the server is running and acces-
sible, file system space is available, etc). Web Application Develop-
ers need to provide additional information to system administrators
if they are expected to intelligently monitor application resources as
well.
Backup/Recovery
Perhaps the most important (and least appreciated) job of the
administrator is to perform backup procedures that guarantee that a
system can be recovered and data restored in the event of a system
failure. Archives can be created to back up essential files (tar, zip,
cpio) and the rsync utility can be used to mirror files on another
server. There are also a variety of proprietary tools for performing
such tasks.
/proc/cpuinfo
/proc/meminfo
/proc/version
To access a specific bit of information from one of these files, pipe the
results to grep and filter on the specific bit of information desired. For
example, to see Physical RAM:
% fuser -n tcp 80
Other UNIX utilities (netstat, iptab, netcat) are available for net-
work administrative and security concerns.
The curl and wget utilities can be used in scripts used to monitor
web applications or copy web resources. They can be useful for creat-
ing web crawlers and automating web application testing. The follow-
ing is a simple example that could be used to populate a string after
a URL returned an HTTP status of 200. If an empty string was found,
an error would be reported.
#!/bin/bash
result=`curl -sIL localhost:8080/AFASWebConsole | grep ‘HTTP/1.1 200 OK’`
if [ -z “$result” ]; then
echo “ERROR: HTTP Response 200 expected.”
fi
Command Description
ps Display process informa-
tion
pidof Returns the pid of a speci-
fied process
54
Command Description
top Display process informa-
tion interactively
vmstat Display memory statistics
report
free Display information about
system memory (e.g. RAM,
swap space)
du Display file space usage.
sudo du -hs /* will show
the directory sizes of the
root directories.
df Display file system space
usage
watch Periodically execute a
command
last List most recently logged
in users
lastlog List the last login time for
each user
ping Send an echo request to a
server
fuser Display a process using a
file or socket
nmap Display network hosts and
services
wget Non interactive download
of files from the web
curl Command line browser
typically used to copy
URLs
55
Monitoring File Space
Although monitoring the file space of a system generally is the
responsibility of a system administrator, you may need to monitor
specific file systems that are heavily used by your application. The
following example sends an email if the /home filesystem’s disk space
usage exceeds 90%. The script could be added to a user’s crontab
and scheduled to run on a periodic basis.
#!/usr/bin/bash
i=`df | grep /home | awk ‘{sub(“%”,””,$5);print $5 }’`
if [ “$i” -gt “90” ]
then
str=”`hostname` File System at $i”
echo $str | mail -s “`hostname` Alert” [email protected]
fi
Monitoring a Process
The ps command can be used to view running processes. To get all
information available (all processes and full listing) you can specify
a number of different options by including -ef. The resulting listing
can be limited by piping to grep. The process id (PID) is listed and is
helpful for other monitoring activities or stopping the process.
56
to window’s task manager. Various key strokes can be used to modify
the display while the program is running.
If you find yourself repeating the same command over and over (e.g.
running ls -l to check if a file has been created or changed), the
watch command can be used to periodically execute the command
in question. If a file is being changed by another program and you
would like to watch the file as it changes (a typical situation when
monitoring log files), the tail -f command can be used.
A job can be put into the foreground (fg), background (bg, Ctrl-Z,
&). A signal can be sent to a process—usually to stop it—using kill.
The kill -9 option sends a KILL signal that cannot be caught or
ignored. Because it does not allow for graceful termination, it should
be used sparingly.
Scripting
The Job That Ate My Brain
Normally we think about running programs by executing them at the
command line. You can start a process in the background by typing
an & after the command.
% command &
57
Command Description
CTRL-Z Stop a running job
bg Put a job in the back-
ground
fg Bring a job into the fore-
ground
<command> & Run a job in the back-
ground
crontab View or modify the list of
scheduled jobs
at Execute a job at a speci-
fied future time
watch Execute a job periodically
and display the output to
the whole screen
Here are a few lines from a cron configuration file (the first line is
commented):
review
• cron task (http://www.redhat.com/docs/manuals/linux/RHL-7.2-Manual/custom-
guide/cron-task.html)
• cron (http://www.unixgeeks.org/security/newbie/unix/cron-1.html)
Debugging
58
Shell scripting is an activity that you will naturally come to as you
develop sets of commands that can be organized into monolithic
tasks. Because the bash shell serves as an interactive interpreter, the
commands and syntax you are already using can be leveraged when
writing a script. This process is much like the process used by rubyists
to solve problems—experiment in Irb (the interactive ruby shell) and
commit final ideas to a script.
• ~/.profile
• ~/.bash_profile
• ~/.bash_login
The simplest (and most common) technique for debugging scripts is
to use display output using echo and redirect output at various points
during a script’s output for subsequent analysis. The shell might be
invoked with a -v option (verbose) which prints shell input lines as
they are read. However, there are a options available that can mini-
mize these activities using the set built-in. The set -x option will
display each command as it is executed. This enables you to narrow
down what is happening at a given point in a script.
The set options can be specified within the script like so:
% set -ex
59
They also can be turned off (by changing the – to +) if you only want
to set behavior for a portion of the script.
% sh -ex <script_to_debug>.sh
Scripts, like other programs, can suffer from performance issues. The
time command can be used to measure how much time a program
takes to complete.
% time <script_to_time>.sh
60
Appendix: Online Resources
chapter 8
linux
• The Linux Documentation Project (http://tldp.org/LDP/abs/html)
• UNIX Philosophy (http://catb.org/~esr/faqs/loginataka.html)
• FAQs (http://www.faqs.org/docs/artu/ch01s06.html)
utilities
• sed FAQ (http://www.linuxtopia.org/online_books/linux_tool_guides/the_sed_faq/
sedfaq4_005.html)
• top and tail (http://www.brunningonline.net/simon/blog/archives/002192.html)
• Network Settings (http://www.faqs.org/docs/linux_network/index.html)
bash scripting
• IBM Article Part 1 (http://www.ibm.com/developerworks/library/l-bash.html)
• IBM Article Part 2 (http://www.ibm.com/developerworks/library/l-bash2.html)
• IBM Article Part 3 (http://www.ibm.com/developerworks/library/l-bash3.html)
linux on oracle technology network
• Filtering (http://www.oracle.com/technology/pub/articles/saternos-filtering.html)
• Scripting (http://www.oracle.com/technology/pub/articles/saternos_scripting.html)
• Kickstart (http://www.oracle.com/technology/pub/articles/saternos_kickstart.html)
pipelines
• Wikipedia on Pipelines (http://en.wikipedia.org/wiki/Pipeline_%28Unix%29)
61
Appendix: Info Script
chapter 9
#!/bin/bash
echo “ “
echo “---------------------------------”
echo “- S E R V E R -”
echo “---------------------------------”
echo “ “
echo “Node name : `uname -n`”
echo “Machine name : `uname -m`”
echo “Operating System : `uname -o`”
echo “Kernel name : `uname -s`”
echo “Kernel version : `uname -v`”
echo “Kernel release : `uname -r`”
echo “Processor Type : `uname -p`”
echo “Hardware Platform: `uname -i`”
echo “ “
echo “---------------------------------”
echo “- M E M O R Y -”
echo “---------------------------------”
echo “ “
cat /proc/meminfo | grep MemTotal
echo “ “
echo “---------------------------------”
echo “- P R O C E S S O R S -”
echo “---------------------------------”
echo “ “
cat /proc/cpuinfo |grep “model name”| nl
echo “ “
echo “---------------------------------”
echo “- D I S K S P A C E -”
echo “---------------------------------”
echo “ “
df -h
echo “ “
62
63