Stream-Oriented, Non-Interactive

EDitor sed

Lecturer: Prof. Andrzej (AJ) Bieszczad

Email: [email protected]
Phone: 818-677-4954

“UNIX for Programmers and Users”

Third Edition, Prentice-Hall, GRAHAM GLASS, KING ABLES

Slides partially adapted from Kumoh National University of Technology (Korea) and NYU
 Stream-Oriented Non-Interactive Editor sed
Transforming Files: sed
• Stream-oriented, non-interactive, text editor

• Look for patterns one line at a time and change lines accordingly
– like awk.

• Non-interactive text editor

– editing commands come in as script
– there is an interactive editor ed which accepts the same commands

• A Unix filter
– superset of previously mentioned tools

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 2

 Stream-Oriented Non-Interactive Editor sed
sed vs. awk
• sed is a pattern-action language, like awk
• awk processes fields while sed only processes lines

• sed +
– regular expressions
– fast
– concise

• sed –
– hard to remember text from one line to another
– not possible to go backward in the file
– no way to do forward references like /..../+1
– no facilities to manipulate numbers
– cumbersome syntax

• awk +
– convenient numeric processing
– variables and control flow in the actions
– convenient way of accessing fields within lines
– flexible printing
– C-like syntax

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 3

 Stream-Oriented Non-Interactive Editor sed
Sed Usage
• Edit files too large for interactive editing

• Edit any size files where editing sequence is too complicated to type in
interactive mode

• Perform “multiple global” editing functions efficiently in one pass through the
input

• Edit multiples files automatically

• Good tool for writing conversion programs

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 4

 Stream-Oriented Non-Interactive Editor sed
Conceptual overview
• A script is read which contains a list of editing commands
– Can be specified in a file or as an argument

• Before any editing is done, all editing commands are compiled into a form to be
more efficient during the execution phase.

• All editing commands in a sed script are applied in order to each input line.

• If a command changes the input, subsequent command address will be applied

to the current (modified) line in the pattern space, not the original input line.

• The original input file is unchanged (sed is a filter), and the results are sent to
standard output (but can be redirected to a file).

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 5

 Stream-Oriented Non-Interactive Editor sed
Sed Architecture

Input

scriptfile
Input line
(Pattern Space)

Hold Space

Output

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 6

 Stream-Oriented Non-Interactive Editor sed
sed Syntax

• sed [-n] [-e] [‘command’] [file…]

• sed [-n] [-f scriptfile] [file…]

• -n - only print lines specified with the print command (or the ‘p’ flag of the
substitute (‘s’) command)

• -f scriptfile - next argument is a filename containing editing commands

• -e command - the next argument is an editing command rather than a filename,

useful if multiple commands are specified

• If the first line of a scriptfile is “#n”, sed acts as though -n had been specified

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 7

 Stream-Oriented Non-Interactive Editor sed
Scripts
• A script is nothing more than a file of commands

• Each command consists of an address and an action, where the address can be
a regular expression or line number.

address action command

address action
address action
address action
address action

script

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 8

 Stream-Oriented Non-Interactive Editor sed
Scripts (continued)
• As each line of the input file is read, sed reads the first command of the script
and checks the address against the current input line:
– if there is a match, the command is executed
– if there is no match, the command is ignored
– sed then repeats this action for every command in the script file

• When it has reached the end of the script, sed outputs the current line (pattern
space) unless the -n option has been set

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 9

 Stream-Oriented Non-Interactive Editor sed
Sed Flow of Control
• sed then reads the next line in the input file and restarts from the beginning of
the script file

• All commands in the script file are compared to, and potentially act on, all lines
in the input file

script

cmd 1 cmd 2 ... cmd n

print cmd
output
output
input only without -n

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 10

 Stream-Oriented Non-Interactive Editor sed
sed Commands
• sed commands have the general form

[address[, address]][!]command [arguments]

• sed copies each input line into a pattern space

– if the address of the command matches the line in the pattern space, the command is a
pplied to that line

– if the command has no address, it is applied to each line as it enters pattern space

– if a command changes the line in pattern space, subsequent commands operate on the
modified line

• When all commands have been read, the line in pattern space is written to
standard output and a new line is read into pattern space

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 11

 Stream-Oriented Non-Interactive Editor sed
Addressing
• An address can be either a line number or a pattern, enclosed in slashes (
/pattern/ )
• A pattern is described using regular expressions (BREs, as in grep)

• If no pattern is specified, the command will be applied to all lines of the input file

• To refer to the last line: $

• Most commands will accept two addresses

– If only one address is given, the command operates only on that line
– If two comma separated addresses are given, then the command operates on a range
of lines between the first and second address, inclusively

• The ! operator can be used to negate an address; i.e., address!command

causes command to be applied to all lines that do not match address

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 12

 Stream-Oriented Non-Interactive Editor sed
Commands
• command is a single letter

• Example:

Deletion: d

[address1][,address2]d

• Delete the addressed line(s) from the pattern space; line(s) not passed to
standard output.

• A new line of input is read and editing resumes with the first command of the
script.

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 13

 Stream-Oriented Non-Interactive Editor sed
Address and Command Examples: delete
d deletes all lines

6d deletes line 6

/^$/d deletes all blank lines

1,10d deletes lines 1 through 10

1,/^$/d deletes from line 1 through the first blank line

/^$/,$d deletes from the first blank line through

the last line of the file

/^$/,10d deletes from the first blank line through line 10

/^ya*y/,/[0-9]$/d deletes from the first line that begins

with yay, yaay, yaaay, etc through
the first line that ends with a digit

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 14

 Stream-Oriented Non-Interactive Editor sed
Multiple Commands

• Braces {} can be used to apply multiple commands to an address

[address][,address]{
command1
command2
command3
}

• The opening brace must be the last character on a line

• The closing brace must be on a line by itself
• Make sure there are no spaces following the braces

• Alternatively, use “;” after each command:

[address][,address]{command1; command2; command3; }

• Or:

‘[address][,address]command1; command2; command3’

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 15

 Stream-Oriented Non-Interactive Editor sed
Sed Commands

• Although sed contains many editing commands, we are only going to cover the
following subset:

s - substitute
a - append
i - insert
c - change
d – delete
p - print
r - read
w - write
y - transform
= - display line number
N - append the next line to the current one
q - quit

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 16

 Stream-Oriented Non-Interactive Editor sed
Print
• The print command (p) can be used to force the pattern space to be output,
useful if the -n option has been specified

• Syntax:

[address1[,address2]]p

• Note: if the -n or #n option has not been specified, p will cause the line to be
output twice!

• Examples:

1,5p will display lines 1 through 5

/^$/,$p will display the lines from the first

blank line through the last line of the file

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 17

 Stream-Oriented Non-Interactive Editor sed
Substitute
• Syntax:

[address(es)]s/pattern/replacement/[flags]

• pattern - search pattern

• replacement - replacement string for pattern
• flags - optionally any of the following

n a number from 1 to 512 indicating which

occurrence of pattern should be replaced

g global, replace all occurrences of pattern

in pattern space

p print contents of pattern space

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 18

 Stream-Oriented Non-Interactive Editor sed
Substitute Examples

s/Puff Daddy/P. Diddy/

Substitute P. Diddy for the first occurrence
of Puff Daddy in pattern space

s/Tom/Dick/2
Substitutes Dick for the second occurrence
of Tom in the pattern space

s/wood/plastic/p
Substitutes plastic for the first occurrence
of wood and outputs (prints) pattern space

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 19

 Stream-Oriented Non-Interactive Editor sed
Replacement Patterns

• Substitute can use several special characters in the replacement string

& replaced by the entire string matched in the regular

expression for pattern

\n replaced by the nth substring (or subexpression) previously

specified using “\(“ and “\)”

\ used to escape the ampersand (&) and the backslash (\)

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 20

 Stream-Oriented Non-Interactive Editor sed
Replacement Pattern Examples

"the UNIX operating system …"

s/.NI./wonderful &/
--> "the wonderful UNIX operating system …"

$ cat test1
first:second
one:two
$ sed 's/\(.*\):\(.*\)/\2:\1/' test1
second:first
two:one

"unix is fun“
sed 's/\([[:alpha:]]\)\([^ \n]*\)/\2\1ay/g'
--> "nixuay siay unfay"

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 21

 Stream-Oriented Non-Interactive Editor sed
Append, Insert, and Change
• Syntax for these commands is a little strange because they must be specified on
multiple lines

• append

[address]a\
text

• insert

[address]i\
text

• change

[address(es)]c\
text

• append/insert for single lines only, not range

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 22

 Stream-Oriented Non-Interactive Editor sed
Append and Insert
• Append places text after the current line in pattern space

• Insert places text before the current line in pattern space

– each of these commands requires a \ following it

– text must begin on the next line.
– if text begins with whitespace, sed will discard it unless you start the line with a \

• Example:

/<Insert Text Here>/i\

Line 1 of inserted text\
<-- whitespace -> Line discarded by sed\
\ Line 2 of inserted text

would leave the following in the pattern space:

Line 1 of inserted text

Line 2 of inserted text
<Insert Text Here>

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 23

 Stream-Oriented Non-Interactive Editor sed
Change
• Unlike Insert and Append, Change can be applied to either a single line address
or a range of addresses

• When applied to a range, the entire range is replaced by text specified with
change, not each line

• Exception! If the Change command is executed with other commands enclosed

in { } that act on a range of lines, each line will be replaced with text

• No subsequent editing allowed

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 24

 Stream-Oriented Non-Interactive Editor sed
Change Examples
• Remove mail headers from email messages

– the address specifies a range of lines beginning with a line that begins with From until t
he first blank line.

• The first example replaces all lines with a single occurrence of <Mail Header
Removed>.

/^From /,/^$/c\
<Mail Headers Removed>

• The following replaces each line with <Mail Header Removed>

/^From /,/^$/{
s/^From //p
c\
<Mail Header Removed>
}

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 25

 Stream-Oriented Non-Interactive Editor sed
Using !
• If an address is followed by an exclamation point (!), the associated command is
applied to all lines that don’t match the address or address range

• Examples:

1,5!d delete all lines except 1 through 5

/black/!s/cow/horse/
substitute “horse” for “cow” on all lines
except those that contained “black”
• e.g.

“The brown cow” -> “The brown horse”

“The black cow” -> “The black cow”

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 26

 Stream-Oriented Non-Interactive Editor sed
Reading and Writing from/to Files
$ cat tmp
one two three
one three five
two four six
$ sed 'r tmp'
My first line of input ---> no read until the first line is taken from the input
My first line of input
one two three
one three five
two four six
My next line
My next line^D
$ sed 'w tmp1'
hello 1
hello 1
hello 2
hello 2
hello 3
hello 3^D
$ cat tmp1
hello 1
hello 2
hello 3
$_

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 27

 Stream-Oriented Non-Interactive Editor sed
Playing with Line numbers
$ sed '=' tmp
1
hello 1
2
hello 2
3
hello 3
$ sed -n '=' tmp1
1
2
3
$_

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 28

 Stream-Oriented Non-Interactive Editor sed
Concatenating a Line with the Next Line
• Concatenating subsequent lines is done with the N command that appends the
following line into the pattern space together with the separating new line
character.

$ cat tmp1
hello 1
hello 2
hello 3
$ sed '=' tmp1 | sed '{N; s/\n/<NL>/g;}'
1<NL>hello 1
2<NL>hello 2
3<NL>hello 3
$_

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 29

 Stream-Oriented Non-Interactive Editor sed
Transform
• The Transform command (y) operates like tr, it does a one-to-one or character-
to-character replacement

• Transform accepts zero, one or two addresses

[address[,address]]y/abc/xyz/

• every a within the specified address(es) is transformed to an x, b to y and c to z

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

changes all lower case characters on

the addressed line to upper case

• if you only want to transform specific characters (or a word) in the line, it is much
more difficult and requires use of the hold space

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 30

 Stream-Oriented Non-Interactive Editor sed
Pattern and Hold spaces
• Pattern space: Workspace or temporary buffer where a single line of input is
held while the editing commands are applied

• Hold space: Secondary temporary buffer for temporary storage only

in

h – put pattern space into the hold space

H – append Pattern

g – get the hold space into the pattern space Hold

G - append out

x – exchange the contents of the hold space and the pattern space

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 31

 Stream-Oriented Non-Interactive Editor sed
Hold Buffer Example
• Edit a line, print the change and show the original

$ sed ‘/Unix/{h; s/.* Unix \(.*\) .*/\1:/; p; x; }’

This describes the Unix ls command.
ls:
This describes the Unix ls command.
This describes the Unix cp command.
cp:
This describes the Unix cp command.
$ sed '/Unix/{G;}'
This describes the Unix ls command.
This describes the Unix ls command.
---> G takes empty line from the hold buffer.
Something else.
Something else.
Again.
Again.

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 32

 Stream-Oriented Non-Interactive Editor sed
Quit
• Quit causes sed to stop reading new input lines and stop sending them to
standard output

• It takes at most a single line address

– Once a line matching the address is reached, the script will be terminated

– This can be used to save time when you only want to process some portion of the begi
nning of a file

• Example

• to print the first 100 lines of a file (like head)

sed '100q' filename

• sed will, by default, send the first 100 lines of filename to standard output and
then quit processing

Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 33

Stream-Oriented Non-Interactive Editor sed

sed challenge