Back Story

Generator Tricks
For Systems Programmers

I think generators are wicked cool

An extremely useful language feature
Yet, they still seem a rather exotic
I still don't think I've fully wrapped my brain

David Beazley
http://www.dabeaz.com

around the best approach to using them

Presented at PyCon'2008

1- 1

Copyright (C) 2008, http://www.dabeaz.com

1- 5

Copyright (C) 2008, http://www.dabeaz.com

An Introduction

A Complaint
The coverage of generators in most Python
books is lame (mine included)

Generators are cool!

But what are they?
And what are they good for?
That's what this tutorial is about

Look at all of these cool examples!

Fibonacci Numbers
Squaring a list of numbers
Randomized sequences
Wow!
Blow me over!

1- 2

Copyright (C) 2008, http://www.dabeaz.com

About Me

This Tutorial
Some more practical uses of generators
Focus is "systems programming"
Which loosely includes files, file systems,

First started using Python with version 1.3

Author : Python Essential Reference
Responsible for a number of open source
I'm a long-time Pythonista

parsing, networking, threads, etc.

My goal : To provide some more compelling

examples of using generators

Python-related packages (Swig, PLY, etc.)

Planting some seeds

1- 3

Copyright (C) 2008, http://www.dabeaz.com

1- 7

Copyright (C) 2008, http://www.dabeaz.com

My Story

Support Files

My addiction to generators started innocently

enough. I was just a happy Python
programmer working away in my secret lair
when I got "the call." A call to sort through
1.5 Terabytes of C++ source code (~800
weekly snapshots of a million line application).
That's when I discovered the os.walk()
function. I knew this wasn't going to end well...

Copyright (C) 2008, http://www.dabeaz.com

1- 6

Copyright (C) 2008, http://www.dabeaz.com

Files used in this tutorial are available here:

http://www.dabeaz.com/generators/

Go there to follow along with the examples

1- 4

Copyright (C) 2008, http://www.dabeaz.com

1- 8

Disclaimer

Iterating over a Dict

This isn't meant to be an exhaustive tutorial

If you loop over a dictionary you get keys

on generators and related theory

>>> prices = { 'GOOG' : 490.10,

...
'AAPL' : 145.23,
...
'YHOO' : 21.71 }
...
>>> for key in prices:
...
print key
...
YHOO
GOOG
AAPL
>>>

Will be looking at a series of examples

I don't know if the code I've written is the

"best" way to solve any of these problems.

So, let's have a discussion

1- 9

Copyright (C) 2008, http://www.dabeaz.com

1- 13

Iterating over a String

Performance Details
There are some later performance numbers
Python 2.5.1 on OS X 10.4.11
All tests were conducted on the following:
Mac Pro 2x2.66 Ghz Dual-Core Xeon
3 Gbytes RAM
WDC WD2500JS-41SGB0 Disk (250G)
Timings are 3-run average of 'time' command

If you loop over a string, you get characters

>>> s = "Yow!"
>>> for c in s:
...
print c
...
Y
o
w
!
>>>

1- 10

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

1- 14

Iterating over a File

If you loop over a file you get lines
>>> for line in open("real.txt"):
...
print line,
...
Real Programmers write in FORTRAN

Part I

Maybe they do now,

in this decadent era of
Lite beer, hand calculators, and "user-friendly" software
but back in the Good Old Days,
when the term "software" sounded funny
and Real Computers were made out of drums and vacuum tubes,
Real Programmers wrote in machine code.
Not FORTRAN. Not RATFOR. Not, even, assembly language.
Machine Code.
Raw, unadorned, inscrutable hexadecimal numbers.
Directly.

Introduction to Iterators and Generators

1- 11

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

Iteration

Consuming Iterables

As you know, Python has a "for" statement

You use it to loop over a collection of items

Many functions consume an "iterable" object

Reductions:

>>> for x in [1,4,5,10]:

...
print x,
...
1 4 5 10
>>>

sum(s), min(s), max(s)

Constructors
list(s), tuple(s), set(s), dict(s)

And, as you have probably noticed, you can

in operator

iterate over many different kinds of objects

(not just lists)

Copyright (C) 2008, http://www.dabeaz.com

1- 15

item in s

Many others in the library

1- 12

Copyright (C) 2008, http://www.dabeaz.com

1- 16

Iteration Example

Iteration Protocol
The reason why you can iterate over different
objects is that there is a specific protocol

Example use:

>>> items = [1, 4, 5]

>>> it = iter(items)
>>> it.next()
1
>>> it.next()
4
>>> it.next()
5
>>> it.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>>

>>> c =
>>> for
...
...
5 4 3 2
>>>

1- 17

Copyright (C) 2008, http://www.dabeaz.com

Iteration Protocol

countdown(5)
i in c:
print i,
1

1-21

Copyright (C) 2008, http://www.dabeaz.com

Iteration Commentary

An inside look at the for statement

for x in obj:
# statements

There are many subtle details involving the

Underneath the covers

_iter = iter(obj)
while 1:
try:
x = _iter.next()
except StopIteration:
break
# statements
...

design of iterators for various objects

# Get iterator object

However, we're not going to cover that

This isn't a tutorial on "iterators"
We're talking about generators...

# Get next item

# No more items

Any object that supports iter() and next() is

said to be "iterable."

Copyright (C) 2008, http://www.dabeaz.com

1-18

1-22

Copyright (C) 2008, http://www.dabeaz.com

Supporting Iteration

Generators
A generator is a function that produces a

User-defined objects can support iteration

Example: Counting down...

sequence of results instead of a single value

def countdown(n):
while n > 0:
yield n
n -= 1
>>> for i in countdown(5):
...
print i,
...
5 4 3 2 1
>>>

>>> for x in countdown(10):

...
print x,
...
10 9 8 7 6 5 4 3 2 1
>>>

To do this, you just have to make the object

implement __iter__() and next()

Instead of returning a value, you generate a

series of values (using the yield statement)

Copyright (C) 2008, http://www.dabeaz.com

1-19

Supporting Iteration

Generators
Behavior is quite different than normal func
Calling a generator function creates an

Sample implementation

generator object. However, it does not start

running the function.

class countdown(object):
def __init__(self,start):
self.count = start
def __iter__(self):
return self
def next(self):
if self.count <= 0:
raise StopIteration
r = self.count
self.count -= 1
return r

Copyright (C) 2008, http://www.dabeaz.com

1-23

Copyright (C) 2008, http://www.dabeaz.com

def countdown(n):
print "Counting down from", n
while n > 0:
yield n
Notice that no
n -= 1
output was
produced
>>> x = countdown(10)
>>> x
<generator object at 0x58490>
>>>

1-20

Copyright (C) 2008, http://www.dabeaz.com

1-24

Generator Functions

Generator Expressions
A generated version of a list comprehension

The function only executes on next()

>>> x = countdown(10)
>>> x
<generator object at 0x58490>
>>> x.next()
Counting down from 10
10
>>>

Function starts
executing here

yield produces a value, but suspends the function

Function resumes on next call to next()
>>> x.next()
9
>>> x.next()
8
>>>
Copyright (C) 2008, http://www.dabeaz.com

>>> a = [1,2,3,4]
>>> b = (2*x for x in a)
>>> b
<generator object at 0x58760>
>>> for i in b: print b,
...
2 4 6 8
>>>

This loops over a sequence of items and applies

an operation to each item

However, results are produced one at a time

using a generator

1-25

Generator Functions

Copyright (C) 2008, http://www.dabeaz.com

1-29

Generator Expressions
Important differences from a list comp.

When the generator returns, iteration stops

>>> x.next()
1
>>> x.next()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
>>>

Copyright (C) 2008, http://www.dabeaz.com

Does not construct a list.

Only useful purpose is iteration
Once consumed, can't be reused

Example:
>>> a = [1,2,3,4]
>>> b = [2*x for x in a]
>>> b
[2, 4, 6, 8]
>>> c = (2*x for x in a)
<generator object at 0x58760>
>>>

1-26

Generator Functions

Copyright (C) 2008, http://www.dabeaz.com

1-30

Generator Expressions
General syntax
(expression for i in s if cond1
for j in t if cond2
...
if condfinal)

A generator function is mainly a more

convenient way of writing an iterator

What it means

You don't have to worry about the iterator

for i in s:
if cond1:
for j in t:
if cond2:
...
if condfinal: yield expression

protocol (.next, .__iter__, etc.)

It just works
Copyright (C) 2008, http://www.dabeaz.com

1-27

Generators vs. Iterators

1-31

A Note on Syntax

A generator function is slightly different

The parens on a generator expression can

than an object that supports iteration

Copyright (C) 2008, http://www.dabeaz.com

dropped if used as a single function argument

A generator is a one-time operation. You

can iterate over the generated data once,
but if you want to do it again, you have to
call the generator function again.

Example:
sum(x*x for x in s)

This is different than a list (which you can

Generator expression

iterate over as many times as you want)

Copyright (C) 2008, http://www.dabeaz.com

1-28

Copyright (C) 2008, http://www.dabeaz.com

1-32

Interlude

A Non-Generator Soln

We now have two basic building blocks

Generator functions:

Just do a simple for-loop

wwwlog = open("access-log")
total = 0
for line in wwwlog:
bytestr = line.rsplit(None,1)[1]
if bytestr != '-':
total += int(bytestr)

def countdown(n):
while n > 0:
yield n
n -= 1

Generator expressions

print "Total", total

squares = (x*x for x in s)

We read line-by-line and just update a sum

However, that's so 90s...

In both cases, we get an object that

generates values (which are typically

consumed in a for loop)
1-33

Copyright (C) 2008, http://www.dabeaz.com

1-37

Copyright (C) 2008, http://www.dabeaz.com

A Generator Solution
Let's use some generator expressions
wwwlog
= open("access-log")
bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
bytes
= (int(x) for x in bytecolumn if x != '-')

Part 2

print "Total", sum(bytes)

Processing Data Files

Whoa! That's different!

Less code
A completely different programming style

(Show me your Web Server Logs)

1- 34

Copyright (C) 2008, http://www.dabeaz.com

Programming Problem

Generators as a Pipeline
To understand the solution, think of it as a data

Find out how many bytes of data were

transferred by summing up the last column
of data in this Apache web server log

processing pipeline

access-log
81.107.39.38 81.107.39.38 81.107.39.38 81.107.39.38 81.107.39.38 66.249.72.134 -

...
...
...
...
...
...

"GET
"GET
"GET
"GET
"GET
"GET

/ply/ HTTP/1.1" 200 7587

/favicon.ico HTTP/1.1" 404 133
/ply/bookplug.gif HTTP/1.1" 200 23903
/ply/ply.html HTTP/1.1" 200 97238
/ply/example.html HTTP/1.1" 200 2359
/index.html HTTP/1.1" 200 4447

bytecolumn

bytes

sum()

total

wwwlog
= open("access-log")
bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
bytes
= (int(x) for x in bytecolumn if x != '-')
print "Total", sum(bytes)

1-35

1-39

Copyright (C) 2008, http://www.dabeaz.com

The Log File

Being Declarative
At each step of the pipeline, we declare an

Each line of the log looks like this:

81.107.39.38 -

wwwlog

Each step is defined by iteration/generation

Oh yeah, and the log file might be huge (Gbytes)

Copyright (C) 2008, http://www.dabeaz.com

1-38

Copyright (C) 2008, http://www.dabeaz.com

... "GET /ply/ply.html HTTP/1.1" 200 97238

The number of bytes is the last column

operation that will be applied to the entire

input stream

access-log

wwwlog

bytecolumn

bytes

sum()

total

bytestr = line.rsplit(None,1)[1]

It's either a number or a missing value (-)

81.107.39.38 -

bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)

... "GET /ply/ HTTP/1.1" 304 -

Converting the value

This operation gets applied to

every line of the log file

if bytestr != '-':
bytes = int(bytestr)

Copyright (C) 2008, http://www.dabeaz.com

1-36

Copyright (C) 2008, http://www.dabeaz.com

1-40

Being Declarative

Commentary
Not only was it not slow, it was 5% faster
And it was less code
And it was relatively easy to read
And frankly, I like it a whole better...

Instead of focusing on the problem at a

line-by-line level, you just break it down

into big operations that operate on the
whole file

This is very much a "declarative" style

The key : Think big...

"Back in the old days, we used AWK for this and

we liked it. Oh, yeah, and get off my lawn!"
1-41

Copyright (C) 2008, http://www.dabeaz.com

Iteration is the Glue

1-45

Copyright (C) 2008, http://www.dabeaz.com

Performance Contest

The glue that holds the pipeline together is the

iteration that occurs in each step

wwwlog
= open("access-log")
bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
bytes
= (int(x) for x in bytecolumn if x != '-')

wwwlog

print "Total", sum(bytes)

= open("access-log")

bytes

Time
25.96

bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)

= (int(x) for x in bytecolumn if x != '-')

print "Total", sum(bytes)

% awk '{ total += $NF } END { print total }' big-access-log

The calculation is being driven by the last step

The sum() function is consuming values being

Note:extracting the last

column may not be
awk's strong point

pushed through the pipeline (via .next() calls)

1-42

Copyright (C) 2008, http://www.dabeaz.com

Performance

37.33
1-46

Copyright (C) 2008, http://www.dabeaz.com

Food for Thought

Surely, this generator approach has all

At no point in our generator solution did

sorts of fancy-dancy magic that is slow.

we ever create large temporary lists

Let's check it out on a 1.3Gb log file...

Thus, not only is that solution faster, it can

be applied to enormous data files

% ls -l big-access-log
-rw-r--r-- beazley 1303238000 Feb 29 08:06 big-access-log

It's competitive with traditional tools

1-43

Copyright (C) 2008, http://www.dabeaz.com

1-47

Copyright (C) 2008, http://www.dabeaz.com

Performance Contest
wwwlog = open("big-access-log")
total = 0
for line in wwwlog:
bytestr = line.rsplit(None,1)[1]
if bytestr != '-':
total += int(bytestr)

Time

More Thoughts
The generator solution was based on the

Time

concept of pipelining data between

different components

27.20

What if you had more advanced kinds of

print "Total", total

components to work with?

Perhaps you could perform different kinds

wwwlog
= open("big-access-log")
bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
bytes
= (int(x) for x in bytecolumn if x != '-')
print "Total", sum(bytes)

of processing by just plugging various

pipeline components together

Time
25.96

Copyright (C) 2008, http://www.dabeaz.com

1-44

Copyright (C) 2008, http://www.dabeaz.com

1-48

This Sounds Familiar

find
Generate all filenames in a directory tree
that match a given filename pattern

Have a collection of useful system utils

Can hook these up to files or each other
Perform complex tasks by piping data
The Unix philosophy

import os
import fnmatch
def gen_find(filepat,top):
for path, dirlist, filelist in os.walk(top):
for name in fnmatch.filter(filelist,filepat):
yield os.path.join(path,name)

Examples
pyfiles = gen_find("*.py","/")
logs
= gen_find("access-log*","/usr/www/")

1-49

Copyright (C) 2008, http://www.dabeaz.com

1-53

Copyright (C) 2008, http://www.dabeaz.com

Performance Contest
pyfiles = gen_find("*.py","/")
for name in pyfiles:
print name

Wall Clock Time

% find / -name '*.py'

Wall Clock Time

559s

Part 3
Fun with Files and Directories

468s
Performed on a 750GB file system
containing about 140000 .py files
1- 50

Copyright (C) 2008, http://www.dabeaz.com

Programming Problem

A File Opener
Open a sequence of filenames

You have hundreds of web server logs scattered

across various directories. In additional, some of
the logs are compressed. Modify the last program
so that you can easily read all of these logs

import gzip, bz2

def gen_open(filenames):
for name in filenames:
if name.endswith(".gz"):
yield gzip.open(name)
elif name.endswith(".bz2"):
yield bz2.BZ2File(name)
else:
yield open(name)

foo/
access-log-012007.gz
access-log-022007.gz
access-log-032007.gz
...
access-log-012008

This is interesting.... it takes a sequence of

filenames as input and yields a sequence of open

file objects

bar/
access-log-092007.bz2
...
access-log-022008

1-51

Copyright (C) 2008, http://www.dabeaz.com

1-54

Copyright (C) 2008, http://www.dabeaz.com

cat

os.walk()
A very useful function for searching the

Concatenate items from one or more

file system

source into a single sequence of items

import os

def gen_cat(sources):
for s in sources:
for item in s:
yield item

for path, dirlist, filelist in os.walk(topdir):

# path
: Current directory
# dirlist : List of subdirectories
# filelist : List of files
...

Example:

lognames = gen_find("access-log*", "/usr/www")

logfiles = gen_open(lognames)
loglines = gen_cat(logfiles)

This utilizes generators to recursively walk

through the file system

Copyright (C) 2008, http://www.dabeaz.com

1-55

Copyright (C) 2008, http://www.dabeaz.com

1-52

Copyright (C) 2008, http://www.dabeaz.com

1-56

Programming Problem

grep
Generate a sequence of lines that contain

Web server logs consist of different columns of

data. Parse each line into a useful data structure
that allows us to easily inspect the different fields.

a given regular expression

import re

def gen_grep(pat, lines):

patc = re.compile(pat)
for line in lines:
if patc.search(line): yield line

81.107.39.38 - - [24/Feb/2008:00:08:59 -0600] "GET ..." 200 7587

Example:
lognames
logfiles
loglines
patlines

=
=
=
=

gen_find("access-log*", "/usr/www")
gen_open(lognames)
gen_cat(logfiles)
gen_grep(pat, loglines)

host referrer user [datetime] "request" status bytes

1-57

Copyright (C) 2008, http://www.dabeaz.com

Example

Parsing with Regex

Let's route the lines through a regex parser

Find out how many bytes transferred for a

specific pattern in a whole directory of logs

pat
logdir

= r"somepattern"
= "/some/dir/"

filenames
logfiles
loglines
patlines
bytecolumn
bytes

=
=
=
=
=
=

logpats = r'(\S+) (\S+) (\S+) \[(.*?)\] '\

r'"(\S+) (\S+) (\S+)" (\S+) (\S+)'
logpat = re.compile(logpats)

gen_find("access-log*",logdir)
gen_open(filenames)
gen_cat(logfiles)
gen_grep(pat,loglines)
(line.rsplit(None,1)[1] for line in patlines)
(int(x) for x in bytecolumn if x != '-')

print "Total", sum(bytes)

1-58

Copyright (C) 2008, http://www.dabeaz.com

Important Concept

= (logpat.match(line) for line in loglines)

= (g.groups() for g in groups if g)

This generates a sequence of tuples

('71.201.176.194', '-', '-', '26/Feb/2008:10:30:08 -0600',
'GET', '/ply/ply.html', 'HTTP/1.1', '200', '97238')

1-62

Copyright (C) 2008, http://www.dabeaz.com

Let's turn tuples into dictionaries

code that uses the results of the iteration

In the last example, we're performing a
calculation on a sequence of lines

colnames

= ('host','referrer','user','datetime',
'method','request','proto','status','bytes')

log

= (dict(zip(colnames,t)) for t in tuples)

This generates a sequence of named fields

It doesn't matter where or how those

{ 'status' :
'proto'
:
'referrer':
'request' :
'bytes'
:
'datetime':
'host'
:
'user'
:
'method' :

lines are generated

Thus, we can plug any number of

components together up front as long as

they eventually produce a line sequence
1-59

Copyright (C) 2008, http://www.dabeaz.com

groups
tuples

Tuples to Dictionaries

Generators decouple iteration from the

1-61

Copyright (C) 2008, http://www.dabeaz.com

'200',
'HTTP/1.1',
'-',
'/ply/ply.html',
'97238',
'24/Feb/2008:00:08:59 -0600',
'140.180.132.213',
'-',
'GET'}

Copyright (C) 2008, http://www.dabeaz.com

1-63

Field Conversion
Map specific dictionary fields through a function
def field_map(dictseq,name,func):
for d in dictseq:
d[name] = func(d[name])
yield d

Part 4

Example: Convert a few field values

Parsing and Processing Data

log = field_map(log,"status", int)

log = field_map(log,"bytes",
lambda s: int(s) if s !='-' else 0)

Copyright (C) 2008, http://www.dabeaz.com

1- 60

Copyright (C) 2008, http://www.dabeaz.com

1-64

Field Conversion

Example Use

Creates dictionaries of converted values

{ 'status': 200,
'proto': 'HTTP/1.1',
Note
'referrer': '-',
'request': '/ply/ply.html',
'datetime': '24/Feb/2008:00:08:59 -0600',
'bytes': 97238,
'host': '140.180.132.213',
'user': '-',
'method': 'GET'}

It's easy

conversion

lines = lines_from_dir("access-log*","www")
log
= apache_log(lines)
for r in log:
print r

Different components have been subdivided

Again, this is just one big processing pipeline

according to the data that they process

1-65

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

The Code So Far

lognames
logfiles
loglines
groups
tuples

=
=
=
=
=

A Query Language
Now that we have our log, let's do some queries
Find the set of all documents that 404

gen_find("access-log*","www")
gen_open(lognames)
gen_cat(logfiles)
(logpat.match(line) for line in loglines)
(g.groups() for g in groups if g)

stat404 = set(r['request'] for r in log

if r['status'] == 404)

colnames = ('host','referrer','user','datetime','method',
'request','proto','status','bytes')
log
log

Print all requests that transfer over a megabyte

= (dict(zip(colnames,t)) for t in tuples)

= field_map(log,"bytes",
lambda s: int(s) if s != '-' else 0)
= field_map(log,"status",int)

log

large = (r for r in log

if r['bytes'] > 1000000)
for r in large:
print r['request'], r['bytes']

1-66

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

Packaging

Find the largest data transfer

parts of the code into functions

print "%d %s" % max((r['bytes'],r['request'])

for r in log)

def lines_from_dir(filepat, dirname):

names
= gen_find(filepat,dirname)
files
= gen_open(names)
lines
= gen_cat(files)
return lines

Collect all unique host IP addresses

hosts = set(r['host'] for r in log)

This is a generate purpose function that reads all

lines from a series of files in a directory

Packaging

Copyright (C) 2008, http://www.dabeaz.com

1-71

A Query Language

Parse an Apache log

Find out who has been hitting robots.txt

def apache_log(lines):
groups
= (logpat.match(line) for line in lines)
tuples
= (g.groups() for g in groups if g)

addrs = set(r['host'] for r in log

if 'robots.txt' in r['request'])

colnames

= ('host','referrer','user','datetime','method',
'request','proto','status','bytes')

log
log

= (dict(zip(colnames,t)) for t in tuples)

= field_map(log,"bytes",
lambda s: int(s) if s != '-' else 0)
= field_map(log,"status",int)

log

Find the number of downloads of a file

sum(1 for r in log
if r['request'] == '/ply/ply-2.3.tar.gz')

1-67

Copyright (C) 2008, http://www.dabeaz.com

1-70

A Query Language

To make it more sane, you may want to package

1-69

import socket
for addr in addrs:
try:
print socket.gethostbyaddr(addr)[0]
except socket.herror:
print addr

return log

Copyright (C) 2008, http://www.dabeaz.com

1-68

Copyright (C) 2008, http://www.dabeaz.com

1-72

Performance Study

Infinite Sequences

Sadly, the last example doesn't run so fast on a

Tailing a log file results in an "infinite" stream

It constantly watches the file and yields lines as

huge input file (53 minutes on the 1.3GB log)

But, the beauty of generators is that you can plug

filters in at almost any stage
lines
lines
log
addrs
...

=
=
=
=

soon as new data is written

But you don't know how much data will actually

lines_from_dir("big-access-log",".")
(line for line in lines if 'robots.txt' in line)
apache_log(lines)
set(r['host'] for r in log)

be written (in advance)

And log files can often be enormous

That version takes 93 seconds

1-73

Copyright (C) 2008, http://www.dabeaz.com

Some Thoughts

Tailing a File
A Python version of 'tail -f'

I like the idea of using generator expressions as a

import time
def follow(thefile):
thefile.seek(0,2)
# Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
# Sleep briefly
continue
yield line

pipeline query language

You can write simple filters, extract data, etc.

You you pass dictionaries/objects through the
pipeline, it becomes quite powerful

Feels similar to writing SQL queries

Idea : Seek to the end of the file and repeatedly

try to read new lines. If new data is written to

the file, we'll pick it up.

1-74

Copyright (C) 2008, http://www.dabeaz.com

1-77

Copyright (C) 2008, http://www.dabeaz.com

1-78

Copyright (C) 2008, http://www.dabeaz.com

Example
Using our follow function

Part 5

logfile = open("access-log")
loglines = follow(logfile)
for line in loglines:
print line,

Processing Infinite Data

This produces the same output as 'tail -f'

1- 75

Copyright (C) 2008, http://www.dabeaz.com

1-79

Copyright (C) 2008, http://www.dabeaz.com

Question

Example

Have you ever used 'tail -f' in Unix?

Turn the real-time log file into records

% tail -f logfile
...
... lines of output ...
...

logfile = open("access-log")
loglines = follow(logfile)
log
= apache_log(loglines)

This prints the lines written to the end of a file

The "standard" way to watch a log file
I used this all of the time when working on

Print out all 404 requests as they happen

r404 = (r for r in log if r['status'] == 404)
for r in r404:
print r['host'],r['datetime'],r['request']

scientific simulations ten years ago...

Copyright (C) 2008, http://www.dabeaz.com

1-76

Copyright (C) 2008, http://www.dabeaz.com

1-80

Commentary

Generating Connections
Generate a sequence of TCP connections

We just plugged this new input scheme onto

the front of our processing pipeline

import socket
def receive_connections(addr):
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind(addr)
s.listen(5)
while True:
client = s.accept()
yield client

Everything else still works, with one caveat-

functions that consume an entire iterable won't

terminate (min, max, sum, set, etc.)

Nevertheless, we can easily write processing

Example:

steps that operate on an infinite data stream

for c,a in receive_connections(("",9000)):

c.send("Hello World\n")
c.close()

1-81

Copyright (C) 2008, http://www.dabeaz.com

Thoughts

Copyright (C) 2008, http://www.dabeaz.com

1-85

Generating Messages
Receive a sequence of UDP messages
import socket
def receive_messages(addr,maxsize):
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.bind(addr)
while True:
msg = s.recvfrom(maxsize)
yield msg

Captures a lot of common systems problems

Especially consumer-producer problems

This data pipeline idea is really quite powerful

Example:
for msg, addr in receive_messages(("",10000),1024):
print msg, "from", addr

1-82

Copyright (C) 2008, http://www.dabeaz.com

Copyright (C) 2008, http://www.dabeaz.com

1-86

I/O Multiplexing
Generating I/O events on a set of sockets
import select
def gen_events(socks):
while True:
rdr,wrt,err = select.select(socks,socks,socks,0.1)
for r in rdr:
yield "read",r
for w in wrt:
yield "write",w
for e in err:
yield "error",e

Part 6
Feeding the Pipeline

Note: Using this one is little tricky

Example : Reading from multiple client sockets
Copyright (C) 2008, http://www.dabeaz.com

1- 83

Feeding Generators

Copyright (C) 2008, http://www.dabeaz.com

1-87

I/O Multiplexing
clientset = []

In order to feed a generator processing

def acceptor(sockset,addr):
for c,a in receive_connections(addr):
sockset.append(c)

pipeline, you need to have an input source

So far, we have looked at two file-based inputs

Reading a file

acc_thr = threading.Thread(target=acceptor,
args=(clientset,("",12000))
acc_thr.setDaemon(True)
acc_thr.start()
for evt,s in gen_events(clientset):
if evt == 'read':
data = s.recv(1024)
if not data:
print "Closing", s
s.close()
clientset.remove(s)
else:
print s,data

lines = open(filename)

Tailing a file
lines = follow(open(filename))

Copyright (C) 2008, http://www.dabeaz.com

1-84

Copyright (C) 2008, http://www.dabeaz.com

1-88

Consuming a Queue

Pickler/Unpickler
Turn a generated sequence into pickled objects

Generate a sequence of items from a queue

def gen_pickle(source):
for item in source:
yield pickle.dumps(item)

def consume_queue(thequeue):
while True:
item = thequeue.get()
if item is StopIteration: break
yield item

def gen_unpickle(infile):
while True:
try:
item = pickle.load(infile)
yield item
except EOFError:
return

Note: Using StopIteration as a sentinel

Might be used to feed a generator pipeline as a
consumer thread

Now, attach these to a pipe or socket

1-89

Copyright (C) 2008, http://www.dabeaz.com

Consuming a Queue

1-93

Copyright (C) 2008, http://www.dabeaz.com

Sender/Receiver
Example: Sender

Example:

def sendto(source,addr):
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(addr)
for pitem in gen_pickle(source):
s.sendall(pitem)
s.close()

import Queue, threading

def consumer(q):
for item in consume_queue(q):
print "Consumed", item
print "Done"

Example: Receiver

in_q = Queue.Queue()
con_thr = threading.Thread(target=consumer,args=(in_q,))
con_thr.start()

def receivefrom(addr):
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind(addr)
s.listen(5)
c,a = s.accept()
for item in gen_unpickle(c.makefile()):
yield item
c.close()

for i in xrange(100):
in_q.put(i)
in_q.put(StopIteration)

1-90

Copyright (C) 2008, http://www.dabeaz.com

1-94

Copyright (C) 2008, http://www.dabeaz.com

Example Use
Example: Read log lines and parse into records
# netprod.py

Part 7

lines = follow(open("access-log"))
log
= apache_log(lines)
sendto(log,("",15000))

Extending the Pipeline

Example: Pick up the log on another machine

# netcons.py
for r in receivefrom(("",15000)):
print r

1- 91

Copyright (C) 2008, http://www.dabeaz.com

1-95

Copyright (C) 2008, http://www.dabeaz.com

Multiple Processes

Fanning Out

Can you extend a processing pipeline across

In all of our examples, the processing pipeline is

driven by a single consumer

processes and machines?

for item in gen:

# Consume item

Can you expand the pipeline to multiple

socket
pipe

consumers?

process 2

generator

process 1
consumer1
Copyright (C) 2008, http://www.dabeaz.com

1-92

Copyright (C) 2008, http://www.dabeaz.com

consumer2

consumer3
1-96

Broadcasting

Network Consumer
Example Usage:

Consume a generator and send items to a set

of consumers

class Stat404(NetConsumer):
def send(self,item):
if item['status'] == 404:
NetConsumer.send(self,item)

def broadcast(source, consumers):

for item in source:
for c in consumers:
c.send(item)

lines = follow(open("access-log"))
log
= apache_log(lines)

This changes the control-flow

The broadcaster is what consumes items
Those items have to be sent to consumers for

stat404 = Stat404(("somehost",15000))
broadcast(log, [stat404])

The 404 entries will go elsewhere...

processing

1-97

Copyright (C) 2008, http://www.dabeaz.com

Consumers

Consumer Thread
Example:

To create a consumer, define an object with a

send method on it

import Queue, threading

class ConsumerThread(threading.Thread):
def __init__(self,target):
threading.Thread.__init__(self)
self.setDaemon(True)
self.in_queue = Queue.Queue()
self.target = target
def send(self,item):
self.in_queue.put(item)
def generate(self):
while True:
item = self.in_queue.get()
yield item
def run(self):
self.target(self.generate())

class Consumer(object):
def send(self,item):
print self, "got", item

Example:
c1 = Consumer()
c2 = Consumer()
c3 = Consumer()
lines = follow(open("access-log"))
broadcast(lines,[c1,c2,c3])

1-98

Copyright (C) 2008, http://www.dabeaz.com

1-102

Copyright (C) 2008, http://www.dabeaz.com

Consumers

Consumer Thread

Sadly, inside consumers, it is not possible to

Sample usage (building on earlier code)

def find_404(log):
for r in (r for r in log if r['status'] == 404):
print r['status'],r['datetime'],r['request']

continue the same processing pipeline idea

In order for it to work, there has to be a single

def bytes_transferred(log):
total = 0
for r in log:
total += r['bytes']
print "Total bytes", total

iteration that is driving the pipeline

With multiple consumers, you would have to be

iterating in more than one location at once

c1 = ConsumerThread(find_404)
c1.start()
c2 = ConsumerThread(bytes_transferred)
c2.start()

You can do this with threads or distributed

processes however

Copyright (C) 2008, http://www.dabeaz.com

1-101

Copyright (C) 2008, http://www.dabeaz.com

lines = follow(open("access-log")) # Follow a log

log
= apache_log(lines)
# Turn into records
broadcast(log,[c1,c2])
# Broadcast to consumers

1-99

Network Consumer

1-103

Copyright (C) 2008, http://www.dabeaz.com

Multiple Sources

Example:

In all of our examples, the processing pipeline is

import socket,pickle
class NetConsumer(object):
def __init__(self,addr):
self.s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
self.s.connect(addr)
def send(self,item):
pitem = pickle.dumps(item)
self.s.sendall(pitem)
def close(self):
self.s.close()

being fed by a single source

But, what if you had multiple sources?

source1

source2

source3

This will route items to a network receiver

Copyright (C) 2008, http://www.dabeaz.com

1-100

Copyright (C) 2008, http://www.dabeaz.com

1-104

Concatenation

Multiplexing Generators
def gen_multiplex(genlist):
item_q = Queue.Queue()
def run_one(source):
for item in source: item_q.put(item)

Concatenate one source after another

def concatenate(sources):
for s in sources:
for item in s:
yield item

def run_all():
Each generator runs in a
thrlist = []
thread and drops items
for source in genlist:
t = threading.Thread(target=run_one,args=(source,))
onto a queue
t.start()
thrlist.append(t)
for t in thrlist: t.join()
item_q.put(StopIteration)

This generates one big sequence

Consumes each generator one at a time
Only works with generators that terminate

threading.Thread(target=run_all).start()
while True:
item = item_q.get()
if item is StopIteration: return
yield item

1-105

Copyright (C) 2008, http://www.dabeaz.com

Parallel Iteration

1-109

Copyright (C) 2008, http://www.dabeaz.com

Multiplexing Generators
def gen_multiplex(genlist):
item_q = Queue.Queue()
def run_one(source):
for item in source: item_q.put(item)

Zipping multiple generators together

import itertools

def run_all():
thrlist = []
for source in genlist:
t = threading.Thread(target=run_one,args=(source,))
t.start()
thrlist.append(t)
for t in thrlist: t.join()
Pull items off the queue
item_q.put(StopIteration)

z = itertools.izip(s1,s2,s3)

This one is only marginally useful

Requires generators to go lock-step
Terminates when the first exits

and yield them

threading.Thread(target=run_all).start()
while True:
item = item_q.get()
if item is StopIteration: return
yield item

1-106

Copyright (C) 2008, http://www.dabeaz.com

Multiplexing

Multiplexing Generators
def gen_multiplex(genlist):
Run all of the
item_q = Queue.Queue()
def run_one(source):
generators, wait for
for item in source: item_q.put(item)

Consumer from multiple generators in real-

them
to terminate, then put a
sentinel on the queue
(StopIteration)

time--producing values as they are generated

def run_all():
thrlist = []
for source in genlist:
t = threading.Thread(target=run_one,args=(source,))
t.start()
thrlist.append(t)
for t in thrlist: t.join()
item_q.put(StopIteration)

Example use
log1 = follow(open("foo/access-log"))
log2 = follow(open("bar/access-log"))
lines = gen_multiplex([log1,log2])

There is no way to poll a generator. So, how do

you do this?

Copyright (C) 2008, http://www.dabeaz.com

1-110

Copyright (C) 2008, http://www.dabeaz.com

1-107

threading.Thread(target=run_all).start()
while True:
item = item_q.get()
if item is StopIteration: return
yield item

1-111

Copyright (C) 2008, http://www.dabeaz.com

Multiplexing Generators
def gen_multiplex(genlist):
item_q = Queue.Queue()
def run_one(source):
for item in source: item_q.put(item)
def run_all():
thrlist = []
for source in genlist:
t = threading.Thread(target=run_one,args=(source,))
t.start()
thrlist.append(t)
for t in thrlist: t.join()
item_q.put(StopIteration)

Part 8
Various Programming Tricks (And Debugging)

threading.Thread(target=run_all).start()
while True:
item = item_q.get()
if item is StopIteration: return
yield item
Copyright (C) 2008, http://www.dabeaz.com

1-108

Copyright (C) 2008, http://www.dabeaz.com

1-112

Putting it all Together

Shutting Down
Generators can be shut down using .close()

import time
def follow(thefile):
thefile.seek(0,2)
# Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
# Sleep briefly
continue
yield line

This data processing pipeline idea is powerful

But, it's also potentially mind-boggling
Especially when you have dozens of pipeline
stages, broadcasting, multiplexing, etc.

Example:

Let's look at a few useful tricks

lines = follow(open("access-log"))
for i,line in enumerate(lines):
print line,
if i == 10: lines.close()

1-113

Copyright (C) 2008, http://www.dabeaz.com

Creating Generators

Shutting Down

Any single-argument function is easy to turn

In the generator, GeneratorExit is raised

into a generator function

import time
def follow(thefile):
thefile.seek(0,2)
# Go to the end of the file
try:
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
# Sleep briefly
continue
yield line
except GeneratorExit:
print "Follow: Shutting down"

def generate(func):
def gen_func(s):
for item in s:
yield func(item)
return gen_func

Example:

gen_sqrt = generate(math.sqrt)
for x in gen_sqrt(xrange(100)):
print x

This allows for resource cleanup (if needed)

1-114

Copyright (C) 2008, http://www.dabeaz.com

Debug Tracing

import time
def follow(thefile):
thefile.seek(0,2)
# Go to the end of the file
while True:
try:
line = thefile.readline()
if not line:
time.sleep(0.1)
# Sleep briefly
continue
yield line
except GeneratorExit:
print "Forget about it"

This can easily be placed around any generator

lines = follow(open("access-log"))
log
= trace(apache_log(lines))
= trace(r for r in log if r['status'] == 404)

Answer: No. You'll get a RuntimeError

Note: Might consider logging module for this

Copyright (C) 2008, http://www.dabeaz.com

1-118

Question: Can you ignore GeneratorExit?

def trace(source):
for item in source:
print item
yield item

Copyright (C) 2008, http://www.dabeaz.com

Ignoring Shutdown

A debugging function that will print items going

through a generator

r404

1-117

Copyright (C) 2008, http://www.dabeaz.com

1-115

Copyright (C) 2008, http://www.dabeaz.com

Recording the Last Item

Shutdown and Threads

Store the last item generated in the generator

Question : Can a thread shutdown a generator

class storelast(object):
def __init__(self,source):
self.source = source
def next(self):
item = self.source.next()
self.last = item
return item
def __iter__(self):
return self

1-119

running in a different thread?

lines = follow(open("foo/test.log"))
def sleep_and_close(s):
time.sleep(s)
lines.close()
threading.Thread(target=sleep_and_close,args=(30,)).start()

This can be easily wrapped around a generator

for line in lines:

print line,

lines = storelast(follow(open("access-log")))
log
= apache_log(lines)
for r in log:
print r
print lines.last
Copyright (C) 2008, http://www.dabeaz.com

1-116

Copyright (C) 2008, http://www.dabeaz.com

1-120

Shutdown and Threads

Shutdown

Separate threads can not call .close()

Output:

Example:
import threading,signal

Exception in thread Thread-1:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.5/
lib/python2.5/threading.py", line 460, in __bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.5/
lib/python2.5/threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "genfollow.py", line 31, in sleep_and_close
lines.close()
ValueError: generator already executing

1-121

Copyright (C) 2008, http://www.dabeaz.com

shutdown = threading.Event()
def sigusr1(signo,frame):
print "Closing it down"
shutdown.set()
signal.signal(signal.SIGUSR1,sigusr1)
lines = follow(open("access-log"),shutdown)
for line in lines:
print line,

1-125

Copyright (C) 2008, http://www.dabeaz.com

Shutdown and Signals

Can you shutdown a generator with a signal?
import signal
def sigusr1(signo,frame):
print "Closing it down"
lines.close()

Part 9

signal.signal(signal.SIGUSR1,sigusr1)

Co-routines

lines = follow(open("access-log"))
for line in lines:
print line,

From the command line

% kill -USR1 pid

1-122

Copyright (C) 2008, http://www.dabeaz.com

Shutdown and Signals

1-126

Copyright (C) 2008, http://www.dabeaz.com

The Final Frontier

In Python 2.5, generators picked up the ability

This also fails:

to receive values using .send()

Traceback (most recent call last):

File "genfollow.py", line 35, in <module>
for line in lines:
File "genfollow.py", line 8, in follow
time.sleep(0.1)
File "genfollow.py", line 30, in sigusr1
lines.close()
ValueError: generator already executing

def recv_count():
try:
while True:
n = (yield)
# Yield expression
print "T-minus", n
except GeneratorExit:
print "Kaboom!"

Sigh.

Think of this function as receiving values rather

than generating them

1-123

Copyright (C) 2008, http://www.dabeaz.com

Shutdown

Example Use

The only way to externally shutdown a

generator would be to instrument with a flag or

some kind of check
def follow(thefile,shutdown=None):
thefile.seek(0,2)
while True:
if shutdown and shutdown.isSet(): break
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line

Copyright (C) 2008, http://www.dabeaz.com

1-127

Copyright (C) 2008, http://www.dabeaz.com

1-124

Using a receiver
>>> r = recv_count()
>>> r.next()
>>> for i in range(5,0,-1):
...
r.send(i)
...
T-minus 5
T-minus 4
T-minus 3
T-minus 2
T-minus 1
>>> r.close()
Kaboom!
>>>

Copyright (C) 2008, http://www.dabeaz.com

Note: must call .next() here

1-128

Co-routines

Coroutine Pipelines

This form of a generator is a "co-routine"

Also sometimes called a "reverse-generator"
Python books (mine included) do a pretty poor

Co-routines also set up a processing pipeline

Instead of being defining by iteration, it's
defining by pushing values into the pipeline
using .send()

job of explaining how co-routines are supposed

to be used
.send()

I like to think of them as "receivers" or

"consumer". They receive values sent to them.

Copyright (C) 2008, http://www.dabeaz.com

.send()

.send()

We already saw some of this with broadcasting

1-129

Setting up a Coroutine

1-133

Copyright (C) 2008, http://www.dabeaz.com

Broadcasting (Reprise)

To get a co-routine to run properly, you have to

ping it with a .next() operation first

Consume a generator and send items to a set

of consumers

def recv_count():
try:
while True:
n = (yield)
# Yield expression
print "T-minus", n
except GeneratorExit:
print "Kaboom!"

def broadcast(source, consumers):

for item in source:
for c in consumers:
c.send(item)

Notice that send() operation there

The consumers could be co-routines

Example:
r = recv_count()
r.next()

This advances it to the first yield--where it will

receive its first value

Copyright (C) 2008, http://www.dabeaz.com

1-130

@consumer decorator

Example
@consumer
def find_404():
while True:
r = (yield)
if r['status'] == 404:
print r['status'],r['datetime'],r['request']

The .next() bit can be handled via decoration

def consumer(func):
def start(*args,**kwargs):
c = func(*args,**kwargs)
c.next()
return c
return start

@consumer
def bytes_transferred():
total = 0
while True:
r = (yield)
total += r['bytes']
print "Total bytes", total

Example:

@consumer
def recv_count():
try:
while True:
n = (yield)
# Yield expression
print "T-minus", n
except GeneratorExit:
print "Kaboom!"

Copyright (C) 2008, http://www.dabeaz.com

lines = follow(open("access-log"))
log
= apache_log(lines)
broadcast(log,[find_404(),bytes_transferred()])

1-131

Discussion

Using the decorated version

In last example, multiple consumers

However, there were no threads
Further exploration along these lines can take

>>> r = recv_count()
>>> for i in range(5,0,-1):
...
r.send(i)
...
T-minus 5
T-minus 4
T-minus 3
T-minus 2
T-minus 1
>>> r.close()
Kaboom!
>>>

you into co-operative multitasking, concurrent

programming without using threads

That's an entirely different tutorial!

Don't need the extra .next() step here

Copyright (C) 2008, http://www.dabeaz.com

1-135

Copyright (C) 2008, http://www.dabeaz.com

@consumer decorator

1-134

Copyright (C) 2008, http://www.dabeaz.com

1-132

Copyright (C) 2008, http://www.dabeaz.com

1-136

Pitfalls
I don't think many programmers really
understand generators yet

Wrap Up

Springing this on the uninitiated might cause

their head to explode

Error handling is really tricky because you have

lots of components chained together

Need to pay careful attention to debugging,

reliability, and other issues.

1-137

Copyright (C) 2008, http://www.dabeaz.com

The Big Idea

Thanks!

Generators are an incredibly useful tool for a

I hope you got some new ideas from this class

Please feel free to contact me

variety of "systems" related problem

1-141

Copyright (C) 2008, http://www.dabeaz.com

Power comes from the ability to set up

processing pipelines

Can create components that plugged into the

http://www.dabeaz.com

pipeline as reusable pieces

Can extend the pipeline idea in many directions

(networking, threads, co-routines)

1-138

Copyright (C) 2008, http://www.dabeaz.com

Code Reuse
I like the way that code gets reused with
generators

Small components that just process a data

stream

Personally, I think this is much easier than what

you commonly see with OO patterns

1-139

Copyright (C) 2008, http://www.dabeaz.com

Example
SocketServer Module (Strategy Pattern)
import SocketServer
class HelloHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.request.sendall("Hello World\n")
serv = SocketServer.TCPServer(("",8000),HelloHandler)
serv.serve_forever()

My generator version
for c,a in receive_connections(("",8000)):
c.send("Hello World\n")
c.close()

Copyright (C) 2008, http://www.dabeaz.com

1-140

Copyright (C) 2008, http://www.dabeaz.com

1-142