FlumeJava: Easy, Efficient Data-Parallel Pipelines

Craig Chambers, Ashish Raniwala, Frances Perry,

Stephen Adams, Robert R. Henry,
Robert Bradshaw, Nathan Weizenbaum
Google, Inc.
{chambers,raniwala,fjp,sra,rrh,robertwb,nweiz}@google.com

Abstract MapReduce works well for computations that can be broken

MapReduce and similar systems significantly ease the task of writ- down into a map step, a shuffle step, and a reduce step, but for many
ing data-parallel code. However, many real-world computations re- real-world computations, a chain of MapReduce stages is required.
quire a pipeline of MapReduces, and programming and managing Such data-parallel pipelines require additional coordination code
such pipelines can be difficult. We present FlumeJava, a Java li- to chain together the separate MapReduce stages, and require addi-
brary that makes it easy to develop, test, and run efficient data- tional work to manage the creation and later deletion of the inter-
parallel pipelines. At the core of the FlumeJava library are a cou- mediate results between pipeline stages. The logical computation
ple of classes that represent immutable parallel collections, each can become obscured by all these low-level coordination details,
supporting a modest number of operations for processing them in making it difficult for new developers to understand the computa-
parallel. Parallel collections and their operations present a simple, tion. Moreover, the division of the pipeline into particular stages
high-level, uniform abstraction over different data representations becomes baked in to the code and difficult to change later if the
and execution strategies. To enable parallel operations to run effi- logical computation needs to evolve.
ciently, FlumeJava defers their evaluation, instead internally con- In this paper we present FlumeJava, a new system that aims to
structing an execution plan dataflow graph. When the final results support the development of data-parallel pipelines. FlumeJava is a
of the parallel operations are eventually needed, FlumeJava first op- Java library centered around a few classes that represent parallel
timizes the execution plan, and then executes the optimized opera- collections. Parallel collections support a modest number of par-
tions on appropriate underlying primitives (e.g., MapReduces). The allel operations which are composed to implement data-parallel
combination of high-level abstractions for parallel data and compu- computations. An entire pipeline, or even multiple pipelines, can
tation, deferred evaluation and optimization, and efficient parallel be implemented in a single Java program using the FlumeJava ab-
primitives yields an easy-to-use system that approaches the effi- stractions; there is no need to break up the logical computation into
ciency of hand-optimized pipelines. FlumeJava is in active use by separate programs for each stage.
hundreds of pipeline developers within Google. FlumeJavas parallel collections abstract away the details of
Categories and Subject Descriptors D.1.3 [Concurrent Pro- how data is represented, including whether the data is represented
gramming]: Parallel Programming as an in-memory data structure, as one or more files, or as an ex-
ternal storage service such as a MySql database or a Bigtable [5].
General Terms Algorithms, Languages, Performance Similarly, FlumeJavas parallel operations abstract away their im-
Keywords data-parallel programming, MapReduce, Java plementation strategy, such as whether an operation is implemented
as a local sequential loop, or as a remote parallel MapReduce invo-
cation, or (in the future) as a query on a database or as a streaming
1. Introduction computation. These abstractions enable an entire pipeline to be ini-
Building programs to process massive amounts of data in parallel tially developed and tested on small in-memory test data, running
can be very hard. MapReduce [68] greatly eased this task for data- in a single process, and debugged using standard Java IDEs and de-
parallel computations. It presented a simple abstraction to users buggers, and then run completely unchanged over large production
for how to think about their computation, and it managed many of data. They also confer a degree of adaptability of the logical Flume-
the difficult low-level tasks, such as distributing and coordinating Java computations as new data storage mechanisms and execution
the parallel work across many machines, and coping robustly with services are developed.
failures of machines, networks, and data. It has been used very To achieve good performance, FlumeJava internally implements
successfully in practice by many developers. MapReduces success parallel operations using deferred evaluation. The invocation of a
in this domain inspired the development of a number of related parallel operation does not actually run the operation, but instead
systems, including Hadoop [2], LINQ/Dryad [20], and Pig [3]. simply records the operation and its arguments in an internal exe-
cution plan graph structure. Once the execution plan for the whole
computation has been constructed, FlumeJava optimizes the exe-
cution plan, for example fusing chains of parallel operations to-
Permission to make digital or hard copies of all or part of this work for personal or gether into a small number of MapReduce operations. FlumeJava
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
then runs the optimized execution plan. When running the exe-
on the first page. To copy otherwise, to republish, to post on servers or to redistribute cution plan, FlumeJava chooses which strategy to use to imple-
to lists, requires prior specific permission and/or a fee. ment each operation (e.g., local sequential loop vs. remote parallel
PLDI10, June 510, 2010, Toronto, Ontario, Canada MapReduce, based in part on the size of the data being processed),
Copyright c 2010 ACM 978-1-4503-0019-3/10/06. . . $10.00 places remote computations near the data they operate on, and per-

363
forms independent operations in parallel. FlumeJava also manages or more replacement values to associate with the input key.
the creation and clean-up of any intermediate files needed within Oftentimes, the Reducer performs some kind of aggregation
the computation. The optimized execution plan is typically sev- over all the values with a given key. For other MapReduces,
eral times faster than a MapReduce pipeline with the same logical the Reducer is just the identity function. The key/value pairs
structure, and approaches the performance achievable by an expe- emitted from all the Reducer calls are then written to an output
rienced MapReduce programmer writing a hand-optimized chain sink, e.g., a sharded file, Bigtable, or database.
of MapReduces, but with significantly less effort. The FlumeJava For Reducers that first combine all the values with a given key
program is also easier to understand and change than the hand- using an associative, commutative operation, a separate user-
optimized chain of MapReduces. defined Combiner function can be specified to perform partial
As of March 2010, FlumeJava has been in use at Google for combining of values associated with a given key during the
nearly a year, with 175 different users in the last month and many Map phase. Each Map worker will keep a cache of key/value
pipelines running in production. Anecdotal reports are that users pairs that have been emitted from the Mapper, and strive to
find FlumeJava significantly easier to work with than MapReduce. combine locally as much as possible before sending the com-
Our main contributions are the following: bined key/value pairs on to the Shuffle phase. The Reducer will
We have developed a Java library, based on a small set of typically complete the combining step, combining values from
composable primitives, that is both expressive and convenient. different Map workers.
We show how this API can be automatically transformed into By default, the Shuffle phase sends each key-and-values group
an efficient execution plan, using deferred evaluation and opti- to a deterministically but randomly chosen Reduce worker ma-
mizations such as fusion. chine; this choice determines which output file shard will hold
that keys results. Alternatively, a user-defined Sharder func-
We have developed a run-time system for executing optimized tion can be specified that selects which Reduce worker machine
plans that selects either local or parallel execution automatically should receive the group for a given key. A user-defined Sharder
and which manages many of the low-level details of running a can be used to aid in load balancing. It also can be used to
pipeline. sort the output keys into Reduce buckets, with all the keys
We demonstrate through benchmarking that our system is ef- of the ith Reduce worker being ordered before all the keys of
fective at transforming logical computations into efficient pro- the i+1st Reduce worker. Since each Reduce worker processes
grams. keys in lexicographic order, this kind of Sharder can be used to
Our system is in active use by many developers, and has pro- produce sorted output.
cessed petabytes of data. Many physical machines can be used in parallel in each of these
three phases.
The next section of this paper gives some background on
MapReduce automatically handles the low-level issues of se-
MapReduce. Section 3 presents the FlumeJava library from the
lecting appropriate parallel worker machines, distributing to them
users point of view. Section 4 describes the FlumeJava optimizer,
the program to run, managing the temporary storage and flow of
and Section 5 describes the FlumeJava executor. Section 6 assesses
intermediate data between the three phases, and synchronizing the
our work, using both usage statistics and benchmark performance
overall sequencing of the phases. MapReduce also automatically
results. Section 7 compares our work to related systems. Section 8
copes with transient failures of machines, networks, and software,
concludes.
which can be a huge and common challenge for distributed pro-
grams run over hundreds of machines.
2. Background on MapReduce The core of MapReduce is implemented in C++, but libraries
FlumeJava builds on the concepts and abstractions for data-parallel exist that allow MapReduce to be invoked from other languages.
programming introduced by MapReduce. A MapReduce has three For example, a Java version of MapReduce is implemented as a
phases: JNI veneer on top of the C++ version of MapReduce.
MapReduce provides a framework into which parallel computa-
1. The Map phase starts by reading a collection of values or tions are mapped. The Map phase supports embarrassingly parallel,
key/value pairs from an input source, such as a text file, binary element-wise computations. The Shuffle and Reduce phases sup-
record-oriented file, Bigtable, or MySql database. Large data port cross-element computations, such as aggregations and group-
sets are often represented by multiple, even thousands, of files ing. The art of programming using MapReduce mainly involves
(called shards), and multiple file shards can be read as a single mapping the logical parallel computation into these basic opera-
logical input source. The Map phase then invokes a user-defined tions. Many computations can be expressed as a MapReduce, but
function, the Mapper, on each element, independently and in many others require a sequence or graph of MapReduces. As the
parallel. For each input element, the user-defined function emits complexity of the logical computation grows, the challenge of map-
zero or more key/value pairs, which are the outputs of the Map ping it into a physical sequence of MapReduces increases. Higher-
phase. Most MapReduces have a single (possibly sharded) input level concepts such as count the number of occurrences or join
source and a single Mapper, but in general a single MapReduce tables by key must be hand-compiled into lower-level MapReduce
can have multiple input sources and associated Mappers. operations. In addition, the user takes on the additional burdens of
2. The Shuffle phase takes the key/value pairs emitted by the writing a driver program to invoke the MapReduces in the proper
Mappers and groups together all the key/value pairs with the sequence, managing the creation and deletion of intermediate files
same key. It then outputs each distinct key and a stream of all holding the data passed between MapReduces, and handling fail-
the values with that key to the next phase. ures across MapReduces.
3. The Reduce phase takes the key-grouped data emitted by the
Shuffle phase and invokes a user-defined function, the Reducer, 3. The FlumeJava Library
on each distinct key-and-values group, independently and in In this section we present the interface to the FlumeJava library,
parallel. Each Reducer invocation is passed a key and an iterator as seen by the FlumeJava user. The FlumeJava library aims to
over all the values associated with that key, and emits zero offer constructs that are close to those found in the users logical

364
computation, and abstract away from the lower-level physical for ordered PCollections and tableOf(keyEncoding,
details of the different kinds of input and output storage formats valueEncoding ) for PTables. emitFn is a call-back function
and the appropriate partitioning of the logical computation into a FlumeJava passes to the users process(...) method, which
graph of MapReduces. should invoke emitFn.emit(outElem ) for each outElem that
should be added to the output PCollection. FlumeJava includes
3.1 Core Abstractions subclasses of DoFn, e.g., MapFn and FilterFn, that provide
The central class of the FlumeJava library is PCollection<T>, simpler interfaces in special cases. There is also a version of
a (possibly huge) immutable bag of elements of type T. A parallelDo() that allows multiple output PCollections to
PCollection can either have a well-defined order (called a se- be produced simultaneously from a single traversal of the input
quence), or the elements can be unordered (called a collection). PCollection.
Because they are less constrained, collections are more efficient parallelDo() can be used to express both the map and reduce
to generate and process than sequences. A PCollection<T> parts of MapReduce. Since they will potentially be distributed
can be created from an in-memory Java Collection<T>. A remotely and run in parallel, DoFn functions should not access
PCollection<T> can also be created by reading a file in one of
several possible formats. For example, a text file can be read as a any global mutable state of the enclosing Java program. Ideally,
PCollection<String>, and a binary record-oriented file can be they should be pure functions of their inputs. It is also legal for
read as a PCollection<T>, given a specification of how to decode DoFn objects to maintain local instance variable state, but users
each binary record into a Java object of type T. Data sets repre- should be aware that there may be multiple DoFn replicas operating
sented by multiple file shards can be read in as a single logical concurrently with no shared state. These restrictions are shared by
PCollection. For example:1 MapReduce as well.
A second primitive, groupByKey(), converts a multi-map of
PCollection<String> lines = type PTable<K,V> (which can have many key/value pairs with the
readTextFileCollection("/gfs/data/shakes/hamlet.txt"); same key) into a uni-map of type PTable<K, Collection<V>>
PCollection<DocInfo> docInfos = where each key maps to an unordered, plain Java Collection of
readRecordFileCollection("/gfs/webdocinfo/part-*", all the values with that key. For example, the following computes
recordsOf(DocInfo.class)); a table mapping URLs to the collection of documents that link to
In this code, recordsOf(...) specifies a particular way in which them:
a DocInfo instance is encoded as a binary record. Other pre- PTable<URL,DocInfo> backlinks =
defined encoding specifiers are strings() for UTF-8-encoded docInfos.parallelDo(new DoFn<DocInfo,
text, ints() for a variable-length encoding of 32-bit integers, and Pair<URL,DocInfo>>() {
pairsOf(e1,e2 ) for an encoding of pairs derived from the en- void process(DocInfo docInfo,
codings of the components. Users can specify their own custom EmitFn<Pair<URL,DocInfo>> emitFn) {
encodings. for (URL targetUrl : docInfo.getLinks()) {
emitFn.emit(Pair.of(targetUrl, docInfo));
A second core class is PTable<K,V>, which represents
}
a (possibly huge) immutable multi-map with keys of type }
K and values of type V. PTable<K,V> is a subclass of }, tableOf(recordsOf(URL.class),
PCollection<Pair<K,V>>, and indeed is just an unordered bag recordsOf(DocInfo.class)));
of pairs. Some FlumeJava operations apply only to PCollections PTable<URL,Collection<DocInfo>> referringDocInfos =
of pairs, and in Java we choose to define a subclass to capture this backlinks.groupByKey();
abstraction; in another language, PTable<K,V> might better be de-
groupByKey() captures the essence of the shuffle step of MapRe-
fined as a type synonym of PCollection<Pair<K,V>>.
The main way to manipulate a PCollection is to invoke a duce. There is also a variant that allows specifying a sorting order
data-parallel operation on it. The FlumeJava library defines only for the collection of values for each key.
a few primitive data-parallel operations; other operations are im- A third primitive, combineValues(), takes an input
plemented in terms of these primitives. The core data-parallel PTable<K, Collection<V>> and an associative combining
primitive is parallelDo(), which supports elementwise compu- function on Vs, and returns a PTable<K, V> where each input
tation over an input PCollection<T> to produce a new output collection of values has been combined into a single output value.
PCollection<S>. This operation takes as its main argument a For example:
DoFn<T, S>, a function-like object defining how to map each PTable<String,Integer> wordsWithOnes =
value in the input PCollection<T> into zero or more values to words.parallelDo(
appear in the output PCollection<S>. It also takes an indication new DoFn<String, Pair<String,Integer>>() {
of the kind of PCollection or PTable to produce as a result. For void process(String word,
example: EmitFn<Pair<String,Integer>> emitFn) {
PCollection<String> words = emitFn.emit(Pair.of(word, 1));
lines.parallelDo(new DoFn<String,String>() { }
void process(String line, EmitFn<String> emitFn) { }, tableOf(strings(), ints()));
for (String word : splitIntoWords(line)) { PTable<String,Collection<Integer>>
emitFn.emit(word); groupedWordsWithOnes = wordsWithOnes.groupByKey();
} PTable<String,Integer> wordCounts =
} groupedWordsWithOnes.combineValues(SUM_INTS);
}, collectionOf(strings()));
combineValues() is semantically just a special case of
In this code, collectionOf(strings()) specifies that parallelDo(), but the associativity of the combining function al-
the parallelDo() operation should produce an unordered lows it to be implemented via a combination of a MapReduce com-
PCollection whose String elements should be encoded using biner (which runs as part of each mapper) and a MapReduce re-
UTF-8. Other options include sequenceOf(elemEncoding ) ducer (to finish the combining), which is more efficient than doing
all the combining in the reducer.
1 Some of these examples have been simplified in minor ways from the real A fourth primitive, flatten(), takes a list of
versions, for clarity and compactness. PCollection<T>s and returns a single PCollection<T> that

365
contains all the elements of the input PCollections. flatten()
does not actually copy the inputs, but rather creates a view of them
as one logical PCollection.
A pipeline typically concludes with operations that write the
final result PCollections to external storage. For example:
wordCounts.writeToRecordFileTable(
"/gfs/data/shakes/hamlet-counts.records");
Because PCollections are regular Java objects, they can be
manipulated like other Java objects. In particular, they can be
passed into and returned from regular Java methods, and they
can be stored in other Java data structures (although they can-
not be stored in other PCollections). Also, regular Java con-
trol flow constructs can be used to define computations involving
PCollections, including functions, conditionals, and loops. For
example:
Collection<PCollection<T2>> pcs =
new Collection<...>();
for (Task task : tasks) {
PCollection<T1> p1 = ...;
PCollection<T2> p2;
if (isFirstKind(task)) {
p2 = doSomeWork(p1);
} else {
p2 = doSomeOtherWork(p1);
} Figure 1. Initial execution plan for the SiteData pipeline.
pcs.add(p2);
}

3.2 Derived Operations function. This operation is implemented on top of parallelDo(),

groupByKey(), and combineValues().
The FlumeJava library includes a number of other operations on The operations mentioned above to read multiple file shards
PCollections, but these others are derived operations, imple-
mented in terms of these primitives, and no different than helper as a single PCollection are derived operations too, implemented
functions the user could write. For example, the count() function using flatten() and the single-file read primitives.
takes a PCollection<T> and returns a PTable<T, Integer>
mapping each distinct element of the input PCollection to the 3.3 Deferred Evaluation
number of times it occurs. This function is implemented in terms
of parallelDo(), groupByKey(), and combineValues(), using In order to enable optimization as described in the next section,
the same pattern as was used to compute wordCounts above. That FlumeJavas parallel operations are executed lazily using deferred
code could thus be simplified to the following: evaluation. Each PCollection object is represented internally ei-
PTable<String,Integer> wordCounts = words.count(); ther in deferred (not yet computed) or materialized (computed)
state. A deferred PCollection holds a pointer to the deferred
Another library function, join(), implements a kind of operation that computes it. A deferred operation, in turn, holds
join over two or more PTables sharing a common key type. references to the PCollections that are its arguments (which
When applied to a multi-map PTable<K, V1> and a multi- may themselves be deferred or materialized) and the deferred
map PTable<K, V2>, join() returns a uni-map PTable<K, PCollections that are its results. When a FlumeJava operation
Tuple2<Collection<V1>, Collection<V2>>> that maps each like parallelDo() is called, it just creates a ParallelDo de-
key in either of the input tables to the collection of all values with ferred operation object and returns a new deferred PCollection
that key in the first table, and the collection of all values with that that points to it. The result of executing a series of FlumeJava op-
key in the second table. This resulting table can be processed fur- erations is thus a directed acyclic graph of deferred PCollections
ther to compute a traditional inner- or outer-join, but oftentimes and operations; we call this graph the execution plan.
it is more efficient to be able to manipulate the value collections Figure 1 shows a simplified version of the execution plan con-
directly without computing their cross-product. join() is imple- structed for the SiteData example used in Section 4.5 when dis-
mented roughly as follows: cussing optimizations and in Section 6 as a benchmark. This
1. Apply parallelDo() to each input PTable<K, Vi> to pipeline takes four different input sources and writes two outputs.
convert it into a common format of type PTable<K, (For simplicity, we usually elide PCollections from execution
TaggedUnion2<V1,V2>>. plan diagrams.)
2. Combine the tables using flatten(). Input1 is processed by parallelDo() A.
3. Apply groupByKey() to the flattened table to produce a Input2 is processed by parallelDo() B, and Input3 is pro-
PTable<K, Collection<TaggedUnion2<V1,V2>>>. cessed by parallelDo() C. The results of these two operations
4. Apply parallelDo() to the key-grouped table, converting are flatten()ed together and fed into parallelDo() D.
each Collection<TaggedUnion2<V1,V2>> into a Tuple2 of Input4 is counted using the count() derived operation, and
a Collection<V1> and a Collection<V2>. the result is further processed by parallelDo() E.
Another useful derived operation is top(), which takes a com- The results of parallelDo()s A, D, and E are joined together
parison function and a count N and returns the greatest N ele- using the join() derived operation. Its result is processed
ments of its receiver PCollection according to the comparison further by parallelDo() F.

366
 Finally, the results of parallelDo()s A and F are written to OperateFn should return a list of Java objects, which operate()
output files. wraps inside of PObjects and returns as its results. Using this
primitive, arbitrary computations can be embedded within a Flume-
To actually trigger evaluation of a series of parallel operations, Java pipeline and executed in deferred fashion. For example, con-
the user follows them with a call to FlumeJava.run(). This first sider embedding a call to an external service that reads and writes
optimizes the execution plan and then visits each of the deferred files:
operations in the optimized plan, in forward topological order, and // Compute the URLs to crawl:
evaluates them. When a deferred operation is evaluated, it converts PCollection<URL> urlsToCrawl = ...;
its result PCollection into a materialized state, e.g., as an in- // Crawl them, via an external service:
memory data structure or as a reference to a temporary intermediate PObject<String> fileOfUrlsToCrawl =
file. FlumeJava automatically deletes any temporary intermediate urlsToCrawl.viewAsFile(TEXT);
files it creates when they are no longer needed by later operations PObject<String> fileOfCrawledDocs =
in the execution plan. Section 4 gives details on the optimizer, and operate(fileOfUrlsToCrawl, new OperateFn() {
Section 5 explains how the optimized execution plan is executed. String operate(String fileOfUrlsToCrawl) {
return crawlUrls(fileOfUrlsToCrawl);
3.4 PObjects }
});
To support inspection of the contents of PCollections during PCollection<DocInfo> docInfos =
and after the execution of a pipeline, FlumeJava includes a class readRecordFileCollection(fileOfCrawledDocs,
PObject<T>, which is a container for a single Java object of recordsOf(DocInfo.class));
type T. Like PCollections, PObjects can be either deferred or // Use the crawled documents.
materialized, allowing them to be computed as results of deferred This example uses operations for converting between
operations in pipelines. After a pipeline has run, the contents of PCollections and PObjects containing file names. The
a now-materialized PObject can be extracted using getValue(). viewAsFile() operation applied to a PCollection and a
PObject thus acts much like a future [10]. file format choice yields a PObject<String> containing
For example, the asSequentialCollection() operation ap- the name of a temporary sharded file of the chosen format
plied to a PCollection<T> yields a PObject<Collection<T>>, where the PCollections contents may be found during
which can be inspected after the pipeline has run to read out all
the elements of the computed PCollection as a regular Java in- execution of the pipeline. File-reading operations such as
memory Collection:2 readRecordFileCollection() are overloaded to allow reading
files whose names are contained in PObjects.
PTable<String,Integer> wordCounts = ...; In much the same way, the contents of PObjects can also
PObject<Collection<Pair<String,Integer>>> result = be examined inside a DoFn by passing them in as side inputs to
wordCounts.asSequentialCollection(); parallelDo(). When the pipeline is run and the parallelDo()
... operation is eventually evaluated, the contents of any now-
FlumeJava.run(); materialized PObject side inputs are extracted and provided to the
for (Pair<String,Integer> count : result.getValue()) { users DoFn, and then the DoFn is invoked on each element of the
System.out.print(count.first + ": " + count.second); input PCollection. For example:
}
PCollection<Integer> values = ...;
As another example, the combine() operation applied to a PObject<Integer> pMaxValue = values.combine(MAX_INTS);
PCollection<T> and a combining function over Ts yields a PCollection<DocInfo> docInfos = ...;
PObject<T> representing the fully combined result. Global sums PCollection<Strings> results = docInfos.parallelDo(
and maximums can be computed this way. pMaxValue,
These features can be used to express a computation that needs new DoFn<DocInfo,String>() {
to iterate until the computed data converges: private int maxValue;
void setSideInputs(Integer maxValue) {
PCollection<Data> results = this.maxValue = maxValue;
computeInitialApproximation(); }
for (;;) { void process(DocInfo docInfo,
results = computeNextApproximation(results); EmitFn<String> emitFn) {
PCollection<Boolean> haveConverged = ... use docInfo and maxValue ...
results.parallelDo(checkIfConvergedFn(), }
collectionOf(booleans())); }, collectionOf(strings()));
PObject<Boolean> allHaveConverged =
haveConverged.combine(AND_BOOLS);
FlumeJava.run(); 4. Optimizer
if (allHaveConverged.getValue()) break; The FlumeJava optimizer transforms a user-constructed, modular
} FlumeJava execution plan into one that can be executed efficiently.
... continue working with converged results ... The optimizer is written as a series of independent graph transfor-
The contents of PObjects also can be examined within the ex- mations.
ecution of a pipeline. One way is using the operate() Flume-
Java primitive, which takes a list of argument PObjects and an 4.1 ParallelDo Fusion
OperateFn, and returns a list of result PObjects. When evaluated,
operate() will extract the contents of its now-materialized argu- One of the simplest and most intuitive optimizations is
ment PObjects, and pass them in to the argument OperateFn. The ParallelDo producer-consumer fusion, which is essentially func-
tion composition or loop fusion. If one ParallelDo opera-
2 Of course, asSequentialCollection() should be invoked only on rel- tion performs function f , and its result is consumed by an-
atively small PCollections that can fit into memory. FlumeJava includes other ParallelDo operation that performs function g, the two
additional operations such as asIterable() that can be used to inspect ParallelDo operations are replaced by a single multi-output
parts of larger PCollections. ParallelDo that computes both f and g f . If the result of the f

367


Figure 3. A MapShuffleCombineReduce (MSCR) operation with 3

input channels, 2 grouping output channels, and 1 pass-through
Figure 2. ParallelDo Producer-Consumer and Sibling Fusion. output channel.

ParallelDo is not needed by other operations in the graph, fusion

has rendered it unnecessary, and the code to produce it is removed faults to the identity operation), and then writes the results to Or
as dead. output PCollections, or (b) writes its input directly as its output.
ParallelDo sibling fusion applies when two or more The former kind of output channel is called a grouping channel,
ParallelDo operations read the same input PCollection. They while the latter kind of output channel is called a pass-through
are fused into a single multi-output ParallelDo operation that channel; a pass-through channel allows the output of a mapper to
computes the results of all the fused operations in a single pass be a result of an MSCR operation.
over the input. MSCR generalizes MapReduce by allowing multiple reducers
Both producer-consumer and sibling fusion can apply to ar- and combiners, by allowing each reducer to produce multiple out-
bitrary trees of multi-output ParallelDo operations. Figure 2 puts, by removing the requirement that the reducer must produce
shows an example execution plan fragment where ParallelDo outputs with the same key as the reducer input, and by allowing
operations A, B, C, and D can be fused into a single ParallelDo pass-through outputs, thereby making it a better target for our op-
A+B+C+D. The new ParallelDo creates all the leaf outputs from timizer. Despite its apparent greater expressiveness, each MSCR op-
the original graph, plus output A.1, since it is needed by some eration is implemented using a single MapReduce.
other non-ParallelDo operation Op. Intermediate output A.0 is Figure 3 shows an MSCR operation with 3 input channels per-
no longer needed and is fused away. forming ParallelDos M1, M2, and M3 respectively, two grouping
As mentioned earlier, CombineValues operations are special output channels, each with a GroupByKey, CombineValues, and
cases of ParallelDo operations that can be repeatedly applied reducing ParallelDo, and one pass-through output channel.
to partially computed results. As such, ParallelDo fusion also
applies to CombineValues operations. 4.3 MSCR Fusion
An MSCR operation is produced from a set of related GroupByKey
4.2 The MapShuffleCombineReduce (MSCR) Operation operations. GroupByKey operations are considered related if they
The core of the FlumeJava optimizer transforms combinations of consume (possibly via Flatten operations) the same input or
ParallelDo, GroupByKey, CombineValues, and Flatten op- inputs created by the same (fused) ParallelDo operations.
erations into single MapReduces. To help bridge the gap be- The MSCRs input and output channels are derived from the re-
tween these two abstraction levels, the FlumeJava optimizer in- lated GroupByKey operations and the adjacent operations in the
cludes an intermediate-level operation, the MapShuffleCombineRe- execution plan. Each ParallelDo operation with at least one out-
duce (MSCR) operation. An MSCR operation has M input chan- put consumed by one of the GroupByKey operations (possibly via
nels (each performing a map operation) and R output channels Flatten operations) is fused into the MSCR, forming a new input
(each optionally performing a shuffle, an optional combine, and channel. Any other inputs to the GroupByKeys also form new input
a reduce). Each input channel m takes a PCollection<Tm > as channels with identity mappers. Each of the related GroupByKey
input and performs an R-output ParallelDo map operation operations starts an output channel. If a GroupByKeys result
(which defaults to the identity operation) on that input to pro- is consumed solely by a CombineValues operation, that opera-
duce R outputs of type PTable<Kr ,Vr >s; the input channel can tion is fused into the corresponding output channel. Similarly, if
choose to emit only to one or a few of its possible output chan- the GroupByKeys or fused CombineValuess result is consumed
nels. Each output channel r Flattens its M inputs and then either soleby by a ParallelDo operation, that operation is also fused into
(a) performs a GroupByKey shuffle, an optional CombineValues the output channel, if it cannot be fused into a different MSCRs input
combine, and a Or -output ParallelDo reduce (which de- channel. All the PCollections internal to the fused ParallelDo,

368


Figure 4. An example of MSCR fusion seeded by three GroupByKey operations. Only the starred PCollections are needed by later
operations.

GroupByKey, and CombineValues operations are now unneces- GroupByKey records that fact. The original CombineValues is
sary and are deleted. Finally, each output of a mapper ParallelDo left in place, and is henceforth treated as a normal ParallelDo
that flows to an operation or output other than one of the related operation and subject to ParallelDo fusion.
GroupByKeys generates its own pass-through output channel. 3. Insert fusion blocks. If two GroupByKey operations are
Figure 4 shows how an example execution plan is fused into an connected by a producer-consumer chain of one or more
MSCR operation. In this example, all three GroupByKey operations ParallelDo operations, the optimizer must choose which
are related, and hence seed a single MSCR operation. GBK1 is related ParallelDos should fuse up into the output channel of the
to GBK2 because they both consume outputs of ParallelDo M2. earlier GroupByKey, and which should fuse down into the in-
GBK2 is related to GBK3 because they both consume PCollection put channel of the later GroupByKey. The optimizer estimates
M4.0. The ParallelDos M2, M3, and M4 are incorporated as MSCR the size of the intermediate PCollections along the chain of
input channels. Each of the GroupByKey operations becomes a ParallelDos, identifies one with minimal expected size, and
grouping output channel. GBK2s output channel incorporates the marks it as boundary blocking ParallelDo fusion.
CV2 CombineValues operation. The R2 and R3 ParallelDos are
also incorporated into output channels. An additional identity in- 4. Fuse ParallelDos.
put channel is created for the input to GBK1 from non-ParallelDo 5. Fuse MSCRs. Create MSCR operations. Convert any remaining
Op1. Two additional pass-through output channels (shown as edges unfused ParallelDo operations into trivial MSCRs.
from mappers to outputs) are created for the M2.0 and M4.1
PCollections that are used after the MSCR. The resulting MSCR 4.5 Example: SiteData
operation has 4 input channels and 5 output channels. In this section, we show how the optimizer works on the SiteData
After all GroupByKey operations have been transformed into pipeline introduced in Section 3.3. Figure 5 shows the execution
MSCR operations, any remaining ParallelDo operations are also plan initially and after each major optimization phase.
transformed into trivial MSCR operations with a single input chan-
nel containing the ParallelDo and a single pass-through output 1. Initially. The initial execution plan is constructed from calls to
channel. The final optimized execution plan contains only MSCR, primitives like parallelDo() and flatten() and derived op-
Flatten, and Operate operations. erations like count() and join() which are themselves imple-
mented by calls to lower-level operations. In this example, the
4.4 Overall Optimizer Strategy count() call expands into ParallelDo C:Map, GroupByKey
C:GBK, and CombineValues C:CV, and the join() call ex-
The optimizer performs a series of passes over the execution plan, pands into ParallelDo operations J:TagN to tag each of the N
with the overall goal to produce the fewest, most efficient MSCR input collections, Flatten J:Fltn, GroupByKey J:GBK, and
operations in the final optimized plan: ParallelDo J:Untag to process the results.
1. Sink Flattens. A Flatten operation can be pushed down 2. After sinking Flattens and lifting CombineValues.
through consuming ParallelDo operations by duplicating the Flatten operation Fltn is pushed down through con-
ParallelDo before each input to the Flatten. In symbols, suming ParallelDo operations D and JTag:2. A copy of
h(f (a) + g(b)) is transformed to h(f (a)) + h(g(b)). This CombineValues operation C:CV is associated with C:GBK.
transformation creates opportunities for ParallelDo fusion, 3. After ParallelDo fusion. Both producer-consumer and sibling
e.g., (h f )(a) + (h g)(b). fusion are applied to adjacent ParallelDo operations. Due to
2. Lift CombineValues operations. If a CombineValues op- fusion blocks, CombineValues operation C:CV is not fused
eration immediately follows a GroupByKey operation, the with ParallelDo operation E+J:Tag3.

369
(1) Initially: (3) After ParallelDo fusion:

(2) After sinking Flattens

and lifting CombineValues: (4) After MSCR fusion:

Figure 5. Optimizations applied to the SiteData pipeline to go from 16 original data-parallel operations down to 2 MSCR operations.

370
4. After MSCR fusion. GroupByKey operation C:GBK and surround- and also to allocate relatively more parallel workers to jobs that
ing ParallelDo operations are fused into a first MSCR opera- have a higher ratio of CPU to I/O.
tion. GroupByKey operations iGBK and J:GBK become the core FlumeJava automatically creates temporary files to hold the
operations of a second MSCR operation, which includes the re- outputs of each operation it executes. It automatically deletes these
maining ParallelDo operations. temporary files as soon as they are no longer needed by some
unevaluated operation later in the pipeline.
The original execution plan had 16 data-parallel operations FlumeJava strives to make building and running pipelines feel
(ParallelDos, GroupByKeys, and CombineValues). The final, as similar as possible to running a regular Java program. Using
optimized plan has two MSCR operations. local, sequential evaluation for modest-sized inputs is one way.
Another way is by automatically routing any output to System.out
4.6 Optimizer Limitations and Future Work
or System.err from within a users DoFn, such as debugging print
The optimizer does no analysis of the code within user-written statements, from the corresponding remote MapReduce worker
functions (e.g., the DoFn arguments to parallelDo() operations). to the main FlumeJava programs output streams. Likewise, any
It bases its optimization decisions on the structure of the execution exceptions thrown within a DoFn running on a remote MapReduce
plan, plus a few optional hints that users can provide giving some worker are captured, sent to the main FlumeJava program, and
information about the behavior of certain operations, such as an rethrown.
estimate of the size of a DoFns output data relative to the size When developing a large pipeline, it can be time-consuming
of its input data. Static analysis of user code might enable better to find a bug in a late pipeline stage, fix the program, and then
optimization and/or less manual user guidance. reexecute the revised pipeline from scratch, particularly when it
Similarly, the optimizer does not modify any user code as part is not possible to debug the pipeline on small-size data sets. To
of its optimizations. For example, it represents the result of fused aid in this cyclic process, the FlumeJava library supports a cached
DoFns via a simple AST-like data structure that explains how to execution mode. In this mode, rather than recompute an operation,
run the users code. Better performance could be achieved by gen- FlumeJava first attempts to reuse the result of that operation from
erating new code to represent the appropriate composition of the the previous run, if it was saved in a (internal or user-visible) file
users functions, and then applying traditional optimizations such and if FlumeJava determines that the operations result has not
as inlining to the resulting code. changed. An operations result is considered to be unchanged if
Users find it so easy to write FlumeJava pipelines that they (a) the operations inputs have not changed, and (b) the operations
often write large and sometimes inefficient programs, contain- code and captured state have not changed. FlumeJava performs an
ing duplicate and/or unnecessary operations. The optimizer could automatic, conservative analysis to identify when reuse of previous
be augmented with additional common-subexpression elimina- results is guaranteed to be safe; the user can direct additional
tion to avoid duplications. Additionally, users tend to include previous results to be reused. Caching can lead to quick edit-
groupByKey() operations more often than necessary, simply be- compile-run-debug cycles, even for pipelines that would normally
cause it makes logical sense to them to keep their data grouped by take hours to run.
key. The optimizer should be extended to identify and remove un- FlumeJava currently implements a batch evaluation strategy, for
necessary groupByKey() operations, such as when the result of a single pipeline at a time. In the future, it would be interesting
one groupByKey() is fed into another (perhaps in the guise of a to experiment with a more incremental, streaming, or continuous
join() operation). execution of pipelines, where incrementally added input leads to
quick, incremental update of outputs. It also would be interesting
5. Executor to investigate optimization across pipelines run by multiple users
over common data sources.
Once the execution plan is optimized, the FlumeJava library runs
it. Currently, FlumeJava supports batch execution: FlumeJava tra-
verses the operations in the plan in forward topological order, and 6. Evaluation
executes each one in turn. Independent operations are executed si- We have implemented the FlumeJava library, optimizer, and execu-
multaneously, supporting a kind of task parallelism that comple- tor, building on MapReduce and other lower-level services avail-
ments the data parallelism within operations. able at Google.
The most interesting operation to execute is MSCR. FlumeJava In this section, we present information about how FlumeJava
first decides whether the operation should be run locally and se- has been used in practice, and demonstrate experimentally that the
quentially, or as a remote, parallel MapReduce. Since there is over- FlumeJava optimizer and executor make modular, clear Flume-
head in launching a remote, parallel job, local evaluation is pre- Java programs run nearly as well as their hand-optimized raw-
ferred for modest-size inputs where the gain from parallel process- MapReduce-based equivalents.
ing is outweighed by the start-up overheads. Modest-size data sets
are common during development and testing, and by using local, 6.1 User Adoption and Experience
in-process evaluation for these data sets, FlumeJava facilities the One measure of the utility of the FlumeJava system is the extent to
use of regular IDEs, debuggers, profilers, and related tools, greatly which real developers find it worth converting to from systems they
easing the task of developing programs that include data-parallel already know and are using. This is the principal way in which we
computations. evaluate the FlumeJava programming abstractions and API.
If the input data set appears large, FlumeJava chooses to launch Since its initial release in May 2009, FlumeJava has seen sig-
a remote, parallel MapReduce. It uses observations of the input data nificant user adoption and production use within Google. To mea-
sizes and estimates of the output data sizes to automatically choose sure usage, we instrumented the FlumeJava library to log a usage
a reasonable number of parallel worker machines. Users can assist record every time a FlumeJava program is run. The following table
in estimating output data sizes, for example by augmenting a DoFn presents some statistics derived from these logs, as of mid-March
with a method that returns the expected ratio of output data size 2010:3
to input data size, based on the computation represented by that
DoFn. In the future, we would like to refine these estimates through 3 The FlumeJava usage logs themselves are processed using a FlumeJava
dynamic monitoring and feedback of observed output data sizes, program.

371
 1-day active users 62 6.3 Execution Performance
7-day active users 106 The goal of FlumeJava is to allow a programmer to express his
30-day active users 176 or her data-parallel computation in a clear, modular way, while
Total users 319 simultaneously executing it with performance approaching that of
The N -day active users numbers give the number of distinct user the best possible hand-optimized programs written directly against
ids that ran a FlumeJava program (excluding canned tutorial pro- MapReduce APIs. While high optimizer compression is good, the
grams) in the previous N days. real goal is small execution time.
Hundreds of FlumeJava programs have been written and To assess how well FlumeJava achieves this goal, we first con-
checked in to Googles internal source-code repository. Individual structed several benchmark programs, based on real pipelines writ-
FlumeJava programs have been run successfully on thousands of ten by FlumeJava users. These benchmarks performed different
machines over petabytes of data. computational tasks, including analyzing ads logs (Ads Logs), ex-
In general, users seem to be very happy with the FlumeJava tracting and joining data about websites from various sources (Site-
abstractions. They are not always as happy with some aspects Data and IndexStats), and computing usage statistics from logs
of Java or FlumeJavas use of Java. In particular, Java provides dumped by internal build tools (Build Logs).
poor support for simple anonymous functions and heterogeneous We wrote each benchmark in three different ways:
tuples, which leads to verbosity and some loss of static type safety. in a modular style using FlumeJava,
Also, FlumeJavas PCollection-based data-parallel model hides
many of the details of the individual parallel worker machines in a modular style using Java MapReduce, and
and the subtle differences between Mappers and Reducers, which in a hand-optimized style using Java MapReduce.
makes it difficult to express certain low-level parallel-programming
techniques used by some advanced MapReduce users. For two of the benchmarks, we also wrote in a fourth way:
FlumeJava is now slated to become the primary Java-based API in a hand-optimized style using Sawzall [17], a domain-specific
for data-parallel computation at Google. logs-processing language implemented on top of MapReduce.
6.2 Optimizer Effectiveness The modular Java MapReduce style mirrors the logical structure
found in the FlumeJava program, but it is not the normal way
In order to study the effectiveness of the FlumeJava optimizer at such computations would be expressed in MapReduce. The hand-
reducing the number of parallel MapReduce stages, we instru- optimized style represents an efficient execution strategy for the
mented the FlumeJava system so that it logs the structure of the computation, and as such is much more common in practice than
users pipeline, before and after optimization. The scatterplot below the modular version, but as a result of being hand-optimized and
shows the results extracted from these logs. Each point in the plot represented directly in terms of MapReduces, the logical com-
depicts one or more user pipelines with the corresponding number putation can become obscured and hard to change. The hand-
of stages. To aid the readability of the plot, we removed data on optimized Sawzall version likewise intermixes logical computation
about 10 larger pipelines with more than 120 unoptimized stages. with lower-level implementation details, in an effort to get better
performance.
The following table shows the number of lines of source it took
to write each version of each benchmark:

Benchmark FlumeJava MapReduce MapReduce Sawzall

(Modular) (Hand-Opt) (Hand-Opt)
Ads Logs 320 465 399 158
IndexStats 176 296 336 -
Build Logs 276 476 355 -
SiteData 465 653 625 261
For each case, the FlumeJava version is more concise than the
equivalent version written using raw Java MapReduce. Sawzall is
more concise than Java.
The following table presents, for the FlumeJava version, the
number of FlumeJava operations in the pipeline (both before and
Looking at the sizes of the user pipelines both before and after after optimization), and for the Java MapReduce and Sawzall ver-
optimization, it is evident that FlumeJava has been used for writing sions, the number of MapReduce stages:
small as well as fairly large pipelines. In fact, the largest of the
pipeline so far (not plotted) had 820 unoptimized stages and 149 Benchmark FlumeJava MapReduce MapReduce Sawzall
optimized stages. This data further underscores the usability of the (Modular) (Hand-Opt) (Hand-Opt)
FlumeJava API. Ads Logs 14 1 4 1 4
Looking at the optimizers compression ratio (the ratio of IndexStats 16 2 3 2 -
number of stages before and after the optimization), the optimizer Build Logs 71 3 1 -
SiteData 12 2 5 2 6
appears to achieve on average a 5x reduction in the number of
stages, and some pipelines had compression ratios over 30x. One For each benchmark, the number of automatically optimized opera-
pipeline (not plotted) had 207 unoptimized stages which were fused tions in the FlumeJava version matches the number of MapReduce
into a single optimized stage. stages in the corresponding hand-optimized MapReduce-based ver-
The FlumeJava optimizer itself runs quickly, especially com- sion. As a higher-level, domain-specific language, Sawzall does
pared to the actual execution that follows optimization. For not provide the programmer sufficient low-level access to enable
pipelines having up to dozens of operations, the optimizer takes them to hand-optimize their programs into this minimum number
less than a second or two. of MapReduce stages, nor does it include an automatic optimizer.

372
 The following table shows, for each benchmark, the size of the make developing a FlumeJava program similar to developing a reg-
input data set and the number of worker machines we used to run ular single-process Java program.
it: Sawzall [17] is a domain-specific logs-processing language that
Benchmark Input Size Number of Machines is implemented as a layer over MapReduce. A Sawzall program can
Ads Logs 550 MB 4 flexibly specify the mapper part of a MapReduce, as long as the
IndexStats 3.3 TB 200 mappers are pure functions. Sawzall includes a library of a dozen
Build Logs 34 GB 15 standard reducers; users cannot specify their own reducers. This
SiteData 1.3 TB 200 limits the Sawzall users ability to express efficient execution plans
for some computations, such as joins. Like MapReduce, Sawzall
We compared the run-time performance of the different versions does not provide help for multi-stage pipelines.
of each benchmark. We ensured that each version used equivalent
Hadoop [2] is an open-source Java-based re-implementation
numbers of machines and other resources. We measured the total
of MapReduce, together with a job scheduler and distributed file
elapsed wall-clock time spent when MapReduce workers were run-
system akin to the Google File System [9]. As such, Hadoop has
ning; we excluded the coordination time of starting up the main
similar limitations as MapReduce when developing multi-stage
controller program, distributing compiled binaries to worker ma-
pipelines.
chines, and cleaning up temporary files. Since execution times can
vary significantly across runs, we ran each benchmark version five Cascading [1] is a Java library built on top of Hadoop. Like
times, and took the minimum measured time as an approximation FlumeJava, Cascading aims to ease the challenge of programming
of the true time undisturbed by unrelated effects of running on a data-parallel pipelines, and provides abstractions similar to those
shared cluster of machines. of FlumeJava. Unlike FlumeJava, a Cascading program explicitly
The chart below shows the elapsed time for each version of each constructs a dataflow graph. In addition, the values flowing through
benchmark, relative to the elapsed time for the FlumeJava version a Cascading pipeline are special untyped tuple values, and Cas-
(shorter bars are better): cading operations focus on transforms over tuples; in contrast, a
FlumeJava pipeline computes over arbitrary Java objects using ar-
bitrary Java computations. Cascading performs some optimizations
of its dataflow graphs prior to running them. Somewhat akin to
FlumeJavas executor, the Cascading evaluator breaks the dataflow
graph into pieces, and, if possible, runs those in parallel, using the
underlying Hadoop job scheduler. There is a mechanism for elid-
ing computation if input data is unchanged, akin to FlumeJavas
caching mechanism.
Pig [3] compiles a special domain-specific language called Pig
Latin [16] into code that is run on Hadoop. A Pig Latin program
combines high-level declarative operators similar to those in SQL,
together with named intermediate variables representing edges in
the dataflow graph between operators. The language allows for
user-defined transformation and extraction functions, and provides
support for co-grouping and joins. The Pig system has a novel
debugging mechanism, wherein it can generate sample data sets
Comparing the two MapReduce columns and the Sawzall column that illustrate what the various operations do. The Pig system has
shows the importance of optimizing. Without optimizations, the an optimizer that tries to minimize the amount of data materialized
set-up overheads for the workers, the extra I/O in order to store between Hadoop jobs, and is sensitive to the size of the input data
the intermediate data, extra data encoding and decoding time, and sets.
other similar factors increase the overall work required to pro- The Dryad [11] system implements a general-purpose data-
duce the output. Comparing the FlumeJava and the hand-optimized parallel execution engine. Dryad programs are written in C++ us-
MapReduce columns demonstrates that a modular program written ing overloaded operators to specify an arbitrary acyclic dataflow
in FlumeJava runs at close to the performance of a hand-optimized graph, somewhat akin to Cascadings model of explicit graph con-
version using the lower-level MapReduce APIs. struction. Like MapReduce, Dryad handles the details of commu-
nication, partitioning, placement, concurrency and fault tolerance.
Unlike stock MapReduce but similar to the FlumeJava optimizers
7. Related Work MSCR primitive, computation nodes can have multiple input and
In this section we briefly describe related work, and compare output edge channels. Unlike FlumeJava, Dryad does not have
FlumeJava to that work. an optimizer to combine or rearrange nodes in the dataflow graph,
Language and library support for data-parallel programming has since the nodes are computational black boxes, but Dryad does in-
a long history. Early work includes *Lisp [13], C* [18], C** [12], clude a notion of run-time graph refinement through which users
and pH [15]. can perform some kinds of optimizations.
MapReduce [68] combines simple abstractions for data- The LINQ [14] extension of C# 3.0 adds a SQL-like construct
parallel processing with an efficient, highly scalable, fault-tolerant to C#. This construct is syntactic sugar for a series of library calls,
implementation. MapReduces abstractions directly support com- which can be implemented differently over different kinds of data
putations that can be expressed as a map step, a shuffle step, and a being queried. The SQL-like construct can be used to express
reduce step. MapReduces can be programmed in several languages, queries over traditional relational data (and shipped out to remote
including C++ and Java. FlumeJava builds on Java MapReduce, database servers), over XML data, and over in-memory C# ob-
offering higher-level, more-composable abstractions, and an opti- jects. It can also be used to express parallel computations and exe-
mizer for recovering good performance from those abstractions. cuted on Dryad [20]. The manner in which the SQL-like construct
FlumeJava builds in support for managing pipelines of MapRe- is desugared into calls that construct an internal representation of
duces. FlumeJava also offers additional conveniences that help the original query is similar to how FlumeJavas parallel operations

373
implicitly construct an internal execution plan. DryadLINQ also representation, which in our implementation was interpreted but
includes optimizations akin to those in FlumeJava. The C# lan- which we planned to eventually dynamically translate into Java
guage was significantly extended in order to support LINQ; in con- bytecode or native machine code. Lumberjacks parallel run-time
trast, FlumeJava is implemented as a pure Java library, with no lan- system shared many of the characteristics of FlumeJavas run-time
guage changes. DryadLINQ requires a pipeline to be expressed via system.
a single SQL-like statement. In contrast, calls to FlumeJava opera- While the Lumberjack-based version of Flume offered a number
tions can be intermixed with other Java code, organized into func- of benefits for programmers, it suffered from several important
tions, and managed with traditional Java control-flow operations; disadvantages relative to the FlumeJava version:
deferred evaluation enables all these calls to be coalesced dynam- Since Lumberjack was specially designed for the task, Lumber-
ically into a single pipeline, which is then optimized and executed
jack programs were significantly more concise than the equiv-
as a unit.
alent FlumeJava programs. However, the implicitly parallel,
SCOPE [4] is a declarative scripting language built on top of mostly functional programming model was not natural for many
Dryad. Programs are written in a variant of SQL, with extensions of its intended users. FlumeJavas explicitly parallel model,
to call out to custom extractors, filters, and processors that are which distinguishes Collection from PCollection and
written in C#. The C# extensions are intermixed with the SQL iterator() from parallelDo(), coupled with its mostly
code. As with Pig Latin, SQL queries are broken down into a imperative model that disallows mutable shared state only
series of distinct steps, with variables naming intermediate streams. across DoFn boundaries, is much more natural for most of these
The SQL framework provides named data fields, but there appears programmers.
to be little support for those names in the extension code. The
optimizer transforms SQL expressions using traditional rules for Lumberjacks optimizer was a traditional static optimizer,
query optimization, together with new rules that take into account which performed its optimization over the programs internal
data and communication locality. representation before executing any of it. Since FlumeJava is a
Map-Reduce-Merge [19] extends the MapReduce model by pure library, it cannot use a traditional static optimization ap-
adding an additional Merge step, making it possible to express ad- proach. Instead, we adopted a more dynamic approach to op-
ditional types of computations, such as relational algebra, in a sin- timization, where the running user program first constructs an
gle execution. FlumeJava supports more general pipelines. execution plan (via deferred evaluation), and then optimizes the
plan before executing it. FlumeJava does no static analysis of
FlumeJavas optimizer shares many concepts with tradi- the source program nor dynamic code generation, which im-
tional compiler optimizations, such as loop fusion and common- poses some costs in run-time performance; those costs have
subexpression elimination. FlumeJavas optimizer also bears some turned out to be relatively modest. On the other hand, being
resemblance to a database query optimizer: they both produce an able to simply run the FlumeJava program to construct the fully
optimized execution plan from a higher-level decription of a logical expanded execution plan has turned out to be a tremendous ad-
computation, and both can optimize programs that perform joins. vantage. The ability of Lumberjacks optimizer to deduce the
However, a database query optimizer typically uses run-time infor- programs execution plan was always limited by the strength
mation about input tables in a relational database, such as their sizes of its static analysis, but FlumeJavas dynamic optimizer has
and available indices, to choose an efficient execution plan, such as no such limits. Indeed, FlumeJava programmers routinely use
which of several possible algorithms to use to compute joins. In complex control structures and Collections and Maps storing
contrast, FlumeJava provides no built-in support for joins. Instead, PCollections in their code expressing their pipeline compu-
join() is a derived library operation that implements a particu- tation. These coding patterns would defeat any static analysis
lar join algorithm, hash-merge-join, which works even for simple, we could reasonably develop, but the FlumeJava dynamic opti-
file-based data sets lacking indices and which can be implemented mizer is unaffected by this complexity. Later, we can augment
using MapReduce. Other join algorithms could be implemented by FlumeJava with a dynamic code generator, if we wish the re-
other derived library operations. FlumeJavas optimizer works at a duce the remaining overheads.
lower level than a typical database query optimizer, applying fu-
sion and other simple transformations to the primitives underlying Building an efficient, complete, usable Lumberjack-based sys-
the join() library operation. It chooses how to optimize without tem is much more difficult and time-consuming than building an
reference to the sizes or other representational properties of its in- equivalently efficient, complete, and usable FlumeJava system.
puts. Indeed, in the context of a join embedded in a large pipeline, Indeed, we had built only a prototype Lumberjack-based sys-
such information may not become available until after the opti- tem after more than a years effort, but we were able to change
mized pipeline has been partly run. FlumeJavas approach allows directions and build a useful FlumeJava system in only a couple
the operations implementing the join to be optimized in the con- of months.
text of the surrounding pipeline; in many cases the joining opera- Novelty is an obstacle to adoption. By being embedded in
tions are completely fused into the rest of the computation (and vice a well-known programming language, FlumeJava focuses the
versa). This was illustrated by the SiteData example in section 4.5. potential adopters attention on a few new features, namely the
Before FlumeJava, we were developing a system based on Flume abstractions and the handful of Java classes and methods
similar abstractions, but made available to users in the context implementing them. Potential adopters are not distracted by a
of a new programming language, named Lumberjack. Lumber- new syntax or a new type system or a new evaluation model.
jack was designed to be particularly good for expressing data- Their normal development tools and practices continue to work.
parallel pipelines, and included features such as an implicitly par- All the standard libraries they have learned and rely on are still
allel, mostly functional programming model, a sophisticated poly- available. They need not fear that Java will go away and leave
morphic type system, local type inference, lightweight tuples and their project in the lurch. By comparison, Lumberjack suffered
records, and first-class anonymous functions. Lumberjack was sup- greatly along these dimensions. The advantages of its specially
ported by a powerful optimizer that included both traditional op- designed syntax and type system were insufficient to overcome
timizations such as inlining and value flow analysis, and non- these real-world obstacles.
traditional optimizations such as fusion of parallel loops. Lum-
berjack programs were transformed into a low-level intermediate

374
8. Conclusion [9] S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system.
In ACM Symposium on Operating Systems Principles (SOSP), 2003.
FlumeJava is a pure Java library that provides a few simple abstrac-
tions for programming data-parallel computations. These abstrac- [10] R. H. Halstead Jr. New ideas in parallel Lisp: Language design,
implementation, and programming tools. In Workshop on Parallel
tions are higher-level than those provided by MapReduce, and pro-
Lisp, 1989.
vide better support for pipelines. FlumeJavas internal use of a form
of deferred evaluation enables the pipeline to be optimized prior to [11] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Dis-
tributed data-parallel programs from sequential building blocks. In
execution, achieving performance close to that of hand-optimized
EuroSys, 2007.
MapReduces. FlumeJavas run-time executor can select among al-
ternative implementation strategies, allowing the same program to [12] J. R. Larus. C**: A large-grain, object-oriented, data-parallel pro-
gramming language. In Languages and Compilers for Parallel Com-
execute completely locally when run on small test inputs and using
puting (LCPC), 1992.
many parallel machines when run on large inputs. FlumeJava is in
active, production use at Google. Its adoption has been facilitated [13] C. Lasser and S. M. Omohundro. The essential Star-lisp manual.
by being a mere library in the context of an existing, well-known, Technical Report 86.15, Thinking Machines, Inc., 1986.
expressive language. [14] E. Meijer, B. Beckman, and G. Bierman. LINQ: reconciling objects,
relations and XML in the .NET framework. In ACM SIGMOD Inter-
national Conference on Management of Data, 2006.
References [15] R. S. Nikhil and Arvind. Implicit Parallel Programming in pH.
[1] Cascading. http://www.cascading.org. Academic Press, 2001.
[2] Hadoop. http://hadoop.apache.org. [16] C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig
[3] Pig. http://hadoop.apache.org/pig. Latin: A not-so-foreign language for data processing. In ACM SIG-
MOD International Conference on Management of Data, 2008.
[4] R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib,
S. Weaver, and J. Zhou. SCOPE: Easy and efficient parallel processing [17] R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the
of massive data sets. Proceedings of the VLDB Endowment (PVLDB), data: Parallel analysis with Sawzall. Scientific Programming, 13(4),
1(2), 2008. 2005.
[5] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Bur- [18] J. R. Rose and G. L. Steele Jr. C*: An extended C language. In C++
rows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed Workshop, 1987.
storage system for structured data. In USENIX Symposium on Operat- [19] H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Map-reduce-
ing Systems Design and Implementation (OSDI), 2006. merge: simplified relational data processing on large clusters. In ACM
[6] J. Dean. Experiences with MapReduce, an abstraction for large-scale SIGMOD International Conference on Management of Data, 2007.
computation. In Parallel Architectures and Compilation Techniques [20] Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and
(PACT), 2006. J. Currey. DryadLINQ: A system for general-purpose distributed data-
[7] J. Dean and S. Ghemawat. MapReduce: Simplified data processing on parallel computing using a high-level language. In USENIX Sympo-
large clusters. Communications of the ACM, 51, no. 1, 2008. sium on Operating Systems Design and Implementation (OSDI), 2008.
[8] J. Dean and S. Ghemawat. MapReduce: Simplified data processing on
large clusters. In USENIX Symposium on Operating Systems Design
and Implementation (OSDI), 2004.

375