Chapter 5

DATA FLOW ARCHITECTURES

 Objectives
 Introduce the concepts of Data Flow
architectures
 Describe the Data Flow Architecture in UML
 Discuss the application domains of the Data
Flow architectural approach
 Discuss the benefits and limitations of the Data
Flow architecture approach
 Demonstrate the Batch Sequential and Pipe &
Filter architectures in OS and Java scripts
 Overview
• The data flow software architectural style
– is characterized by viewing the whole system as a
series of transformations on successive sets of data,
where data and operations on it are independent of
each other.
• The software system is decomposed into data
processing elements
– where data directs and controls the order of data
computation processing.
• Each component in this architecture transforms
its input data into corresponding output data.
• The connection
– between the sub-system components
• may be implemented as I/O streams, I/O files,
buffers, piped streams, or other types of
connections.
• Data can flow
– in a graph topology with cycles
– in a linear structure without cycles
– in a tree type structure.
• The sub-systems are independent of each other
– One sub-system can be substituted by other sub-
system
• without affecting the rest of the system as long as the new
sub-system is compatible with the corresponding input and
output data format.
• Important property attributes of the data flow
architecture.
– Modifiability and
– Reusability
Because each sub-system does not need to know the
identity of any other sub-system
Block Diagram of Data Flow
Architecture
We will discuss about 3 sub-categories in
the data flow architectural styles:
– Batch Sequential
– Pipe & Filter
– Process Control
 Batch Sequential
• The batch sequential architectural style
– Represents a traditional data processing model
– widely used in 1950 - 1970.
• RPG and COBOL are two typical programming
languages working on this model.
• In batch sequential architecture
– each data transformation sub-system or module
• Cannot start its process
– until its previous sub-system completes its computation.
• Data flow carries a batch of data as a whole
– to move from one sub-system to another sub-system.
Batch Sequential Architecture

Validate Sort Update Report

Data flow data

transformation
 Batch Sequential Architecture
• Typical applications are Business data
processing such as banking and utility
billing
• A script is often used
– to make the batch sequence of the sub-
systems in the system.
 Batch Sequential in Business Data Processing
Rejected
transaction

-123 U
Validated Sorted
Transaction File
transaction transaction

222 U
222 U 111 I
111 I
Validate 111 I sort 222 U
333 D
333 D 333 D
-123 U

100 ----
Update
111 ----
200 ----
222 ----
Generate Report 444 ----

Reports 100 ---

Updated 200 ---
Master file 222 ---
333 ---
444 ---

Master file
 Batch Sequential Architecture
• Applicable Design Domains
– Data are batched.
– Intermediate file is a sequential access file.
– Each sub-system reads related input files and writes output files.

• Benefits:
– Simple divisions on sub-systems.
– Each sub-system can be a stand-alone program working on input data and
producing output data.

• Limitations:
– Implementation requires external control.
– Does no provide interactive interface.
– Concurrency is not supported and hence throughput remains low
– High latency.
 Pipe & Filter Architecture
• Is another type of data flow architecture where the flow is
driven by data.
• This architecture decomposes the whole system into
components of
– Data source
– Filters
– Pipes
– Data sinks
• The connections between components are data streams.

• The particular property attribute is concurrent and

incremented execution.
 A Data stream
• A data stream is a first-in-first-out buffer can be
– A stream of bytes
– A stream of characters
– A stream of records of XML
– Or any other type.
• Most operating systems and programming
languages provide a data stream mechanism
• -> a very important tool for marshalling and
unmarshalling in any distributed system.
 A Filter
• Is an independent data stream transformer
– reads data from its input data stream,
– transforms and processes it,
– writes the transformed data stream over a pipe for the
next filter to process.
• Does not need to wait for batched data as a
whole.
• As soon as the data arrives through the connected pipe, the
filter can start working right away.
• Does not even know the identity of data
upstream or data downstream.
– A filter is just working in a local incremental mode.
 A Pipe
• Moves data stream from one filter to another
filter.
• Can carry
– binary streams
– character streams.
• An object-type data must be serialized to be
able to go over a stream.
• Is placed between two filters;
– these filters can run in separate threads of the same
process as Java I/O streams.
 Data transfer types
3 ways to make the data flow:
• Push only (Write only)
A data source may pushes data in a downstream
A filter may push data in a downstream
• Pull only (Read only)
A data sink may pull data from an upstream
A filter may pull data from an upstream
• Pull/Push (Read/Write)
A filter may pull data from an upstream and push
transformed data in a downstream
 Filter types
• An active filter
– Pulls in data and push out the transformed data (pull/push);
– Works with a passive pipe which provides read/write
mechanisms for pulling and pushing.
– The pipe & filter mechanism in Unix adopts this mode.
– The PipedWriter and PipedReader pipe classes in Java are also
passive pipes
• A passive filter
– Lets connected pipes to push data in and pull data out.
– Works with active pipes that pull data out from a filter and push
data into the next filter.
– Must provide the read/write mechanisms in this case. This is
very similar to data flow architecture.
 Pipe and Filter Class Diagram
Pipe
Filter
+Read()
+Write()

Data Source Data Sink

+read() +write()
Pipe and Filter Block Diagram and Sequence Diagram

data source: filter1: pipe: filter2: data sink:

Data Source Filter pipe Filter Data Sink

read

Filter1

W
Filter2
R

write

Data Source Filter1 Filter2 Data Sink

pipe
 Fig Pipelined Pipe and Filter

e
a
b
c
e
A2
c Upper Case ECBAE DEF Matching (A , E ) AE (Sort * Count )
def ac E2
a Conversion Filter Filter Filter

f
e
d
Data Source
Data Sink
• The conversion filter converts all
characters from lower case to upper case
and the matching filter selects character
“A” and “E” in the stream.
• They are working concurrently; the
matching filter can start its job before
conversion completes its transformation of
the whole input stream.
• The sort and count filter counts the number
of occurrences of “A” and “E” and sort
them in alphabetical order.
• It can take its input before the matching
filter finishes the job but cannot output data
until it gets all data in.
Pipe and Filter Sequence Diagram
 Pipe & Filter in Unix
• The pipe operator “|” moves the stdout from its
predecessor to the stdin of its successor.
• The successor can start data transformation
over the data stream before the predecessor
completes its transformation or processing.
• For example, the pipeline of Unix commands
reports the number of users who are logged on
the system currently.
who | wc –l
 Pipe & Filter in Java
• The Java API provides the PipedWriter
and PipedReader classes in the java.io
package
– allows for filters to push data into downstream
or to pull data from the upstream.
• Filters
– can run in separate threads
– can be synchronized in a parallel manner by
PipedWriter and PipedReader.
Filter1.java:

package pf;
import java.io.*;
public class Filter1 extends Thread {
PipedWriter myPw;
public Filter1(PipedWriter pw) { myPw=pw; }
public void run() {
int j;
try {
for (int j = 1; j<100; j++) pw.write(j);
pw.write(-1);
}
catch(Exception e){. . .}
}
}
Filter2.java:

package pf;
import java.io.*;
class Filter2 extends Thread {
PipedReader myPr;
public Filter2(PipedReader pr) { myPr = pr; }
public void run() {
int j;
try {
while (myPr.read()!= -1){ . . . }
}
catch(Exception e){. . .}
}
}
 Contd…..
pipeFilter.java:

import pf.*;
import java.io.*;
public class pipeFilter {
public static void main(String[] args) {
try {
PipedWriter pw = new PipedWriter();
PipedReader pr = new PipedReader(pw);
Filter1 f1 = new Filter1(pw);
Filter2 f2 = new Filter2(pr);
f2.start();
f1.start();
}
catch(Exception e){ . . . }
}
}
• The pipeFilter Java main program runs two threads in
addition to itself: Filter1 and Filter2.
• They are connected by a synchronized pipe (PipedWriter
and PipedReader).
• Filter1 writes to the pipe and Filter2 reads from the pipe.
Do we need to start Filter1 before starting Filter2? The
answer is No.
• You can see that the main program actually starts Filter2
before Filter1.
• It works perfectly. This example demonstrates that the
order of filters in the sequence is not important as long as
the synchronization is taken care of.
 Pipe & Filter Architecture
• Applicable Design Domain
– Whenever the system can be broken into a series of
processing steps over data streams
• in each step filters consume and move data incrementally.
– Whenever the data format on the data streams is
• simple and stable,
• and easy to be adapted if it is necessary.
– Whenever there is significant work
• Which can be pipelined to gain increased performance
– Suitable for producer/consumer type of problems.
 Pipe & Filter Architecture
• Benefits of Pipe & Filter
– Concurrency: High overall throughput for excessive
data processing
– Reusability: Encapsulation of filters makes it easy to
plug and play and to substitute.
– Modifiability: Low coupling between filters, less impact
from adding new filters and modifying the
implementation of any existing filters as long as the
I/O interfaces are unchanged.
– Simplicity: Clear division between any two filters
connected by a pipe.
– Flexibility: It supports both sequential and parallel
execution.
 Pipe & Filter Architecture
• Limitations of Pipe & Filter
– Not suitable for dynamic interactions.
– Low Common Denominator
• Is required for data transmission in the ASCII formats
– since filters may need to handle data streams in different
formats such as record type or XML type rather than character
type.
– Overhead of data transformation among filters such
as parsing repeated in two consecutive filters.
– Difficult to configure a P&F system dynamically.
 Process-Control Architecture
• Is suitable for the embedded system software
design
– where the system is manipulated by a process
control variable data.
• Decomposes the whole system into
– sub-systems (modules) and
– connections between sub-systems.
• 2 types of sub-systems:
– executor processing unit for changing process
control variables
– controller unit for calculating the amounts of the
changes.
 A process control system must have the following process
control data:
• Controlled variable: target controlled variable such as
speed in a cruise control system or the temperature in an
auto H/A system. It has a set point which is the goal to
reach. The controlled variable data should be measured by
sensors as a feedback reference to re-calculate
manipulated variables.
• Input variable: measured input data such as the
temperature of return air in a temperature control system.
• Manipulated variable: can be adjusted by the controller.
• The input variables and manipulated variables
are applied to the execution processor which
results in a controlled variable.
• The set point and controlled variables are the
input data to the controller, the difference of the
controlled variable value from the set point value
is used to get a new manipulated value.
• Cars cruise-control systems and building
temperature control systems are good examples
of this process control software architecture type
of applications.
Data Flow in the Process Control
Architecture.
Input variables

C ontroller Process
Manipulated C ontrolled
Set Point ( s )
variables ( m) variable (c )

Æ= s Ğ c, m = f(Æ)
 Process Control Architecture
• Applicable Domains
– Embedded software system involving
continuing actions.
– The system needs to maintain an output data
at a stable level.
– The system can have a set point which is the
goal the system will reach and stay at that
level.
• Benefits ( loop-back architecture )
– Better solution to the control system where no
precise formula can be used to decide the
manipulated variable.
– The software can be completely embedded in
the devices.
 Summary
• The data flow architecture decomposes a system into a fixed sequence of
transformations and computations.
– There is no direct interaction between any two consecutive sub-systems except
for the exchange of data through data flow links.
– There is no data sharing among sub-systems in data flow architecture.
– It is not suitable for interactive business processing.
• 3 types of data flow architectures were discussed.
– Reading and writing I/O files drive the data flow in batch sequential architecture;
its control flow is thus explicit.
– The batch sequential architecture may cause bottlenecks because it requires
batched data as input and output.
– The pipe & filter is an incremental data transformation processing model and
runs concurrently.
– The data flow and the control flow in pipe & filter are implicit.
– The process control architecture is another type of data flow architecture where
the data is neither batched sequential nor pipelined streamed.
– The mechanism to drive the data flow comes from a set of variables that controls
the process execution.
 The design guidelines for a data flow based
software system is listed here:
• Decompose the system into series of process
steps; each step takes the output of its previous
step.
• Define the output and input data formats for
each step.
• Define the data transformation in each step.
• Design pipelines if concurrency is necessary.