Scribd
Upload
81 views

Modular Composition

The document discusses a project to modularize imaging applications on commercial-off-the-shelf programmable hardware platforms. It aims to replace proprietary FPGA-based platforms with COTS platforms while maintaining safety and performance. This will enable easy support for product variants and reuse of components through performance modeling and simulation. The new design integrates a performance analysis component and pipeline configuration tool. It models GStreamer pipelines as synchronous dataflow graphs to analyze throughput, latency and optimize configurations. Future work includes validating models against execution and supporting more complex pipelines.

Uploaded by

Sorin Crăcană
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
81 views

Modular Composition

The document discusses a project to modularize imaging applications on commercial-off-the-shelf programmable hardware platforms. It aims to replace proprietary FPGA-based platforms with COTS platforms while maintaining safety and performance. This will enable easy support for product variants and reuse of components through performance modeling and simulation. The new design integrates a performance analysis component and pipeline configuration tool. It models GStreamer pipelines as synchronous dataflow graphs to analyze throughput, latency and optimize configurations. Future work includes validating models against execution and supporting more complex pipelines.

Uploaded by

Sorin Crăcană
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 32

Modular composition of imaging application on commercial-off-the-shelf

programmable hardware platforms

23.05.2014
S.C. Cracana

Outline
Context

Background
Existing work
Problems to be solved

Current work

Identified problems
Performance modeling
Experiments

Future work

Project background
Barco

Display hardware manufacturer specialized in projectors, video displays


etc. (image processing products)

Active markets

Entertainment
Defense & aerospace
Industrial
Health-care: Interactive patient care

Products

Integrated solution for interactive patient care that includes devices for
patients aimed at information delivery and entertainment

Current practice
General

Display platforms are based on proprietary hardware (i.e. FPGAs)

Characteristics

Cost-effective for a limited number of products


Huge R&D effort for developing & maintaining an increased number of
products variants
HW development requires typically longer development times

Image display

An input signal needs several transformations before being displayed


The sequence of transformations is called image pipeline
The output has to comply with several regulations

[1]

Envisioned new design


Main modifications

Replace proprietary FPGA-based platforms by a COTS platforms


Keep safety and performance
Enable easy support for product variants
Include techniques to simulate and define the optimal architecture avoiding
expensive hardware development/test loops
Reuse of components and models
Reuse of already complying components

System components
Software

User Interface

Image Processing
Pipeline

Custom Algorithms

Define interface in QML language

Define underlying imaging processing using


GStreamer

Proprietary algorithms for various signal


transformations

Hardware
IMX 6

Atom N270

...

Various COTS platforms

Identified problems

1. The system does not have a performance analysis


component
2. Configuration of pipelines must be done manually

Use case
Input video stream formats to be
supported

SD - 576i: MPEG2 Transport Stream up to 6 Mbit/s


HD - 720p: H.264 Transport Stream up to 15 Mbit/s
HD - 1080i: H.264 Transport Stream up to 20 Mbit/s

Input video transport formats to


be supported

Multicast UDP
RTP

Output screen resolutions

1366x768
1920x1080

Hardware

Intel Atom N270


Intel Cedarview D2550
Freescale iMX6 dual and quad core
Intel Baytrail DN2820

Example
Medical Video Player

Network

Demux

Scaling

Player

Subtitles

Decoding

Buttons

Video/Audio

H264

Play/Pauze
Channel Select

[2]

Current new design


User Interface

qt-quickstreamer

Image Processing
Pipeline
C Wrapper for Matlab

Custom Algorithms

IMX 6

Atom N270

...

New Design and Performance


Analysis
User Interface

qt-quickstreamer

Pipeline Configuration

Image Processing
Pipeline

Performance Analysis

C Wrapper for Matlab

Custom Algorithms

IMX 6

Atom N270

...

Requirements

Expected Performance

New Design and Performance


Analysis
User Interface

qt-quickstreamer

Pipeline Configuration PC1

Image Processing
Pipeline

Performance Analysis

C Wrapper for Matlab

Custom Algorithms

IMX 6

Atom N270

...

Requirements
R1

Expected Performance EP1

New Design and Performance


Analysis
User Interface

qt-quickstreamer

Pipeline Configuration PC2

Image Processing
Pipeline

Performance Analysis

C Wrapper for Matlab

Custom Algorithms

IMX 6

Atom N270

...

Requirements
R2

Expected Performance EP2

Questions
A. How to model a GStreamer pipeline?
B. What are the steering parameters of a model?
C. How to evaluate the performance of a pipeline
on hardware?
D. What are the configurations that a pipeline
has?
E. How to integrate the modeling in the system?

Gstreamer
General

Graph based multimedia framework


Open Source
Cross Platform
Bindings to different languages (C, C++, Python etc.)

Provides

Pipeline architecture
Various processing elements
Tools for testing and profiling
Verbose debugging

GStreamer Pipeline

A number of plugins can be connected to attain the requisite media


processing
The processing unit in GStreamer is called a pipeline
It handles the clocking, the synchronizations, scheduling and the control
message flow between elements

[3]
[1]

Inter Plugin Communication

Managed using prespecified media stubs in the plugins called pads


Data flows out of an element through source pads
Data flows in an element through sink pads
Pads have a set of proprieties called capabilities or Caps

pads

[3]
[1]

Communication Content
Control:

Events: messages used for controlling media processing e.g. QOS, EOS,
Latency
Content:

Buffers: actual media content that needs to be processed;


Application

Queries: messages for asking information about the processing e.g.


duration of streaming;

Messages: information sent to the application by the plugins e.g. warnings,


errors, tags

[4]

[2]

Queues

[5]

Reference Pipeline

Takes a encoded video (h264) from a file, decodes it and displays it on the
screen
file src filter decoder colorspace ximage sink

To analyze the behavior and the throughput of each element queues are
placed in between elements

file src Q_filt filter Q_dec decoder Q_csp colorspace Q_sink ximage sink

Element

Configuration

File Src

Chunk

Queue

Maximum Size
Back pressure policy

Decoder

Max worker threads

Sink

Maximum lateness

Runtime
Input stream
1920x1080
X264 encoding
Bitrate 20 Mbit/s
100 fps, 10 seconds
Running environment
Intel i7: #Cores 4, #Threads 8
OS: Ubuntu 12.04 64bits
Gstreamer-0.10
Nice: -20
Schedulling: SCHED_NORMAL

Performance of pipelines
Throughput = the amount of data data passes through
an element per second
End2end Latency = the time it takes to a buffer from the
first element of the pipeline until it's sent to the display
driver
Element latency = the time it takes to a buffer to be
processed by an element in a pipeline

Experiments ...

Measurements

For measuring the pipeline performance we


use gst-tracelib
It uses LD_PRELOAD environment variables
for overriding a number of methods from
GStreamer
The new functions keep the old functionality
and add code for measurements and statistics

SDF components
Actors: functionality, code, that must be executed
Rates: no of tokens produced while firing
Tokens: data units
Initial tokens: initial state
Channels: data dependencies/execution order
Firing rules: number of tokens that must be present on actors' input
for execution

Gstreamer SDF model mapping


Plugins(elements) = actors
Pad linking = channels
Rate / firing rules = production/consumption rates of buffers per
element
Buffers = tokens

[3]

Analysis
Consumption/production rate
Expected performance

Topology

Algorithms
Execution Time
Requirements

Size configuration

Determining
production/consumption rate

Runs for different video sequences would return different production rate
for the decoder
Filesrc plugin has a default size of 4Kb for each block it reads
4Kb blocks are then transformed into Buffers of 4Kb size and passed
further on the pipeline
Therefore, depending on the size of the file a different number of buffers
will be produced and fed into the decoder but the decoder will produce the
same amount of buffers (corresponding to the number of rendered frames)
Solution
Change the block size of the Filesrc for each file so that a input buffer
contains enough data for decoding a frame => (1 to 1
production/consumption)

Future steps

Run algorithms and compare results with


execution results
Model more different pipelines:

With branches

With live sources

Bibliography
[1] CRitical SYSTem Engineering AcceLeration Use Case Definition D405.010
[2]
[3]
[4]
[5]
[6]

You might also like