BROUGHT TO YOU IN PARTNERSHIP WITH

Data Pipeline CONTENTS

•  Introduction

Essentials •  What Is a Data Pipeline?

•  Deploying a Data Pipeline
•  Challenges of Implementing a Data

Strategies for Successful Deployment and Collecting Pipeline

•  Advanced Strategies for Modern Data

Analytical Insights Pipelines

•  Conclusion

SUDIP SENGUPTA
TECHNICAL WRITER AT JAVELYNN

Modern data-driven applications are based on various data sources BATCH PIPELINES
and complex data stacks that require well-designed frameworks to In batch pipelines, data sets are collected over time in batches and
deliver operational efficiency and business insights. The result is a then fed into storage clusters for future use. These pipelines are
flexible, dynamic, and scalable application that enables businesses mostly considered applicable for legacy systems that cannot deliver
to predict, influence, and optimize their business outcomes based on data in streams, or in use cases that deal with colossal amounts of
real-time recommendations. Based on the numerous benefits that data. Batch pipelines are usually deployed when there’s no need
data-driven applications offer to businesses, Gartner predicts that for real-time analytics and are popular for use cases such as billing,
the role of data in achieving agility and collaboration is expected to payroll processing, and customer order management.
grow further over the next five years.
STREAMING PIPELINES
In this Refcard, we delve into the fundamentals of a data pipeline and In contrast to batch pipelines, streaming data pipelines continuously
the problems it solves for modern enterprises, along with its benefits ingests data, processing it as soon as it reaches the storage layer.
and challenges. Such pipelines rely on highly efficient frameworks that support the
ingestion and processing of a continuous stream of data within a
WHAT IS A DATA PIPELINE?
sub-second time frame. As a result, stream data pipelines are mostly
A data pipeline comprises a collection of tools and processes
for efficient transfer, storage, and processing of data across
multiple systems. With data pipelines, organizations can automate
information extraction from distributed sources while consolidating
data into high-performance storage for centralized access. A data See Ya Later,
pipeline essentially forms the foundation to build and manage
analytical tools for critical insights and strategic business decisions.
Star Schema
Acquire, enrich, analyze and act on data.
By building reliable pipelines for the consolidation and management No data warehouse required.
of data flows, development and DataOps teams can also efficiently
train, analyze, and deploy machine learning models. GET STARTED FOR FREE

DATA PIPELINE TYPES

Data pipelines are broadly categorized into the following types:
incorta.com
SEE NEXT COLUMN

1
See Ya Later,
Star Schema
Acquire, enrich, analyze and act on data.
No data warehouse required.

GET STARTED FOR FREE

incorta.com
 REFCARD | DATA PIPELINE ESSENTIALS

suitable for operations that require quicker analysis and real-time CORRECTION
insights of smaller data sets. Typical use cases include social media This process involves cleansing the data to eliminate errors and
engagement analysis, log monitoring, traffic management, user pattern anomalies. When performing correction, data engineers
experience analysis, and real-time fraud detection. typically use rules to identify a violation of data expectation, then
modify it to meet the organization’s needs. Unaccepted values can
DATA PIPELINE PROCESSES then be ignored, reported, or cleansed according to pre-defined
Though the underlying framework of data pipelines differ based on
business or technical rules.
use cases, they mostly rely on a number of common processes and
elements for efficient data flow. Some key processes of data pipelines LOADS
include: Once data has been extracted, standardized, and cleansed, it is
loaded into the destination system, such as a data warehouse or
SOURCES relational database, for storage or analysis.
In a data pipeline, a source acts as the first point of the framework
that feeds information into the pipeline. These include NoSQL AUTOMATION
databases, application APIs, cloud sources, Apache Hadoop, Data pipelines often involve multiple iterations of administrative
relational databases, and many more. and executive tasks. Automation involves monitoring the workflows
to help identify patterns for scheduling tasks and executing them
JOINS with minimal human intervention. Comprehensive automation of a
A join represents an operation that enables the establishment of a
data pipeline also involves the detection of errors and notification
connection between disparate data sets by combining tables. While
mechanisms to maintain consistent data sanity.
doing so, join specifies the criteria and logic for combining data from
different sources into a single pipeline. DEPLOYING A DATA PIPELINE
Considered one of the most crucial components of modern data-
Joins in data processing are categorized as:
driven applications, a data pipeline automates the extraction,
•  INNER Join — Retrieves records whose values match in both correlation, and analysis of data for seamless decision-making. When
tables building a data pipeline that is production-ready, consistent, and
•  LEFT (OUTER) Join — Retrieves all records from the left table reproducible, there are plenty of factors to consider that make it a
plus matching values from the right table highly technical affair. This section explores the key considerations,
components, and options available when building a data pipeline in
•  RIGHT (OUTER) Join — Retrieves all records from the right
production.
table plus matching records from the left table

•  FULL (OUTER) Join — Retrieves all records, whether there COMPONENTS OF A DATA PIPELINE
is a match or not in any of the two tables. In SQL tables with The data pipeline relies on a combination of tools and methodologies
star schema, full joins are typically implemented through to enable efficient extraction and transformation of data. These
conformed dimensions to link fact tables, creating fact-to-fact include:
joins.
Common components of a data pipeline
EXTR ACTION
Source data remains in a raw format that requires processing for
further analysis. Extraction is the first step of data ingestion where
the data is crawled and analyzed to ensure information relevancy
before it is passed to the storage layer for transformation.

STANDARDIZATION
Once data has been extracted, it is converted into a uniform
format that enables efficient analysis, research, and utilization.
Standardization is the process of formulating data with disparate
EVENT FR AMEWORKS
variables on the same scale to enable easier comparison and trend
Event processing encompasses the analysis and decision-making
analysis. Data standardization is commonly used for attributes such
based on data streamed continuously from applications. These
as dates, units of measure, color, size, etc.

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DATA PIPELINE ESSENTIALS

systems extract information from data points that respond to tasks data pipelines. The traditional approach of doing so is to build
performed by users or various application services. Any identifiable in-house data pipelines that require provisioning infrastructure in a
task or process that causes a change in the system is marked as an self-managed, private data center setup. This offers various benefits,
event, which is recorded in an event log for processing and analysis. including flexible customization and complete control over the
handling of data.
MESSAGE BUS
A message bus is a combination of a messaging infrastructure and However, self-managed orchestration frameworks rely on a number
data model that both receives and queues data sent between different of various tools and niche skills. Such platforms are also considered
systems. Leveraging an asynchronous messaging mechanism, less flexible to handle pipelines that require constant scaling or high
applications use a message bus to instantaneously exchange data availability. On the other hand, unified data orchestration platforms
between systems without having to wait for an acknowledgement. are supported by the right tools and skills that offer higher computing
A well-architected message bus also allows disparate systems to power and replication that enables organizations to scale quickly
communicate using their own protocols without worrying about while maintaining minimum latency.
system inaccessibility, errors, or conflicts.
ONLINE TR ANSACTION PROCESSING (OLTP) VS ONLINE

DATA PERSISTENCE ANALYTICAL PROCESSING (OLAP)

OLTP and OLAP are the two primary data processing mechanisms.
Persisting data refers to the ability of an application to store and
An OLTP system captures, stores, and processes user transactions in
retrieve information, so it can be processed in batches. Data
real-time where every transaction is made up of individual records
persistence can be achieved in several ways, such as by storing it on
consisting of multiple fields and columns.
the block, object, or file storage devices that ensure data is durable
and resilient in the event of system failure.
An OLAP system relies on large amounts of historical data to perform
high-speed, multidimensional analysis. This data typically comes
Data persistence also includes backup drives that provide readily
from a combination of sources, including OLTP databases, data
available replicas for automatic recovery when a server crashes. The
marts, data warehouses, or any other centralized data store. OLAP
data persistence layer creates the foundation for unifying multiple
systems are considered ideal for business intelligence, data mining,
data sources and destinations into a single pipeline.
and other use cases that require complicated analytical calculations.
WORKFLOW MANAGEMENT
In data pipelines, a workflow comprises a set of tasks with directional
QUERY OPTIONS
These are a set of query string parameters that help to fine-tune the
dependencies. These tasks filter, transform, and move data across
order and amount of data a service will return for objects identified
systems, often triggering events. Workflow management tools, such
by the Uniform Resource Identifier (URI). These options essentially
as Apache Airflow, structure these tasks within the pipeline, making
define a set of data transformations that are to be applied before
it easier to automate, supervise, and manage tasks.
returning the result.
SERIALIZATION FR AMEWORKS
Serialization tools convert data objects into byte streams that can These options can be applied to any task except the DELETE operation.

easily be stored, processed, or transmitted. Most firms operate

Some commonly used query options include:
with multiple data pipelines built using different approaches and
•  Filter: Enables the client to exclude a collection of resources
technologies. Data serialization frameworks define storage formats
addressed by the URI
that make it easy to identify and access relevant data, then write it to
another location. •  Expand: Specifies a list of resources related to the data stream
that will be included in the response
KEY CONSIDERATIONS
•  Select: Allows the client to request a specific set of
Key factors to consider when building and deploying a modern data
properties for each resource type
pipeline include:
•  OrderBy: Sorts resources in a specified order
SELF-MANAGED VS. UNIFIED DATA ORCHESTR ATION
PLATFORM DATA PROCESSING OPTIONS
Organizations can choose whether to leverage third-party enterprise There are two primary approaches to cleaning, enriching, and
services or self-managed orchestration frameworks for deploying transforming data before integration into the pipeline.

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DATA PIPELINE ESSENTIALS

In ETL (Extract - Transform - Load), data is first transformed in the GROWING TALENT GAP
staging server before it is loaded to the destination storage or data With the growth of emerging disciplines such as data science and
warehouse. ETL is easier to implement and is suited for on-premises deep learning, companies require more personnel resources and
data pipelines running mostly structured, relational data. expertise than job markets can offer. Combined with this is the fact
that a typical data pipeline implementation requires a huge learning
On the other hand, in ELT (Extract - Load - Transform), data is
curve, thereby requiring organizations to dedicate resources to either
loaded directly into the destination system before processing or
upskill existing staff or hire skilled experts.
transformation. When compared to ETL, ELT is more flexible and
scalable, making it suitable for both structured and unstructured SLOW DEVELOPMENT OF RUNNABLE DATA
cloud workloads. TRANSFORMATIONS
With modern data pipelines, organizations are able to build
CHALLENGES OF IMPLEMENTING A functional data models based on the recorded data definitions.
DATA PIPELINE However, developing functional transformations from these models
A data pipeline includes a series of steps that are executed comes with its own challenges as the process is expensive, slow,
sequentially on each dataset in order to generate a final output. and error-prone. Developers are often required to manually create
The entire process usually involves complex stages of extraction, executable codes and runtimes for data models, thereby resulting in
processing, storage, and analysis. As a result, each stage as well as ad-hoc, unstable transformations.
the entire framework requires diligent management and adoption of
best practices. Some common challenges while implementing a data ADVANCED STRATEGIES FOR MODERN
pipeline include: DATA PIPELINES
Some best practices to implement useful data pipelines include:
COMPLEXITY IN SECURING SENSITIVE DATA
Organizations host petabytes of data for multiple users with GRADUAL BUILD USING MINIMUM VIABLE
different data requirements. Each of these users has different access PRODUCT PRINCIPLES

permissions for different services, requiring restrictions on how data When developing a lean data pipeline, it is important to implement

can be accessed, shared, or modified. Assigning access rights to an architecture that scales to meet growing needs while still being

every individual manually is often a herculean task, which if not done easy to manage. As a recommended best practice, organizations

right, may lead to the access of sensitive information to malicious should apply a modular approach while incrementally developing

individuals. functionalities to handle more advanced data processing needs.

SLOWER PIPELINES DUE TO MULTIPLE JOINS INCORPORATE AI CAPABILITIES FOR TASK

AND STAR SCHEMA AUTOMATION

Joins allow data teams to combine data from two separate tables and DataOps teams should leverage auto-provisioning, auto-scaling, and

extract insights. Given the number of sources, modern data pipelines auto-tuning to reduce design time and simplify routing. Autoscaling

use multiple joins for end-to-end orchestration. These joins consume is crucial since big data workloads have data intake requirements

computing resources, thereby slowing down data operations. Besides that vary dramatically within short durations.

this, large data warehouses rely on star schemas to join DIMENSION

Example: The following snippet outlines a sample automation script
tables to FACT tables. On account of its highly denormalized state,
for the ingestion of logs in a Python-based data pipeline.
star schemas are considered less flexible to enforce the data integrity
of dynamic data models. =
f_a = open(LOG_FILE_A, 'r')
NUMEROUS SOURCES AND ORIGINS f_b = open(LOG_FILE_B, 'r')

The dynamic nature of data-driven applications requires constant while True:

where_a = f_a.tell()
evolution and are often ingesting data from a growing number of
line_a = f_a.readline()
sources. Managing these sources and the processes they run is
where_b = f_b.tell()
often challenging as these expose data with different formats. A
line_b = f_b.readline()
large number of sources also makes it difficult to document the if not line_a and not line_b:
data pipeline’s configuration details, which hampers cross-domain time.sleep(1)
collaboration in software teams.
CODE CONTINUES ON NEXT PAGE

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DATA PIPELINE ESSENTIALS

f_a.seek(where_a) TRACK AND LOG CHANGES TO THE DATA

f_b.seek(where_b)
Logging and tracking changes to data help data teams to find
continue
problems within the pipeline as soon as possible. As an extension
else:
of the versioning mechanism, logging helps operations and security
if line_a:
line = line_a teams to optimize strategies for root cause analysis and gather
else: insightful metrics of application usage.
line = line_b
CONCLUSION
Modern data-driven applications need more than just data. As such,
The above script opens log files and reads them one line at a time. Each
niche purposes of emerging tech such as data science and AI rely on a
line is parsed into fields, following which the lines and fields are both
complex framework of data sources, storage, and computing power.
parsed to the database. The script also ensures that the database does
The benefits of data pipelines are often considered proportional to
not accept duplicate lines.
the weightage of its accumulated data.
PARAMETERIZE THE DATA PIPELINE
By leveraging the power of these data-driven solutions, businesses
Parameters help data teams to make predictions using data models
can take faster business decisions, maximize their bottom line, and
and estimate the effectiveness of the models in analytics. By referring
satisfy customer requirements. In this Refcard, we delved into the
objects defined within a function and using them as parameters to
benefits, challenges, and configuration strategies of data pipelines
pass external values, parameterization ensures clean code in data
for emerging businesses. As the technology landscape evolves
pipelines while maintaining a common standard.
further, it will be interesting to see how businesses churn, store, and
USE NO-CODE/LOW-CODE ETL TO SIMPLIFY process data in the years to come.
DATA OPERATIONS
As a recommended best practice, organizations should leverage low-
code or no-code ETL platforms that learn how to transform data from
previous data sets based on available boilerplates and formulas. Such WRITTEN BY SUDIP SENGUPTA,
TECHNICAL WRITER AT JAVELYNN
platforms typically include built-in actions that eliminate manual coding
work, enabling rapid setup of data integration and transformation. Sudip Sengupta is a TOGAF Certified Solutions
Architect with more than 15 years of experience
working for global majors such as CSC, Hewlett
MINIMIZE DEPENDENCIES BY USING ATOMIC
Packard Enterprise, and DXC Technology. Sudip now works as
WORKFLOWS a full-time tech writer, focusing on Cloud, DevOps, SaaS, and
Data pipelines undertake multiple data transformations, such cybersecurity. When not writing or reading, he’s likely on the
squash court or playing chess.
as enrichment and format validation. As a best practice, it is
recommended that these transformations are broken down into
smaller, reproducible tasks with deterministic outputs. This makes it
easy to test changes while ensuring data quality and reliability.

IMPLEMENT DATA PIPELINE VERSIONING

Versioning pipelines enables data teams to determine how and when DZone, a Devada Media Property, is the resource software developers,
engineers, and architects turn to time and again to learn new skills, solve
data was ingested, transformed, or modified. It is important for data software development problems, and share their expertise. Every day,
hundreds of thousands of developers come to DZone to read about the latest
engineers and operators to know which version was used to create a technologies, methodologies, and best practices. That makes DZone the ideal
place for developer marketers to build product and brand awareness and drive
specific dataset so that incident root causes and action items can be sales. DZone clients include some of the most innovative technology and tech-
enabled companies in the world including Red Hat, Cloud Elements, Sensu, and
reproduced easily. Versioning also enables rollback while making it
Sauce Labs.
easy to evaluate whether new changes to the pipeline are effective.
Devada, Inc.
ENABLE MONITORING AND ALERTS FOR ALL 600 Park Offices Drive
Suite 300
TRANSACTIONS Research Triangle Park, NC 27709
888.678.0399 | 919.678.0300
For proactive security and data consistency, it is crucial to implement
Copyright © 2021 Devada, Inc. All rights reserved. No part of this publication
end-to-end observability of data pipelines. This also enables data may be reproduced, stored in a retrieval system, or transmitted, in any form or
by means of electronic, mechanical, photocopying, or otherwise, without prior
teams to validate data introduced into the pipeline, aiding in faster written permission of the publisher.
troubleshooting and vulnerability management.

6 BROUGHT TO YOU IN PARTNERSHIP WITH