Scribd
Upload
301 views

Building Data Pipelines in Python

This document discusses building data pipelines in Python. It covers extracting, transforming, and loading data; breaking pipelines into modular components; using workflow managers like Luigi and Airflow for orchestration; implementing testing, logging, and static type checking; and applying engineering principles to research and development work. Overall it argues that treating data pipelines as software engineering projects can help make them more robust, reusable, and maintainable.

Uploaded by

bender bender
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
301 views

Building Data Pipelines in Python

This document discusses building data pipelines in Python. It covers extracting, transforming, and loading data; breaking pipelines into modular components; using workflow managers like Luigi and Airflow for orchestration; implementing testing, logging, and static type checking; and applying engineering principles to research and development work. Overall it argues that treating data pipelines as software engineering projects can help make them more robust, reusable, and maintainable.

Uploaded by

bender bender
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 49

Building Data

Pipelines in Python
Marco Bonzanini

QCon London 2017


Nice to meet you
R&D ≠ Engineering
R&D ≠ Engineering
R&D results in production = high value
Big Data Problems
vs
Big Data Problems
Data Pipelines (from 30,000ft)

Data ETL Analytics


Data Pipelines (zooming in)

ETL
{ { Extract
Transform
Load
Clean
Augment
Join
Good Data Pipelines

Easy to
{Reproduce
Productise
Towards Good Data Pipelines
Towards Good Data Pipelines (a)

Your Data is Dirty


unless proven otherwise

“It’s in the database, so it’s already good”


Towards Good Data Pipelines (b)

All Your Data is Important


unless proven otherwise
Towards Good Data Pipelines (b)

All Your Data is Important


unless proven otherwise

Keep it. Transform it. Don’t overwrite it.


Towards Good Data Pipelines (c)

Pipelines vs Script Soups


Tasty, but not a pipeline

Pic: Romanian potato soup from Wikipedia


Anti-pattern: the script soup

$ ./do_something.sh
$ ./do_something_else.sh
$ ./extract_some_data.sh
$ ./join_some_other_data.sh
...
Script soups kill replicability
Anti-pattern: the master script
$ cat ./run_everything.sh
./do_something.sh
./do_something_else.sh
./extract_some_data.sh
./join_some_other_data.sh
$ ./run_everything.sh
Towards Good Data Pipelines (d)

Break it Down
setup.py and conda
Towards Good Data Pipelines (e)

Automated Testing
i.e. why scientists don’t write unit tests
Intermezzo

Let me rant about testing

Icon by Freepik from flaticon.com


(Unit) Testing
Unit tests in three easy steps:
• import unittest
• Write your tests
• Quit complaining about lack of time to write tests
Benefits of (unit) testing

• Safety net for refactoring


• Safety net for lib upgrades
• Validate your assumptions
• Document code / communicate your intentions
• You’re forced to think
Testing: not convinced yet?
Testing: not convinced yet?
Testing: not convinced yet?

f1 = fscore(p, r)
min_bound, max_bound = sorted([p, r])
assert min_bound <= f1 <= max_bound
Testing: I’m almost done
• Unit tests vs Defensive Programming
• Say no to tautologies
• Say no to vanity tests
• The Python ecosystem is rich: 

py.test, nosetests, hypothesis, coverage.py, …
</rant>
Towards Good Data Pipelines (f)

Orchestration
Don’t re-invent the wheel
You need a workflow manager

Think: 

GNU Make + Unix pipes + Steroids
Intro to Luigi

• Task dependency management


• Error control, checkpoints, failure recovery
• Minimal boilerplate
• Dependency graph visualisation
$ pip install luigi
Luigi Task: unit of execution

class MyTask(luigi.Task):
def requires(self):
return [SomeTask()]
def output(self):
return luigi.LocalTarget(…)
def run(self):
mylib.run()
Luigi Target: output of a task

class MyTarget(luigi.Target):
def exists(self):
... # return bool

Great off the shelf support 



local file system, S3, Elasticsearch, RDBMS
(also via luigi.contrib)
Intro to Airflow

• Like Luigi, just younger


• Nicer (?) GUI
• Scheduling
• Apache Project
Towards Good Data Pipelines (g)

When things go wrong


The Joy of debugging
import logging
Who reads the logs?

You’re not going to read the logs, unless…


• E-mail notifications (built-in in Luigi)
• Slack notifications
$ pip install luigi_slack # WIP
Towards Good Data Pipelines (h)

Static Analysis
The Joy of Duck Typing
If it looks like a duck,
swims like a duck,
and quacks like a duck,
then it probably is a duck.
— somebody on the Web
>>> 1.0 == 1 == True
True
>>> 1 + True
2
>>> '1' * 2
'11'
>>> '1' + 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't convert 'int' object
to str implicitly
PEP 3107 — Function Annotations

(since Python 3.0)

def do_stuff(a: int,


b: int) -> str:
...
return something

(annotations are ignored by the interpreter)


PEP 484 — Type Hints

(since Python 3.5)

typing module: semantically coherent

(still ignored by the interpreter)


pip install mypy
• Add optional types
• Run:
mypy --follow-imports silent mylib
• Refine gradual typing (e.g. Any)
Summary

Basic engineering principles help



(packaging, testing, orchestration, logging, static analysis, ...)
Summary

R&D is not Engineering:



can we meet halfway?
Vanity Slide

• speakerdeck.com/marcobonzanini

• github.com/bonzanini

• marcobonzanini.com

• @MarcoBonzanini

You might also like