Building Data Pipelines - 3

On the importance
of tests
B U I L D I N G D ATA E N G I N E E R I N G P I P E L I N E S I N P Y T H O N
Oliver Willekens
Data Engineer at Data Minded
Software tends to change
Common reasons for change:
new functionality desired
bugs need to get squashed
performance needs to be improved
Core functionality rarely evolves
How to ensure stability in light of changes?
BUILDING DATA ENGINEERING PIPELINES IN PYTHON

Rationale behind testing
improves chance of code being correct in the future
prevent introducing breaking changes
raises con dence (not a guarantee) that code is correct now

assert actuals match expectations
most up-to-date documentation

form of documentation that is always in sync with what’s running

The test pyramid: where to invest your efforts
Testing takes time
thinking what to test
writing tests
running tests
Testing has a high return on investment
when targeted at the correct layer
when testing the non-trivial parts, e.g.

distance between 2 coordinates ? uppercasing
a rst name © Martin Fowler “TestPyramid”

Testing takes time
writing tests
running tests


Testing takes time
writing tests
running tests


Testing takes time
writing tests
running tests


Let’s have this sink
in!
Writing unit tests for
PySpark
Oliver Willekens
Our earlier Spark application is an ETL pipeline



Separate transform from extract and load
prices_with_ratings = spark.read.csv(…) # extract
exchange_rates = spark.read.csv(…) # extract
unit_prices_with_ratings = (prices_with_ratings
.join(…) # transform
.withColumn(…)) # transform

Solution: construct DataFrames in-memory
# Extract the data - depends on input/output (network access,
df = spark.read.csv(path_to_file) lesystem permissions, …)
- unclear how big the data is
- unclear what data goes in
from pyspark.sql import Row + inputs are clear

purchase = Row("price",
"quantity", + data is close to where it is being used (“code-
"product") proximity”)
record = purchase(12.99, 1, "cake")
df = spark.createDataFrame((record,))

Create small, reusable and well-named functions
unit_prices_with_ratings = (prices_with_ratings
.join(exchange_rates, ["currency", "date"])
.withColumn("unit_price_in_euro",
col("price") / col("quantity")
* col("exchange_rate_to_euro"))
def link_with_exchange_rates(prices, rates):

return prices.join(rates, ["currency", "date"])
def calculate_unit_price_in_euro(df):
return df.withColumn(
"unit_price_in_euro",
col("price") / col("quantity") * col("exchange_rate_to_euro"))

Create small, reusable and well-named functions
def link_with_exchange_rates(prices, rates):
return prices.join(rates, ["currency", "date"])
def calculate_unit_price_in_euro(df):
return df.withColumn(
"unit_price_in_euro",
col("price") / col("quantity") * col("exchange_rate_to_euro"))
unit_prices_with_ratings = (
calculate_unit_price_in_euro(
link_with_exchange_rates(prices, exchange_rates)
)
)

Testing a single unit
def test_calculate_unit_price_in_euro():
record = dict(price=10,
quantity=5,
exchange_rate_to_euro=2.)
df = spark.createDataFrame([Row(**record)])

quantity=5,
result = calculate_unit_price_in_euro(df)

quantity=5,
expected_record = Row(**record, unit_price_in_euro=4.)

expected = spark.createDataFrame([expected_record])

quantity=5,
expected_record = Row(**record, unit_price_in_euro=4.)

expected = spark.createDataFrame([expected_record])
assertDataFrameEqual(result, expected)

Take home messages
1. Interacting with external data sources is costly
2. Creating in-memory DataFrames makes testing easier

the data is in plain sight,
focus is on just a small number of examples.
3. Creating small and well-named functions leads to more reusability and easier testing.

Let’s practice!
Continuous testing
Oliver Willekens
Running a test suite
Execute tests in Python, with one of:
in stdlib 3rd party
unittest pytest
doctest nose
Core task: assert or raise
Examples:
assert computed == expected
with pytest.raises(ValueError): # pytest specific

Manually triggering tests
In a Unix shell:
cd ~/workspace/my_good_python_project
pytest .
# Lots of output…
== 19 passed, 2 warnings in 36.80 seconds ==
cd ~/workspace/my_bad_python_project
pytest .
# Lots of output…
== 3 failed, 1 passed in 6.72 seconds ==
Note: Spark increases time to run unit tests.

Automating tests
Problem:
forget to run unit tests when making changes
Solution:
Automation
How:
Git -> con gure hooks
Con gure CI/CD pipeline to run tests automatically

CI/CD

Continuous Integration:
get code changes integrated with the master branch regularly.
Continuous Delivery:
Create “artifacts” (deliverables like documentation, but also programs) that can be deployed into
production without breaking things.

Con guring a CI/CD tool
CircleCI looks for .circleci/con g.yml.
Example:
Often:
jobs:
test:
1. checkout code
docker:
- image: circleci/python:3.6.4 2. install test & build requirements
steps:
3. run tests
- checkout
- run: pip install -r requirements.txt 4. package/build the software artefacts
- run: pytest .
5. deploy the artefacts (update docs / install
app / …)

Let’s practice!

Building Data Pipelines - 3

Uploaded by

Copyright:

Available Formats

Building Data Pipelines - 3

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Building Data Pipelines - 3

Uploaded by

Copyright:

Available Formats

What are some reasons for testing code?

What are some reasons for testing code?

What are the benefits of testing code?

What are the benefits of testing code?

On the importance

new functionality desired

bugs need to get squashed

performance needs to be improved

Core functionality rarely evolves

How to ensure stability in light of changes?

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

raises con dence (not a guarantee) that code is correct now

most up-to-date documentation

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

thinking what to test

Testing has a high return on investment

when targeted at the correct layer

when testing the non-trivial parts, e.g.

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

thinking what to test

Testing has a high return on investment

when targeted at the correct layer

when testing the non-trivial parts, e.g.

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

thinking what to test

Testing has a high return on investment

when targeted at the correct layer

when testing the non-trivial parts, e.g.

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

thinking what to test

Testing has a high return on investment

when targeted at the correct layer

when testing the non-trivial parts, e.g.

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

- unclear how big the data is

- unclear what data goes in

from pyspark.sql import Row + inputs are clear

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

def link_with_exchange_rates(prices, rates):

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

expected_record = Row(**record, unit_price_in_euro=4.)

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

expected_record = Row(**record, unit_price_in_euro=4.)

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

2. Creating in-memory DataFrames makes testing easier

focus is on just a small number of examples.

BUILDING DATA ENGINEERING PIPELINES IN PYTHON

in stdlib 3rd party

Core task: assert or raise

assert computed == expected

with pytest.raises(ValueError): # pytest specific

BUILDING DATA ENGINEERING PIPELINES IN PYTHON