How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium

https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Search Write

DBSQL SME Engineering

How to build an end-to-end testing

pipeline with dbt on Databricks
Databricks SQL SME 12 min read · Oct 15, 2024

Data Pipelines with Embedded Data Quality & Testing

Author: Tobi Sam, Solutions Architect @ Databricks

Introduction
In previous articles, the DBSQL SME group has introduced how to perform
basic performant ETL on DBT for all things Databricks (here and here). Now

1 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

we dive into the next stage: data quality & pipeline testing. Data quality is
essential in any analytics pipeline. This blog post outlines a robust approach
to ensuring data integrity throughout your dbt workflow. We will explore a
series of tests such as anomaly detection, unit tests, and data contracts that
will help you maintain high-quality data from the source to the final output.

Databricks provides a unified platform for data processing and analytics that
allows users to build, test, deploy, and monitor data products all in one
place! We will leverage dbt (data build tool) — an open-source command-line
tool that helps analysts and engineers transform data in their warehouse
more effectively — to implement robust testing techniques to our data
pipeline.

We will use the medallion architecture, a data design pattern used to

logically organize data in a lakehouse, with the goal of incrementally and
progressively improving the structure and quality of data as it flows through
each layer of the architecture (from Bronze/raw ⇒ Silver/cleaned ⇒ Gold/
serving layer tables).

Prerequisites & Configurations

dbt comes in two flavors dbt-core and dbt-cloud, and setting either one up is
pretty straightforward. We will use dbt-core in this article. Please follow the
installation instructions here and you should have a dbt project enabled.

Databricks

• Ensure you have a Databricks SQL warehouse created and ready to be

2 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

used

• Setup OAuth or Personal Token

Your Local Machine

• On your local machine, create a virtual environment and install the dbt-
databricks adapter

• Clone this repository as we will use it throughout this article https://

github.com/kamalendubiswas-db/dbt-dss-demo/tree/dev

• Regarding the profiles.yml file, the best practice is to gitignore this file,
as it relates to an individuals configuration and is created automatically
when you set up dbt locally. This dbt YAML file lives in the .dbt/ directory
of your user/home directory. Update your profiles.yml file from the repo
to point to the Databricks SQL warehouse you created above.

Now test that you can connect to your Databricks SQL warehouse from your
terminal with dbt debug. If all goes well, you should see this output “All
checks passed!” If not, please troubleshoot using the error messages (I find
them very helpful actually).

Configure your dbt Project

Sync your dbt metadata with Unity Catalog

To ensure that all your dbt table and column-level descriptions are
synchronized with Unity Catalog, you can enable persist_docs in your
dbt_projects.yml file. This really enriches the Unity Catalog view, adds rich

3 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

lineage, and enhances the GenAI capabilities of the Databricks platform.

How to sync table metadata from DBT to Unity Catalog

Once you enable this, all your metadata is synchronized with Unity Catalog
in Databricks

Syncing metadata from DBT automatically to UC

4 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Optimise your Databricks Compute Resources

In your profiles.yml file, you can increase the value of the “threads”
configuration to increase your project’s parallelism and fully take advantage
of compute concurrency on the Databricks Lakehouse.

Threads help parallelize node execution in the dbt’s (DAG). The default
number of threads is currently 4, but there is no maximum, and dbt will
allow you to go up to your Databricks SQL maximum limit. As a starting
point, increase this number to 10 with a medium SQL warehouse but check
out this in-depth analysis for more details on the best combination. To see
how Databricks SQL manages multiple queries, click here.

5 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Testing your models in dbt

Now that we have Databricks and dbt setup, we will break down our testing
strategy into 3 phases (you can customise this based on your use case).

• Validating the source data

• Validating the ingested data

• Validating the data transformations and output

Validate Source Data

Before diving into transformations, it’s crucial to verify the integrity of your
source data. This step ensures that you’re working with fresh, accurate data
from the start.

Freshness Checks
One of the first things to check is the freshness of your data. To check the
freshness of your source data in your pipeline in dbt, you can use dbt’s
freshness check block. A freshness block is defined within your models/
sources.yml file and is used to define the acceptable amount of time
between the most recent record in the source, and now, for a table to be
considered “fresh”.

Example record of freshness

6 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Let’s assume that it is currently 15:00 on September 15th, 2024, and we

expect one new row every day. We can configure the freshness block to
throw an error if the most recent row is older than 1 day and warn us if we
haven’t received a new row within the last 12 hours.

Freshness check on DBSQL

When we run

dbt source freshness

7 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

on the table, our check will fail as expected.

Freshness checks are very helpful as they notify you when your data
becomes stale so you can address them before proceeding further in your
pipeline.

Validate Ingested Data

Once you’ve confirmed the source data’s integrity, the next step is to ensure
that data is loaded correctly into your warehouse. Validation at this stage
ensures a solid foundation and eliminates a stage in debugging any
subsequent issues.

Row Count Comparison

One of the first things I like to do at this stage is to compare the rows of the
source data to the bronze layer. Note that this is only valid when doing a full
data load or if the source data is never deleted. You could write custom SQL
to compare this but there are robust testing packages such as dbt-
expectations that have a lot of pre-built tests ready to use. Using the
expect_table_row_count_to_equal_other_table test, all we have to do is
specify the two tables to compare:

8 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

DBT tests

When you run

dbt test

dbt will pass if both tables have equal row counts or it would fail otherwise.
Easy!

To install the dbt-expectations package, include this in your packages.yml

file and run the following:

dbt init

9 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Installing expectations packages

Feel free to add other tests at this stage that may be relevant to your project.

Validate Data Transformations & Output

With clean source data and a successful load, it’s time to validate your
transformations. This step ensures that your business logic is correctly
applied and that the resulting data meets your quality standards.

One-off Data Tests

Sometimes, you may have those scenarios which require a test that is very
specific and cannot be easily reused. In dbt, these are called Singular data
tests. These are simply custom SQL queries that check for specific data
quality issues or business rules in your models. However, if you find yourself

10 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

reusing your singular tests in multiple models, then it may be time to

“upgrade” them to Generic data tests which we will come to shortly.

Let us create a simple singular test that asserts that there are no future order
dates in the order table, as that would be strange.

Data Quality Tests

Create this file and save it in the tests directory. Now run

dbt test —select assert_no_future_order_dates

to run this test.

11 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

I included 99 records in the future and thus, my test failed as expected:

DBT run with test fail

Reusable Data Tests

Now, when it comes to checking for nulls, uniqueness etc, these are tests
you’d typically apply to several dbt models; and this brings us to Generic data
tests. These are tests in dbt that take in certain parameters and can be reused
across all models across your project. Think about them like a function in
any programming language that expects a parameter and returns a result.
This makes generic tests DRY and easily reusable.

dbt comes with 4 generic tests out of the box, but you can build your own
custom generic tests. However, before you go build yours, check out these
open-source packages ( dbt-utils and dbt-expectations which we looked at
already) to see if the test you have in mind hasn’t already been created.

In dbt, a good practice is to include test definitions directly in the

schema.yml file alongside your model definitions, centralizing your data
quality checks, making them easier to manage and maintain.

12 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Below is an example of the 4 built-in generic tests being used for the orders
model in a schema.yml file:

In plain English, these data tests translate to:

• unique: the order_id column in the orders model should be unique

• not_null: the order_id column in the orders model should not contain
null values

• accepted_values: the status column in the orders table should be one of

13 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

‘placed’, ‘shipped’, ‘completed’, or ‘returned’

• relationships: each customer_id in the orders model exists as an id in the

customers table (also known as referential integrity)

Enforcing Referential Integrity

An important test to point out here is the “relationships” generic data test. In
Databricks, primary key and foreign key constraints are not enforced as they
are only informational. If your application requires enforcing referential
integrity, including this test in your pipeline will ensure that only datasets
that meet this constraint are published into your schema.

Anomaly Detection in Time Series Data

dbt-expectations has another handy test, that I call “Z-sigma”. It’s a simple
anomaly test based on the assumption that differences between periods in a
given time series follow a log-normal distribution. This statistically measures
a certain data point’s relationship to the mean of the entire dataset. Try it
out, very easy to use.

14 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Constraints & Data Contracts

dbt constraints apply additional validation on data as it is being inserted into
a new or pre-existing table using Databricks constraints. When enforced, a
constraint guarantees that you will never see invalid data in the table
materialized by your model.

To enable constraints, you need to enable dbt contracts, which are

configurations that guarantee the shape of a model while it is being built to
avoid surprises or breaking changes for downstream queries.

Let’s take a look at an example of some constraints embedded within a

15 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

model contract:

DBT contracts

You may notice that constraints are very similar to data tests, and in some
cases, you can replace data tests with their equivalent constraint. Data tests
validate the content of your model after it is built while constraints check
this at build time. See here and here for more details.

At the moment with dbt-databricks, once you implement constraints, you get
an error message; however, dbt only validates the data once it has already
been inserted into the table, meaning that if the constraints and/or
constraint_check fails, the table with the failing data will still exist in Unity

16 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Catalog/Databricks SQL. There is an open issue to update the

implementation.

Unit Tests
Unit tests examine the smallest testable parts of your models. We implement
unit tests to validate specific transformations or calculations in our models,
especially when there is complex logic. These can be written as singular
tests or using dbt’s recently released unit test framework.

Let’s consider a scenario where you’re calculating Customer Lifetime Value

(CLV) in an e-commerce dataset. This calculation involves multiple steps and
business rules, making it an ideal candidate for unit testing.

17 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Testable Code Logic in SQL

To ensure the CLV calculation is correct, you can create unit tests where we
pass our sample data (“given”) and then our expected output (“Expect”):

18 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Inputs & Expected outputs in DBT for Unit Testing of pipeline logic

To run the test, run this command in your terminal:

dbt test - select test_type:unit

This unit test verifies that:

• The annual value calculation is correct: (1000/4) * 12 =3000

• The lifetime value calculation is correct: (3000 *3) =9000

19 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Unit tests like this are invaluable when dealing with complex calculations,
ensuring that your dbt models produce accurate and reliable results. Since
the inputs of unit tests are static, it is recommended that they be run only in
development or CI environments.

Additional Tips
To enhance your testing strategy, consider implementing these additional
features:

Test Severity
Another feature that I particularly like is test severity. This allows you to
configure your tests to return a warning instead of an error. Generally, if
your test query returns at least one row, this will return an error.

I recently tackled a project involving an important model named

“opportunity” with a crucial field called “customer_id.” Naturally, it was
important for all opportunities to be linked to a customer_id. Unfortunately,
the sales team overlooked the customer_id for 33 opportunity records.
Despite this, I still wanted dbt to run while ensuring the integrity of this
field.

To achieve this, I configured a test to “warn” me if there were any failing

rows, and “fail” when the records exceeded 33, as this would suggest the
identification of new records that were not initially taken into account.
Below is a sample of what the configuration would look like:

20 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Here is a simple breakdown:

• Not_null: A generic not_null test

• Where: in the test configuration, you can include a filter to apply the
tests to. In this case, it was only important that “closed won”
opportunities had a customer ID assigned to them.

• Config: with severity set to error (which is the default), dbt will check the
error_if condition first. If the error condition is met, the test will return
and error, if not, it checks the warn_if condition and “warns” if that
condition is met, or else, the test passes if neither the “error” nor the
“warn” conditions are met.

To run the test in your environment, simply run

21 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

dbt test

in your terminal.

We will see how you can add alerts using Databricks Workflows in the
Monitoring and Alerting in the next part of this article.

Failing Fast
Sometimes, during development, if there is a failure during your build or
test, you may want to exit dbt immediately instead of waiting for all the
models to complete. This will help you save some time and money on your
warehouse especially when you have a lot of models. dbt has a little-known
flag called — fail-fast or -x which immediately exits dbt if any resource fails
to build. You can find out more about it here.

Conclusion
We looked at many approaches to implementing data quality checks in your
data pipeline in Databricks. Remember that data quality is not a one-time
effort but an ongoing process. It is essential to regularly review and update
your tests as your data models evolve. In summary, you should now be
familiar with the following:

• How to set up a basic dbt project for Databricks including a rule of thumb
for compute sizing,

22 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

• How to implement unit tests and custom one-off tests on dbt models,

• How to validate data and perform freshness checks of sources,

• How to implement constraints and data contracts to prevent models from

being populated with invalid data,

• The importance of testing at each stage of your data lifecycle strategy to

isolate errors and identify if they are occurring at source, during
ingestion or during transformation, and

• The benefits of leveraging persist_docs to push metadata from your dbt

models directly into Unity Catalog for data discovery.

In the next article, we will look at how to automate monitoring and alerting
using Databricks Workflows and Databricks SQL in a CI/CD pipeline.

Databricks Sql Databricks Dbt Data Quality Data Mesh

Published in DBSQL SME Engineering

1.1K followers · Last published 3 days ago

Publication for content from the DBSQL SME group and the surrounding
community on DBSQL best practices, new design patterns, creative solutions,
and real world user stories all in one place.

Written by Databricks SQL SME

3.5K followers · 42 following

One stop shop for all technical how-tos, demos, and best practices for building
on Databricks SQL

23 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Responses (4)

Sava Matic

Marian R
Oct 16, 2024

Very helpful, thank you!

Franz Wöllert
Oct 16, 2024

Very comprehensive article - thanks!

marlanbar
Mar 27

Is it possible to run dbt unit tests on CI/CD without connecting to the actual workspace?

More from Databricks SQL SME and DBSQL SME Engineering

24 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

In DBSQL SME Engineeri… by Databricks SQL S… In DBSQL SME Engineering by Franco Patano

Introducing the Power BI on A primer for metadata-driven

Databricks Best Practices Cheat… frameworks with Databricks…
Authors Metadata-driven frameworks are a proven
way to control data ingestion at scale. Instea…

Apr 21 151 4 Apr 29 98 3

In DBSQL SME Engineeri… by Databricks SQL S… In DBSQL SME Engineeri… by Databricks SQL S…

Databricks SQL Orchestration Chat Your Way to Cost Visibility

Patterns with For Each and… with Databricks Genie
Author Author

Apr 16 29 Apr 16 43 6

See all from Databricks SQL SME See all from DBSQL SME Engineering

25 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Recommended from Medium

In revodata by René Luijk In DBSQL SME Engineeri… by Databricks SQL S…

Unit Testing PySpark Using Databricks SQL Orchestration

Databricks Connect Patterns with For Each and…
Using Databricks Connect and serverless for Author
automated unit testing

Apr 1 Apr 16 29

26 of 27 5/24/2025, 10:20 AM
How to build an end-to-end testing pipeline with dbt on Databricks | by Databricks SQL SME | DBSQL SME Engineering | Medium https://medium.com/dbsql-sme-engineering/how-to-build-an-end-to-end-testing-pipeline-with-dbt-on-databricks-cb6e179e646c

Nitasha Pursnani Matt Weingarten

Automating Metadata Tagging and Databricks Q2 Roadmap: W2W4

Data Security in Databricks using…
Scalable RLS, masking, and metadata
tagging with PySpark and Unity Catalog

May 6 85 1 May 16 33

Badrish Davay In Dev Genius by Maksim Pachkovskiy

Dataproducts Development uisng 15 Critical Databricks Mistakes

Databricks , Asset Bundles ,… Advanced Developers Make:…
Today’s data engineering shifted from Uncover 15 critical Databricks mistakes
building monolithic data pipeline structures… experienced developers frequently…

Jan 27 Apr 14 60 2

See more recommendations

Help Status About Careers Press Blog Privacy Rules Terms Text to speech

27 of 27 5/24/2025, 10:20 AM