Scribd
Upload
117 views8 pages

09-Working With Databricks Delta Live Tables (DLT)

Uploaded by

Madhusudhan Velijala
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
117 views8 pages

09-Working With Databricks Delta Live Tables (DLT)

Uploaded by

Madhusudhan Velijala
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

Delta Live Tables ScholarNest

DLT – Delta Live Tables

Databricks Cloud
Introduction to DLT
DLT End to End Scenario
DLT Scenario Data Setup
Create DLT Workload using SQL
Create and run DLT Pipeline
Create DLT Workload using Python
ScholarNest

Databricks Delta Live Table


DLT is a framework for building data processing pipelines
What is Databricks We use this framework in two steps.
Delta Live Table (DLT)
1. Write Declarative pipeline code
2. Create a DLT pipeline using Databricks Workflow UI

Declarative ETL Tool


ETL Pipeline – How Declarative ETL – What

Write code for how to do Define what to do


1. What to extract from the given source
1. How to extract data from a given
source system. 2. What are your data quality expectations
2. How to transform or validate data 3. What actions to take when expectations
are met
after extracting.
4. What actions to take when expectations
3. How to load valid and invalid data are not met
after transforming.
Integrated orchestration
Stitch tasks together using orchestration
tools
What is Databricks
Delta Live Table (DLT)

Declarative ETL Tool


ScholarNest

Databricks Delta Live Table


Lakehouse

Data Engineering Workspace SQL WS


Spark Spark Spark

Unity Catalog - Data Governance

Bronze Silver Gold

/ Raw Processed Analysis

Azure Storage Account

Azure Cloud Platform


Quality Data SCD Type 2
Landing Zone

Quality Data Change Data Capture Full Refresh

Bronze Silver Gold

You might also like