Getting started with

Databricks
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Compute cluster refresh

DATABRICKS CONCEPTS
Create your first cluster
The first step is to create a cluster for your
data processing!

Configuration options:

DATABRICKS CONCEPTS
Create your first cluster
The first step is to create a cluster for your
data processing!

Configuration options:

Cluster policies and access

DATABRICKS CONCEPTS
Cluster Access

DATABRICKS CONCEPTS
Create your first cluster
The first step is to create a cluster for your
data processing!

Configuration options:

Cluster policies and access

Databricks Runtime

Photon Acceleration

DATABRICKS CONCEPTS
Create your first cluster
The first step is to create a cluster for your
data processing!

Configuration options:

Cluster policies and access

Databricks Runtime

Photon Acceleration

Node instance types and number

Auto-scaling / Auto-termination

DATABRICKS CONCEPTS
 Data Explorer
Get familiar with the Data Explorer! In this UI,
you can:

1. Browse available catalogs/schemas/tables

2. Look at sample data and summary

statistics

3. View data lineage and history

You can also upload new data by clicking the

"plus" icon!

1 Photo by Jakub Zerdzicki: https://www.pexels.com/photo/magnifier-loupe-17284804/

DATABRICKS CONCEPTS
Create a notebook
Databricks notebooks:

Standard interface for Databricks

Improvements on open-source Jupyter

Support for many languages
Python, R, Scala, SQL

Magic commands (%sql)

Built-in visualizations

Real-time commenting and collaboration

DATABRICKS CONCEPTS
Let's practice!
D ATA B R I C K S C O N C E P T S
 Data Engineering
foundations in
Databricks
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Medallion architecture

DATABRICKS CONCEPTS
Reading data
Spark is a highly flexible framework and can
read from various data sources/types.

Common data sources and types:

Delta tables

File formats (CSV, JSON, Parquet, XML)

Databases (MySQL, Postgres, EDW)

Streaming data

Images / Videos

DATABRICKS CONCEPTS
Reading data
Spark is a highly flexible framework and can #Delta table
read from various data sources/types. spark.read.table()
#CSV files
Common data sources and types:
spark.read.format('csv').load('*.csv')
Delta tables #Postgres table
spark.read.format("jdbc")
File formats (CSV, JSON, Parquet, XML)
.option("driver", driver)
Databases (MySQL, Postgres, EDW)
.option("url", url)
Streaming data .option("dbtable", table)
.option("user", user)
Images / Videos
.option("password", password)
.load()

DATABRICKS CONCEPTS
Structure of a Delta table
A Delta table provides table-like qualities to an open file format.

Feels like a table when reading

Access to underlying files (Parquet and JSON)

DATABRICKS CONCEPTS
Explaining the Delta Lake structure

DATABRICKS CONCEPTS
DataFrames
DataFrames are two-dimensional id customerName bookTitle
representations of data. 1 John Data Guide to Spark

Look and feel similar to tables 2 Sally Bricks SQL for Data
Engineering
Similar concept for many different data
3 Adam Delta Keeping Data Clean
tools
Spark (default), pandas, dplyr, SQL df = (spark.read
queries .format("csv")
Underlying construct for most data .option("header", "true")
processes .option("inferSchema", "true")
.load("/data.csv"))

DATABRICKS CONCEPTS
Writing data
Kinds of tables in Databricks df.write.saveAsTable(table_name)

1. Managed tables
CREATE TABLE table_name
Default type
USING delta
Stored with Unity Catalog AS ...
Databricks managed

2. External tables df.write

Stored in another location .location('').saveAsTable(table_name)

Set LOCATION
CREATE TABLE table_name
Customer managed USING delta
LOCATION "<path>"
AS ...

DATABRICKS CONCEPTS
Let's practice!
D ATA B R I C K S C O N C E P T S
 Data
transformations in
Databricks
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
SQL for data engineering
SQL -- Creating a new table in SQL

Familiar for Database Administrators

CREATE TABLE table_name
(DBAs)
USING delta
Great for standard manipulations AS (
Execute pre-defined UDFs SELECT *
FROM source_table
WHERE date >= '2023-01-01'
)

DATABRICKS CONCEPTS
Other languages for data engineering
Python, R, Scala #Creating a new table in Pyspark

Familiar for software engineers

spark
Standard and complex transformations .read
Use and define custom functions .table('source_table')
.filter(col('date') >= '2023-01-01')
.write
.saveAsTable('table_name')

DATABRICKS CONCEPTS
Common transformations
Schema manipulation #Pyspark

Add and remove columns df

Redefine columns .withColumn(col('newCol'), ...)
.drop(col('oldCol'))

Filtering #Pyspark

Reduce DataFrame to subset of data

df
Pass multiple criteria .filter(col('date') >= target_date)
.filter(col('id') IS NOT NULL)

DATABRICKS CONCEPTS
Common transformations (continued)
Nested data df
.explode(col('arrayCol')) #wide to long
Arrays or Struct data
.flatten(col('items')) #long to wide
Expand or contract

Aggregation df
.groupBy(col('region'))
Group data based on columns
.agg(sum(col('sales')))
Calculate data summarizations

DATABRICKS CONCEPTS
 Auto Loader
Auto Loader processes new data files as they
land in a data lake.

Incremental processing

Efficient processing

Automatic

spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load(file_path)

1 https://www.databricks.com/blog/2020/02/24/introducing-databricks-ingest-easy-data-ingestion-into-delta-
lake.html

DATABRICKS CONCEPTS
Structured Streaming
spark.readStream
.format("kafka")
.option("subscribe", "<topic>")
.load()
.join(table_df,
on="<id>", how="left")
.writeStream
.format("kafka")
.option("topic", "<topic>")
.start()

DATABRICKS CONCEPTS
Let's practice!
D ATA B R I C K S C O N C E P T S
 Orchestration in
Databricks
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Analytics Practitioner
What is data orchestration?
Data orchestration is a form of automation!

Enables data engineers to automate the end-to-end data life cycle

DATABRICKS CONCEPTS
 Databricks Workflows
Databricks Workflows is a collection of built-in capabilities to orchestrate all your data
processes, at no additional cost!

Example Databricks Workflow

1 https://docs.databricks.com/workflows

DATABRICKS CONCEPTS
What can we orchestrate?
Data engineers/data scientists Data analysts

DATABRICKS CONCEPTS
 Databricks Jobs
Workflows UI
Users can create jobs directly from the
Databricks UI:

Directly from a notebook

In the Workflows section

1 https://docs.databricks.com/workflows/jobs

DATABRICKS CONCEPTS
Databricks Jobs
Programmatic {
Users can also programmatically create jobs "name": "A multitask job",
using the Jobs CLI or Jobs API with the "tags": {},
Databricks platform. "tasks": [],
"job_clusters": [],
"format": "MULTI_TASK",
}

DATABRICKS CONCEPTS
Delta Live Tables

DATABRICKS CONCEPTS
Delta Live Tables

DATABRICKS CONCEPTS
Delta Live Tables

DATABRICKS CONCEPTS
Let's practice!
D ATA B R I C K S C O N C E P T S
 End-to-end data
pipeline example in
Databricks
D ATA B R I C K S C O N C E P T S

Kevin Barlow
Data Practitioner
Let's practice!
D ATA B R I C K S C O N C E P T S