Module 1:

Data Factory Fundamentals

What is it and why use it?
Resource Components
Common Activities
Execution Dependencies
Azure Data Factory –
What is it?
Why use it?
A Quick History Lesson

v1 v2 Data Flows (Alpha)

Sept
July Sept March
2020
2016 2017 2018
What is Azure Data Factory (ADF)?

https://azure.microsoft.com/en-gb/services/data-factory
What is Azure Data Factory (ADF)?

Copy Transform
Resource Components
Data Factory Components

Copy Transform
Data Factory Components
Data Factory Components

1 Linked Services – What to interact with and how?

SQLDBLinkedService

ConnectionString: Server=MyServer;Database=myDataBase
UserName: “MrPaulAndrew”
Password: ***************
Data Factory Components

1 Linked Services

2 Datasets – Where is my data? What format? What file path/table do I need?

[dbo].[SalesOrders]

/RAW/Orders/2018/01/01/SalesOrders.csv
Data Factory Components

1 Linked Services

2 Datasets
Databricks Notebook Activity

3 Activities – What do we notebookPath: /Playground/Playing

want to happen when we baseParameters: Testing
libraries[ jar]: dbfs:/lib1.jar
invoke a Linked Service?
linkedServiceName: BricksOfData01
With what conditions?
Data Factory Components

Extract Transform
1 Linked Services

2 Datasets

3 Activities

4 Pipelines – Logical groups of

work that can be executed.
Execute Pipeline
Activity
Data Factory Components

Extract
1 Linked Services

2 Datasets

3 Activities

4 Pipelines – Logical groups of Transform & Load

work that can be executed.
Data Factory Components

Extract Transform
1 Linked Services

2 Datasets Manually
Programmatically
3 Activities Schedule
Tumbling Windows
4 Pipelines
File Event
5 Triggers – Telling our when pipelines to run.
Data Factory Components

1 Linked Services

2 Datasets

3 Activities

4 Pipelines

5 Triggers
Data Factory Control Flow Components

1 Linked Services

2 Datasets

3 Activities

4 Pipelines

5 Triggers
Common Activities

Paul’s Favourites
Data Factory Common Activities

1 Linked Services

2 Datasets

3 Activities

4 Pipelines

5 Triggers
Copy

Dataset Dataset
(Source) (Sink)

Copy Data

Auto Scaling
Transactional Restarts
Handle Zip Compression
Attribute Mapping and Schema Drift
Handle Failed Rows
Add Custom Attributes
Parse Excel & JSON Files
 Lookup
Get value to support other control flow activities
Single Value
Or
Many Values
Dataset [array]

Lookup

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity
 ForEach IsSequential:
true
[array]
Scaling Out Control Flow Activities [0]

[1]

Many Values [2]

[array]
[3]

[i]

ForEach
[array]

Lookup

[0] [1] [2] [3] [4] [5] [6] [i]

Copy Data Do Stuff

Batch Count Default: 20

@item().
Batch Count Max: 50

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity
 Switch

@case Switch

Default: Dev: Test: Prod:

Run Notebook Run Notebook

Raise on Small on Medium
Run Notebook
Error on Big Cluster
Cluster Cluster

https://mrpaulandrew.com/2020/01/22/using-the-azure-data-factory-switch-activity/
Execute Pipeline

Execute Pipeline

Call Child
Pipeline
 Custom
Extend Data Factory with Custom Code
References Objects
Datasets: []
Linked Services: []

Custom

Linked Services
Azure Batch ???
Azure Blob Storage

https://mrpaulandrew.com/2018/11/12/creating-an-azure-data-factory-v2-custom-activity/
Azure Function
Web

Do Stuff
Extend Data Factory with Rest Calls
Web Hook

GET Do Stuff
POST
PUT
etc...

Do Stuff ???

Headers
Body
Execution Dependencies
Execution Dependency Options

Success

Fail

Get
Values
Complete

Skip
Execution On Failure

Get
Values

Error Handler
Execution On Failure or On Success

Do Stuff

Get
Values

Error Handler
Execution On ???

Run Stored
Do Stuff Procedure

Get
Values
AND AN
D
Error Handler
Execution On Failure or On Success

Run Stored
Do Stuff Procedure

Get
Values

OR OR OR
Error Handler Error Handler Error Handler
Module 1:
Data Factory Fundamentals
What is it and why use it?
Resource Components
Common Activities
Execution Dependencies