Using Azure DevOps

for Azure Data Factory

Paul Andrew | Principal Consultant & Solution Architect

@MrPaulAndrew In/MrPaulAndrew MrPaulAndrew.com

https://github.com/mrpaulandrew

{Event/Location}-{Month}-{Year}
Azure Data Factory
A Quick Overview
What is Azure Data Factory?

Extract Transform Load

What is Azure Data Factory?

Extract Transform Load

Data Factory Components
Extract Transform Load

1 Linked Services

2 Data Sets

3 Activities

4 Pipelines

5 Triggers
Data Factory DevOps – CI/CD
Data Factory Continuous Integration

Debug
Git branch Service
Developer
save
merge

master publish
Deployed
Developer Service

Dev Prod

Developer
Data Factory Continuous Delivery

Debug
Git branch Service

save
merge

master publish
Deployed
Developer Service

Dev Prod
Data Factory Continuous Delivery
Option 1 – Single Data Factory Service Debug
Git branch Service

save
merge

master publish
Deployed
Developer Service

Dev Prod
Option 2 – ARM Templates for Multiple Data Factory Services
Debug
Test Service Live Service
branch Service
Git

save
merge

master ARM
Template
Developer
Dev Test Prod
Data Factory Publish

Git

master publish
Deployed
Developer Service

Debug Dev
Data Factory Publish

Git

master
adf_publish Deployed
Developer Service

Debug Dev

ARMTemplateForFactory.json
Data Factory Continuous Delivery Release-20190101

Master
Azure DevOps

Feature 101
Build Artifacts Feature 203

Releases Scoped Variables

Test

Azure Resource Group Deployment ARMTemplateForFactory.json

▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production

Azure Resource Group Deployment

Data Factory Continuous Delivery
1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities

Releases Scoped Variables 4 Pipelines

Test 5 Triggers

Azure Resource Group Deployment ARMTemplateForFactory.json

▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production

Azure Resource Group Deployment

Data Factory Continuous Delivery
1 Linked Services
Azure DevOps

Build Artifacts

Releases Scoped Variables

Test

Azure Resource Group Deployment ARMTemplateForFactory.json

▪ Override Template Parameters
-factoryName "$(DataFactory.Name)"
Release
Approver
Production

Azure Resource Group Deployment

Data Factory Continuous Delivery
1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities

Releases Scoped Variables 4 Pipelines

Test Set-AzDataFactoryV2Pipeline
5 Triggers
Set-AzDataFactoryV2Dataset
Set-AzDataFactoryV2LinkedService
Azure Resource Group Deployment ARMTemplateForFactory.json
▪ Override Template Parameters params.json
linkedservices.json
-factoryName "$(DataFactory.Name)"
Release
Approver pipelines & activites.json
Production
Set-AzDataFactoryV2IntegrationRuntime triggers.json
Set-AzDataFactoryV2Trigger
datasets.json
Azure Resource Group Deployment
Data Factory Continuous Delivery - Bonus
Option 3 1 Linked Services
Azure DevOps
2 Data Sets
Build Artifacts
3 Activities

Releases Scoped Variables 4 Pipelines

Test 5 Triggers
Run PowerShell params.json
linkedservices.json
Set-AzDataFactoryV2Pipeline pipelines & activites.json
Set-AzDataFactoryV2Dataset datasets.json
Set-AzDataFactoryV2LinkedService triggers.json

Release
Approver
Production
Run PowerShell params.json
Set-AzDataFactoryV2IntegrationRuntime
Set-AzDataFactoryV2Trigger
 Data Factory DevOps Summary
What is your Which source How many What deployment What built artefacts are
code branching control tool to environments method do we we going to use?...
strategy? use? do we want? want to use? OR
How much control do
Feature you want?

ARMTemplate
ForFactory.json
Release

linkedservices.json
pipelines & activites.json
Feature datasets.json
triggers.json

Master
Thank you for listening…
Paul Andrew

Blog: mrpaulandrew.com
Email: [email protected]

Twitter: @mrpaulandrew
LinkedIn: In/mrpaulandrew

GitHub: github.com/mrpaulandrew