Scribd
Upload
48 views14 pages

PASS ADF CICD With GitHub

This document outlines a session on Continuous Integration and Delivery (CI/CD) in Azure Data Factory. The session plan includes an introduction to Azure Data Factory, an overview of CI/CD, a case study of an ADF pipeline, integrating ADF with CI/CD, integrating ADF with GitHub, reviewing Azure DevOps, creating integration of ADF code and DevOps pipelines, how ADF pipeline amendments affect DevOps pipelines, and a conclusion. The document provides details on demonstrations of integrating ADF with GitHub and Azure DevOps to automate deploying ADF code through CI/CD pipelines.

Uploaded by

PMP Aspirants IMT
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
48 views14 pages

PASS ADF CICD With GitHub

This document outlines a session on Continuous Integration and Delivery (CI/CD) in Azure Data Factory. The session plan includes an introduction to Azure Data Factory, an overview of CI/CD, a case study of an ADF pipeline, integrating ADF with CI/CD, integrating ADF with GitHub, reviewing Azure DevOps, creating integration of ADF code and DevOps pipelines, how ADF pipeline amendments affect DevOps pipelines, and a conclusion. The document provides details on demonstrations of integrating ADF with GitHub and Azure DevOps to automate deploying ADF code through CI/CD pipelines.

Uploaded by

PMP Aspirants IMT
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 14

Rayis Imayev

June 20th, 2019

Continuous Integration and Delivery (CI/CD)


in Azure Data Factory
"Coming together is a beginning,
staying together is progress,
and working together is success.“

Henry Ford
Continuous Integration and Delivery (CI/CD) in Azure Data Factory

Session plan:
o Introduction to the Azure Data Factory (ADF)
o Continuous Integration and Delivery (CI/CD) overview
o ADF pipeline case study
o Merging ADF with CI/CD
o ADF integration with a GitHub repository
o Azure DevOps review
o Creating integration of GitHub based ADF code and DevOps pipelines
o ADF pipeline amendments and affect on DevOps pipeline
o Conclusion
Introduction to the Azure Data Factory (ADF)
Cloud platform to compose data-driven workflows with
steps to move and transform data.

 Control Flow
 Pipeline
 Activity
 Datasets
 Linked Services
 Triggers

 Mapping Data Flows


 Source/Sink
 Transformation Tasks

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Continuous Integration and Delivery overview
CI/CD process helps to establish a good
software development practice and
aims to build a healthy relationship
between development, quality
assurance, and other supporting teams.

1. Develop code
2. Unit test
3. Integrate
4. Acceptance test
5. Deploy to production

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF pipeline case study (Demo)
Sample ADF pipeline

Copy all files across multiple containers


from one file-based storage store to
another.

Activities:
- Get Metadata
- For Each Loop container
- Copy Data
- … more items

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Merging ADF with CI/CD
Questions:

1. Can multiple people work within the same ADF?


2. Can I integrate my ADF changes into other people's ADF code?
3. Can I have a process to frequently ship the ADF solution code to various environments?
4. Can a continuous deployment be automated to a production environment?

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF integration with a GitHub repository (Demo)

1. Setting a GitHub Repository for your Azure Data Factory


2. Saving your ADF code in a GitHub repository
3. Adding changes to the existing ADF code
4. Publishing changes to a live Data Factory and creating ARM templates

http://datanrg.blogspot.com/2019/02/azure-data-factory-integration-with.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Azure DevOps review
Plan, track, and discuss work across your teams.

Collaborate on code development using free Git public


and private repositories, pull requests, and code review.

Azure Pipelines helps you implement a build, test, and


deployment pipeline for any app.

Using testing services for your apps.

Code once and share packages across your organization

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Creating DevOps project to deploy ADF code (Demo)

1. New DevOps project


2. New Release Pipeline
3. Variable groups for my deployment solution
4. Adding Git artifact for the deployment
5. Deployment stages and deployment steps within each stage
6. Release pipeline variables
7. Testing a deployment release

http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF code change and its affect in DevOps pipeline (Demo)

1. Changing ADF code


2. Merging the ADF code to your GitHub repository
3. Publishing / Creaing ARM templates to deploy ADF code
4. Monitoring ADF code deployment
5. Validating code changes in other environments

http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Continuous Integration and Delivery (CI/CD) in Azure Data Factory

Session plan:
o Introduction to the Azure Data Factory (ADF)
o Continuous Integration and Delivery (CI/CD) overview
o ADF pipeline case study
o Merging ADF with CI/CD
o ADF integration with a GitHub repository
o Azure DevOps review
o Creating integration of GitHub based ADF code and DevOps pipelines
o ADF pipeline amendments and affect on DevOps pipeline
o Conclusion
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Conclusion:
1) ADF code deployment could be automated with the help of Azure DevOps.
2) Don’t forget to include Disable/Enable deployment steps if your ADF has triggers.
3) Make sure to use Azure Key Vault to store your secrets and reuse them during deployment.

Links:
http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Thank you for attending my session!

Rayis Imayev
Toronto, Canada

Consultant / Big Data Integration & Analytics

Senior BI & ETL Developer at Mosaic North America


http://datanrg.blogspot.com/
https://twitter.com/RayisImayev

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

You might also like