Azure Data Factory

Course Contents
• Introduction of Azure
• Introduction of Azure Data Factory
• Data Factory components
• Differences between v1 and v2
• Triggers
• Control Flow
• SSIS in ADFv2
• Demo
Introduction of Azure
• Azure is Microsoft's cloud computing platform, provides cloud services that gives you the freedom to build, manage, and
deploy applications on a massive global network using your favorite tools and frameworks.
A quick explanation on how Azure works
• Cloud computing is the delivery of computing services over the Internet using a pay-as-you-go pricing model. In
other words it's a way to rent compute power and storage from someone’s data center.
• Microsoft categorizes Azure cloud services into below product types:
• Compute
• Storage
• Networking
• Web
• Databases
• Analytics and IOT
• Artificial Intelligence
• DevOps
Introduction of Azure
Introduction of Azure Data Factory
• Azure Data Factory is a cloud-based data integration service to compose data storage, movement, and
processing services into automated data pipelines.
• It compose of data processing, storage, and movement services to create and manage analytics pipelines,
also provides orchestration, data movement and monitoring services.
• In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage
systems, big data requires service that can orchestrate and operationalize processes to refine these
enormous stores of raw data into actionable business insights.
• Azure Data Factory is a managed cloud service that's built for these complex hybrid extract-transform-load
(ETL), extract-load-transform (ELT), and data integration projects.
• Azure Data Factory is a data ingestion and transformation service that allows you to load raw data from over
70 different on-premises or cloud sources. The ingested data can be cleaned, transformed, restructured, and
loaded back into a data warehouse.
• Currently, there are two versions of the service: version 1 (V1) and version 2 (V2).
Introduction of Azure Data Factory
• The pipelines (data-driven workflows) in Azure Data Factory typically perform the following four steps:

• Connect and collect: The first step in building an information production system is to connect to all the
required sources of data and processing, such as software-as-a-service (SaaS) services, databases, file shares,
and FTP web services. The next step is to move the data as needed to a centralized location for subsequent
processing.
• Transform and enrich: After data is present in a centralized data store in the cloud, process or transform the
collected data by using compute services such as HDInsight Hadoop, Spark, Data Lake Analytics, and Machine
Learning.
• Publish: After the raw data has been refined into a business-ready consumable form, load the data into Azure
Data Warehouse, Azure SQL Database, Azure Cosmos DB, or whichever analytics engine your business users
can point to from their business intelligence tools.
• Monitor: After you have successfully built and deployed your data integration pipeline, providing business
value from refined data, monitor the scheduled activities and pipelines for success and failure rates.
Data Factory Components
• Azure Data Factory is composed of four key components. These components work together to provide the
platform on which you can compose data-driven workflows with steps to move and transform data.
• Pipeline: A data factory might have one or more pipelines. A pipeline is a logical grouping of activities that
performs a unit of work. For example, a pipeline can contain a group of activities that ingests data from an
Azure blob, and then runs a Hive query on an HDInsight cluster to partition the data.
• Activity: Activities represent a processing step in a pipeline. For example, you might use a copy activity to
copy data from one data store to another data store. Data Factory supports three types of activities: data
movement activities, data transformation activities, and control activities.
• Datasets: Datasets represent data structures within the data stores, which simply point to or reference the
data you want to use in your activities as inputs or outputs.
• Linked services: Linked services are much like connection strings, which define the connection information
that's needed for Data Factory to connect to external resources. For example, an Azure Storage-linked
service specifies a connection string to connect to the Azure Storage account.
• Linked services are used for two purposes in Data Factory :
• To represent a data store that includes, but isn't limited to, an on-premises SQL Server database, Oracle database, file
share, or Azure blob storage account.
• To represent a compute resource that can host the execution of an activity. For example, the HDInsight Hive activity
runs on an HDInsight Hadoop cluster.
Data Factory Components
• Overview of Data Factory flow
Data Factory Components
• Overview of Data Factory flow

My Pipeline1 My Pipeline2

For Each…
Trigger
Activity 3
params params params
Event Wall
Activity 1 Activity 2
Clock Activity 4
On Demand

param “OnError”
…
s Activity1
Data Factory Components
• Other components of Data Factory.

• Triggers: Triggers represent the unit of processing that determines when a pipeline execution needs to be
kicked off. There are different types of triggers for different types of events.
• Pipeline runs: A pipeline run is an instance of the pipeline execution. Pipeline runs are typically instantiated
by passing the arguments to the parameters that are defined in pipelines. The arguments can be passed
manually or within the trigger definition.
• Parameters: Parameters are key-value pairs of read-only configuration. Parameters are defined in the
pipeline. Activities within the pipeline consume the parameter values.
• Control flow: Control flow is an orchestration of pipeline activities that includes chaining activities in a
sequence, branching, defining parameters at the pipeline level, and passing arguments while invoking the
pipeline on-demand or from a trigger. It also includes custom-state passing and looping containers, that is,
For-each iterators.
Differences between v1 and v2
Feature Version 1 Version 2
Datasets A named view of data that references Datasets are the same in the current
the data, can be utilized in activities as version. However, you do not need to
inputs and outputs. define availability schedules for
Datasets identify data within different datasets.
data stores, such as tables, files, folders,
and documents

Availability defines the processing

window slicing model for the dataset
(for example, hourly, daily, and so on).
Linked services Linked services are much like connection Linked services are the same as in Data
strings, which define the connection Factory V1, but with a
information that's necessary for Data new connectVia property to utilize the
Factory to connect to external resources. Integration Runtime compute
environment of the current version of
Data Factory.
Differences between v1 and v2
Feature Version 1 Version 2
Pipelines A data factory can have one or more Pipelines are groups of activities that are
pipelines. A pipeline is a logical grouping performed on data. However, the
of activities that together perform a scheduling of activities in the pipeline
task. has been separated into new trigger
resources.
The Data Factory V1 concepts of
startTime, endTime, and isPaused are no
longer present in the current version of
Data Factory.
Activities Activities define actions to perform on In this version of Data Factory, activities
your data within a pipeline. Data still are defined actions within a pipeline
movement (copy activity) and data The current version of Data Factory
transformation activities (such as Hive, introduces new control flow activities.
Pig, and MapReduce) are supported.
Differences between v1 and v2
Feature Version 1 Version 2
Hybrid data movement Now called Integration Runtime, Data Data Management Gateway is now called
and activity dispatch Management Gateway supported Self-Hosted Integration Runtime. It
moving data between on-premises and provides the same capability as it did in
cloud. V1.
The Azure-SSIS Integration Runtime in
the current version of Data Factory also
supports deploying and running SQL
Server Integration Services (SSIS)
packages in the cloud.
Parameters NA Parameters are key-value pairs of read-
only configuration settings that are
defined in pipelines.
Differences between v1 and v2
Feature Version 1 Version 2
Expressions Data Factory V1 allows to use functions In this version of Data Factory, one can
and system variables in data selection use expressions anywhere in a JSON
queries and activity/dataset properties. string value.
Pipeline runs NA A single instance of a pipeline execution.
Each pipeline run has a unique pipeline
run ID. The pipeline run ID is a GUID that
uniquely defines that particular pipeline
run.
Activity runs NA An instance of an activity execution
within a pipeline.
Trigger runs NA An instance of a trigger execution. For
more information.
Scheduling Scheduling is based on pipeline Scheduler trigger or execution via
start/end times and dataset availability. external scheduler.
Differences between v1 and v2
Feature Version 1 Version 2
Chaining activities In V1, must configure the output of an In this version of Data Factory, in the
activity as an input of another activity to current version, one can chain activities
chain them. in a sequence within a pipeline, by using
the dependsOn property in an activity
definition to chain it with an upstream
activity.
Branching activities NA Can branch activities within a
pipeline. The If-condition activity
provides the same functionality that an if
statement provides in programming
languages.
Custom state passing NA Activity outputs including state can be
consumed by a subsequent activity in the
pipeline. By using this feature, we can
build workflows where values can pass
through activities.
Differences between v1 and v2
Feature Version 1 Version 2
Looping containers NA The ForEach activity defines a repeating
control flow in your pipeline. This activity
iterates over a collection and runs
specified activities in a loop.
Trigger-based flows NA Pipelines can be triggered by on-demand
(event-based, i.e. blob post) or wall-clock
time.
Invoking a pipeline from NA The Execute Pipeline activity allows a
another pipeline Data Factory pipeline to invoke another
pipeline.
Delta flows NA A key use case in ETL patterns is “delta
loads”. New capabilities in this current
version, such as lookup activity, flexible
scheduling, and control flow, enable this
use case.
Differences between v1 and v2
Feature Version 1 Version 2
Other control flow NA ForEach activity, Web activity, Lookup
activities activity, Get metadata activity, Wait
activity.
Deploy SSIS packages to NA We can Azure-SSIS if you want to move
Azure our SSIS workloads to the cloud, create a
data factory by using the current version,
and provision an Azure-SSIS Integration
Runtime.
Custom activities In V1, we implement (custom) DotNet In a custom activity in this version, you
activity code by creating a .NET class don't have to implement a .NET
library project with a class that interface. You can directly run
implements the Execute method of the commands, scripts, and your own
IDotNetActivity interface. Therefore, you custom code compiled as an executable.
need to write your custom code in .NET
Framework 4.5.2 and run it on
Windows-based Azure Batch Pool nodes.
Triggers

How do pipelines getstarted

1. on-demand
2. Wall-clock Schedule
3. Tumbling Window (akatime-slices inv1)
4. Event
Triggers
How do pipelines getstarted
1.Power Shell:
Invoke-AzureRmDataFactoryV2Pipeline +Parameters

2.Rest API: https://management.azure.com/subscriptions/mySubId/resourceGroups/myResou

rceGroup/providers/Microsoft.DataFactory/factories/{yourDataFactory}/pipelines
/{yourPipeline}/createRun?api-version=2017-03-01-preview

3..NET:
client.Pipelines.CreateRunWithHttpMessagesAsync(+ parameters)

4. AzurePortal
(Data factory -> <Author & Monitor> -> Pipeline runs)
Triggers

Run pipeline by schedule

Triggers

Tumbling Window
Tumbling window triggers are a type of trigger that fires at a
periodic time interval from a specified start time, while
retaining state. Tumbling windows are a series of fixed-sized,
non-overlapping, and contiguous time intervals.
Triggers

Event Based Trigger

Data integration scenarios often require Data Factory
customers to trigger pipelines based on events. Data Factory
is now integrated with Azure Event Grid, which lets you
trigger pipelines on an event.
Control Flow
Activities Known from v1 -Data TransformationActivities
Data transformationactivity Compute environment

Hive HDInsight [Hadoop]

Pig HDInsight [Hadoop]

MapReduce HDInsight [Hadoop]

Hadoop Streaming HDInsight [Hadoop]

Spark HDInsight [Hadoop]

Machine Learning activities: Batch Executionand Azure VM

Update Resource

Stored Procedure Azure SQL, Azure SQL Data Warehouse, orSQL

Server

U-SQL Azure Data LakeAnalytics

Control Flow
New! Control FlowActivities in v2
Control activity Description
Execute Pipeline allows a Data Factory pipeline to invoke anotherpipeline.
Activity
ForEachActivity used to iterate over a collection and executes specified activities ina loop.

WebActivity call a custom REST endpoint and pass datasets and linked services

LookupActivity look up a record/ table name/ value from any external source to be referencedby
succeeding activities. Could be used for incrementalloads!
Get Metadata retrieve metadata of any data in Azure Data Factory e.g. did another pipelinefinish
Activity
Do UntilActivity similar to Do-Until looping structure in programminglanguages.
If Condition do something based on condition that evaluates to true orfalse.
Activity
Control Flow
New! Control FlowActivities in v2
Control activity Description
Append Variable to add a value to an existing array variable defined in a Data Factory pipeline.
Activity
Filter activity to apply a filter expression to an input array.

Set Variable to set the value of an existing variable of type String, Bool, or Array defined in a Data Factory
Activity pipeline.
Validation activity to ensure the pipeline only continues execution once it has validated the attached dataset
reference exists
Wait activity the pipeline waits for the specified period of time before continuing with execution of subsequent
activities.
Webhook activity to control the execution of pipelines through your custom code.
Data flow to run your ADF data flow in pipeline debug (sandbox) runs and in pipeline triggered runs. (This
activity Activity is in public preview)
SSIS in ADFv2
Managed Cloud
Environment
Pick# nodes & node size
Resizable
SQL Standard Edition, Enterprise coming soon
Azure
SSIS
Project
Integration
Runtime
Compatible
Same SSIS runtime across Windows, Linux, Azure
Cloud

SSIS + SQL Server

SQL Managed instance + SSIS (in
ADFv2) Access on premises data
via VNet

Get Started
Hourly pricing (no SQL Server license
SSIS in ADFv2
Integration runtime - Different capabilities
1. DataMovement
Move data between data stores, built-in connectors, format
conversion, column mapping, and performantand scalable data
transfer
2. Activity Dispatch
Dispatch and monitor transformationactivities (e.g.Stored Proc on
SQLServer,Hive on HD Insight..)

3. SSIS package execution

Execute SSIS packages
51
SSIS in ADFv2
Combinations of IRtypes, networks and
capabilities
IR type Public network Private network

Azure Data movement

Activity dispatch

Self-hosted Data movement Data movement

Activity dispatch Activity dispatch

Azure-SSIS SSIS package execution SSIS package execution

https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime 52
 SSIS in ADFv2
Integrationruntimes
1.Azure IntegrationRuntime

• move data between cloud data stores

• fully managed
• serverless compute service (PaaS) on Azure
• cost will occur only for time of duration
• user could define data movement units 29

• compute size auto scaled for copy jobs

 SSIS in ADFv2
Integrationruntimes
2. Self-hosted IntegrationRuntime
• perform data integration securely in a private network environment w/o
direct line-of-sight from the public cloud environment
• Installed on-premises in your environment
• Supports copy activity between a cloud data stores and a data store
in private network
• Supports dispatching the transform activities
30
• Works in your corporate network or virtual private network
• Only Outbound http based connections to open internet
• Scale out supported
SSIS in ADFv2
Integrationruntimes
3.Azure-SSIS Integration Runtime
• fully managed cluster of Azure VMs for native execution of SSIS packages.
• Access to on-premises data access using Vnet (classic in preview)
• SSIS Catalog on Azure SQL DB or SQL Managed Instance
• scale up: set node size
• scale out: number of nodes
• reduce costs by start/stop of service
SSIS in ADFv2
 SSIS in ADFv2
Determining which IRto use
• IRis referenced aslinked servicein thedatafactory
• Transformation Activity: target compute needslinked
service
• Copy Activity: source and sink need linked service, the computation is
determined automatically (see detailson msdn)
• Integrationruntimelocations candiffer fromitsData Factory location
which usesit

https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
SSIS in ADFv2
SamplesADF and IRlocations
SSIS in ADFv2
Scaleable IntegrationServices
How to scale up/out using 3 Settings on Azure SSIS IR

1. Configurable number of nodes on which SSIS is executed

$AzureSSISNodeSize = "Standard_A4_v2" # minimum size,
others avail.*
2. Configurable size ofnodes
$AzureSSISNodeNumber = 2 #between 1 and 10 nodes*
3. Configurable maximum parallel executions pernode
$AzureSSISMaxParallelExecutionsPerNode = 2 # between 1-8*
In Powershell CommandLet: Set-
AzureRmDataFactoryV2IntegrationRuntime
SSIS in ADFv2
SSIS in ADFv2
Notes from thefield
1. Connect in SSMS directly to the DB SSISDB to see SSIS
Catalog
2. Deploy from Visual Studio only in Project Deployment
Mode, workaround SSMS Import
3. In Preview no SSIS 3rd Party components supported (e.g.
Theobald for SAP, cozyroc..)
4. V2 IR supports only V2 pipelines and V1 IR supports only V1. Both
cannot be used interchangeably. Even though it is the same installer
37
5. Use same location for Azure-SSIS IR and the used (SQL
Azure) DB for SSIS Catalog
SSIS in ADFv2
ExecutionMethods
1. SSIS packages can be executed via SSMS

2. SSIS packages can be executed via CLI

› Run dtexec.exe from the command prompt

3. SSIS packages can be executed via custom code/PSH using SSIS MOM .NET
SDK/API
› Microsoft.SqlServer.Management.IntegrationServices.dll is installed in .NET GAC
with SQL Server/SSMS installation

4. SSIS packages can be executed via T-SQL scripts executing SSISDB sprocs
› Execute SSISDB sprocs [catalog].[create_execution] +
[catalog].[set_execution_parameter_value] + [catalog].[start_execution]
SSIS in ADFv2
Scheduling Methods
1. SSIS package executions can be directly/explicitly scheduled via ADFv2 App (Work in
Progress)
› For now, SSIS package executions can be indirectly/implicitly scheduled via ADFv1/v2
Sproc
Activity

2. If you use Azure SQL MI server to host SSISDB

› SSIS package executions can also be scheduled via Azure SQL MI Agent (Extended
Private
Preview)

3. If you use Azure SQL DB server to host SSISDB

› SSIS package executions can also be scheduled via Elastic Jobs (Private Preview)

4. If you keep on-prem SQL Server

› SSIS package executions can also be scheduled via on-prem SQL Server Agent
Demo

comments

Azure Data Factory Azure SQL Database Power BI Dashboard

ingest copy and transform visualize

(just for fun)
storage blob
copy Data Management
Gateway
SQL database
(on-premises)
What is Azure Data Factory?
• Data Factory is a fully managed, cloud-based, data-integration service that
automates the movement and transformation of data. Like a factory that runs
equipment to transform raw materials into finished goods, Azure Data Factory
orchestrates existing services that collect raw data and transform it into ready-to-
use information.
• By using Azure Data Factory, you can create data-driven workflows to move data
between on-premises and cloud data stores. And you can process and transform
data by using compute services such as Azure HDInsight, Azure Data Lake
Analytics, and the SQL Server Integration Services (SSIS) integration runtime.
• With Data Factory, you can execute your data processing either on an Azure-
based cloud service or in your own self-hosted compute environment, such as
SSIS, SQL Server, or Oracle. After you create a pipeline that performs the action
you need, you can schedule it to run periodically (hourly, daily, or weekly, for
example), time window scheduling, or trigger the pipeline from an event
occurrence.
Introduction to Azure Data Factory
• The data landscape is more varied than ever with unstructured and
structured data originating from many cloud and on-premises sources.
• Data Factory enables you to process on-premises data like SQL Server,
together with cloud data like Azure SQL Database, Blobs, and Tables.
• These data sources can be composed, processed, and monitored through
simple, highly available, fault-tolerant data pipelines.
• Combining and shaping complex data can take more than one try to get it
right, and changing data models can be costly and time consuming.
• Using Data Factory you can focus on transformative analytics while the
service 'takes care of the plumbing'.
What are the top-level concepts of Azure Data
Factory?
• An Azure subscription can have one or more Azure Data Factory
instances (or data factories).
• Azure Data Factory contains four key components that work together
as a platform on which you can compose data-driven workflows with
steps to move and transform data.
Pipelines
• A data factory can have one or more pipelines. A pipeline is a logical grouping of activities to perform a unit of work. Together, the
activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingest data from an Azure blob
and then run a Hive query on an HDInsight cluster to partition the data. The benefit is that you can use a pipeline to manage the
activities as a set instead of having to manage each activity individually. You can chain together the activities in a pipeline to
operate them sequentially, or you can operate them independently, in parallel.
Activities
• Activities represent a processing step in a pipeline. For example, you can use a Copy activity to copy data from one data store to
another data store. Similarly, you can use a Hive activity, which runs a Hive query on an Azure HDInsight cluster to transform or
analyze your data. Data Factory supports three types of activities: data movement activities, data transformation activities, and
control activities.
Datasets
• Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your
activities as inputs or outputs.
Linked services
• Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to
external resources. Think of it this way: A linked service defines the connection to the data source, and a dataset represents the
structure of the data. For example, an Azure Storage linked service specifies the connection string to connect to the Azure Storage
account. And an Azure blob dataset specifies the blob container and the folder that contains the data.
Linked services have two purposes in Data Factory:
• To represent a data store that includes, but is not limited to, an on-premises SQL Server instance, an Oracle database instance, a file share, or an Azure
Blob storage account..
• To represent a compute resource that can host the execution of an activity. For example, the HDInsight Hive activity runs on an HDInsight Hadoop
cluster.
Triggers
• Triggers represent units of processing that determine when a pipeline execution is kicked off. There are different types of triggers for different types of
events.
Pipeline runs
• A pipeline run is an instance of a pipeline execution. You usually instantiate a pipeline run by passing arguments to the parameters that are defined in
the pipeline. You can pass the arguments manually or within the trigger definition.
Parameters
• Parameters are key-value pairs in a read-only configuration. You define parameters in a pipeline, and you pass the arguments for the defined
parameters during execution from a run context. The run context is created by a trigger or from a pipeline that you execute manually. Activities within
the pipeline consume the parameter values.
• A dataset is a strongly typed parameter and an entity that you can reuse or reference. An activity can reference datasets, and it can consume the
properties that are defined in the dataset definition.
• A linked service is also a strongly typed parameter that contains connection information to either a data store or a compute environment. It's also an
entity that you can reuse or reference.
• Control flows
• Control flows orchestrate pipeline activities that include chaining
activities in a sequence, branching, parameters that you define at the
pipeline level, and arguments that you pass as you invoke the pipeline
on demand or from a trigger. Control flows also include custom state
passing and looping containers (that is, foreach iterators).