DATA ENGINEERING 101

Azure Data Factory

Custom Expressions

Shwetank Singh
GritSetGrow - GSGLearn.com

gsglearn.com
Azure Data Factory Expressions
Dynamic Schema Mapping Based on
Parameter
@if(equals(pipeline().parameters.useExten
dedSchema, true), 'ExtendedSchema',
'BasicSchema')

Schema property of the dataset or activity

where the schema needs to be dynamically set.

Choose between 'ExtendedSchema' and

'BasicSchema' based on the
'useExtendedSchema' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Conditional Copy Behavior

@if(contains(pipeline().parameters.copyM
ode, 'Incremental'), 'IncrementalCopy',
'FullCopy')

Settings of the Copy Data activity to determine

the copy behavior.

Determine copy behavior ('IncrementalCopy' vs.

'FullCopy') based on the 'copyMode' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Blob Storage Path Based on File
Type
@concat('storage/account/',
pipeline().parameters.fileType, '/',
pipeline().parameters.fileName)

File path field in the sink dataset (e.g., Azure

Blob Storage).

Store files in different folders based on 'fileType'

(e.g., 'images', 'documents').

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Setting Timeout Based on Environment

@if(equals(pipeline().parameters.environ
ment, 'Prod'), '01:00:00', '00:30:00')

Timeout settings of activities such as Copy Data

or Web activities.

Set different timeout values for production and

non-production environments.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic API Endpoint Selection

@if(equals(pipeline().parameters.environ
ment, 'Prod'),
'https://api.prod.example.com',
'https://api.dev.example.com')
Endpoint URL field in Web activities or HTTP
datasets.

Switch API endpoints based on the

'environment' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Retry Count Based on Parameter

@if(greater(pipeline().parameters.retryAt
tempts, 5), 5,
pipeline().parameters.retryAttempts)

Retry policy settings of activities like Copy Data

or Web activities.

Limit retry attempts to a maximum of 5.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Conditional Logging Level

@if(equals(pipeline().parameters.isDebug,
true), 'Verbose', 'Error')

Logging configurations in activities or custom

logging solutions.

Set logging levels based on a debug flag.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Linked Service Selection

@if(equals(pipeline().parameters.useNewS
ervice, true), 'NewLinkedService',
'OldLinkedService')

Linked service reference in datasets or

activities.

Switch between linked services based on

'useNewService' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Column Selection in SQL Query

@concat('SELECT ',
pipeline().parameters.columns, ' FROM
YourTable')

Source settings in Copy Data activity where SQL

queries are defined.

Select specific columns based on the 'columns'

parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Setting Dynamic Partition Columns

@split(pipeline().parameters.partitionCo
lumns, ',')

Partition settings in data flow activities.

Define partition columns dynamically using a

comma-separated string.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Join Conditions in Data Flow

@concat('A.',
pipeline().parameters.joinColumn, ' = B.',
pipeline().parameters.joinColumn)

Join conditions in Mapping Data Flow activities.

Define join conditions based on the 'joinColumn'

parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Pipeline Trigger Time

@concat('2025-01-',
pipeline().parameters.triggerDay, 'T',
pipeline().parameters.triggerTime, ':00Z')

Trigger settings when scheduling pipeline runs.

Set a dynamic trigger time using 'triggerDay'

and 'triggerTime' parameters.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Service Bus Topic Name

@concat('topic-',
pipeline().parameters.topicSuffix)

Topic name field in Service Bus activities or

configurations.

Create service bus topic names dynamically

based on a suffix.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Lookup Table Name

@concat('LookupTable_',
pipeline().parameters.lookupType)

Table name field in Lookup activities.

Select different lookup tables based on

'lookupType'.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Data Partitioning

@pipeline().parameters.partitionCount

Partition count settings in activities like Copy

Data or data flows.

Set the number of data partitions dynamically.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Retry Interval

@mul(pipeline().parameters.retryIntervalS
econds, 1000)

Retry interval settings in retry policies of

activities.

Convert retry interval from seconds to

milliseconds.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Notification Email Addresses

@pipeline().parameters.emailList

Email recipients field in Notification activities

like Send Email.

Send notifications to a dynamic list of email

addresses.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Row Count Validation

@greaterOrEquals(activity('SourceActi
vity').output.rowCount,
pipeline().parameters.minRowCount)

Validation activities or conditions to check row

counts.

Validate that the source activity returns at least

'minRowCount' rows.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Schema Validation

@if(equals(activity('SourceActivity').ou
tput.schema,
pipeline().parameters.expectedSchema),
'Valid', 'Invalid')

Validation steps or conditional activities based

on schema.

Check if the source schema matches the

expected schema.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic File Extension Handling

@concat('datafile_',
pipeline().parameters.fileVersion, '.',
pipeline().parameters.fileExtension)

File name field in sink datasets or activities.

Generate filenames with dynamic extensions.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Source Connection String

@concat('Server=',
pipeline().parameters.serverName,
';Database=',
pipeline().parameters.databaseName, ';User
Id=', pipeline().parameters.userId,
';Password=',
pipeline().parameters.password, ';')

Connection string field in linked services for

data sources.
Build a connection string dynamically based on
parameters.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Sink Configuration Based on Type

@if(equals(pipeline().parameters.sinkType,
'Blob'), 'AzureBlob', 'AzureDataLake')

Sink type configurations in Copy Data

activities.

Choose between Azure Blob Storage and Azure

Data Lake as the sink based on 'sinkType'.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic HTTP Headers in Web Activity

@json(pipeline().parameters.httpHeaders)

Headers field in Web or HTTP activities.

Pass dynamic HTTP headers to a Web activity.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Resource Group Selection

@pipeline().parameters.resourceGroup

Resource group field in linked services or

deployment activities.

Deploy resources to different Azure resource

groups dynamically.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Storage Account Selection

@pipeline().parameters.storageAccountName

Storage account field in linked services or

datasets.

Select different storage accounts based on the

'storageAccountName' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic API Versioning

@concat('v',
pipeline().parameters.apiVersion)

API version field in Web activities or HTTP

datasets.

Set API version dynamically in HTTP requests.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Conditional Split in Data Flow

@pipeline().parameters.splitCondition

Split transformation conditions in Mapping

Data Flow activities.

Define split conditions in a Mapping Data Flow

dynamically.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Lookup Condition

@concat('WHERE ',
pipeline().parameters.lookupColumn, ' = ''',
pipeline().parameters.lookupValue, '''')

Lookup query field in Lookup activities.

Create dynamic lookup conditions based on

parameters.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Transformation Logic

@pipeline().parameters.transformationScript

Script or transformation field in data flow

activities.

Apply different transformation scripts based on

parameters.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Compression Type for Sink

@pipeline().parameters.compressionType

Compression settings in sink datasets or

activities.

Set the compression type (e.g., 'gzip', 'zip')

dynamically for output files.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic SQL Server Instance Selection

@pipeline().parameters.sqlServerInstance

SQL Server instance field in linked services or

datasets.

Connect to different SQL Server instances based

on the 'sqlServerInstance' parameter.

Shwetank Singh
GritSetGrow - GSGLearn.com
Azure Data Factory Expressions
Dynamic Pipeline Variable Initialization

@pipeline().parameters.initialValue

Variable initialization field in pipeline

variables.

Initialize pipeline variables with dynamic values.

Shwetank Singh
GritSetGrow - GSGLearn.com
Shwetank Singh
GritSetGrow - GSGLearn.com