Scribd
Upload
14 views24 pages

Azure

The document outlines a step-by-step process for using Azure Data Factory's Copy Activity to transfer a .csv file while capturing custom metadata such as file last modified, pipeline name, and Azure Data Factory account name. It includes instructions for setting up containers in Azure Data Lake Storage, creating a new pipeline, and adding metadata properties during the copy process. The final outcome allows users to view the transferred file along with the specified custom metadata in the target container.

Uploaded by

Disha S
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views24 pages

Azure

The document outlines a step-by-step process for using Azure Data Factory's Copy Activity to transfer a .csv file while capturing custom metadata such as file last modified, pipeline name, and Azure Data Factory account name. It includes instructions for setting up containers in Azure Data Lake Storage, creating a new pipeline, and adding metadata properties during the copy process. The final outcome allows users to view the transferred file along with the specified custom metadata in the target container.

Uploaded by

Disha S
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 24

Azure Data Factory Real Time Scenario

Metadata Property in Copy Activity

Praveen Patel
Azure Data Engineer
Business Requirement

Let's assume using Copy Activity Data movement takes place between source and sink (we
are copying a .csv file from source to sink).

As per requirement i have been assigned a task by product manager while moving data to
sink some metadata must be captured like file last modified, format, pipeline name and
Azure Data Factory account name etc.

To carry out this requirement metadata property comes to work it helps you to set custom
metadata when copying data to sink like pipeline name, file lastModified, pipeline run id
and azure data factory account name etc.

Further you will get better understanding to learn metadata property in the practical
demonstration.

Praveen Patel | Azure Data Engineer


Step 1
Open Azure Portal go to ADLS storage account then go to container create a new
container called input to upload a new .csv file.

Praveen Patel | Azure Data Engineer


Step 2
Go to input container click on upload to upload any .csv file from my computer as i
uploaded education.csv file.

Praveen Patel | Azure Data Engineer


Step 3
Create another new container called target to store source file with custom metadata.

Praveen Patel | Azure Data Engineer


Step 4
Go back to Azure Portal and open Azure Data Factory.

Praveen Patel | Azure Data Engineer


Step 5
After Azure Data Factory is opened go to Author Tab.

Praveen Patel | Azure Data Engineer


Step 6
Click on + icon and create a new pipeline.

Select format as
Delimited Text click on
continue.

Praveen Patel | Azure Data Engineer


Step 7

In activity search bar search copy activity and drag and drop it.

Praveen Patel | Azure Data Engineer


Step 8
Go to Copy Activity sink create a new source dataset for ADLS Gen2 click on continue.

Praveen Patel | Azure Data Engineer


Step 9
Select a new dataset format as Delimited Text and click on continue.

Praveen Patel | Azure Data Engineer


Step 10
In set properties create a new linked service for ADLS Gen2 if you already created linked
service then select it.

Praveen Patel | Azure Data Engineer


Step 11
Click on browse button to select source file that stored in input container.

Praveen Patel | Azure Data Engineer


Step 12
Go to input container select source file that you uploaded as i selected education.csv file
and click on ok.

Praveen Patel | Azure Data Engineer


Step 13
Once source file is selected click on ok.

Praveen Patel | Azure Data Engineer


Step 14
Go to copy activity sink and similarly create a new sink dataset for ADLS Gen2 click on
continue and select dataset format as Delimited Text and click on continue.

Praveen Patel | Azure Data Engineer


Step 15
In set properties select same linked service for ADLS Gen2 click on browse button to select
target container to store source file.

Praveen Patel | Azure Data Engineer


Step 16
Select target container and click on ok.

Praveen Patel | Azure Data Engineer


Step 17
Go to copy activity sink you see metadata property click on new metadata.

Praveen Patel | Azure Data Engineer


Step 18
Click on add dynamic content select any meta custom data that you want to show in sink as
i went to system variables click on Data Factory name custom metadata and write custom
metadata name as Data Factory like add i added Pipeline run id metadata you can add more
more.

Praveen Patel | Azure Data Engineer


Step 19
Once setting metadata execute pipeline using debug mode and wait for five to ten seconds
until pipeline executes successfully.

Praveen Patel | Azure Data Engineer


Step 20
Once pipeline is executed successfully go back to ADLS storage and open target container.

Praveen Patel | Azure Data Engineer


Step 21
Click on refresh you see .csv file with same record existed in input container open .csv file
go to overview go down you see custom metadata with value Data Factory Name and
PipelineRunId that we added in sink metadata.

Praveen Patel | Azure Data Engineer


Praveen Patel
Azure Data Engineer

Follow Me to Get Such


More Content Like This

You might also like