Scribd
Upload
164 views

Restful API

The document provides instructions for setting up a REST API pagination scenario in Azure Data Factory. It describes creating linked services, datasets, and a pipeline to retrieve data from a REST API in pages and load it into a target database table. The pipeline uses variables to track the current page number and record count to iterate through each page of data from the API using an Until loop activity until no more records are returned.

Uploaded by

NS R
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
164 views

Restful API

The document provides instructions for setting up a REST API pagination scenario in Azure Data Factory. It describes creating linked services, datasets, and a pipeline to retrieve data from a REST API in pages and load it into a target database table. The pipeline uses variables to track the current page number and record count to iterate through each page of data from the API using an Until loop activity until no more records are returned.

Uploaded by

NS R
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 5

A 

RESTful API is an application program interface (API) that uses HTTP requests to GET,
PUT, POST and DELETE data. An API for a website is code that allows two software programs
to communicate with each other. a RESTful API will define one or more URL endpoints with a
domain, port, path, and/or querystring — for example, https://mydomain/user/123?format=json .
The HTTP method. The primary or most-commonly-used HTTP verbs (or methods, as they are
properly called) are POST, GET, PUT, PATCH, and DELETE. These correspond to create,
read, update, and delete (or CRUD) operations, respectively.

General Notes
 This setup is for unique Pagination Rest scenarios where the next page URL isn't returned in
the response. For Pagination Rest APIs where the next page URL is returned, look into Azure
built in support for Pagination Azure Data Factory Documentation

Setup Steps
Database Setup
Refer to database section Database Setup

Trigger Setup
Refer to trigger section Trigger Setup

Rest API Linked Services Setup


1. Open Azure Data Factory.

2. Click on Author (Pencil on left side)

3. Click on Connections (bottom left).

4. Under Linked Services tab click New.

5. Select REST data source.

6. Give Link Service a Name.


Link Services Naming Strategy

7. Enter in the needed credentials and select the integration runtime. (Use


"AutoResolveIntegrationRuntime" for non internal systems)
Credentials Storage (lastpass)

8. Specify the Base URL.


9. Add an Annotations showing the direction of the connection i.e. "Source Connection" then
click Create.

Dataset Setup
1. Create a new Dataset in the Author section by clicking the three dots next to Datasets.

2. Give the Dataset a Name.


Datasets Naming Strategy

3. Setup the Dataset with the REST Linked Service.

4. Add a Page Parameter to the dataset.

5. Setup the Relative URL using the relative path and Page Parameter.
Example Dynamic content:

Template: @concat('api/v2/advisors/report/{reportID}?
api_key={apiKey}&username={userName}&page=',dataset().Page)

Example: @concat('api/v2/advisors/report/25268?
api_key=123&username=name&page=',dataset().Page)

6. Setup Target Dataset as needed.

Pipeline Setup
1. Create a new Pipeline in the Author section by clicking the three dots next to Pipelines.

2. Give the Pipeline a Name.


Pipeline Naming Strategy

3. Create three Pipeline Variables, all with the value of 1.

1. Page [Used as a parameter to hit the API, also used to set PageCounter]
2. PageCounter [Used to set Page variable]
3. RecordCount [Used to check if ADF is at the end of API pages]

4. Add an Until Activity to the pipeline. Name it "UntilNoRecords".


5. Create a Stored Procedure action, rename it to SwapTable. Change timeout to 5 seconds
and retry to 5. Select the Targeted Link service, then clicked the SwapTable SP. Import
parameters and set the parameter to the table name i.e “[dbo].[Agents]”.

6. Create a "TruncateTable" Lookup Activity and use it as a 'hack' to create a truncate


statement.

7. Clone the SwapTable action, and rename the clone to SwapTableBack.

8. Connect the Activities in the following order SwapTable, TruncateTable, UntilNoRecords on


success then SwapTableBack on UntilNoRecords failure.

9. Edit the Until Activity and change the Expression to the following:

@lessOrEquals(int(variables('RecordCount')),0)

This expression will check if there are more records left from the API.

10. Add a Copy data Activity inside of the Until Activity.

11. Change the following on the Copy Data:


11. To handle complex values (i.e Json Array) we can parse those as string by adding the
following to the translator:
"mapComplexValuesToString": true

To add that value click on the edit Code button on the pipeline.

12. Add a Set variable Activity. And rename it to "SetRecordCount".

13. Connect SetRecordCount to CopyData on CopyData success.

14. Setup the SetRecordCount variable from CopyData rows copied

@string(activity('CopyData').output.rowsCopied)

15. Repeat the steps to create two more Set variable


 SetPageCounter
 SetPage
16. Set the "SetPageCounter" value to the following:
@string(add(int(variables('Page')),1))

This incruments the count by 1.

17. Set the "SetPage" value to the following:


@variables('PageCounter')

18. Connect the variable on success in this order

1. SetRecordCount
2. SetPageCounter
3. SetPage
19. Move the newly created Pipeline and Datasets to a the BU folder.
Folder Naming Strategy
20. Added the needed trigger to the pipeline.

You might also like