Copy Data from Rest API to ADL using dynamic parameters at Dataset and Linked Service.

In this article, we will see how to copy data from an API with specific files to an Azure data lake.

Step 1:

Create a folder raw where we will copy data in the data lake.

ADL folder to copy data

Create a file with .json format and add the configuration-related information.

In order to copy these data from HTTP requests we will have the following components from ADF.

1. Dataset -> At the source and destination.
2. Linked Service -> At the source and destination.

Step 2: Datasets

Dataset for the JSON file.

Dataset for the JSON file upload for the list of source file data

Dataset at the source.

Parameters at the source dataset

Dataset to read the source URLs

Dataset at the destination.

Parameters at the destination dataset

Dataset to read the destination filename

Step 3: LinkedService

Linked service at the source.

Source dataset LS

Parameters at the Linked service

Linked service at the destination.

Destination dataset LS

Step4:

Create a pipeline end-to-end dynamically to copy data.

Lookup Activity:- To get a list of files to copy data from the source.

ForEachActivity: Works as a loop to copy data.

Copy Activity

At the source

At the destination

Successfully executed copy activity.

Data copies at the destination ADL raw folder.

Thanks for the read🙏.Do clap👏 if find it useful.

“Keep learning and keep sharing knowledge”