Copy Data from Rest API to ADL using dynamic parameters at Dataset and Linked Service.
In this article, we will see how to copy data from an API with specific files to an Azure data lake.
Create a folder raw where we will copy data in the data lake.
Create a file with .json format and add the configuration-related information.
In order to copy these data from HTTP requests we will have the following components from ADF.
- Dataset -> At the source and destination.
- Linked Service -> At the source and destination.
Step 2: Datasets
Dataset for the JSON file.
Dataset at the source.
Dataset at the destination.
Step 3: LinkedService
Linked service at the source.
Linked service at the destination.
Create a pipeline end-to-end dynamically to copy data.
Lookup Activity:- To get a list of files to copy data from the source.
ForEachActivity: Works as a loop to copy data.
At the source
At the destination
Successfully executed copy activity.
Data copies at the destination ADL raw folder.
Thanks for the read🙏.Do clap👏 if find it useful.
“Keep learning and keep sharing knowledge”