Migrating from Azure Data Factory to Azure Synapse Integration

391

Did you think that it’s straightforward? I mean, did you think that you can simply export the ARM template from Azure Data Factory and import it into Azure Synapse? Well, let me break the bad news for you! There is no provision to import an ARM template into Azure Synapse.

Okay! Okay! Hold on! That’s too fast-paced. Hence, let’s introduce the characters here. Firstly comes the Azure Data Factory. It is Microsoft Azure’s primary ETL/ELT tool. Next comes Azure Synapse Family consisting of two members viz. dedicated SQL pool (Formerly MPP SQLDW) and Synapse workspace. The integration service of Synapse workspace is nothing but the ADF equivalent of the Synapse family.

You may be wondering why does this topic deserve an article in itself? Can’t we export the ARM template from  ADF and import it to Synapse? Well, the answer is NO. So, wherein lies the answer? It’s in two steps:

  1. Download ADF pipeline support files.
  2. Deploying them using PowerShell.

Download ADF support files

I won’t reinvent the wheel by repeating the steps which have been demonstrated so succinctly on this blog: Download Azure Data Factory support files.

Follow the steps mentioned and download the support files on your machine.

Deploying pipelines to Azure Synapse using PowerShell

Before pipeline deployment, we need to understand the way ADF is organized. The most fundamental entity of ADF is a Linked Service. Linked Services set up a connection between ADF and the datastores (sources and sink). On top of linked services, we have Datasets.  A dataset is a logical view on which data processing Activities are built. Finally, the activities combine to create a pipeline. The below diagram is a good summary.

Hence, the deployment/migration of ADF happens in the following order:

  • Linked Service
  • Datasets
  • Pipelines

Now let’s go through the steps for migration of ADF to synapse:

1 – Establish a connection

Make a connection to your Azure tenant:

Connect-AzAccount

Make sure the current subscription content is correctly set  by running the following command:

$context = Get-AzContext
$context

If not, run the following command.

$context = Select-AzSubscription -SubscriptionName "<subscription name>"
$context

However, make sure that you have the Azure Synapse PowerShell module installed before you go further, using these commands:

Install-Module Az.Synapse
Import-Module Az.Synapse

2 – Import a linked service

Create linked services using the following command:

Set-AzSynapseLinkedService -WorkspaceName <synapse workspace name> -Name <name of linked service name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context

3 – Import a dataset

Create dataset using the following command:

Set-AzSynapseDataSet -WorkspaceName <synapse workspace name> -Name <dataset name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context

4 – Import a pipeline

Finally, import the pipeline using the following command:

Set-AzSynapsePipeline -WorkspaceName <synapse workspace name> -Name <pipeline name> -DefinitionFile "<folder path to json file\json file name>.json" -DefaultProfile $context

 

Reference: https://github.com/solliancenet/azure-synapse-analytics-ga-content-packs/blob/main/hands-on-labs/lab-03/README.md

P.C. Azure Synapse Analytics

 



I am a Data Analytics professional with 5+ years of experience.


Leave a Reply