Category: Big Data

Managed Identity between Azure Data Factory and Azure storage

Last month Microsoft announced that Data Factory is now a ‘Trusted Service’ in Azure Storage and Azure Key Vault firewall. Accordingly, Data Factory can...

Azure Synapse Analytics: Azure SQL Data Warehouse revamped

Prologue On 5th November 2019, I read an early morning announcement of Azure Synapse Analytics. It was formerly known as Azure SQL Data warehouse. This announcement took...

Log Analytics with Python Pandas Explode

Prologue to Analytics with Python I strongly recommend reading this article before you go ahead with this article: Log Analytics using STRING_SPLIT  function. This...

Making ADF Web activity synchronous with Logic App

Prologue It is a common scenario in ETL jobs to send Email Notifications. With Azure Data Factory, such an example can be found in this article. Accordingly, we use web...

Move Files with Azure Data Factory- End to End

Pre-requisite In the first part of this series, we focused on moving a single file from one blob location to another using ADF. Here is the link for the first part: Move...

Move Files with Azure Data Factory- Part II

Prologue In the first part of this series i.e. Move Files with Azure Data Factory- Part I, we went through the approach and demonstration to move a single file from one...

Move Files with Azure Data Factory- Part I

Why Move Files? A typical ETL process entails extracting the data, transforming it and loading the same in the destination. Firstly, we extract files in a...

Azure Data Factory Webhook activity

Motivating Webhook Before getting into details of a webhook activity, we should analyse its need in ADF. Apart from being an ETL/ELT tool, Azure Data Factory(ADF) has...

An Introduction to Azure IoT with Machine Learning

Motivating Azure IoT Before we dive into Azure IoT, we need to understand the significance of IoT. There was a time when the word ‘interaction’ was...

Spark Dataframe performance benefits

Before we move to Spark Dataframes and its benefits, it is imperative to look at the concept of Dataframe. A Dataframe is based on the concept of a statistical table. It...