Category: Big Data
Log Analytics using STRING_SPLIT function
Motivating Log Analytics Being a Data Engineer comes with its own set of challenges and opportunities since 80 per cent time is spent on cleaning and munging the raw...
Move Files with Azure Data Factory- End to End
Pre-requisite In the first part of this series, we focused on moving a single file from one blob location to another using ADF. Here is the link for the first part: Move...
Move Files with Azure Data Factory- Part II
Prologue In the first part of this series i.e. Move Files with Azure Data Factory- Part I, we went through the approach and demonstration to move a single file from one...
Move Files with Azure Data Factory- Part I
Why Move Files? A typical ETL process entails extracting the data, transforming it and loading the same in the destination. Firstly, we extract files in a...
Azure Data Factory Webhook activity
Motivating Webhook Before getting into details of a webhook activity, we should analyse its need in ADF. Apart from being an ETL/ELT tool, Azure Data Factory(ADF) has...
An Introduction to Azure IoT with Machine Learning
Motivating Azure IoT Before we dive into Azure IoT, we need to understand the significance of IoT in general. There was a time when the word ‘interaction’...
Spark Dataframe performance benefits
Before we move to Spark Dataframes and it’s benefits, it is imperative to take a look at the concept of Dataframe in general. A Dataframe is based on the concept...
Azure Databricks tutorial: end to end analytics
Before jumping to the Azure Databricks tutorial, it is good to know the evolution of the Data and AI space. Knowledge production started in ancient Sumerian...