Tag: Apache Spark, azure databricks, pyspark, spark sql
Cumulative Distribution in Azure Databricks using Spark SQL
Prasad KulkarniMay 24, 2020
We can solve every problem in multiple ways. In our previous article, we motivated the need to fit cumulative distributions. Moreover, we demonstrated the same in Azure...
Cumulative Distribution in Azure Databricks
Prasad KulkarniMay 03, 2020
Imagine that you receive a requirement to calculate the aggregations like average on a range of percentiles and quartiles, for a given dataset. There are two ways to...
Databricks Koalas: bridge between pandas and spark
Prasad KulkarniMar 22, 2020
Imagine that you are an ML engineer. You have a massive task of operationalizing a model trained and tested by your Data Scientists. It is working perfectly well for the...
How to read mismatched schema in apache spark
Prasad KulkarniDec 20, 2019
In today’s world, python dominates the data analysis space, while apache-spark rules the big data paradigm. The former contains a plethora of libraries like pandas...
Lambda Architecture with Azure Databricks
Prasad KulkarniOct 22, 2019
Introducing Lambda Architecture It is imperative to know what is a Lambda Architecture, before jumping into Azure Databricks. The greek symbol lambda(λ) signifies...
Spark Dataframe performance benefits
Prasad KulkarniMay 07, 2019
Before we move to Spark Dataframes and it’s benefits, it is imperative to take a look at the concept of Dataframe in general. A Dataframe is based on the concept...
Azure Databricks tutorial: end to end analytics
Prasad KulkarniMay 02, 2019
Before jumping to the Azure Databricks tutorial, it is good to know the evolution of the Data and AI space. Knowledge production started in ancient Sumerian...