Update Delta table schema

How to change Column type in Delta Table? Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs. In this

Tips for Postman tool

Postman is a great tool to test APIs without writing a single line of code. Here are some functionalities I find very useful: 1. Environment variables To avoid copy-pasting the same token or key across multiple Postman requests, you can use variables. First, click on the eye symbol, on the top right corner This will open a

Data Wrangling and It’s importance

WHAT IS DATA WRANGLING? Data wrangling—also called data cleaning, data remediation, or data munging—refers to a variety of processes designed to transform raw data into more readily used formats. The exact methods differ from project to project depending on the data you’re leveraging and the goal you’re trying to achieve. Some examples of data wrangling include: Merging multiple

Kimball vs Inmon – Data warehouse Architecture

In this blog, we will discuss the basics of a data warehouse, it’s characteristics, and compare the two popular data warehouse approaches- Kimball and Inmon. Concept of Data Warehouse The key data warehouse concept allows users to access a unified version of truth for timely business decision-making, reporting, and forecasting. DWH functions like an information

Slowly Changing Dimensions in Data Warehouse

Slowly Changing Dimensions (SCD) – dimensions that change slowly over time, rather than changing on regular schedule, time-base. In Data Warehouse there is a need to track changes in dimension attributes in order to report historical data. In other words, implementing one of the SCD types should enable users assigning proper dimension’s attribute value for given

How to run SSIS packages using Azure Data Factory(ADF)

 In this article, we will show how to run an SSIS package using the Azure Data Factory. Azure SSIS Integration Runtime Setup Azure SSIS IR is an Azure Data Factory fully managed cluster of virtual machines that are hosted in Azure and dedicated to run SSIS packages in the Data Factory, with the ability to

Data Modelling: Conceptual, Logical & Physical

What is Data Modelling? Data modeling (data modelling) is the process of creating a data model for the data to be stored in a database. This data model is a conceptual representation of Data objects, the associations between different data objects, and the rules. Data modeling helps in the visual representation of data and enforces business

How Logic Apps differs from Data Factory

Azure Logic Apps & Azure Data Factory are both orchestrators, but how do they differ from each other? Azure data Factory is a serverless orchestrator where you can create pipelines that represent a workflow. In these pipelines you have sequences of activities, or steps, and have granular control on what to do if something fails

Azure Logic Apps vs Azure Functions

We were recently asked by a client about how to choose between Microsoft’s Azure Logic Apps or Azure Functions for a recent integration project. This post is the response that we gave to our client and will help you compare and choose between the two. Comparison Solution Azure Logic Apps Azure Functions Summary Azure Logic

Azure Storage redundancy

Azure Storage always stores multiple copies of your data so that it is protected from planned and unplanned events, including transient hardware failures, network or power outages, and massive natural disasters. Redundancy ensures that your storage account meets its availability and durability targets even in the face of failures. When deciding which redundancy option is