DATA PROCESSING

Two mechanisms help us in creating visualization from the data that is structured, unstructured, or semi-structured.

To perform ETL or ELT activities data is the major component i.e., data ingestion from multiple sources, and these sources can be device data, financial transaction data, sales data, and many more…..

In a world where data is the top priority for every industry 🪩, the requirement for data processing is also the top priority. In this article, we will understand the two most important mechanisms for data processing.

(i) Batch Processing

(ii) Stream Processing

To understand data processing let’s first understand Data Extraction/Data Ingestion from multiple sources:

Data Ingestion(Extract)

To bring data to a single centralized place is called Data Ingestion.

Data processing(Transform)

To process or transform this raw data and create one business model is called Data Processing ie, clean and trustable data. We have two ways to process our data:

(i) Batch Processing: Batch processing considers data in large volume.No real time data processing.Performs complex analytics.Example: credit card bills, Tax data.

ELT (Extract, Load, and Transform): ELT makes use of Batch oriented approach to process data.Mainly suitable for cloud Ex: ADF (Azure Data Factory).Works on complex models.