DATA PROCESSING

There are two mechanism’s that help us in creating visualization from the data that is structured, unstructured, or semi-structured.

In order to perform ETL or ELT activities data is the major component i.e., data ingestion from multiple sources, and these sources can be device data, financial transaction data, sales data, and many more…..

In a world where data is the top priority for every industry 🪩, so the requirement for data processing is also the top priority. In this article, we will understand the two most important mechanism’s for data processing.

(i) Batch Processing

(ii) Stream Processing

In order to understand data processing let’s first understand Data Extraction/Data Ingestion from multiple sources:

Data Ingestion(Extract)

To bring data to a one-single centralized place is called Data Ingestion.

Data processing(Transform)

In order to process or transform this raw data and create one business model is called Data Processing ie, clean and trustable data. We have two ways to process our data:

(i) Batch Processing: Batch processing considers data in large volume.No real time data processing.Performs complex analytics.Example: credit card bills, Tax data.

ELT (Extract, Load, and Transform): ELT makes use of Batch oriented approach to process data.Mainly suitable for cloud Ex: ADF (Azure Data Factory).Works on complex models.

ELT(Extract, Transform, and Load)

(ii) Stream Processing:Stream processing considers data in small volumn and process data in real timePerforms simple logic, aggregation or calculations.Example: Stock Market, Heat alarm system, Youtube

ETL (Extract, Transform and Load): ETL makes use of a stream-oriented approach to process data.Suitable for on-premises activity Ex: SSIS.Works on simple models.

ETL(Extract, Transform and Load)

ETL vs ELT

Thanks for the read. Do clap 👏👏if find it useful🙂.

“Keep learning and keep sharing knowledge”