Fivetran

After a long 🙂 back with another article. This article is about Fivetran🤔.

No worries 🙃 if you heard this for the first time or it's great if you already know about it. This article is for the people i.e; my audience who heard it for the first time.

As the data is growing so, the urge for data extraction, transformation, and loading using the best tools will always be in demand. Secondly, the data integration demand is also increasing day by day. Fivetran is one of the ELT tools that provides a fully automated cloud-managed data pipeline for extraction, loading, and transformation. So this makes Fivetran popular among various tools.

Fivetran Architecture

Note: Data in Fivetran passes in an encrypted manner only.

In Fivetran architecture, we have a data source, Fivetran UI, and Destination:

(i) Source:- Data comes from multiple sources like databases, cloud, events, files, etc…

(ii)Fivetran:- The UI of Fivetran has three major components: Connector, Core, and Writer.

(iii)Destination: Last stage where data is actually stored.

Why Fivetran?

1. One of the ELT tools.
2. No setup required the complete browser managed.
3. A central repository of data that pulls data from different sources.
4. Helps us in extracting and loading data to the destination.
5. Helps to centralize data from varied sources.

Fivetran provides connections from 165(example: from the cloud, drives, social media ) different sources.

step1: Set up the connection with help of connectors from the source.

step2: Once the connection is set up try to load the database tables and make sure to enable tracking changes or data capture changes from the database in order to have table sync.

We have two types of accounts set up for destinations in Fivetran:

1. Use our own destination (example:- snowflake, Postgres rds, etc…)
2. Fivetran managed destination (example:- Fivetran (Bigquery data warehouse)

step3: After loading data, we can perform the transformation.

We have two ways of transformation:

1. SQL transformation
2. Transformation for dbt core

step4: We can sync data from source to destination based on your requirements.

We have three ways to manage sync

1. Initial sync
2. Incremental sync
3. Re-sync

The best resource to learn Fivetran:

https://www.udemy.com/share/106hy63@PL6GOcC48URxOmWePlC5LcRHRNNplZhjHUrLccE9YRjKkY6So4BqmCdvlq5rd573/

Thanks for the read 😊 Do clap if find it useful✌️!!!

“Keep learning and keep sharing knowledge”