📄️ About pipelines
Data pipeline is a structured workflow that facilitates the movement and transformation of data from various data sources into your central warehouse and also in another data source. A pipeline can be compared to a conveyor belt that picks up data, performs specific actions on it, and then deposits it in its designated location.
📄️ Creating Standard pipeline
1. On the DataGOL Home page, from the left navigation panel, click Lakehouse > Pipelines.
📄️ Creating Custom pipeline
1. On the DataGOL Home page, from the left navigation panel, click Lakehouse > Pipelines.
📄️ Creating Dedup pipeline
1. On the DataGOL Home page, from the left navigation panel, click Lakehouse > Pipelines.
📄️ Last mile pipeline flow
The Last Mile Pipeline transfers data from S3 storage to a database (DB), changing the storage type for easier access and use. Its primary function is to copy data from S3 into the designated DB.
📄️ Pipeline sync modes
While creating a pipelilne, you can choose any of the following sync mode options:
📄️ Managing pipeline
You can manage and monitor the pipelines from the Pipelines page.
📄️ Pipeline details
Click the link of a pipeline in the Pipeline list for a detailed view.