You can add users to an orchestration. If they do not already have access to associated pipelines, they can be granted permissions. Only Lakehouse members can be added, and they will receive creator permissions by default.
Orchestration in DataGOL allows you to define and manage the execution order of multiple pipelines. This is crucial when you have dependencies between pipelines, where one pipeline needs to complete successfully before another can begin. Without orchestration, scheduling pipelines to run independently might lead to failures or unexpected results if a dependent pipeline runs before its required data is ready.
To delete an orchestration, do the following:
To edit an orchestration, do the following:
You can manage and monitor the orchestrations from the Orchestrations page. The main Orchestration page display a list of all created orchestrations.
1. On the DataGOL Home page, from the left navigation panel, click Lakehouse > Orchestration. The list of created orchestrations is displayed.
1. On the DataGOL Home page, from the left navigation panel, click Lakehouse > Orchestration.
From the Pipelines tab of an orchestration, you can view the specifics of each pipeline within an orchestration. Click a pipeline link to access its comprehensive details, including its data source and destination. Furthermore, you can find information about the pipeline's creator and its designated stage order. For granular insights, the status and job stage details are readily available. To explore the data lineage, scroll right and click the more options icon.
The Graphs tab within an orchestration provides a clear visual representation of the pipeline's execution order, illustrating the sequence of stages. To view the orchestration graph do the following:
The Runs tab provides a comprehensive view of each orchestration execution. Here, you can readily see details such as the run's name, the user who initiated it, the start and end times, the total duration, and the current status (e.g., completed, failed). To investigate data lineage, click the more options icon. Additionally, clicking the View Snapshot link offers a detailed, point-in-time overview of that specific run.