📄️ About Lakehouse
Lakehouse is a hybrid data architecture, which provides a unified platform for storing diverse data (structured, semi-structured, unstructured) and performing advanced analytics. Key capabilities include:
📄️ Building the golden layer
In the DataGOL ecosystem, the Golden Layer represents the pinnacle of data maturity. It is the transition from raw, noisy information into a polished, high-performance asset. Creating this layer is a journey that begins in the Lakehouse, matures in the Playground, and culminates in the Workbook.
📄️ Lakehouse workflow
1. Connecting to data Sources: The journey begins with establishing connections to various data sources. Think of these as the starting points where your raw data lives. These could be relational databases like SQL Server, PostgreSQL, cloud data warehouses like Redshift, or even file storage systems such as S3 or Azure blob.. The crucial function here is to enable the Lakehouse to access the data that needs to be processed and analyzed.
🗃️ Data sources
5 items
🗃️ Pipelines
8 items
🗃️ Orchestrations
11 items
📄️ Schema change detection
DataGOL's Schema Change Detection feature monitors the source tables involved in your pipelines for any modifications to their structure (e.g., new columns, deleted columns, data type changes). This helps you stay informed about potential impacts on your data pipelines and warehouse. When a schema change is detected, it is recorded, indicating the table, the modified columns, and the nature of the change (e.g., data type, size).
📄️ Jobs
Navigate to Jobs: Under the Lakehouse section, go to Jobs.
📄️ Agent Ops
Agent ops is a feature designed for company administrators or super admins to monitor, manage, and analyze the usage and cost associated with your company's large language model (LLM) providers (e.g., OpenAI, Anthropic, Gemini).