irst and most important: very detailed instruction by the author and good to check if you need to explore the concepts!

https://colab.research.google.com/drive/1FiAHNFenM8RyptyTPtDTfqPCi5W6KX_V?usp=sharing#scrollTo=ew7-fr5EuyOu

Step Snap 1: [Data Lakehouse - Understanding Schema and Metadata Management Principles 🏗️]

The Data Lakehouse architecture combines the best features of data lakes and data warehouses through intelligent metadata management:

1. Schema Management Approaches

a) Write-time Schema Definition:
   - Pre-defined schema like traditional warehouses
   - Strict data validation during ingestion
   - Example:
     table.create("events", schema=("id STRING", "timestamp LONG"))

b) Read-time Schema Inference:
   - Dynamic schema detection like data lakes
   - Flexible data ingestion
   - Example:
     table.create("events", schema="infer")

c) Hybrid Mode (Most Common):
   - Combines both approaches
   - Allows schema evolution

2. Metadata Layer Operations

Data Flow:
Raw Data → Schema Check/Inference → Metadata Update → Storage
                      ↓
            Metadata Catalog Update
                      ↓
            Statistics Generation

3. Schema Evolution Mechanism

// Initial Data Structure
{
  "user_id": "123",
  "action": "login",
  "timestamp": "2024-02-16T10:00:00"
}

// Evolution with New Fields
{
  "user_id": "124",
  "action": "login",
  "timestamp": "2024-02-16T10:05:00",
  "device": "mobile",    // New field
  "location": "CN"       // New field
}

4. Core Metadata Management Features

  1. Schema Evolution Management
  1. Data Lineage Tracking
Raw Logs → Cleansed Data → Aggregated Stats → Analytics
    ↓           ↓              ↓               ↓
Metadata1   Metadata2      Metadata3       Metadata4
  1. Performance Optimization Data