Step Snap 1: [dbt refresh (freshness)]

Refresh (freshness) is an important data quality control concept in dbt. Here is more specific how you could use it in daily usage.

1. Freshness Definition

sources:
  - name: staging
    database: production
    schema: trip_data_all
    tables:
      - name: green_tripdata
      - name: yellow_tripdata
        freshness:
          error_after: {count: 6, period: hour}

This configuration means:

• Sets up data freshness checks
• Will report an error if data hasn't been updated for 6 hours
• This is a data quality control mechanism

1. Why do we need Freshness checks?

• Ensure data is up-to-date
• Quickly identify data ingestion issues
• Prevent decision-making with stale data

1. How it works

• dbt checks the latest timestamp of source data
• Compares it with current time
• Triggers warning or error if difference exceeds threshold (6 hours in this case)

1. Common time period options

freshness:
  error_after:
    count: 6
    period: hour
    # Other options:
    # period: minute
    # period: day
    # period: week

1. How to check

# Run freshness check command
dbt source freshness