Data Quality and Governance Fundamentals

Bad data is worse than no data. A dashboard showing wrong revenue numbers leads to wrong business decisions. A model trained on dirty data makes bad predictions. Data quality and governance are not optional — they are what separates a production data platform from a spreadsheet someone emailed around.

The six dimensions of data quality

1. Accuracy

Does the data reflect reality? If a customer’s address changed last week, does the warehouse show the new address?

2. Completeness

Are required fields populated? If 30% of your orders have null customer_id, your revenue-by-segment report is unreliable.

3. Consistency

Does the same data agree across systems? If the payments service says $1M in revenue and the orders table says $950K, something is wrong.

4. Timeliness

Is the data fresh enough? A fraud detection system needs sub-second data. A monthly board report can tolerate a day of delay.

5. Validity

Does the data conform to business rules? An order quantity of -5 or an email address without an @ is invalid.

6. Uniqueness

No duplicate records. If the same order appears twice, revenue is double-counted.

Implementing quality checks

dbt tests — the simplest starting point

dbt has built-in tests for the most common quality checks:

# models/staging/_staging.yml
version: 2
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ['pending', 'shipped', 'delivered', 'cancelled']
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('stg_customers')
              field: customer_id
      - name: total_amount
        tests:
          - not_null

Custom SQL tests for business rules:

-- tests/assert_no_negative_revenue.sql
SELECT order_id, total_amount
FROM {{ ref('fact_orders') }}
WHERE total_amount < 0

This test fails if any rows are returned — any order with negative revenue triggers an alert.

Great Expectations — Python-based quality framework

For more complex validations:

import great_expectations as gx

context = gx.get_context()

validator = context.sources.pandas_default.read_csv("orders.csv")

validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_unique("order_id")
validator.expect_column_values_to_be_between(
    "quantity", min_value=1, max_value=10000
)
validator.expect_column_values_to_be_in_set(
    "status", ["pending", "shipped", "delivered", "cancelled"]
)
validator.expect_column_pair_values_a_to_be_greater_than_b(
    "ship_date", "order_date", or_equal=True
)

results = validator.validate()
if not results.success:
    raise Exception("Data quality check failed")

The circuit breaker pattern

Stop bad data from flowing downstream. If quality thresholds are breached, halt the pipeline.

def quality_gate(df, table_name):
    """Halt pipeline if data quality thresholds are breached."""
    null_rate = df.isnull().sum().max() / len(df)
    duplicate_rate = 1 - df['id'].nunique() / len(df)
    row_count = len(df)

    if null_rate > 0.05:
        raise DataQualityError(
            f"{table_name}: null rate {null_rate:.2%} exceeds 5% threshold"
        )
    if duplicate_rate > 0.01:
        raise DataQualityError(
            f"{table_name}: duplicate rate {duplicate_rate:.2%} exceeds 1%"
        )
    if row_count == 0:
        raise DataQualityError(
            f"{table_name}: zero rows — source may be down"
        )

    return df

Integrate this into your Airflow DAG:

extract >> quality_gate_task >> transform >> load

If quality_gate_task fails, the pipeline stops. No bad data reaches the warehouse.

Data governance

Quality checks catch problems row by row. Governance is the organizational system that prevents problems at scale.

The four pillars

1. Data catalog — an inventory of all data assets. What tables exist, what they contain, who owns them, when they were last updated.

Tools: DataHub, OpenMetadata, Atlan, Alation.

2. Data lineage — track data from source to consumption. When a metric looks wrong, lineage tells you which upstream table, transformation, or source system to investigate.

Source: PostgreSQL.orders → Airflow DAG: daily_orders
  → Staging: stg_orders → dbt model: fact_orders
    → Dashboard: Revenue by Region

3. Access control — who can see what. PII (names, emails, SSNs) should be restricted. Column-level security lets analysts query the table without seeing sensitive columns.

-- Snowflake row access policy example
CREATE ROW ACCESS POLICY region_policy AS (region VARCHAR)
RETURNS BOOLEAN ->
  CASE
    WHEN CURRENT_ROLE() = 'ADMIN' THEN TRUE
    WHEN CURRENT_ROLE() = 'US_ANALYST' AND region = 'US' THEN TRUE
    ELSE FALSE
  END;

4. Data ownership — every dataset has an accountable owner. When the fact_orders table breaks, there is a specific person or team responsible for fixing it.

Classification

Tag data by sensitivity level:

• Public — aggregate metrics, published reports.
• Internal — business data accessible to employees.
• Confidential — customer PII, financial records, credentials.
• Restricted — regulated data (HIPAA, GDPR, SOC2).

Classification drives access control. Restricted data gets encryption at rest, audit logging, and limited access.

Governance for small teams

You do not need a data catalog tool on day one. Start with:

1. dbt docs — dbt docs generate creates a searchable documentation site with lineage graphs for free.
2. YAML schema files — document every model and column in dbt YAML files.
3. Naming conventions — stg_ for staging, int_ for intermediate, fct_ for facts, dim_ for dimensions. Consistent naming is the cheapest governance tool.
4. CODEOWNERS — use Git’s CODEOWNERS file to assign ownership to dbt model directories.
5. Tests — dbt tests are your first quality layer. Start with unique and not_null on every primary key.

Scale up to dedicated tools when your data platform has more than 50 models or more than 10 consumers.

Quality monitoring tools

Tool	Type	Best for
dbt tests	SQL assertions	Teams already using dbt
Great Expectations	Python expectations	Complex Python pipelines
Elementary	dbt-native observability	dbt-centric teams wanting automated monitoring
Monte Carlo	Data observability platform	Enterprise-scale anomaly detection
Soda	YAML-based checks	Multi-platform, simple configuration

Next steps

• Data Warehouse Concepts — understand the schemas your quality checks protect.
• Data Modeling for Analytics — build models worth governing.
• What Is Apache Airflow? — orchestrate quality checks in your pipelines.