Use Data Pipeline Prover with your AI.

Validate data pipeline

Bad pipelines do not crash. they silently produce wrong data that drives wrong decisions. The worst pipeline failure is one that looks correct. You must: (1) define the EXACT schema contract at pipeline boundaries. field names, types, required/optional, validation library, failure behavior (reject to dead-letter queue, not silently drop). "We have schemas" is not a contract. show the fields, types, and what happens when validation fails, (2) describe the IDEMPOTENCY mechanism. how are re-runs safe? Upserts with composite key, deduplication via event_id, exactly-once via Kafka transactions, checkpoint-based resumption. "It is idempotent" without naming the mechanism is a claim, not a guarantee, (3) set a FRESHNESS SLA with a NUMBER and monitoring. "data no older than X minutes/hours." Monitored via last_updated_at timestamp. Alert when breached. Without a number, there is no SLA. "current" means nothing, (4) describe data LINEAGE. source system, extraction method, transformation steps (each named), output destination, data ownership (which team). When a number looks wrong, can you trace it to the raw source record through every transformation? If rejected, your pipeline has an architecture flaw that will silently corrupt downstream data. Structured reflection capability for data pipeline architecture based on Data Mesh (Dehghani 2022), Data Contracts (Jones 2023), and Designing Data-Intensive Applications (Kleppmann 2017). Forces the agent to define schema contracts at pipeline boundaries, prove idempotency mechanisms, set freshness SLAs with monitoring, and trace data lineage end-to-end. Agents are brutal consumers of data. bad data produces confidently wrong actions. Catches Schema Absence (no data contract. pipeline accepts any shape and silently corrupts downstream consumers when a source adds a column or changes a type), Non-Idempotency (re-running a pipeline creates duplicates. no upsert keys, no dedup mechanism, no exactly-once guarantee. "Just run it again" doubles the data), Stale Unawareness (no freshness SLA. "the data is current" without a number. Is it 15 minutes old? 15 hours? The dashboard shows yesterday's revenue as today's and nobody notices), and Lineage Blindness (a number looks wrong but you cannot trace it back to the source record. no transformation log, no ownership, no way to debug). Call once per pipeline design or data architecture review