"A local-first guardrail system that prevents silent data failures in batch pipelines."

Batch pipelines often fail silently—bad data enters, is processed, and leads to incorrect business reports hours later. This system introduces a strict validation gate that checks data against a contract before downstream jobs run.

• Docker & Docker Compose
• Python 3.11+
• Make

1. Start Infrastructure:

   make up

   (Wait for Postgres & Grafana to be ready)

2. Run a "Good" Pipeline:

   make demo-good

   • Ingests valid data.
   • Validates (Pass).
   • Runs Downstream Job.

3. Run a "Bad" Pipeline:

   make demo-bad

   • Injects a schema failure or freshness breach.
   • Validates (Fail).
   • Creates Incident.
   • Blocks Downstream Job.

4. Replay & Fix:

   make replay

   • Simulates fixing the data.
   • Reruns validation.
   • Unblocks downstream.

• Grafana: http://localhost:3000 (admin/admin)
• Prometheus: http://localhost:9090

See DESIGN.md for details.

# Ingest data
python -m drg.cli ingest --run-id <uuid> --output data/raw/ <scenarios>

# Validate
python -m drg.cli validate --run-id <uuid> --dataset data/raw/file.parquet

# Check Gate & Run Downstream
python -m drg.cli downstream run --run-id <uuid>

# Replay
python -m drg.cli replay --run-id <uuid>