Production-grade distributed task queue built from scratch in Python with zero external dependencies.
- Task lifecycle: pending → running → success/failed/retry → dead-letter
- Min-heap priority queue: O(log n) enqueue/dequeue with O(1) lookup by task ID
- Worker pool: ThreadPoolExecutor with timeout enforcement and graceful shutdown
- Exponential backoff:
min(max_delay, base × 2^retry) + jitter - Write-ahead log: append-only with fsync, periodic checkpoint, crash recovery
- Bloom filter + hash map: deduplication with configurable TTL
- Cron parser: wildcards, ranges, steps, lists; next-occurrence calculation
- DAG executor: Kahn's topological sort, cycle detection, parallel execution levels
- Token bucket rate limiter: per-queue with burst allowance
- Metrics: counters, gauges, histograms with P50/P95/P99
- Graceful shutdown: SIGTERM/SIGINT signal handling, drain in-flight tasks
- Circuit breaker: open/half-open/closed states for cascading failure prevention
from dtq import Broker, BrokerConfig, task, set_default_broker
@task(name="add")
def add(a, b):
return a + b
config = BrokerConfig(num_workers=4)
broker = Broker(config)
set_default_broker(broker)
broker.start()
result = add.delay(2, 3)
print(result.get(timeout=10)) # 5
broker.stop()┌─────────┐ ┌──────────────┐ ┌─────────────┐
│ Client │───▶│ Priority Queue│───▶│ Worker Pool │
│ @task │ │ (min-heap) │ │ (ThreadPool)│
└─────────┘ └──────────────┘ └──────┬──────┘
│
┌───────────────┐ ┌──────▼──────┐
│ Dead Letter │◀───│ Retry Policy│
│ Queue │ │ (exp backoff)│
└───────────────┘ └─────────────┘
Supporting: WAL, Dedup, Circuit Breaker, Rate Limiter, Cron, DAG, Metrics
python cli.py start --workers 4 # Start broker
python cli.py stats # Show statistics
python cli.py health # Health check
python cli.py dead-letter # Inspect DLQpip install pytest
python -m pytest tests/ -vdtq/
├── broker.py # Central coordinator
├── client.py # @task decorator, .delay(), AsyncResult
├── config.py # BrokerConfig
├── core/ # Task, Queue, Worker, Scheduler, Registry, Result
├── storage/ # WAL, MemoryStore, FileStore
├── reliability/ # Retry, DeadLetter, Dedup, CircuitBreaker
├── scheduling/ # Cron, DAG, RateLimiter
└── monitoring/ # Metrics, Health