A production-grade URL shortener built on Flask · Peewee · PostgreSQL · Redis · Nginx · Gunicorn, with full observability (Prometheus + Grafana), load testing (k6), and chaos engineering.

Stack: Flask · Peewee ORM · PostgreSQL 16 · Redis 7 · Gunicorn · Nginx · Prometheus · Grafana · Docker Compose · Docker Swarm · uv

# 1. Clone
git clone <repo-url> && cd ProdBreaker

# 2. Configure
cp .env.example .env   # add DISCORD_WEBHOOK_URL if you want alerts

# 3. Start everything
docker compose up --build -d

# 4. Seed the database (users → urls → events, order matters for FK constraints)
docker compose cp users.csv web:/app/users.csv
docker compose cp urls.csv  web:/app/urls.csv
docker compose cp events.csv web:/app/events.csv
docker compose exec web python load_csv.py users.csv urls.csv events.csv
# → Loaded 400 rows into User ...
# → Loaded 2000 rows into Url ...
# → Loaded 3422 rows into Event ...

# 5. Verify
curl http://localhost:${APP_PORT:-8000}/health
# → {"status": "ok"}

Key endpoints to verify the stack is working:

Log entries look like:

[
  {"timestamp": "2026-04-04 22:43:09,858", "level": "INFO", "logger": "app", "message": "request started", "method": "GET", "path": "/health"},
  {"timestamp": "2026-04-04 22:43:09,862", "level": "INFO", "logger": "app", "message": "request finished", "method": "GET", "path": "/health", "status": 200, "duration_ms": 4.07}
]

Logs are written to a Docker volume (app_logs) so they persist across container restarts.

All errors return JSON:

{"error": "Not Found", "message": "User 99 not found"}

All datetimes are ISO 8601 strings: "2025-09-19T22:25:05"

Cache header on /products: X-Cache: HIT or X-Cache: MISS

Prerequisites: Python 3.13+, PostgreSQL, Redis, uv

uv sync                        # install dependencies
cp .env.example .env           # set DATABASE_HOST=localhost for local Postgres
uv run run.py                  # starts on APP_PORT (default 8000)

uv run pytest tests/ -v --cov=app --cov-report=term-missing

53 tests, 86% coverage. Tests use in-memory SQLite — no Postgres or Redis required.

tests/test_health.py     — health endpoint
tests/test_errors.py     — 404/405 return JSON not HTML
tests/test_products.py   — products route + Redis cache + chaos
tests/test_users.py      — full user CRUD + bulk CSV
tests/test_urls.py       — full URL CRUD + short code + redirect + events
tests/test_events.py     — event listing and detail parsing

k6 run load_test.js

Ramps to 500 concurrent users over 90 seconds. Each run uses a unique timestamp prefix for generated usernames (via k6's setup() function), so re-running against a populated database does not produce 409 conflicts — do not remove the setup() function from load_test.js.

Results (10-core MacBook, 3 web replicas):

Outputs load-summary.html (HTML report) and load-summary.json.

Three CSV files are included as seed data. They must be loaded in this order (FK constraint: urls reference users, events reference both):

docker compose exec web python load_csv.py users.csv urls.csv events.csv

load_csv.py auto-detects the model from the CSV headers, preserves explicit IDs, and resets PostgreSQL sequences after each bulk insert. Multiple files can be passed in one invocation.

Important: DATABASE_HOST=db in .env is the Docker-internal service name. Running load_csv.py directly from the host will fail with a DNS error — always run it via docker compose exec web.

The product table requires no seed data. /products returns [] by design until product rows are inserted.

Grafana dashboard auto-loads at http://localhost:3000 with 10 panels:

• Latency — p50/p95/p99 time series + p95 gauge (red at >500ms)
• Traffic — req/s by status code + total RPS
• Errors — 5xx rate (red at >5%)
• Saturation — Postgres connections vs max, Redis memory
• Cache — Redis hit rate
• Uptime — App status (UP/DOWN)

Alerts fire to Discord on:

• p95 latency > 500ms
• 5xx error rate > 5%
• Zero traffic for 2 minutes
• Postgres connections > 80% of max
• Redis memory > 80%
• Flask app unreachable

Set your webhook URL directly in monitoring/alertmanager/alertmanager.yml — replace the url: value under the discord receiver, then docker compose restart alertmanager.

ProdBreaker/
├── app/
│   ├── __init__.py              # App factory, error handlers, metrics
│   ├── cache.py                 # Redis helpers (cache_get, cache_set)
│   ├── database.py              # PooledPostgresqlDatabase, BaseModel
│   ├── models/
│   │   ├── product.py           # Product model
│   │   ├── user.py              # User model
│   │   ├── url.py               # Url model (short codes)
│   │   └── event.py             # Event model (analytics)
│   ├── logging_config.py        # JSON structured logging (stdout + file)
│   └── routes/
│       ├── products.py          # GET /products
│       ├── users.py             # /users CRUD + bulk
│       ├── urls.py              # /urls CRUD
│       ├── events.py            # GET /events
│       └── logs.py              # GET /logs (view logs without SSH)
├── monitoring/
│   ├── prometheus/              # prometheus.yml + alerts.yml
│   ├── alertmanager/            # alertmanager.yml (Discord routing)
│   └── grafana/                 # Dashboard JSON + provisioning
├── nginx/
│   └── nginx.conf               # Reverse proxy + stub_status
├── tests/                       # pytest suite (49 tests)
├── Dockerfile                   # Multi-stage: builder + runtime (python:3.13)
├── docker-compose.yml           # Local dev: app + db + redis + nginx + monitoring
├── docker-stack.yml             # Docker Swarm: 3 web replicas, self-healing chaos demo
├── load_test.js                 # k6 load test (500 VUs)
├── load_csv.py                  # CSV seed data loader
├── RUNBOOK.md                   # 3 AM emergency guide
├── DECISION_LOG.md              # Why we chose each technology
├── CAPACITY_PLAN.md             # How many users, what breaks first
├── FAILURE_MODES.md             # What breaks + chaos reproduction
└── PERFORMANCE_REPORT.md        # Load test results + bottleneck analysis

/products returns 503 "Database is unavailable" The product table does not exist. This happens if the container image was built before Product was added to db.create_tables() in app/__init__.py. Rebuild: docker compose up --build -d.

load_csv.py fails with "could not translate host name db" You are running the script from the host. DATABASE_HOST=db only resolves inside Docker. Use docker compose exec web python load_csv.py ... instead.

k6 create user 201 failing at high rate (409 Conflict) The database has users from a previous run and the setup() function was removed from load_test.js. Restore setup() — it generates a per-run runId that prefixes all generated usernames, preventing collisions across runs. Do not truncate the database to work around this.

pytest fails with could not translate host name db init_db() is overwriting the test's SQLite database with a Postgres connection. Ensure app/database.py only calls db.initialize() (and registers request hooks) inside if db.obj is None:. The conftest sets db to SQLite before calling create_app(), so the guard prevents the overwrite.

pytest fails with no such table: user The SQLite in-memory connection was closed between requests. Flask's teardown_appcontext hook calls db.close(), which destroys :memory:. The fix: request lifecycle hooks must be inside the if db.obj is None: guard so they are not registered in test mode. The conftest calls db.connect() once at session scope to keep the connection alive.