## Description Currently, the Helm chart's hook Jobs run **sequentially** due to different `helm.sh/hook-weight` values: | Order | Job | Weight | Phase | |-------|-----|--------|-------| | 1 | kafka-provisioning | 0 | pre-install/pre-upgrade | | 2 | sentry-db-check | -1 | post-install/post-upgrade | | 3 | snuba-db-init | 3 | post-install/post-upgrade | | 4 | snuba-migrate | 5 | post-install/post-upgrade | | 5 | sentry-db-init | 6 | post-install/post-upgrade | | 6 | user-create | 9 | post-install/post-upgrade | ## Question Is there any reason these migrations **cannot** run in parallel? Specifically: 1. **snuba-db-init** (ClickHouse bootstrap) and **sentry-db-init** (PostgreSQL `sentry upgrade`) — these operate on completely different database systems. Can they run simultaneously? 2. **snuba-migrate** (ClickHouse migrations) and **sentry-db-init** (PostgreSQL migrations) �� same question. 3. **kafka-provisioning** (topic creation) is already a pre-install hook — can it run in parallel with the db-check job? Running these in parallel would significantly reduce deployment/upgrade time. The current sequential approach means the total wait time is the **sum** of all migration durations, whereas parallel execution would reduce it to the duration of the **longest** single migration. If there are no technical blockers, would you accept a PR to align the hook-weights so that independent migrations run concurrently? my-sentry-snuba-db-init - snuba create topic in kafka ``` k get job -n sentry NAME STATUS COMPLETIONS DURATION AGE my-sentry-db-check Complete 1/1 6s 25m my-sentry-db-init Complete 1/1 2m36s 3m4s my-sentry-kafka-provisioning Complete 1/1 11m 25m my-sentry-snuba-db-init Complete 1/1 6s 14m my-sentry-snuba-migrate Complete 1/1 10m 13m my-sentry-user-create Complete 1/1 11s 26s ``` # Job Dependency Graph — Sentry Helm Chart Hooks ## Current Sequential Execution Order Jobs run sequentially due to different `helm.sh/hook-weight` values: | Order | Job | Weight | Hook Phase | Database | Command | |-------|-----|--------|------------|----------|---------| | 1 | `kafka-provisioning` | 0 | pre-install, pre-upgrade | Kafka | `topicctl apply` | | 2 | `sentry-db-check` | -1 | post-install, post-upgrade | ClickHouse + Kafka | `nc -z` (readiness probe) | | 3 | `snuba-db-init` | 3 | post-install, post-upgrade | ClickHouse + Kafka | `snuba bootstrap --no-migrate [--kafka]` | | 4 | `snuba-migrate` | 5 | post-install, post-upgrade | ClickHouse | `snuba migrations migrate --force` | | 5 | `sentry-db-init` | 6 | post-install, post-upgrade | PostgreSQL | `sentry upgrade --noinput` | | 6 | `user-create` | 9 | post-install, post-upgrade | PostgreSQL | `sentry createuser` | | 7 | `geoip-install` | 9 | post-install, post-upgrade | — | GeoIP DB download | **Total time** = sum of all durations (~25 min in observed run) ## Dependency Graph (Mermaid) ```mermaid graph TD KP["kafka-provisioning
weight: 0 | pre-install
~11m"] DBC["sentry-db-check
weight: -1 | post-install
~6s"] SNI["snuba-db-init
weight: 3 | post-install
~6s"] SNM["snuba-migrate
weight: 5 | post-install
~10m"] SDBI["sentry-db-init
weight: 3 | post-install
~2m36s"] UC["user-create
weight: 9 | post-install
~11s"] GEO["geoip-install
weight: 9 | post-install"] KP -->|creates Kafka topics| SNI DBC -->|ClickHouse ready| SNI DBC -->|Kafka ready| SNI DBC -->|ClickHouse ready| SNM SNI -->|ClickHouse schema bootstrapped| SNM SNM -->|ClickHouse migrated| GEO SDBI -->|PostgreSQL migrated| UC ``` ## Explicit Dependencies ### Hard dependencies (must complete first): 1. **snuba-db-init → snuba-migrate**: `snuba bootstrap` must create ClickHouse schema before `snuba migrations migrate` can apply migrations 2. **sentry-db-init → user-create**: PostgreSQL must be migrated (`sentry upgrade`) before creating the admin user (`sentry createuser`) 3. **kafka-provisioning → snuba-db-init**: Kafka topics must exist before `snuba bootstrap --kafka` (when `hooks.snubaInit.kafka.enabled=true`) ### Soft dependencies (readiness checks, can be replaced by parallel probes): 4. **sentry-db-check → snuba-db-init**: ClickHouse/Kafka readiness (already checked inside `snuba bootstrap` itself) 5. **sentry-db-check → sentry-db-init**: PostgreSQL readiness (Sentry retries internally) ### Independent (no dependencies): 6. **geoip-install** — downloads MaxMind GeoIP database, no relation to other jobs 7. **sentry-db-init** and **snuba-db-init** — operate on different databases (PostgreSQL vs ClickHouse) 8. **sentry-db-init** and **snuba-migrate** — same: PostgreSQL vs ClickHouse ## Proposed Parallel Execution ### Group A: Pre-install (runs before main install) ``` kafka-provisioning (weight: 0) ``` ### Group B: Post-install — Readiness gate ``` sentry-db-check (weight: -1) ``` ### Group C: Post-install — Independent DB init (parallel) ``` snuba-db-init (weight: 3) ← ClickHouse bootstrap sentry-db-init (weight: 3) ← PostgreSQL migration ``` ### Group D: Post-install — Dependent migrations ``` snuba-migrate (weight: 5) ← depends on snuba-db-init ``` ### Group E: Post-install — Final jobs (parallel, after all migrations) ``` user-create (weight: 9) ← depends on sentry-db-init geoip-install (weight: 9) ← independent, always last ``` **Proposed total time** = max(kafka-provisioning) + db-check + max(snuba-db-init, sentry-db-init) + snuba-migrate + max(user-create, geoip-install) **≈ 11m + 6s + 2m36s + 10m + max(11s, ?) ≈ 23m42s+** vs current **~25m** > **Note**: The biggest win comes from running `snuba-migrate` (10m) and `sentry-db-init` (2m36s) in parallel instead of sequentially. `user-create` and `geoip-install` always run last (weight 9) after all migrations complete. ## Weight Changes Required | Job | Current Weight | Proposed Weight | Reason | |-----|---------------|----------------|--------| | `kafka-provisioning` | 0 | 0 | Keep as-is (pre-install phase) | | `sentry-db-check` | -1 | -1 | Keep as-is (readiness gate) | | `snuba-db-init` | 3 | **3** | Keep — must complete before snuba-migrate | | `snuba-migrate` | 5 | **5** | Keep — depends on snuba-db-init | | `sentry-db-init` | 6 | **3** | **Change** — run in parallel with snuba-db-init | | `user-create` | 9 | **9** | Keep — always after all migrations | | `geoip-install` | 9 | **9** | Keep — always last | ### Summary of changes: - `sentry-db-init`: weight 6 → **3** (parallel with snuba-db-init, different DB) - `user-create`: weight **9** (unchanged — runs after all migrations) - `geoip-install`: weight **9** (unchanged — always last) ## Risk Assessment | Risk | Severity | Mitigation | |------|----------|------------| | `sentry upgrade` writes to shared Kafka topics | Low | Sentry's upgrade is PostgreSQL-only; Kafka interactions are idempotent | | Race condition in ClickHouse bootstrap + Sentry init | None | Completely different database systems | | `user-create` before Sentry fully initialized | Low | `sentry createuser` retries internally; weight 5 ensures db-init (weight 3) completes first | | GeoIP download fails blocks other jobs | None | GeoIP is independent; failure doesn't affect Sentry startup | ## Files to Modify 1. `charts/sentry/templates/hooks/sentry-db-init.job.yaml:16` — change weight from `6` to `3`

Mokto analysis page

Théo Mathieu