Summary

Authors the two operator runbooks the R4.1 §12 readiness checklist
pinned as outstanding. Closes 2 §12 items.

• kora_docs/15_status_and_roadmap/dr_runbook.md (~162 lines) —
  post-PITR substrate_epoch bump per R4.1 §9.8 P1+P5.
• kora_docs/15_status_and_roadmap/token_rotation_runbook.md
  (~191 lines) — unified KORA_SERVICE_TOKEN (wsk_*) +
  CLAUDE_CODE_OAUTH_TOKEN rotation per R2 §5.

No code changes; pure docs.

DR runbook structure

1. When applies — substrate rewound (PITR / restore / explicit DB
   reset); Kora's kora_known_epoch ahead of substrate_epoch.
2. Symptoms — /api/operational-state PAUSED+degradation_reasons:["substrate"],
   /api/dr-state match_status: mismatch_detected, gate 3b fails in
   /api/boot-status, kora.dr.observed event in chain log.
3. Pre-procedure checks — confirm rewind, confirm operator actor
   class, confirm one Fly machine.
4. Bump procedure — read substrate_epoch + kora_known_epoch
   via read SECDEFs; loop bump_substrate_epoch('<OPERATOR_UUID>'::uuid)
   until substrate_epoch > kora_known_epoch (each call is +1).
5. Clear PAUSED — issue_kora_control with p_level=0,
   p_kind='reset' (operator-only — SECDEF rejects actor_kind='kora'
   and rejects level=0 from non-operators).
6. Verify gate 3b — /api/dr-state clean, latest boot READY with
   gate 3b pass, kora.dr.observed fires on the failed boot only
   (audit) and not on the recovered boot.
7. Failure modes — under-counted bump, re-pause loop (PAUSED-clear
   listener write failed), wrong actor class for reset, higher-level
   open command preempts reset, KoraKnownEpochMonotonicViolation
   escalation (substrate timeline moved backwards beyond
   kora_known_epoch — not a routine PITR).

Token rotation runbook structure

1. Credentials covered — table with Doppler project, cadence,
   current expiry per token.
2. When applies — proactive (T-30/14/7/1 day alerts) +
   reactive symptoms per token (wsk_* lapse: gate 4 fail / kora.boot.failed
   / STOPPED; OAuth lapse: 401/403 at inference / /api/health
worker: degraded / tickets in failed_retryable).
3. wsk_ procedure* — mint via substrate-team → doppler update
   kora-runtime-substrate → flyctl secrets import (preserves rest
   of project's secrets) → restart → verify boot gates 4/6/10 PASS
   via /api/boot-status → revoke OLD only after verified.
4. OAuth procedure — claude setup-token → doppler update
   kora-runtime-anthropic → flyctl import + restart → verify via
   inference probe (boot gates don't check OAuth; needs a real
   Anthropic call) → OAuth expires automatically (no revoke).
5. Joshua's preference — explicit-overlap rotation (mint → verify
   → revoke), per user_joshua memory.
6. T-30/14/7/1 alerts — operator-workstation cron template per
   token.
7. Failure modes — stale Doppler download, wrong scope on new
   wsk_*, trailing whitespace in OAuth, simultaneous-rotation revert
   via doppler activity + version-pinned restore.

R4.1 §12 closure claim

Two §12 readiness items close with this PR:

• "DR runbook mandates the post-restore substrate_epoch bump as a
  non-negotiable step, with explicit post-restore verification that
  gate 3b fired on Kora's next boot." → covered by dr_runbook.md.
• "The two tokens' rotations are documented as one unified runbook (a
  wsk-token lapse while OAuth is valid burns inference with zero
  durable output)." → covered by token_rotation_runbook.md.

The path kora_docs/15_status_and_roadmap/dr_runbook.md matches the
hardcoded reference in web/src/pages/DRStatePage.tsx:377 ("See
kora_docs/15_status_and_roadmap/dr_runbook.md (pending KR-P2-M)").
The pending tag remains in CC#2's surface — once this PR merges, the
panel reference becomes a live link.

No Rule-3 ASKs raised

Investigation phase verified all referenced helpers + SECDEFs exist
on main:

• agent/dr_handler.py (gate 3b) + agent/dr_writer.py (writers) +
  plugins/memory/isokron/dr_epoch.py (read/write helpers + the
  KoraKnownEpochMonotonicViolation exception class).
• packages/db/migrations/0100_kora_dr_epoch_substrate.sql (read
  SECDEFs + bump_substrate_epoch).
• packages/db/migrations/0090_kora_control_secdefs.sql
  (issue_kora_control, operator-only level=0).
• docs/deploy-fly-io.md (3-Doppler-project layout + OAuth rotation
  reference the new runbook extends).

No gaps; no STOPs.

Test plan

• Mechanical: gh pr checks passes (docs-only — no CI gates
  should regress).
• Path verification: DRStatePage.tsx link target resolves once
  this lands.
• PM review: confirm structure + R4.1 §12 closure claim.

🤖 Generated with Claude Code