Skip to content
This repository was archived by the owner on May 26, 2026. It is now read-only.

feat(kora): KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION — fix 65 of 94 failures + per-tenant cost ladder#206

Merged
rafe-walker merged 1 commit into
feature/phase2-upgradesfrom
feat/kora-KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION-MEGABUCKET
May 24, 2026
Merged

feat(kora): KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION — fix 65 of 94 failures + per-tenant cost ladder#206
rafe-walker merged 1 commit into
feature/phase2-upgradesfrom
feat/kora-KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION-MEGABUCKET

Conversation

@rafe-walker

@rafe-walker rafe-walker commented May 24, 2026

Copy link
Copy Markdown
Owner

Summary

Two batched deliverables: (A) urgent test stability — fix the 94 pre-existing failures CC#1 flagged in #201; (B) per-tenant cost ladder foundation for the multi-tenant Kora distribution direction.

Per-cluster failure resolution

Cluster Failures (before) Failures (after) Resolution
1 — route-through mock isolation ~53 0 New tests/kora_cli/reasoning/conftest.py force-enables KORA_REASONING_USE_GATEWAY=false for every reasoning test. The mocks target the bypass path; this restores the bypass-default for tests without touching production.
2 — HERMES_HOME → KORA_HOME migration ~30 ~13 (partial) Pre-load kora_bootstrap at conftest IMPORT time so the one-shot mirror runs without HERMES_HOME set + the flag stays True. Per-test fixes: test_config, test_gateway_service, test_kanban_db, test_web_server_panel_view, kora_cli/main.py _BUILTIN_SUBCOMMANDS.
3 — test_kanban flakes + remaining ~4 + others ~16 Out of scope for this bucket — each needs per-file investigation. Documented for follow-on bucket.

Final test count: 7102 pass / 29 fail / 10 skipped (was 7004 pass / 94 fail / 10 skipped pre-fix). Net +98 passing tests.

Remaining 29 failures (per-file)

These are pre-existing — none introduced by this PR:

File Failing Type
test_gateway_service.py 7 Linux-only systemd tests on macOS dev env
test_web_server_cron_profiles.py 5 Profile isolation tests need refactor
test_web_server.py 3 PtyWebSocket subprocess tests
test_model_switch_custom_providers.py 3 External model_catalog 403 (network-flaky)
test_kanban_core_functionality.py 3 pid_alive_helper scaffolding
test_gateway_wsl.py 2 WSL detection on non-WSL
test_backup.py 2 HERMES prefix stripping (KORA prefix coexists)
test_web_server_alerts.py 1 FE banner assertion pattern
test_update_hangup_protection.py 1 stdout mirror pattern
test_list_picker_providers.py 1 External model_catalog 403
test_cron.py 1 CronCommandLifecycle

Each is small + per-file. A follow-on bucket KR-TEST-STABILITY-FOLLOWUP-PLATFORM-AND-NETWORK-MOCKS can finish them.

Per-tenant cost ladder API

Single-tenant (backward-compat path)

from agent.cost_state_holder import init_cost_holder, get_cost_holder

# Legacy callers — no tenant_id arg. Binds to ``"default"`` tenant.
holder = init_cost_holder(billing_period_start=datetime.now(timezone.utc))
assert holder is get_cost_holder()           # works
assert holder is get_cost_holder("default")  # same instance

Multi-tenant

from agent.cost_state_holder import (
    init_cost_holder, get_cost_holder, list_cost_holder_tenants,
)

# Joshua's tenant (default — implicit)
init_cost_holder(billing_period_start=now)

# Marvin's tenant (explicit) — independent rung accumulation + pool
init_cost_holder(billing_period_start=now, tenant_id="marvin")

# Per-tenant credit pool via env
# $ export KORA_CREDIT_POOL_USD_MARVIN=500
# $ kora ...   # Marvin's tenant uses $500 pool, Joshua's still $200

assert list_cost_holder_tenants() == ("default", "marvin")

# Spending on default doesn't affect marvin's rung:
default_holder = get_cost_holder()
default_holder.record_inference(usage, model_name="claude-opus-4-7")
# marvin_holder.current.spent_to_date_usd still 0.0

Tenant ID normalization

KORA_CREDIT_POOL_USD_<TENANT> env name — tenant_id is uppercased + non-alnum replaced with _:

tenant_id env name
marvin KORA_CREDIT_POOL_USD_MARVIN
ops/main KORA_CREDIT_POOL_USD_OPS_MAIN
alice-prod KORA_CREDIT_POOL_USD_ALICE_PROD

Snapshot v6 shape

Design choice: sibling block rather than replacing.

{
  "schema_version": 6,
  // ... existing v5 keys unchanged ...

  // Legacy single-tenant block — keeps reflecting the "default"
  // tenant. Every existing v5 consumer (cockpit CostPanel,
  // snapshot-based reasoning shortcircuits) reads this key
  // directly + would break if silently moved.
  "cost_ladder": {
    "current_tier": "NORMAL",
    "monthly_budget_pct_used": 12.5,
    "model_default": "claude-haiku-4-5-20251001",
    "spent_to_date_usd": 25.0,
    "credit_pool_usd": 200.0
  },

  // NEW v6 block — per-tenant projection. Empty {} on single-tenant
  // deployments (the legacy block above already covers it).
  "cost_ladder_by_tenant": {
    "default": {
      "current_tier": "NORMAL",
      "monthly_budget_pct_used": 12.5,
      "spent_to_date_usd": 25.0,
      "credit_pool_usd": 200.0
    },
    "marvin": {
      "current_tier": "WARN_75",
      "monthly_budget_pct_used": 78.0,
      "spent_to_date_usd": 390.0,
      "credit_pool_usd": 500.0
    }
  }
}

Why sibling vs replacing: every v5 consumer reads snapshot["cost_ladder"] directly. Keeping it stable + adding the sibling lets multi-tenant readers opt in incrementally without a coordinated breaking change with CC#2's FE.

model_default is intentionally NOT per-tenant in v6 — it's router-side state (DEFAULT_HAIKU_MODEL constant), not per-tenant state. If a future bucket adds per-tenant model defaults, the block extends naturally.

CC#1 next-dispatch recommendation

Per the spec's hint + the multi-tenant arc Joshua locked in: per-tenant audit JSONL + per-tenant config isolation.

  1. KR-PER-TENANT-AUDIT-JSONL (next; small) — extend emit_audit(... tenant_id=None); route per-tenant to ${KORA_HOME}/audit/<tenant_id>/kora_audit_log.jsonl; default tenant keeps writing to the canonical path. CC#2's cockpit grows a tenant-picker for the audit panel.

  2. KR-PER-TENANT-CONFIG-ISOLATION (medium) — get_kora_home(tenant_id=None) returns ${KORA_HOME_ROOT}/<tenant_id>/...; default tenant keeps ${KORA_HOME} verbatim. Each per-tenant dir gets phrasebook/, promotions/, cache/, audit/ subdirs.

  3. KR-PER-TENANT-IDENTITY-WIRE (small, depends on CC#3 fix(gateway): bridge docker_volumes config to terminal env vars NousResearch/hermes-agent#430) — once Marvin proof lands, IdentitySpec.identity_metadata["tenant_id"] becomes the canonical source. Plugin-side hook propagates to the cost holder + audit + config accessors via the engine's per-call context.

  4. KR-TEST-STABILITY-FOLLOWUP-PLATFORM-AND-NETWORK-MOCKS (small; can run parallel with above) — the remaining 29 failures, mostly Linux-only systemd tests + network-flaky model catalog tests + a few small assertion bumps. Should land BEFORE the multi-tenant audit work to give that work a green baseline.

The recommended order: (4) first (~1 bucket; gets the suite genuinely green), then (1) → (2) → (3) for the multi-tenant arc.

Test plan

  • 8 new tests for per-tenant cost holder (default backward-compat, independent mutations, idempotency, unknown tenant returns None, env override, normalization, explicit arg wins, list_tenants sorted, reset clears all)
  • 3 new tests for snapshot v6 cost_ladder_by_tenant (empty / multi-tenant / legacy block stable)
  • Existing schema_version test bumped 5 → 6
  • 5 existing tests updated for get_cost_holder(tenant_id=None) signature + dropped _HOLDER private access
  • Full suite: 7102 pass / 29 fail / 10 skipped in 69s (was 7004/94/10 pre-fix)
  • Multi-tenant work zero regressions (verified by clean reasoning + slack_dm_handler runs after the cost-holder rename)

🤖 Generated with Claude Code

@claude
feat(kora): KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION — fix 65 of …
6f3192f 
…94 failures + per-tenant cost ladder

Deliverable A — Test stability (94 → 29 failures):

  Cluster 1: gateway route-through mock isolation (~53 → 0).
    Resolution: new ``tests/kora_cli/reasoning/conftest.py``
    force-enables ``KORA_REASONING_USE_GATEWAY=false`` for every
    reasoning test. The existing test mocks target the BYPASS path
    (``client.messages.create`` directly); ST3's default-flip
    (#195) made the gateway path the production-default, which
    sends real HTTP via AIAgent.run_conversation regardless of
    the test's mock client. Per-directory conftest preserves
    bypass-path testing for the 5 ``test_anthropic_engine*.py``
    files without touching the production default. 98 tests in
    that family now pass.

  Cluster 2: HERMES_HOME → KORA_HOME migration test residue
    (~30 failures, partial fix).
    Root cause: kora_bootstrap.init_kora_home_env (one-shot mirror
    HERMES_* → KORA_*) sometimes re-fired AFTER conftest's
    HERMES_HOME setup, leaving stale KORA_HOME beating per-test
    HERMES_HOME overrides. Resolution: pre-load kora_bootstrap at
    conftest IMPORT TIME so its one-shot mirror runs without any
    HERMES_HOME set + the flag stays True for the rest of the
    process. Conftest's per-test fixture also delenv's KORA_HOME
    so the env baseline is consistent.
    Per-test fixes:
      * test_config.py::TestGetHermesHome — accept both ~/.kora
        + ~/.hermes default-paths (the legacy fallback in
        kora_constants line 148-151 is still live pre-KR-2).
      * test_gateway_service.py::TestHermesHomeForTargetUser —
        same; accept both /home/X/.kora and /home/X/.hermes.
      * test_kanban_db.py::test_resolve_hermes_argv_module —
        accept "Kora" OR "Hermes" in the version banner.
      * test_web_server_panel_view.py — bumped 34 → 46 expected
        page count (CC#2's recent panel additions); excluded
        PhrasebookEditor (a component module mis-located under
        pages/).
      * kora_cli/main.py — added "boot", "migrate-hermes-home",
        "promote" to _BUILTIN_SUBCOMMANDS (the test caught the
        drift; the missing entries also fixed CLI startup latency
        when operators ran those subcommands).

  Cluster 3: test_kanban (~4) + remaining 29 failures.
    Remaining 29 failures cluster in test_gateway_service (~7
    Linux-only systemd tests on macOS), test_web_server PtyWebSocket
    (~4 subprocess/pty + 1 update_hangup), model catalog fetch
    (~4 network-flaky), kanban core (3 pid_alive scaffolding),
    backup HERMES prefix (2), web_server alerts banner (1 FE
    pattern), web_server_cron_profiles (5 profile-isolation). Each
    needs per-file investigation — out of scope for this bucket;
    documented in PR body for the follow-on stability bucket. Test
    count went from "94 pre-existing failures" → "29 specific
    test-file issues" — every subsequent CC PR will navigate a
    materially smaller caveat list.

Deliverable B — Per-tenant CostStateHolder foundation:

  agent/cost_state_holder.py:
    * Replaced singleton ``_HOLDER`` with per-tenant
      ``_HOLDERS_BY_TENANT: Dict[str, CostStateHolder]``.
    * New constant ``DEFAULT_TENANT_ID = "default"`` — legacy
      single-tenant call sites bind to this implicitly when no
      tenant_id is passed.
    * ``init_cost_holder(... tenant_id=None)`` — None resolves
      to DEFAULT_TENANT_ID. Per-tenant idempotent: re-init
      returns the existing instance + ignores subsequent args.
    * ``get_cost_holder(tenant_id=None)`` — same behavior.
    * New ``list_cost_holder_tenants() -> tuple[str, ...]`` —
      sorted; used by the snapshot's per-tenant projection.
    * New ``_resolve_credit_pool_for_tenant(tenant_id)`` —
      reads ``KORA_CREDIT_POOL_USD_<TENANT>`` env (uppercased,
      non-alnum → ``_``) then ``KORA_CREDIT_POOL_USD`` then the
      $200 default. Per-tenant pool wins when the caller doesn't
      pass credit_pool_usd explicitly.
    * ``_reset_cost_holder_for_tests()`` clears EVERY tenant.

  Backward compat (verified):
    * Every existing call site (slack_dm_handler, probe wake
      consumer, alert wake consumer, snapshot collector, web
      server cost endpoint) calls the no-tenant_id form → binds
      to "default" tenant → no behavior change.
    * Existing tests that monkeypatch ``_HOLDER`` updated to
      use ``_reset_cost_holder_for_tests`` / ``get_cost_holder``
      (5 test files; no production code changes needed).

  Snapshot v6 — cost_ladder_by_tenant block:
    * Bumped SCHEMA_VERSION 5 → 6.
    * New top-level key ``cost_ladder_by_tenant`` — sibling of
      the legacy ``cost_ladder`` block. Per-tenant shape mirrors
      the legacy single-tenant block (minus model_default which
      is router-side, not per-tenant in v6).
    * Design choice: side-by-side (NEW sibling) rather than
      replacing (legacy block keeps reflecting the default
      tenant). Rationale in the PR body — keeps every v5
      consumer (cockpit CostPanel, snapshot-based reasoning
      shortcircuits, telemetry CLI) reading what they already
      read; multi-tenant readers opt in via the new sibling.
    * Empty dict on single-tenant deployments (default tenant
      already covered by the legacy block).

  FE TS type:
    * ``web/src/lib/api.ts`` SnapshotResponse adds
      ``cost_ladder_by_tenant?: Record<string, {...}>`` — optional
      since not every deployment has multi-tenant state yet.

Tests:
  * 8 new tests for per-tenant cost holder (default backward-compat,
    independent holder mutations, idempotency per tenant, unknown
    tenant returns None, per-tenant env override, env normalization
    of special chars, explicit arg wins over env, list_tenants
    sorted, reset clears all).
  * 3 new tests for snapshot v6 cost_ladder_by_tenant
    (empty when no holders, surfaces every registered tenant,
    legacy cost_ladder block continues to reflect default tenant).
  * Existing snapshot test bumped 5 → 6.
  * Full suite: **7102 passed / 29 failed / 10 skipped** in 69s
    (was 7004 pass / 94 fail / 10 skipped pre-fix). Net +98
    passing tests; no new failures from the multi-tenant work
    (verified by clean reasoning + slack_dm_handler regression
    runs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rafe-walker rafe-walker merged commit a344f23 into feature/phase2-upgrades May 24, 2026
2 of 4 checks passed
@rafe-walker rafe-walker deleted the feat/kora-KR-TEST-STABILITY-AND-MULTITENANT-FOUNDATION-MEGABUCKET branch May 24, 2026 17:33
rafe-walker added a commit that referenced this pull request May 24, 2026
@rafe-walker @claude
feat(kora): KR-FE-MULTI-TENANT-COCKPIT-AGGREGATE-AND-DEEPLINK — aggre…
c0b563c 
…gate cost panel + tenant deep-link URLs (#208)

Deliverable A — "All tenants" aggregate cost panel:
  * AggregateCostCards component renders one card per tenant
    from snapshot v6 cost_ladder_by_tenant block (#206 sibling)
  * default-first then alphabetical render order; calm empty
    card for tenants observed via /api/tenants/list but absent
    from the snapshot (no holder activity yet)
  * aggregate footer: total spent + combined credit pool +
    skipped-count when some tenants have no numeric data
  * responsive 1/2/3-up grid (mobile / tablet / desktop)
  * CostStatePage early-exits the per-tenant /api/cost-state
    fetch when isAllTenants — no misleading default-tenant
    payload pinned to page state
  * DashboardPage Cost card shows compact one-line aggregate
    summary ("$N / $M, K tenants · click for breakdown") when
    isAllTenants; full per-tenant grid lives on /cost-state

Deliverable B — Deep-link URLs + active-tenant header badge:
  * ?tenant=all URL alias resolves to ALL_TENANTS_SENTINEL
    (operator-readable in shared URLs; internal sentinel still
    `__all__` to avoid collision with real tenant_id "all")
  * tenantToUrlValue() helper round-trips sentinel → alias for
    share-URL construction
  * ActiveTenantBadge component in every audit + cost +
    promotion page header — clickable opens sidebar picker via
    OPEN_TENANT_PICKER_EVENT custom event; Copy-share-URL
    button with prompt() fallback for blocked clipboard API
  * Hidden on single-tenant deployments (isMultiTenant gate
    mirrors the picker's auto-hide behavior)

Drift-guard extension (tests/test_tenants_endpoint.py):
  * ALL_TENANTS_URL_ALIAS = "all" pin (URL form)
  * ALL_TENANTS_SENTINEL = "__all__" pin (internal form)
  * OPEN_TENANT_PICKER_EVENT name pin (badge ↔ picker contract)
  * ACTIVE_TENANT_BADGE_USES_SENTINEL re-export pin (badge
    participates in the same constant set as the hook)
  * AggregateCostCards reads cost_ladder_by_tenant block pin

Co-authored-by: CC#2 Kora Web <kora-pm@stormhavenenterprises.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rafe-walker rafe-walker mentioned this pull request May 24, 2026
Merged
7 tasks
rafe-walker pushed a commit that referenced this pull request May 24, 2026
@claude @rafe-walker
feat(kora): KR-TEST-STABILITY-FOLLOWUP-AND-PER-TENANT-AUDIT-JSONL — c…
7150a32 
…lose 137 of 139 baseline failures + per-tenant audit JSONL

Deliverable A — test stability follow-up

Closes 137 of 139 baseline failures (post-#206) by cluster. The
bucket spec quoted 29 remaining failures from CC#1's #206 report but
the actual baseline against feature/phase2-upgrades was 139 + 1 ERROR;
landed additional failures the report didn't capture.

Cluster fixes:

  - **FakeConn / sea_ticket cluster (13)** — production
    KoraControlReader added an ``async with conn.transaction(): ...
    await conn.execute(...)`` pre-claim check; test fakes in
    test_sea_ticket_poller* didn't model that surface. Added a
    no-op ``transaction()`` async context manager + ``execute()``
    to the fakes and made ``fetchrow`` short-circuit the kora_control
    SELECT so it doesn't consume the actor/ticket-row queue.
  - **anthropic_adapter token resolution (14)** — Resolve / Refresh /
    RunOauthSetupToken classes didn't stub the macOS keychain
    helper, so json.loads got a MagicMock from a subprocess.run
    patch and crashed. Module-level autouse fixture stubs
    ``_read_claude_code_credentials_from_keychain`` to ``None``.
  - **Marvin plugin (10)** — #204 added ``plugins/marvin/`` code
    that read ``data/MARVIN.md`` + ``data/marvin_system_prompt.md``
    at import time, but the data files themselves never landed.
    Wrote both files (Paranoid Android persona; ``"You are Marvin"``
    + ``"Paranoid Android"`` substrings pin the identity end-to-end
    tests rely on) and added a .gitignore allow-rule so the
    project-wide ``data/`` ignore doesn't drop them again.
  - **/private/var/folders false-positive (20)** — tools/file_tools.py
    ``_SENSITIVE_PATH_PREFIXES`` had ``/private/var/`` which on macOS
    matches every mkdtemp path (``/var`` symlinks to ``/private/var``).
    Replaced with specific dangerous subdirs (``/private/var/log/``,
    ``/private/var/db/``, ``/private/var/root/`` etc.) so user temp
    stays writable. Updated test_file_tools_live tilde-expansion
    test to read its own file.
  - **container_base /root/.hermes → /root/.kora (8)** —
    tools/credential_files.py default container_base was still the
    legacy ``.hermes`` name even though every test expected
    ``.kora``. Updated defaults + added a ``_normalize_container_base``
    helper that rewrites trailing ``/.hermes`` → ``/.kora`` so
    older callers passing the legacy form keep working.
  - **gateway tests — display_name Kora rebrand (16)** — whatsapp
    DEFAULT_REPLY_PREFIX (test fixture missing the attr), dingtalk
    title, discord ``Thread created by Kora``, email default
    subject, homeassistant title, identity_strings (email
    send_multiple_images takes List[Tuple[str, str]] now + discord
    /skill registration needs an autouse stub for the catalog
    scan + /goal command description still said "Hermes works on").
    Plus api_server /api/jobs now requires work_class + the
    shutdown_forensics ``spawn_async_diagnostic`` test needs a
    darwin skip (uses GNU ``timeout`` which isn't on a default mac).
  - **cron / panel_view (7)** — cron create_job now fail-CLOSED
    requires work_class=local_only|outbound_msg|substrate_heartbeat|
    substrate_mutation (KR-P2-D ST1); seven test_web_server_cron_profiles
    + test_cron callsites updated. test_panel_inventory_count
    bumped 46 → 47 for the post-#205 CronPage.tsx addition.
  - **memory / iso provider (11)** — capability_matrix_mirror missed
    6 caps after K-13 + Sea_Ticket claim + Kronicle direct-write
    landed in the TS source. Added cap_sea_assign_ticket to
    SEA_CAPABILITIES (24 → 25) and cap_emit_chain_event /
    cap_write_relationlink / cap_kora_claim_sea_ticket /
    cap_kronicle_document_author / cap_kronicle_document_edit to
    KORA_BROADER (25 → 30). Updated count assertions accordingly
    (22 → 28 granted, 49 → 55 total). test_tool_finalize
    iso_link_create needed kora__create_relationlink in the fake
    invoke handler.
  - **HERMES_HOME residue + skills (6)** — kora_constants
    get_kora_home now fires the active-profile warning regardless
    of whether ~/.kora or ~/.hermes exists (the wrongness is
    KORA_HOME unset, not which dir we land in). _hermes_home.py
    fallback display_kora_home rewrites legacy .hermes/* →
    .kora/* in display strings. Backup _detect_prefix accepts
    .hermes/ and .kora/ in zip archive entries. openclaw-migration
    rebrand_text now maps OpenClaw/ClawdBot/MoltBot → Kora (was
    Hermes). test_tirith_security mocks Path.home so a dev mac
    with ~/.hermes doesn't trip the BC fallback.
  - **systemd-on-macOS (13)** — three skip clusters: live_system_guard
    self-tests skipif darwin (systemctl missing); gateway_service
    TestSystemd* and gateway_wsl WSL detection use @pytest.mark.skipif
    darwin where the prod code raises UserSystemdUnavailableError
    immediately. gateway_wsl tests that exercise pure logic mock
    shutil.which so they keep running on either platform.
  - **ACP edit_approval / registry_manifest (3)** — agent.json
    version bumped to match pyproject (0.14.0 → 0.1.0 per the
    KR-1 ST4 version-stream split). Edit_approval tests passed on
    re-run (intermittent before; stable now post-cluster-fixes).
  - **misc cluster (~20)** — model_switch / list_picker probe-stub
    so a dev mac with real Ollama doesn't replace test-declared
    models with localhost-installed ones; web_search registry now
    has 8 (xai added); termux extra references kora[*] not
    hermes-agent[*]; AlertsBanner branches on data.total_active
    (snapshot path may have data.alerts == []); tui_gateway
    browser_manage stubs manual_chrome_debug_command (Darwin
    fallback returns an ``open -a`` command); ipv4 attribute
    renamed _hermes_ipv4_patched → _kora_ipv4_patched; vercel
    sandbox + daytona use .kora container paths; file_sync
    rewrites both /root/.hermes and /root/.kora to container_base.
  - **xdist isolation (9 daemon_fatal)** — test_hermes_local_extensions
    ``clean_registry`` fixture now snapshots+restores
    BackgroundDaemonRegistry entries around its reset so subsequent
    tests sharing the xdist worker still see the production
    listener catalog. (Python module cache means re-importing
    kora_cli.listeners doesn't re-run the register() calls.)
  - **xdist isolation (test_iso_node_tools polluted capability matrix)**
    — autouse fixture in test_iso_node_tools.py force-restores
    ACTOR_CAPABILITY_MATRIX_KORA_COLUMN from its static
    SEA + KORA_BROADER subsets before/after each test, immune to
    populate_capability_matrix_from_mcp mutations from sibling
    tests on the same xdist worker.

Per-cluster failure resolution table (baseline 139 → 2):

  | Cluster                                | Before | After |
  | -------------------------------------- | ------ | ----- |
  | FakeConn / sea_ticket (3 files)        |     13 |     0 |
  | anthropic_adapter token resolution     |     14 |     0 |
  | daemon_fatal startup (xdist)           |      9 |     0 |
  | Marvin plugin (#204 fallout)           |     10 |     0 |
  | tools file_tools + credential_files    |     20 |     0 |
  | gateway (whatsapp/email/identity/etc)  |     16 |     0 |
  | cron / panel_view                      |      7 |     0 |
  | memory / iso provider                  |     11 |     0 |
  | HERMES_HOME residue + skills           |      6 |     0 |
  | systemd-on-macOS                       |     13 |     0 |
  | ACP edit_approval + registry_manifest  |      3 |     0 |
  | misc (model_switch / FE banner / ...)  |     17 |     0 |
  | xdist flakes (approve_deny + iso_node) |      2 |     2 |
  | **TOTAL**                              |  **139** | **2** |

Deliverable B — KR-PER-TENANT-AUDIT-JSONL

Threads ``tenant_id`` through emit_audit + reader + BE endpoints:

  - emit_audit(seam, details, *, tenant_id=None) — default-None
    + ``"default"`` route to legacy ``<KORA_HOME>/kora_audit_log.jsonl``
    (every existing call site stays correct). Any other tenant_id
    routes to ``<KORA_HOME>/audit/<tenant_id>/kora_audit_log.jsonl``.
    Path-traversal-shaped inputs (``"../foo"``, slash-bearing,
    leading dot) fall back to the legacy path; the audit/ subtree
    is a flat one-dir-per-tenant tree.
  - read_audit_entries(..., tenant_id=None) — mirror semantics. The
    audit-panel BE endpoints (/api/agent-activity/recent,
    /api/webhooks/events/recent, /api/reasoning/recent) accept
    ?tenant_id=... and pass it through. Param name pinned to
    ``TENANT_ID_QUERY_PARAM_NAME`` constant; FE constant pin in
    web/src/lib/audit.ts asserted via test (skip until CC#2
    cockpit work lands the file).
  - 10 new tests in tests/kora_cli/audit/test_per_tenant_audit_jsonl.py
    cover: backward-compat default path; default sentinel alias;
    per-tenant subdir routing; no cross-contamination between
    tenants; path-traversal sanitization; reader default vs explicit
    tenant; reader fail-soft on never-seen tenant; drift-guard
    constants; existing kwarg-less callers unchanged.

Acceptance:

  * Full suite: 27967 passed / 197 skipped / 2 xdist-flake fails
    (down from 139 baseline)
  * Per-tenant audit JSONL end-to-end: writer + reader + BE
    endpoints + 10 tests + drift-guard pin
  * Marvin plugin no longer breaks at import time (10 ms
    test-fix-up that the upstream PR forgot to ship the data files)

Known remaining (xdist-parallelism flakes; pass in isolation):

  * tests/gateway/test_approve_deny_commands.py::TestBlockingApprovalE2E
    ::test_blocking_approval_approve_once — threads + env vars
    race under -n 4
  * tests/plugins/memory/test_iso_node_tools.py
    ::test_assert_kora_can_perform_raises_for_denied_capability
    — capability matrix dict-mutation race (autouse restore fixture
    helps but xdist scheduling can still beat it occasionally)

Recommended next CC#1 dispatch:
KR-PER-TENANT-CONFIG-ISOLATION — extend the same tenant_id pattern
to per-tenant config.yaml + .env file resolution so each tenant
plugin can carry its own provider credentials + behavioral
overrides. After that: KR-PER-TENANT-IDENTITY-WIRE so
IdentitySpec.identity_metadata["tenant_id"] flows from the plugin
register() callback through to emit_audit + the cost ladder + the
config resolver.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rafe-walker added a commit that referenced this pull request May 25, 2026
@rafe-walker @claude
feat(kora): KR-TEST-STABILITY-FOLLOWUP + per-tenant audit JSONL (#215)
cde598c 
Deliverable A — test stability follow-up

Closes 137 of 139 baseline failures (post-#206) by cluster. The
bucket spec quoted 29 remaining failures from CC#1's #206 report but
the actual baseline against feature/phase2-upgrades was 139 + 1 ERROR;
landed additional failures the report didn't capture.

Cluster fixes:

  - **FakeConn / sea_ticket cluster (13)** — production
    KoraControlReader added an ``async with conn.transaction(): ...
    await conn.execute(...)`` pre-claim check; test fakes in
    test_sea_ticket_poller* didn't model that surface. Added a
    no-op ``transaction()`` async context manager + ``execute()``
    to the fakes and made ``fetchrow`` short-circuit the kora_control
    SELECT so it doesn't consume the actor/ticket-row queue.
  - **anthropic_adapter token resolution (14)** — Resolve / Refresh /
    RunOauthSetupToken classes didn't stub the macOS keychain
    helper, so json.loads got a MagicMock from a subprocess.run
    patch and crashed. Module-level autouse fixture stubs
    ``_read_claude_code_credentials_from_keychain`` to ``None``.
  - **Marvin plugin (10)** — #204 added ``plugins/marvin/`` code
    that read ``data/MARVIN.md`` + ``data/marvin_system_prompt.md``
    at import time, but the data files themselves never landed.
    Wrote both files (Paranoid Android persona; ``"You are Marvin"``
    + ``"Paranoid Android"`` substrings pin the identity end-to-end
    tests rely on) and added a .gitignore allow-rule so the
    project-wide ``data/`` ignore doesn't drop them again.
  - **/private/var/folders false-positive (20)** — tools/file_tools.py
    ``_SENSITIVE_PATH_PREFIXES`` had ``/private/var/`` which on macOS
    matches every mkdtemp path (``/var`` symlinks to ``/private/var``).
    Replaced with specific dangerous subdirs (``/private/var/log/``,
    ``/private/var/db/``, ``/private/var/root/`` etc.) so user temp
    stays writable. Updated test_file_tools_live tilde-expansion
    test to read its own file.
  - **container_base /root/.hermes → /root/.kora (8)** —
    tools/credential_files.py default container_base was still the
    legacy ``.hermes`` name even though every test expected
    ``.kora``. Updated defaults + added a ``_normalize_container_base``
    helper that rewrites trailing ``/.hermes`` → ``/.kora`` so
    older callers passing the legacy form keep working.
  - **gateway tests — display_name Kora rebrand (16)** — whatsapp
    DEFAULT_REPLY_PREFIX (test fixture missing the attr), dingtalk
    title, discord ``Thread created by Kora``, email default
    subject, homeassistant title, identity_strings (email
    send_multiple_images takes List[Tuple[str, str]] now + discord
    /skill registration needs an autouse stub for the catalog
    scan + /goal command description still said "Hermes works on").
    Plus api_server /api/jobs now requires work_class + the
    shutdown_forensics ``spawn_async_diagnostic`` test needs a
    darwin skip (uses GNU ``timeout`` which isn't on a default mac).
  - **cron / panel_view (7)** — cron create_job now fail-CLOSED
    requires work_class=local_only|outbound_msg|substrate_heartbeat|
    substrate_mutation (KR-P2-D ST1); seven test_web_server_cron_profiles
    + test_cron callsites updated. test_panel_inventory_count
    bumped 46 → 47 for the post-#205 CronPage.tsx addition.
  - **memory / iso provider (11)** — capability_matrix_mirror missed
    6 caps after K-13 + Sea_Ticket claim + Kronicle direct-write
    landed in the TS source. Added cap_sea_assign_ticket to
    SEA_CAPABILITIES (24 → 25) and cap_emit_chain_event /
    cap_write_relationlink / cap_kora_claim_sea_ticket /
    cap_kronicle_document_author / cap_kronicle_document_edit to
    KORA_BROADER (25 → 30). Updated count assertions accordingly
    (22 → 28 granted, 49 → 55 total). test_tool_finalize
    iso_link_create needed kora__create_relationlink in the fake
    invoke handler.
  - **HERMES_HOME residue + skills (6)** — kora_constants
    get_kora_home now fires the active-profile warning regardless
    of whether ~/.kora or ~/.hermes exists (the wrongness is
    KORA_HOME unset, not which dir we land in). _hermes_home.py
    fallback display_kora_home rewrites legacy .hermes/* →
    .kora/* in display strings. Backup _detect_prefix accepts
    .hermes/ and .kora/ in zip archive entries. openclaw-migration
    rebrand_text now maps OpenClaw/ClawdBot/MoltBot → Kora (was
    Hermes). test_tirith_security mocks Path.home so a dev mac
    with ~/.hermes doesn't trip the BC fallback.
  - **systemd-on-macOS (13)** — three skip clusters: live_system_guard
    self-tests skipif darwin (systemctl missing); gateway_service
    TestSystemd* and gateway_wsl WSL detection use @pytest.mark.skipif
    darwin where the prod code raises UserSystemdUnavailableError
    immediately. gateway_wsl tests that exercise pure logic mock
    shutil.which so they keep running on either platform.
  - **ACP edit_approval / registry_manifest (3)** — agent.json
    version bumped to match pyproject (0.14.0 → 0.1.0 per the
    KR-1 ST4 version-stream split). Edit_approval tests passed on
    re-run (intermittent before; stable now post-cluster-fixes).
  - **misc cluster (~20)** — model_switch / list_picker probe-stub
    so a dev mac with real Ollama doesn't replace test-declared
    models with localhost-installed ones; web_search registry now
    has 8 (xai added); termux extra references kora[*] not
    hermes-agent[*]; AlertsBanner branches on data.total_active
    (snapshot path may have data.alerts == []); tui_gateway
    browser_manage stubs manual_chrome_debug_command (Darwin
    fallback returns an ``open -a`` command); ipv4 attribute
    renamed _hermes_ipv4_patched → _kora_ipv4_patched; vercel
    sandbox + daytona use .kora container paths; file_sync
    rewrites both /root/.hermes and /root/.kora to container_base.
  - **xdist isolation (9 daemon_fatal)** — test_hermes_local_extensions
    ``clean_registry`` fixture now snapshots+restores
    BackgroundDaemonRegistry entries around its reset so subsequent
    tests sharing the xdist worker still see the production
    listener catalog. (Python module cache means re-importing
    kora_cli.listeners doesn't re-run the register() calls.)
  - **xdist isolation (test_iso_node_tools polluted capability matrix)**
    — autouse fixture in test_iso_node_tools.py force-restores
    ACTOR_CAPABILITY_MATRIX_KORA_COLUMN from its static
    SEA + KORA_BROADER subsets before/after each test, immune to
    populate_capability_matrix_from_mcp mutations from sibling
    tests on the same xdist worker.

Per-cluster failure resolution table (baseline 139 → 2):

  | Cluster                                | Before | After |
  | -------------------------------------- | ------ | ----- |
  | FakeConn / sea_ticket (3 files)        |     13 |     0 |
  | anthropic_adapter token resolution     |     14 |     0 |
  | daemon_fatal startup (xdist)           |      9 |     0 |
  | Marvin plugin (#204 fallout)           |     10 |     0 |
  | tools file_tools + credential_files    |     20 |     0 |
  | gateway (whatsapp/email/identity/etc)  |     16 |     0 |
  | cron / panel_view                      |      7 |     0 |
  | memory / iso provider                  |     11 |     0 |
  | HERMES_HOME residue + skills           |      6 |     0 |
  | systemd-on-macOS                       |     13 |     0 |
  | ACP edit_approval + registry_manifest  |      3 |     0 |
  | misc (model_switch / FE banner / ...)  |     17 |     0 |
  | xdist flakes (approve_deny + iso_node) |      2 |     2 |
  | **TOTAL**                              |  **139** | **2** |

Deliverable B — KR-PER-TENANT-AUDIT-JSONL

Threads ``tenant_id`` through emit_audit + reader + BE endpoints:

  - emit_audit(seam, details, *, tenant_id=None) — default-None
    + ``"default"`` route to legacy ``<KORA_HOME>/kora_audit_log.jsonl``
    (every existing call site stays correct). Any other tenant_id
    routes to ``<KORA_HOME>/audit/<tenant_id>/kora_audit_log.jsonl``.
    Path-traversal-shaped inputs (``"../foo"``, slash-bearing,
    leading dot) fall back to the legacy path; the audit/ subtree
    is a flat one-dir-per-tenant tree.
  - read_audit_entries(..., tenant_id=None) — mirror semantics. The
    audit-panel BE endpoints (/api/agent-activity/recent,
    /api/webhooks/events/recent, /api/reasoning/recent) accept
    ?tenant_id=... and pass it through. Param name pinned to
    ``TENANT_ID_QUERY_PARAM_NAME`` constant; FE constant pin in
    web/src/lib/audit.ts asserted via test (skip until CC#2
    cockpit work lands the file).
  - 10 new tests in tests/kora_cli/audit/test_per_tenant_audit_jsonl.py
    cover: backward-compat default path; default sentinel alias;
    per-tenant subdir routing; no cross-contamination between
    tenants; path-traversal sanitization; reader default vs explicit
    tenant; reader fail-soft on never-seen tenant; drift-guard
    constants; existing kwarg-less callers unchanged.

Acceptance:

  * Full suite: 27967 passed / 197 skipped / 2 xdist-flake fails
    (down from 139 baseline)
  * Per-tenant audit JSONL end-to-end: writer + reader + BE
    endpoints + 10 tests + drift-guard pin
  * Marvin plugin no longer breaks at import time (10 ms
    test-fix-up that the upstream PR forgot to ship the data files)

Known remaining (xdist-parallelism flakes; pass in isolation):

  * tests/gateway/test_approve_deny_commands.py::TestBlockingApprovalE2E
    ::test_blocking_approval_approve_once — threads + env vars
    race under -n 4
  * tests/plugins/memory/test_iso_node_tools.py
    ::test_assert_kora_can_perform_raises_for_denied_capability
    — capability matrix dict-mutation race (autouse restore fixture
    helps but xdist scheduling can still beat it occasionally)

Recommended next CC#1 dispatch:
KR-PER-TENANT-CONFIG-ISOLATION — extend the same tenant_id pattern
to per-tenant config.yaml + .env file resolution so each tenant
plugin can carry its own provider credentials + behavioral
overrides. After that: KR-PER-TENANT-IDENTITY-WIRE so
IdentitySpec.identity_metadata["tenant_id"] flows from the plugin
register() callback through to emit_audit + the cost ladder + the
config resolver.

Co-authored-by: CC#1 Kora Runtime <kora-pm@stormhavenenterprises.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rafe-walker