Skip to content

refactor: move session metadata to SQLite#91322

Merged
steipete merged 5 commits into
mainfrom
refactor/session-store-sqlite
Jun 8, 2026
Merged

refactor: move session metadata to SQLite#91322
steipete merged 5 commits into
mainfrom
refactor/session-store-sqlite

Conversation

@steipete

@steipete steipete commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Moves session metadata off legacy sessions.json files and into SQLite-backed session stores.
  • Adds doctor/startup migration for legacy JSON/JSON5 session stores, then keeps runtime reads and writes on the canonical SQLite path.
  • Updates cleanup/reset/delete/status/task/export/gateway/plugin consumers, SQLite target discovery, dry-run read-only previews, and DB handle shutdown before agent directory deletion.

Release Notes

  • Session metadata storage now uses SQLite. Legacy JSON session stores are migrated by doctor/startup migration paths; transient cache state is rebuilt instead of kept as a runtime fallback.

Verification

  • .agents/skills/autoreview/scripts/autoreview --mode local clean: no accepted/actionable findings reported.
  • pnpm plugin-sdk:api:check
  • pnpm build
  • node scripts/run-vitest.mjs src/gateway/server-methods/agents-mutate.test.ts src/commands/agents.delete.test.ts src/state/openclaw-agent-db.test.ts src/commands/session-state-migration.test.ts src/config/sessions/store-read.test.ts src/commands/sessions-cleanup.test.ts src/commands/tasks.test.ts src/commands/status.summary.test.ts src/commands/status.test.ts

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation channel: msteams Channel integration: msteams gateway Gateway runtime cli CLI command changes scripts Repository scripts commands Command implementations docker Docker and sandbox tooling agents Agent runtime and tooling channel: feishu Channel integration: feishu channel: qqbot extensions: qa-lab extensions: codex size: XL maintainer Maintainer-authored PR labels Jun 8, 2026
@clawsweeper

clawsweeper Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 8, 2026, 1:33 AM ET / 05:33 UTC.

Summary
The PR moves session metadata from legacy sessions.json files into SQLite-backed agent session databases and updates migrations, command/gateway/plugin consumers, SDK exports, tests, and docs.

Reproducibility: yes. Source inspection shows the latest PR head removes the current-main existence filter and then calls loadSessionStore in a non-repair doctor scan, while the new loader opens or creates the SQLite backing DB.

Review metrics: 2 noteworthy metrics.

  • Public SDK Session-Store Exports: 1 function, 1 type added. The session-store runtime subpath is plugin-facing API, so target-discovery exports need compatibility review before merge.
  • Session Metadata Backend: 1 backend replaced. Runtime reads move from JSON session-store files to SQLite, which makes upgrade and rollback behavior more important than ordinary refactor coverage.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Fix the doctor preview path so read-only scans cannot create session-store SQLite files.
  • [P1] Add redacted terminal or log proof for a release-style sessions.json migrating through startup, doctor, openclaw sessions, and openclaw status paths.
  • Get maintainer review on the session-store SDK export and storage-backend compatibility contract.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists autoreview, build, API check, and targeted Vitest commands, but no redacted real upgrade/startup/CLI output proving the changed SQLite session-store behavior after the patch. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] A read-only openclaw doctor preview can create a new SQLite session DB for an explicit configured store that does not exist yet, which violates preview expectations and leaves state behind before the user opts into repair.
  • [P1] Existing installs depend on complete legacy sessions.json import before runtime stops reading JSON; missed startup, doctor, CLI, or plugin paths can hide or mis-associate session metadata after upgrade.
  • [P1] The PR expands plugin-sdk/session-store-runtime target-discovery exports while changing the backend, so third-party plugin consumers need compatibility-conscious review.
  • [P1] The PR body lists tests/build/API checks but still lacks redacted real release-style upgrade/startup/CLI proof for the changed storage behavior.

Maintainer options:

  1. Fix Preview Before Merge (recommended)
    Keep non-repair doctor scans on existing JSON or existing SQLite stores using a read-only loader so preview mode cannot create session DB files.
  2. Require Upgrade Proof
    Ask for redacted terminal or log proof showing a release-style sessions.json migrates and remains visible through startup, doctor, sessions, and status commands.
  3. Pause For Storage Contract Review
    Maintainers can pause this branch until session-store backend ownership and the new public SDK target-discovery export are explicitly accepted.

Next step before merge

  • [P1] Protected maintainer label, broad session-state migration, missing real proof, public SDK compatibility, and a source-backed review finding make this a human review item rather than an automated repair lane.

Security
Cleared: No concrete security or supply-chain regression was found; the diff does not add third-party dependencies, workflows, package resolution changes, or new secret handling.

Review findings

  • [P2] Keep doctor preview scans read-only — src/commands/doctor/shared/codex-route-warnings.ts:3046
Review details

Best possible solution:

Land only after the doctor preview path is read-only, a latest-release-style session-store upgrade is proven through startup/doctor/CLI read paths, and maintainers explicitly accept the SDK/storage compatibility change.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows the latest PR head removes the current-main existence filter and then calls loadSessionStore in a non-repair doctor scan, while the new loader opens or creates the SQLite backing DB.

Is this the best way to solve the issue?

No. The SQLite migration direction is plausible, but this is not the best merge shape until preview scans use an existing-store read-only path and release-style upgrade behavior is proven.

Full review comments:

  • [P2] Keep doctor preview scans read-only — src/commands/doctor/shared/codex-route-warnings.ts:3046
    The non-repair scan now calls loadSessionStore for every resolved target. Because this PR makes loadSessionStore open the SQLite backing DB, a read-only doctor preview for an explicit configured store that does not exist can create openclaw-session-store-*.sqlite before the user opts into repair. Use an existing-store/read-only loader or restore an existence guard for preview scans.
    Confidence: 0.9

Overall correctness: patch is incorrect
Overall confidence: 0.84

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against b8adc11977ab.

Label changes

Label justifications:

  • P2: This is a broad but bounded storage migration with normal-priority compatibility and session-state impact, not an emergency runtime outage.
  • merge-risk: 🚨 compatibility: Merging changes the persisted session metadata backend and public SDK subpath exports, which can affect upgrades and plugin consumers.
  • merge-risk: 🚨 session-state: Merging can hide, mis-associate, or leave behind session metadata if legacy JSON import or read-only scan paths are incomplete.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists autoreview, build, API check, and targeted Vitest commands, but no redacted real upgrade/startup/CLI output proving the changed SQLite session-store behavior after the patch. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

What I checked:

  • Protected and compatibility-sensitive PR state: The provided GitHub context shows this PR is open, has the protected maintainer label, and already carries merge-risk: 🚨 compatibility, merge-risk: 🚨 session-state, and status: 📣 needs proof labels. (7759b8699c96)
  • Current main filtered doctor session scans to existing JSON stores: Current main only scans Codex session routes for targets whose sessions.json path exists, which kept the preview path from opening a missing explicit store. (src/commands/doctor/shared/codex-route-warnings.ts:3033, b8adc11977ab)
  • PR head removes that existence filter before preview loading: At PR head, maybeRepairCodexSessionRoutes resolves all targets and the non-repair branch calls loadSessionStore directly for each target. (src/commands/doctor/shared/codex-route-warnings.ts:3046, 7759b8699c96)
  • PR head loadSessionStore now opens the SQLite backend: The PR changes loadSessionStore so reads go through loadSqliteSessionStore, not the JSON reader; this makes ordinary reads capable of opening the per-agent database path. (src/config/sessions/store-load.ts:391, 7759b8699c96)
  • SQLite session reads create/open agent DB files: loadSqliteSessionStore calls openOpenClawAgentDatabase, and that path ensures directories then constructs new DatabaseSync(pathname), so it can create the backing SQLite DB for a missing explicit store. (src/config/sessions/store-sqlite.ts:67, 7759b8699c96)
  • Read-only SQLite loader exists for the repair shape: The PR adds loadExistingSqliteSessionStoreReadOnly, which checks that the database exists before opening it read-only and can be used as the safer preview scan primitive. (src/config/sessions/store-sqlite.ts:87, 7759b8699c96)

Likely related people:

  • steipete: Current-main blame for the session-store loader, Codex doctor session-route scanner, and agent SQLite DB lifecycle points to bab18d567b0cd9811012a50e032f71a9c9beb440, and GitHub maps that commit to steipete. (role: recent area contributor; confidence: high; commits: bab18d567b0c; files: src/config/sessions/store-load.ts, src/commands/doctor/shared/codex-route-warnings.ts, src/state/openclaw-agent-db.ts)
  • obviyus: Recent history includes fix(agents): preserve cli session metadata before transcript persist, which is adjacent to the CLI/session metadata paths this migration touches. (role: recent adjacent session-metadata contributor; confidence: medium; commits: 898fd0482a40; files: src/agents, src/config/sessions)
  • gumadeiras: Recent history includes Gateway: keep outbound session metadata in owner store, which is adjacent to gateway session ownership and storage behavior touched by this PR. (role: recent gateway session-metadata contributor; confidence: medium; commits: e29d3709696b; files: src/gateway, src/config/sessions)
  • Takhoffman: Recent history includes fix: preserve reset acp session metadata, relevant because this PR migrates ACP/session metadata paths into SQLite-backed stores. (role: recent ACP/session metadata contributor; confidence: medium; commits: 22f9c19a3904; files: src/acp, src/gateway, src/config/sessions)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c119b8b089

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 3030 to +3032
const targets = resolveAllAgentSessionStoreTargetsSync(params.cfg, {
env: params.env ?? process.env,
}).filter((target) => fs.existsSync(target.storePath));
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid creating session DBs during doctor preview

When session.store is an explicit non-agent path that does not exist yet, resolveAllAgentSessionStoreTargetsSync still returns it, and the preview branch below calls loadSessionStore, which now opens/creates the SQLite backing file. That means a read-only openclaw doctor preview for Codex route warnings can leave a new openclaw-session-store-*.sqlite beside the configured store even when there are no sessions to scan; keep the old existence/read-only behavior for preview scans or use the read-only SQLite loader here.

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 8, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 8, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 704a521681

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

function resolveSessionStoreDatabaseOptions(storePath: string): OpenClawAgentDatabaseOptions {
const structuralStateDir = resolveStateDirFromSessionStorePath(storePath);
const agentId = resolveAgentIdFromSessionStorePath(storePath) ?? DEFAULT_AGENT_ID;
if (structuralStateDir) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep SQLite stores in the discovered agent directory

When a legacy store lives under an agent directory whose basename does not round-trip through normalizeAgentId (for example agents/Retired Agent/sessions/sessions.json), this structural branch resolves the SQLite DB using the normalized id, so opening/migrating it creates a new sibling agents/retired-agent/agent/openclaw-agent.sqlite. The target discovery code scans every directory under agents/ and dedupes by storePath, so after migration both the original Retired Agent/sessions/sessions.json path and the new normalized retired-agent/sessions/sessions.json path resolve to the same DB and can list or clean the same sessions twice; keep the DB path tied to the actual discovered agent directory or dedupe by resolved SQLite path.

Useful? React with 👍 / 👎.

@steipete

steipete commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Land-ready verification for head 224c24006a0796ad4b76979e036c44cf8f66c6ce.

Local proof run:

  • node scripts/run-vitest.mjs src/agents/command/attempt-execution.cli.test.ts — 1 file, 36 tests passed.
  • OPENCLAW_VITEST_MAX_WORKERS=2 OPENCLAW_TEST_PROJECTS_PARALLEL=2 OPENCLAW_VITEST_SHARD_NAME=agentic-agents-support node scripts/run-vitest.mjs --config test/vitest/vitest.agents-support.config.ts — 92 files, 1122 tests passed.
  • node scripts/run-oxlint.mjs src/agents/command/attempt-execution.cli.test.ts — passed.
  • git diff --check — passed.

Earlier branch proof retained after the SQLite session-store/cache fixes:

  • node scripts/run-vitest.mjs --config test/vitest/vitest.agents-embedded-agent.config.ts — 81 files, 1334 tests passed.
  • node scripts/run-vitest.mjs --config test/vitest/vitest.agents-support.config.ts — 92 files, 1122 tests passed.
  • node scripts/run-vitest.mjs --config test/vitest/vitest.gateway-core.config.ts — 138 files, 2163 tests passed.
  • node scripts/run-vitest.mjs --config test/vitest/vitest.gateway-client.config.ts — 27 files, 238 tests passed.
  • node scripts/run-vitest.mjs src/config/sessions.cache.test.ts src/config/sessions/sessions.test.ts src/gateway/session-utils.test.ts src/gateway/session-utils.subagent.test.ts — passed.
  • node scripts/run-oxlint.mjs $(git diff --name-only), git diff --check, and pnpm check:test-types — passed before the final test-only fixture commit.

Autoreview:

  • .agents/skills/autoreview/scripts/autoreview --mode local — clean, no accepted/actionable findings (overall: patch is correct (0.87)).
  • Prior branch autoreview after the SQLite freshness fix was also clean (overall: patch is correct (0.84)).

CI on this head is clean:

Note: Workflow Sanity / actionlint initially failed on a transient GitHub Releases 504 while downloading rhysd/actionlint; rerun succeeded. Known proof gaps: no live provider/E2E run, not needed for this SQLite session metadata/store migration.

@steipete steipete merged commit 538d36e into main Jun 8, 2026
161 of 162 checks passed
@steipete steipete deleted the refactor/session-store-sqlite branch June 8, 2026 06:17
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 9, 2026
* refactor: move session metadata to sqlite

* test: seed session stores with sqlite fixtures

* test: seed remaining session stores with sqlite fixtures

* fix: stabilize sqlite session cache freshness

* test: seed cli transcript metadata in sqlite
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling channel: feishu Channel integration: feishu channel: msteams Channel integration: msteams channel: qqbot cli CLI command changes commands Command implementations docker Docker and sandbox tooling docs Improvements or additions to documentation extensions: codex extensions: qa-lab gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. scripts Repository scripts size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant