Skip to content

✨ feat(model-runtime): add DatabasePersistError code for failed DB queries#15279

Merged
arvinxx merged 5 commits into
canaryfrom
arvinxu/feat/database-persist-error
May 28, 2026
Merged

✨ feat(model-runtime): add DatabasePersistError code for failed DB queries#15279
arvinxx merged 5 commits into
canaryfrom
arvinxu/feat/database-persist-error

Conversation

@arvinxx

@arvinxx arvinxx commented May 28, 2026

Copy link
Copy Markdown
Member

πŸ’» Change Type

  • ✨ feat
  • βœ… test

πŸ”— Related Issue

Follow-up to the error-classification work (#15262 / #15273). Surfaced while backfilling DC's operation dashboards: a large unknown bucket was Drizzle Failed query: wraps.

πŸ”€ Description of Change

Drizzle stringifies a failed query/transaction as Failed query: <sql> params: <values>. These are harness-side persistence failures (DB write/read/txn could not complete), but:

  1. They reached the dashboards classified as unknown.
  2. The embedded SQL + parameter text (model names, error_logs rows, user messages) contains substrings that match unrelated provider patterns β€” so message-based classification misfiled them as CapabilityNotSupported / InsufficientQuota / ModelNotFound.

This adds a dedicated code:

  • agentRuntime.ts β€” new DatabasePersistError.
  • specs.ts β€” E7004, under the 7xxx Stream / Runtime (harness) bucket alongside StreamChunkError / ConversationParentMissing. attribution: harness, severity: error, countAsFailure: true, httpStatus: 500.
  • patterns.ts β€” Failed query: substring pattern placed first in ERROR_PATTERNS. matchErrorPattern is first-match-wins, so claiming it up front both classifies these correctly and prevents the embedded blob from matching any pattern below it.
  • match.test.ts β€” asserts the wrap β†’ DatabasePersistError, and that a blob embedding InsufficientQuota / context length exceeded still resolves to DatabasePersistError (the false-positive guard).
  • modelRuntime.ts β€” en-US locale copy (other languages auto-translate).

πŸ§ͺ How to Test

  • Tested locally
  • Added/updated tests

```bash
bun run type-check # passes
cd packages/model-runtime && bunx vitest run src/errors/ # 49 pass
```

πŸ“ Additional Information

The agent-gateway mirror is updated in parallel (codes.ts + specs.ts E7004 + the first-position Failed query: pattern).

πŸ€– Generated with Claude Code

…eries

Drizzle stringifies a failed query/transaction as `Failed query: <sql>
params: <values>`. These are harness-side persistence failures, but they
were landing in the operation dashboards as `unknown` β€” and worse, the
embedded SQL/parameter text (model names, error_log rows, user messages)
contains substrings that trip unrelated provider patterns, so naive
message-matching misclassified them as CapabilityNotSupported /
InsufficientQuota / ModelNotFound.

- `agentRuntime.ts` β€” new `DatabasePersistError` code.
- `specs.ts` β€” E7004 under the 7xxx Stream/Runtime (harness) bucket,
  `attribution: harness`, `countAsFailure: true`, httpStatus 500.
- `patterns.ts` β€” `Failed query:` substring pattern placed **first** in the
  registry. matchErrorPattern is first-match-wins, so claiming it up front
  both classifies these correctly and stops the embedded blob from matching
  anything below.
- `match.test.ts` β€” assert the wrap classifies as DatabasePersistError and
  that a blob embedding `InsufficientQuota` / `context length exceeded` still
  resolves to DatabasePersistError.
- `modelRuntime.ts` β€” en-US `DatabasePersistError` copy (others auto-translate).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@vercel

vercel Bot commented May 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
lobehub Ready Ready Preview, Comment May 28, 2026 5:16am

Request Review

@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 28, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've reviewed this pull request using the Sourcery rules engine

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ’‘ Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0448114de8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with πŸ‘.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +57 to +58
code: AgentRuntimeErrorType.DatabasePersistError,
match: sub('Failed query:'),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Route Drizzle Error instances through the new code

When a failed Drizzle query is thrown as a normal Error in the agent-runtime path, this new pattern is never consulted: formatErrorForState still maps all Error instances to ChatErrorType.InternalServerError before publishing/persisting the state (src/server/modules/AgentRuntime/formatErrorForState.ts, lines 78-83, used by AgentRuntimeService). In that scenario the message may be Failed query: ..., but downstream clients and operation records continue to see type 500 rather than DatabasePersistError, so the new code/spec does not actually surface for the DB failures it is meant to classify.

Useful? React with πŸ‘Β / πŸ‘Ž.

@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 71.03%. Comparing base (bcf97d9) to head (1745cf2).
⚠️ Report is 1 commits behind head on canary.

Additional details and impacted files
@@            Coverage Diff            @@
##           canary   #15279     +/-   ##
=========================================
  Coverage   71.02%   71.03%             
=========================================
  Files        3172     3172             
  Lines      317388   317454     +66     
  Branches    27919    34576   +6657     
=========================================
+ Hits       225437   225503     +66     
  Misses      91782    91782             
  Partials      169      169             
Flag Coverage Ξ”
app 61.82% <ΓΈ> (ΓΈ)
database 92.22% <ΓΈ> (ΓΈ)
packages/agent-runtime 80.48% <ΓΈ> (ΓΈ)
packages/builtin-tool-lobe-agent 19.87% <ΓΈ> (ΓΈ)
packages/context-engine 84.13% <ΓΈ> (ΓΈ)
packages/conversation-flow 91.28% <ΓΈ> (ΓΈ)
packages/file-loaders 87.89% <ΓΈ> (ΓΈ)
packages/memory-user-memory 74.99% <ΓΈ> (ΓΈ)
packages/model-bank 99.99% <ΓΈ> (ΓΈ)
packages/model-runtime 84.59% <100.00%> (+0.04%) ⬆️
packages/prompts 72.67% <ΓΈ> (ΓΈ)
packages/python-interpreter 92.90% <ΓΈ> (ΓΈ)
packages/ssrf-safe-fetch 0.00% <ΓΈ> (ΓΈ)
packages/types 35.75% <100.00%> (+0.10%) ⬆️
packages/utils 88.47% <ΓΈ> (ΓΈ)
packages/web-crawler 88.08% <ΓΈ> (ΓΈ)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Ξ”
Store 67.70% <ΓΈ> (ΓΈ)
Services 54.64% <ΓΈ> (ΓΈ)
Server 72.30% <ΓΈ> (ΓΈ)
Libs 56.97% <ΓΈ> (ΓΈ)
Utils 81.43% <ΓΈ> (ΓΈ)
πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • πŸ“¦ JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…edis aborts as provider-network

`Command aborted due to connection close` is an ioredis error β€” the
Redis/Upstash agent-state store dropping a queued command, not the LLM
provider's network. It was mapped to `ProviderNetworkError`, which
misattributed our own infra failures to upstream providers.

- `agentRuntime.ts` β€” new `StateStorePersistError` (sibling of
  `DatabasePersistError`: DB layer vs state-store layer).
- `specs.ts` β€” E7005 under 7xxx Stream/Runtime (harness), countAsFailure true.
- `patterns.ts` β€” repoint `Command aborted due to connection close` to
  StateStorePersistError, and add the other Upstash state-store signatures
  (`max request size exceeded`, `database has been suspended`).
- `match.test.ts` + `modelRuntime.ts` β€” test + en-US locale.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ash patterns

Classify the harness-side crashes that were landing as `unknown`:

- `ContextEnginePipelineError` (E7006, 7xxx Stream/Runtime, harness) β€” the
  context-engine pipeline processor crash, surfaced as "Processor [<name>]
  execution failed". The context-engine throws `PipelineError` (its
  `error.name`), so a CODE_ALIASES entry resolves `PipelineError` β†’
  ContextEnginePipelineError for stored / live records.
- patterns: `Processor [` β†’ ContextEnginePipelineError, placed before the
  generic JS-crash fallbacks so a processor crash with a nested TypeError is
  attributed to the pipeline, not the bare `Cannot read properties` rule.
- patterns: bare V8 crashes (`is not a function`, `Cannot read properties of`,
  `Maximum call stack size exceeded`) β†’ AgentRuntimeError, kept LAST so
  specific provider/harness patterns win first.
- test + en-US locale.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels May 28, 2026
arvinxx and others added 2 commits May 28, 2026 12:56
…user

The broken conversation chain (`parent_id` no longer exists) is usually the
user deleting the topic / parent message mid-operation β€” an expected race,
not a harness bug. Flip attribution harness β†’ user, countAsFailure
true β†’ false (so it drops out of failure metrics), severity error β†’ warning.

numericId 7003 / category `stream` stay put (append-only); attribution and
category are orthogonal, so a stream-bucket code can be user-attributed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ntimeError

A message of literally "[object Object]" means the harness stringified an
error object instead of extracting its message β€” a harness serialization bug.
Add it to the JS-crash fallbacks (last, lowest priority) so it resolves to
AgentRuntimeError instead of staying unknown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@arvinxx arvinxx merged commit 8aa075c into canary May 28, 2026
35 checks passed
@arvinxx arvinxx deleted the arvinxu/feat/database-persist-error branch May 28, 2026 05:21
arvinxx added a commit that referenced this pull request May 29, 2026
# πŸš€ LobeHub Release (20260528)

**Release Date:** May 28, 2026  
**Since v2.2.0:** 220 merged PRs Β· 15 contributors

> This cycle brings heterogeneous "platform agents" you can dispatch to
local or remote devices, a rebuilt onboarding flow, document-centric
chat, and a unified model-runtime error model β€” with new DeepSeek V4 and
Gemini 3.5 Flash support along the way.

---

## ✨ Highlights

- **More Hetero Agents (OpenClaw / Hermes)** β€” Create heterogeneous
agents and dispatch them to local or remote devices through the device
gateway, with an execution-target switcher in the composer and
persistent CLI sessions. (#15065, #15179, #15022)
- **iMessage on Desktop** β€” New iMessage setup and bridge on desktop,
plus bot attachments across every platform. (#15228, #15227, #15029)
- **Skills in the Composer** β€” Drag skill chips into chat, trigger
installed skills from the slash menu mid-line, and surface project-level
skills in the homogeneous agent runtime. (#15095, #15061, #15110)
- **New Models** β€” DeepSeek V4 Flash/Pro and Gemini 3.5 Flash across
providers, with thinking params for structured output and chat cost
estimates. (#15031, #15001, #15051, #14876)
- **Agent Runtime Observability** β€” OpenTelemetry GenAI semantic
conventions plus per-call generation tracing. (#15123, #15124)

---

## πŸ€– Agents & Heterogeneous Runtime

- **Platform agent creation** β€” OpenClaw/Hermes creation UI, device
guard, and remote dispatch backend. (#15065)
- **Execution-target switcher** β€” Pick local vs remote execution
directly in the composer; device-selection UX with actionable guidance.
(#15179, #15111)
- **CLI hetero dispatch** β€” OpenClaw/Hermes dispatch with persistent
sessions and a notify protocol. (#15022)
- **Gateway snapshot as source of truth** β€” Consume the gateway
`uiMessages` snapshot at step boundaries to keep chat state consistent.
(#15153, #15152)
- **Client sub-agent as a normal tool call** β€” Simplifies the sub-agent
execution path. (#15281)
- **Hermes agent chain** β€” Implements the Hermes agent chain logic.
(#15189)
- **Device registry** β€” TRPC endpoints to register, list, update, and
remove devices. (#15299)
- **Desktop device routing** β€” Route gateway agent runs through `lh
hetero exec`; restore `userId` in gateway dispatch and gate local-system
by execution target. (#15132, #15232)
- **Agent signals** β€” Anchor agent-signal receipts to messages and
isolate memory-agent messages into a child thread. (#14969, #14921)

---

## πŸš€ Onboarding

- **Simplified first screen** β€” Defer topic creation to first send.
(#15090)
- **Market Agent Picker** β€” Added as a classic onboarding step, with
template prefetch. (#14980, #15041)
- **Welcome guidance** β€” Show agent welcome guidance on first run.
(#15098)
- **Mobile** β€” Adapt agent onboarding UI and restore Classic-step
padding on mobile. (#15019, #15032)
- **Discovery** β€” Streamline discovery to a single profession question.
(#14987)
- **Analytics** β€” Track onboarding step events and create-agent modal
source. (#15133, #15028)

---

## πŸ“„ Documents, Pages & Knowledge

- **Thread chat in preview** β€” Embed thread chat in the document preview
portal. (#15216)
- **Non-markdown rendering** β€” Render non-markdown docs as a read-only
highlight. (#15272)
- **Multi-select** β€” Multi-select delete in the document tree. (#15125)
- **Page-agent streaming** β€” Preview `initPage` streaming arguments.
(#15039)
- **Per-agent topics** β€” Per-agent topic management page. (#15207)
- **Server-side category** β€” Derive document category server-side and
drop frontend predicates. (#15076)

---

## 🧩 Skills & Tools

- **Drag skill chips** β€” Drag skills into chat input and register
agent-document skills. (#15095)
- **Slash menu** β€” Installed skills appear in the slash menu with a
mid-line trigger. (#15061)
- **Project skills** β€” Recognize project-level skills in the homogeneous
agent runtime and surface them regardless of active device. (#15110,
#15177)
- **VFS archiving** β€” Archive oversized tool results to VFS instead of
truncating. (#15074)
- **@localfile mentions** β€” Drag folders into chat input as `@localFile`
mentions on desktop. (#15071)

---

## 🧠 Model Runtime & Providers

- **Error spec registry** β€” Unify error codes into a spec + pattern
registry, split `ProviderBizError` into finer codes, classify Cloud-only
codes via a tier digit, and add `DatabasePersistError`. (#15262, #15286,
#15278, #15279)
- **New models** β€” DeepSeek V4 Flash/Pro (opencode-go) and Gemini 3.5
Flash; DeepSeek V4 Pro on SiliconCloud. (#15031, #15001, #15017, #15267)
- **Structured output** β€” Thinking params for structured output, Bedrock
structured generation, and DeepSeek `generateObject` tool choice.
(#15051, #15174, #15054)
- **Cost** β€” Chat cost estimate support; preserve usage cost in custom
streams. (#14876, #15218)

---

## πŸ’¬ Chat & User Experience

- **Follow-up chips** β€” Extend follow-up chip suggestions to general
chat with scene-specific model config. (#15101, #14797)
- **Input drafts** β€” Persist unsent input drafts across tab switches and
prevent repeated draft restore. (#14992, #15024)
- **Command menu** β€” Order topic/message search by recency and promote
inline type filters. (#15094, #14986)
- **Zoom HUD** β€” Show a zoom-level HUD on Cmd +/βˆ’ and Cmd 0. (#15294)
- **Copy** β€” Unescape markdown escapes when copying user messages.
(#15253)

---

## πŸ–₯️ Desktop

- **App Nap fix** β€” Prevent App Nap from dropping the gateway WebSocket
during display sleep. (#14994)
- **File preview** β€” Preview `.cjs`/`.mjs`/no-extension files instead of
binary fallback and expand `~` when opening local files. (#15168,
#15284)
- **Cross-platform settings** β€” Open settings via main-window navigation
on Windows/Linux and restore the route after an update restart. (#15036,
#14922)
- **Token refresh** β€” Prevent frequent logout from token-refresh
retries. (#14928)

---

## πŸ“Š Observability

- **OTel GenAI** β€” Instrument Agent Runtime with OpenTelemetry GenAI
semantic conventions. (#15123)
- **Generation tracing** β€” Per-call `llm_generation_tracing` with a
pre-allocated tracingId and recordFeedback router. (#15124, #15146)
- **Error classification** β€” Persist `ERROR_CODE_SPECS` classification
on operation errors. (#15273)

---

## πŸ—ƒοΈ Database Migrations

- **Batch migrations** β€” Topic usage stats, push tokens,
`tasks.editor_data`, and document shares. (#15280)
- **Tracing & eval tables** β€” Add `llm_generation_tracing` and agent
eval experiment tables. (#15126)

> Self-hosted operators should run the database migration (`pnpm
db:migrate`, or restart with auto-migrate enabled) after upgrading. The
changes are additive and backwards-compatible.

---

## πŸ”’ Security & Reliability

- **Security:** Remove the `getPlaintextCred` tool to prevent plaintext
credential exposure. (#14998)
- **Security:** Prompt account selection for Google OAuth and add
`prompt=consent` to the OIDC authorization URL to fix missing refresh
tokens. (#15234, #15010)
- **Reliability:** Preserve streamed content across a mid-stream cancel.
(#15173)
- **Reliability:** Bound the Redis command timeout and configure the
Anthropic client timeout. (#15091, #15042)
- **Reliability:** Prevent infinite recursion in the assistant chain.
(#15288)

---

## πŸ‘₯ Contributors

Huge thanks to **15 contributors** who shipped **220 merged PRs** this
cycle.

@AnotiaWang Β· @sxjeru Β· @algojogacor Β· @hardy-one Β· @arvinxx Β· @Innei Β·
@tjx666 Β· @lijian Β· @AmAzing129 Β· @rdmclin2 Β· @neko Β· @cy948 Β·
@CanisMinor Β· @sudongyuer Β· @rivertwilight

Plus @lobehubbot and renovate[bot] for maintenance.

---

**Full Changelog**: v2.2.0...release/weekly-20260528
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant