β¨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router#15146
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
π‘ Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f07d0210f3
βΉοΈ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with π.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| signal: input.signal, | ||
| source: input.source, | ||
| }); | ||
| return { ok: true as const }; |
There was a problem hiding this comment.
Surface feedback write failures instead of always succeeding
This mutation always returns { ok: true }, but the called service path can silently drop writes (DB init/update errors are swallowed in LLMGenerationTracingService.recordFeedback, and a non-matching id + userId update is a no-op). As a result, callers will treat feedback as persisted even when nothing was stored, which can skew tracing-feedback metrics and prevent reliable retry/error handling.
Useful? React with πΒ / π.
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #15146 +/- ##
===========================================
+ Coverage 70.89% 89.70% +18.81%
===========================================
Files 3144 854 -2290
Lines 312964 102574 -210390
Branches 33148 10002 -23146
===========================================
- Hits 221881 92018 -129863
+ Misses 90917 10390 -80527
Partials 166 166
Flags with carried forward coverage won't be shown. Click here to find out more.
π New features to boost your workflow:
|
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #15146 +/- ##
===========================================
+ Coverage 70.89% 89.25% +18.36%
===========================================
Files 3144 785 -2359
Lines 312964 86881 -226083
Branches 33148 7188 -25960
===========================================
- Hits 221881 77548 -144333
+ Misses 90917 9167 -81750
Partials 166 166
Flags with carried forward coverage won't be shown. Click here to find out more.
π New features to boost your workflow:
|
Codecov Reportβ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## canary #15146 +/- ##
==========================================
- Coverage 70.92% 70.91% -0.02%
==========================================
Files 3152 3153 +1
Lines 314060 314202 +142
Branches 28568 33365 +4797
==========================================
+ Hits 222738 222801 +63
- Misses 91154 91233 +79
Partials 168 168
Flags with carried forward coverage won't be shown. Click here to find out more.
π New features to boost your workflow:
|
β¦ck router
Wire up the per-call feedback loop foundation.
1. **Pre-allocate tracingId (plan A2)**
- `TracingOptions.tracingId?: string` β optional caller-supplied UUID.
- `LLMGenerationTracingService.record` generates one via `randomUUID()`
when the caller doesn't supply one, so the id is always known
before DB insert.
- `LlmGenerationTracingModel.record` accepts an optional `id` and
forwards it to the insert (Drizzle still autogens when omitted).
- `aiChat.outputJSON` allocates the id up-front, threads it through
`tracing.tracingId`, and returns `{ data, tracingId }` so the
client can wire feedback against the id even though
`service.record` runs inside Next's `after()`.
- `aiChatService.generateJSON` consumers (InputEditor, supervisor)
unwrap the envelope.
2. **New `llmGenerationTracingRouter.recordFeedback`**
- Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`.
- Validates `{ tracingId (uuid), signal (positive|negative|neutral),
source, score?, data? }` and forwards to
`LLMGenerationTracingService.recordFeedback`.
Follow-up issues already filed:
- LOBE-9488 β `@lobehub/editor` AutoCompletePlugin needs
`onAccept`/`onReject`/`onCancel` callbacks before the client side can
capture Tab/Esc/keep-typing signals against the returned tracingId.
- LOBE-9489 β session-level signal modeling (multi-suggestion typing
sessions) β deferred until per-row feedback data lands.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
β¦d of silent ok
The recordFeedback mutation used to always return `{ ok: true }` even when
the underlying write was silently dropped β `LLMGenerationTracingService`
swallowed both DB-init/update throws and the no-op case where the WHERE
clause (id + userId) matched zero rows. Callers couldn't tell
"persisted" from "lost", which would skew tracing-feedback metrics and
prevent reasoned retry/error handling.
Fix:
- `LlmGenerationTracingModel.updateFeedback` now returns
`{ updated: boolean }` (via `.returning({ id })`), so the caller knows
whether the WHERE clause actually matched a row.
- `LLMGenerationTracingService.recordFeedback` throws a typed
`LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'`
instead of swallowing β stops logging-only behaviour for DB errors and
promotes the 0-rows case to an explicit signal.
- `llmGenerationTracingRouter.recordFeedback` catches that error and
translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and
`INTERNAL_SERVER_ERROR` for DB outages β `{ ok: true }` only flows
back when a row was actually patched.
Tests:
- Model: assert `{ updated: true/false }` for happy / cross-user / missing-id
- Service: assert throws on both not_found scenarios
- Router: assert TRPCError code translation for both error kinds
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected}
- add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback
- InputEditor: map suggestionIdβtracingId, fire positive on accept, negative on
esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as
neutral/autocomplete_ime so CJK input doesn't poison the signal
Closes LOBE-9488
β¦Service Single trpc mutation didn't warrant a dedicated service file; aiChatService already owns the paired `outputJSON` call that mints the tracingId, so recordTracingFeedback belongs alongside it.
β¦ersion (#15191) * π style(QueueTray): use borderless variant for queued file preview Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * β¨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
a38dddf to
98b432c
Compare
β¦UUID
The `outputJSON` route echoed `tracing.tracingId` back to clients without
checking the shape. Because the surrounding `tracing` record is free-form,
a malformed value passed request validation, then failed DB insertion on
the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`),
so callers could receive a tracingId unusable for the feedback flow.
Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)`
so the validation happens at the request boundary; the route can then drop
the redundant `typeof === 'string'` guard.
β¦k router (lobehub#15146) * β¨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router Wire up the per-call feedback loop foundation. 1. **Pre-allocate tracingId (plan A2)** - `TracingOptions.tracingId?: string` β optional caller-supplied UUID. - `LLMGenerationTracingService.record` generates one via `randomUUID()` when the caller doesn't supply one, so the id is always known before DB insert. - `LlmGenerationTracingModel.record` accepts an optional `id` and forwards it to the insert (Drizzle still autogens when omitted). - `aiChat.outputJSON` allocates the id up-front, threads it through `tracing.tracingId`, and returns `{ data, tracingId }` so the client can wire feedback against the id even though `service.record` runs inside Next's `after()`. - `aiChatService.generateJSON` consumers (InputEditor, supervisor) unwrap the envelope. 2. **New `llmGenerationTracingRouter.recordFeedback`** - Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`. - Validates `{ tracingId (uuid), signal (positive|negative|neutral), source, score?, data? }` and forwards to `LLMGenerationTracingService.recordFeedback`. Follow-up issues already filed: - LOBE-9488 β `@lobehub/editor` AutoCompletePlugin needs `onAccept`/`onReject`/`onCancel` callbacks before the client side can capture Tab/Esc/keep-typing signals against the returned tracingId. - LOBE-9489 β session-level signal modeling (multi-suggestion typing sessions) β deferred until per-row feedback data lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * π fix(llm-generation-tracing): surface feedback write failures instead of silent ok The recordFeedback mutation used to always return `{ ok: true }` even when the underlying write was silently dropped β `LLMGenerationTracingService` swallowed both DB-init/update throws and the no-op case where the WHERE clause (id + userId) matched zero rows. Callers couldn't tell "persisted" from "lost", which would skew tracing-feedback metrics and prevent reasoned retry/error handling. Fix: - `LlmGenerationTracingModel.updateFeedback` now returns `{ updated: boolean }` (via `.returning({ id })`), so the caller knows whether the WHERE clause actually matched a row. - `LLMGenerationTracingService.recordFeedback` throws a typed `LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'` instead of swallowing β stops logging-only behaviour for DB errors and promotes the 0-rows case to an explicit signal. - `llmGenerationTracingRouter.recordFeedback` catches that error and translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and `INTERNAL_SERVER_ERROR` for DB outages β `{ ok: true }` only flows back when a row was actually patched. Tests: - Model: assert `{ updated: true/false }` for happy / cross-user / missing-id - Service: assert throws on both not_found scenarios - Router: assert TRPCError code translation for both error kinds Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * β¨ feat(input-completion): wire Tab/Esc/typing feedback to recordFeedback - bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected} - add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback - InputEditor: map suggestionIdβtracingId, fire positive on accept, negative on esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as neutral/autocomplete_ime so CJK input doesn't poison the signal Closes LOBE-9488 * β»οΈ refactor(input-completion): fold recordTracingFeedback into aiChatService Single trpc mutation didn't warrant a dedicated service file; aiChatService already owns the paired `outputJSON` call that mints the tracingId, so recordTracingFeedback belongs alongside it. * π style(llm-generation-tracing): tag task-handoff scenario + prompt version (lobehub#15191) * π style(QueueTray): use borderless variant for queued file preview Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * β¨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π fix(llm-generation-tracing): validate caller-supplied tracingId as UUID The `outputJSON` route echoed `tracing.tracingId` back to clients without checking the shape. Because the surrounding `tracing` record is free-form, a malformed value passed request validation, then failed DB insertion on the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`), so callers could receive a tracingId unusable for the feedback flow. Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)` so the validation happens at the request boundary; the route can then drop the redundant `typeof === 'string'` guard. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
# π LobeHub Release (20260528) **Release Date:** May 28, 2026 **Since v2.2.0:** 220 merged PRs Β· 15 contributors > This cycle brings heterogeneous "platform agents" you can dispatch to local or remote devices, a rebuilt onboarding flow, document-centric chat, and a unified model-runtime error model β with new DeepSeek V4 and Gemini 3.5 Flash support along the way. --- ## β¨ Highlights - **More Hetero Agents (OpenClaw / Hermes)** β Create heterogeneous agents and dispatch them to local or remote devices through the device gateway, with an execution-target switcher in the composer and persistent CLI sessions. (#15065, #15179, #15022) - **iMessage on Desktop** β New iMessage setup and bridge on desktop, plus bot attachments across every platform. (#15228, #15227, #15029) - **Skills in the Composer** β Drag skill chips into chat, trigger installed skills from the slash menu mid-line, and surface project-level skills in the homogeneous agent runtime. (#15095, #15061, #15110) - **New Models** β DeepSeek V4 Flash/Pro and Gemini 3.5 Flash across providers, with thinking params for structured output and chat cost estimates. (#15031, #15001, #15051, #14876) - **Agent Runtime Observability** β OpenTelemetry GenAI semantic conventions plus per-call generation tracing. (#15123, #15124) --- ## π€ Agents & Heterogeneous Runtime - **Platform agent creation** β OpenClaw/Hermes creation UI, device guard, and remote dispatch backend. (#15065) - **Execution-target switcher** β Pick local vs remote execution directly in the composer; device-selection UX with actionable guidance. (#15179, #15111) - **CLI hetero dispatch** β OpenClaw/Hermes dispatch with persistent sessions and a notify protocol. (#15022) - **Gateway snapshot as source of truth** β Consume the gateway `uiMessages` snapshot at step boundaries to keep chat state consistent. (#15153, #15152) - **Client sub-agent as a normal tool call** β Simplifies the sub-agent execution path. (#15281) - **Hermes agent chain** β Implements the Hermes agent chain logic. (#15189) - **Device registry** β TRPC endpoints to register, list, update, and remove devices. (#15299) - **Desktop device routing** β Route gateway agent runs through `lh hetero exec`; restore `userId` in gateway dispatch and gate local-system by execution target. (#15132, #15232) - **Agent signals** β Anchor agent-signal receipts to messages and isolate memory-agent messages into a child thread. (#14969, #14921) --- ## π Onboarding - **Simplified first screen** β Defer topic creation to first send. (#15090) - **Market Agent Picker** β Added as a classic onboarding step, with template prefetch. (#14980, #15041) - **Welcome guidance** β Show agent welcome guidance on first run. (#15098) - **Mobile** β Adapt agent onboarding UI and restore Classic-step padding on mobile. (#15019, #15032) - **Discovery** β Streamline discovery to a single profession question. (#14987) - **Analytics** β Track onboarding step events and create-agent modal source. (#15133, #15028) --- ## π Documents, Pages & Knowledge - **Thread chat in preview** β Embed thread chat in the document preview portal. (#15216) - **Non-markdown rendering** β Render non-markdown docs as a read-only highlight. (#15272) - **Multi-select** β Multi-select delete in the document tree. (#15125) - **Page-agent streaming** β Preview `initPage` streaming arguments. (#15039) - **Per-agent topics** β Per-agent topic management page. (#15207) - **Server-side category** β Derive document category server-side and drop frontend predicates. (#15076) --- ## π§© Skills & Tools - **Drag skill chips** β Drag skills into chat input and register agent-document skills. (#15095) - **Slash menu** β Installed skills appear in the slash menu with a mid-line trigger. (#15061) - **Project skills** β Recognize project-level skills in the homogeneous agent runtime and surface them regardless of active device. (#15110, #15177) - **VFS archiving** β Archive oversized tool results to VFS instead of truncating. (#15074) - **@localfile mentions** β Drag folders into chat input as `@localFile` mentions on desktop. (#15071) --- ## π§ Model Runtime & Providers - **Error spec registry** β Unify error codes into a spec + pattern registry, split `ProviderBizError` into finer codes, classify Cloud-only codes via a tier digit, and add `DatabasePersistError`. (#15262, #15286, #15278, #15279) - **New models** β DeepSeek V4 Flash/Pro (opencode-go) and Gemini 3.5 Flash; DeepSeek V4 Pro on SiliconCloud. (#15031, #15001, #15017, #15267) - **Structured output** β Thinking params for structured output, Bedrock structured generation, and DeepSeek `generateObject` tool choice. (#15051, #15174, #15054) - **Cost** β Chat cost estimate support; preserve usage cost in custom streams. (#14876, #15218) --- ## π¬ Chat & User Experience - **Follow-up chips** β Extend follow-up chip suggestions to general chat with scene-specific model config. (#15101, #14797) - **Input drafts** β Persist unsent input drafts across tab switches and prevent repeated draft restore. (#14992, #15024) - **Command menu** β Order topic/message search by recency and promote inline type filters. (#15094, #14986) - **Zoom HUD** β Show a zoom-level HUD on Cmd +/β and Cmd 0. (#15294) - **Copy** β Unescape markdown escapes when copying user messages. (#15253) --- ## π₯οΈ Desktop - **App Nap fix** β Prevent App Nap from dropping the gateway WebSocket during display sleep. (#14994) - **File preview** β Preview `.cjs`/`.mjs`/no-extension files instead of binary fallback and expand `~` when opening local files. (#15168, #15284) - **Cross-platform settings** β Open settings via main-window navigation on Windows/Linux and restore the route after an update restart. (#15036, #14922) - **Token refresh** β Prevent frequent logout from token-refresh retries. (#14928) --- ## π Observability - **OTel GenAI** β Instrument Agent Runtime with OpenTelemetry GenAI semantic conventions. (#15123) - **Generation tracing** β Per-call `llm_generation_tracing` with a pre-allocated tracingId and recordFeedback router. (#15124, #15146) - **Error classification** β Persist `ERROR_CODE_SPECS` classification on operation errors. (#15273) --- ## ποΈ Database Migrations - **Batch migrations** β Topic usage stats, push tokens, `tasks.editor_data`, and document shares. (#15280) - **Tracing & eval tables** β Add `llm_generation_tracing` and agent eval experiment tables. (#15126) > Self-hosted operators should run the database migration (`pnpm db:migrate`, or restart with auto-migrate enabled) after upgrading. The changes are additive and backwards-compatible. --- ## π Security & Reliability - **Security:** Remove the `getPlaintextCred` tool to prevent plaintext credential exposure. (#14998) - **Security:** Prompt account selection for Google OAuth and add `prompt=consent` to the OIDC authorization URL to fix missing refresh tokens. (#15234, #15010) - **Reliability:** Preserve streamed content across a mid-stream cancel. (#15173) - **Reliability:** Bound the Redis command timeout and configure the Anthropic client timeout. (#15091, #15042) - **Reliability:** Prevent infinite recursion in the assistant chain. (#15288) --- ## π₯ Contributors Huge thanks to **15 contributors** who shipped **220 merged PRs** this cycle. @AnotiaWang Β· @sxjeru Β· @algojogacor Β· @hardy-one Β· @arvinxx Β· @Innei Β· @tjx666 Β· @lijian Β· @AmAzing129 Β· @rdmclin2 Β· @neko Β· @cy948 Β· @CanisMinor Β· @sudongyuer Β· @rivertwilight Plus @lobehubbot and renovate[bot] for maintenance. --- **Full Changelog**: v2.2.0...release/weekly-20260528
π» Change Type
π Related Issue
Builds on LOBE-9462 (per-call observability, shipped in #15124). Unblocks LOBE-9488 (editor accept/reject/cancel callbacks) and LOBE-9489 (session-level signal modeling).
π Description of Change
Wire up the per-call feedback loop foundation for
llm_generation_tracing.1. Pre-allocate tracingId (plan A2)
The tracing row's id is now allocated before DB insert, so calling routes can return it synchronously to the client even though
service.record()runs inside Next'safter().TracingOptions.tracingId?: stringβ optional caller-supplied UUIDLLMGenerationTracingService.record()generates one viarandomUUID()when the caller doesn't supply oneLlmGenerationTracingModel.record()accepts an optionalidand forwards it to the insert (Drizzle still autogens when omitted)aiChat.outputJSONallocates the id up-front, threads it throughtracing.tracingId, and returns{ data, tracingId }so the client can wire feedback against the idaiChatService.generateJSONconsumers (InputEditor,supervisor) unwrap the envelope2. New
llmGenerationTracingRouter.recordFeedbackScenario-agnostic feedback endpoint at
lambda.llmGenerationTracing. Any caller holding atracingId(returned by the originating generation route) can report a signal.```ts
input: {
tracingId: string; // uuid
signal: 'positive' | 'negative' | 'neutral';
source: string; // free-form
score?: number; // [-1, 1]
data?: Record<string, unknown>; // jsonb extras
}
```
Forwards to the existing `LLMGenerationTracingService.recordFeedback` (which already wrote the DB columns; this just exposes the endpoint).
π§ͺ How to Test
Tested locally
Added/updated tests
`src/server/services/llmGenerationTracing/{index,hook}.test.ts` β caller-supplied `tracingId` + `inputHint` override
`src/server/routers/lambda/tests/aiChat.test.ts` β envelope shape `{ data, tracingId }` and caller-supplied `tracing.tracingId` honored
`src/server/routers/lambda/tests/llmGenerationTracing.test.ts` (new) β recordFeedback procedure + zod validation
`src/store/chat/slices/message/supervisor.test.ts` β envelope unwrap
All 46 tests passing locally, type-check clean within scope.
π Additional Information
API shape change: `aiChat.outputJSON` now returns `{ data, tracingId }` instead of the raw payload. Two client consumers updated:
Follow-up work (out of scope for this PR):
Once LOBE-9488 lands, the cloud-side wiring will be: `onAccept β recordFeedback({ signal: 'positive', source: 'autocomplete_tab' })`, etc.
π€ Generated with Claude Code