💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version#15191
Merged
arvinxx merged 2 commits intoMay 25, 2026
Merged
Conversation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rsion Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat/lobe-tracing-feedback-prep #15191 +/- ##
===================================================================
- Coverage 70.88% 70.88% -0.01%
===================================================================
Files 3145 3145
Lines 313097 313107 +10
Branches 27486 33159 +5673
===================================================================
+ Hits 221948 221951 +3
- Misses 90983 90990 +7
Partials 166 166
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
arvinxx
added a commit
that referenced
this pull request
May 25, 2026
…ersion (#15191) * 💄 style(QueueTray): use borderless variant for queued file preview Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
arvinxx
added a commit
that referenced
this pull request
May 25, 2026
…k router (#15146) * ✨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router Wire up the per-call feedback loop foundation. 1. **Pre-allocate tracingId (plan A2)** - `TracingOptions.tracingId?: string` — optional caller-supplied UUID. - `LLMGenerationTracingService.record` generates one via `randomUUID()` when the caller doesn't supply one, so the id is always known before DB insert. - `LlmGenerationTracingModel.record` accepts an optional `id` and forwards it to the insert (Drizzle still autogens when omitted). - `aiChat.outputJSON` allocates the id up-front, threads it through `tracing.tracingId`, and returns `{ data, tracingId }` so the client can wire feedback against the id even though `service.record` runs inside Next's `after()`. - `aiChatService.generateJSON` consumers (InputEditor, supervisor) unwrap the envelope. 2. **New `llmGenerationTracingRouter.recordFeedback`** - Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`. - Validates `{ tracingId (uuid), signal (positive|negative|neutral), source, score?, data? }` and forwards to `LLMGenerationTracingService.recordFeedback`. Follow-up issues already filed: - LOBE-9488 — `@lobehub/editor` AutoCompletePlugin needs `onAccept`/`onReject`/`onCancel` callbacks before the client side can capture Tab/Esc/keep-typing signals against the returned tracingId. - LOBE-9489 — session-level signal modeling (multi-suggestion typing sessions) — deferred until per-row feedback data lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * 🐛 fix(llm-generation-tracing): surface feedback write failures instead of silent ok The recordFeedback mutation used to always return `{ ok: true }` even when the underlying write was silently dropped — `LLMGenerationTracingService` swallowed both DB-init/update throws and the no-op case where the WHERE clause (id + userId) matched zero rows. Callers couldn't tell "persisted" from "lost", which would skew tracing-feedback metrics and prevent reasoned retry/error handling. Fix: - `LlmGenerationTracingModel.updateFeedback` now returns `{ updated: boolean }` (via `.returning({ id })`), so the caller knows whether the WHERE clause actually matched a row. - `LLMGenerationTracingService.recordFeedback` throws a typed `LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'` instead of swallowing — stops logging-only behaviour for DB errors and promotes the 0-rows case to an explicit signal. - `llmGenerationTracingRouter.recordFeedback` catches that error and translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and `INTERNAL_SERVER_ERROR` for DB outages — `{ ok: true }` only flows back when a row was actually patched. Tests: - Model: assert `{ updated: true/false }` for happy / cross-user / missing-id - Service: assert throws on both not_found scenarios - Router: assert TRPCError code translation for both error kinds Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(input-completion): wire Tab/Esc/typing feedback to recordFeedback - bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected} - add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback - InputEditor: map suggestionId→tracingId, fire positive on accept, negative on esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as neutral/autocomplete_ime so CJK input doesn't poison the signal Closes LOBE-9488 * ♻️ refactor(input-completion): fold recordTracingFeedback into aiChatService Single trpc mutation didn't warrant a dedicated service file; aiChatService already owns the paired `outputJSON` call that mints the tracingId, so recordTracingFeedback belongs alongside it. * 💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version (#15191) * 💄 style(QueueTray): use borderless variant for queued file preview Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * 🐛 fix(llm-generation-tracing): validate caller-supplied tracingId as UUID The `outputJSON` route echoed `tracing.tracingId` back to clients without checking the shape. Because the surrounding `tracing` record is free-form, a malformed value passed request validation, then failed DB insertion on the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`), so callers could receive a tracingId unusable for the feedback flow. Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)` so the validation happens at the request boundary; the route can then drop the redundant `typeof === 'string'` guard. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
hardy-one
pushed a commit
to hardy-one/lobehub
that referenced
this pull request
May 27, 2026
…k router (lobehub#15146) * ✨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router Wire up the per-call feedback loop foundation. 1. **Pre-allocate tracingId (plan A2)** - `TracingOptions.tracingId?: string` — optional caller-supplied UUID. - `LLMGenerationTracingService.record` generates one via `randomUUID()` when the caller doesn't supply one, so the id is always known before DB insert. - `LlmGenerationTracingModel.record` accepts an optional `id` and forwards it to the insert (Drizzle still autogens when omitted). - `aiChat.outputJSON` allocates the id up-front, threads it through `tracing.tracingId`, and returns `{ data, tracingId }` so the client can wire feedback against the id even though `service.record` runs inside Next's `after()`. - `aiChatService.generateJSON` consumers (InputEditor, supervisor) unwrap the envelope. 2. **New `llmGenerationTracingRouter.recordFeedback`** - Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`. - Validates `{ tracingId (uuid), signal (positive|negative|neutral), source, score?, data? }` and forwards to `LLMGenerationTracingService.recordFeedback`. Follow-up issues already filed: - LOBE-9488 — `@lobehub/editor` AutoCompletePlugin needs `onAccept`/`onReject`/`onCancel` callbacks before the client side can capture Tab/Esc/keep-typing signals against the returned tracingId. - LOBE-9489 — session-level signal modeling (multi-suggestion typing sessions) — deferred until per-row feedback data lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * 🐛 fix(llm-generation-tracing): surface feedback write failures instead of silent ok The recordFeedback mutation used to always return `{ ok: true }` even when the underlying write was silently dropped — `LLMGenerationTracingService` swallowed both DB-init/update throws and the no-op case where the WHERE clause (id + userId) matched zero rows. Callers couldn't tell "persisted" from "lost", which would skew tracing-feedback metrics and prevent reasoned retry/error handling. Fix: - `LlmGenerationTracingModel.updateFeedback` now returns `{ updated: boolean }` (via `.returning({ id })`), so the caller knows whether the WHERE clause actually matched a row. - `LLMGenerationTracingService.recordFeedback` throws a typed `LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'` instead of swallowing — stops logging-only behaviour for DB errors and promotes the 0-rows case to an explicit signal. - `llmGenerationTracingRouter.recordFeedback` catches that error and translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and `INTERNAL_SERVER_ERROR` for DB outages — `{ ok: true }` only flows back when a row was actually patched. Tests: - Model: assert `{ updated: true/false }` for happy / cross-user / missing-id - Service: assert throws on both not_found scenarios - Router: assert TRPCError code translation for both error kinds Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(input-completion): wire Tab/Esc/typing feedback to recordFeedback - bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected} - add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback - InputEditor: map suggestionId→tracingId, fire positive on accept, negative on esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as neutral/autocomplete_ime so CJK input doesn't poison the signal Closes LOBE-9488 * ♻️ refactor(input-completion): fold recordTracingFeedback into aiChatService Single trpc mutation didn't warrant a dedicated service file; aiChatService already owns the paired `outputJSON` call that mints the tracingId, so recordTracingFeedback belongs alongside it. * 💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version (lobehub#15191) * 💄 style(QueueTray): use borderless variant for queued file preview Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the generateObject call only set metadata.trigger and that trigger isn't in the registry. Add a TaskHandoff scenario const, version the prompt next to its definition, and pass tracing options explicitly at the call site (mirroring followUpAction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * 🐛 fix(llm-generation-tracing): validate caller-supplied tracingId as UUID The `outputJSON` route echoed `tracing.tracingId` back to clients without checking the shape. Because the surrounding `tracing` record is free-form, a malformed value passed request validation, then failed DB insertion on the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`), so callers could receive a tracingId unusable for the feedback flow. Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)` so the validation happens at the request boundary; the route can then drop the redundant `typeof === 'string'` guard. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💻 Change Type
🔗 Related Issue
Stacked on top of #15146.
🔀 Description of Change
Task topic handoff rows were tracing as
scenario=unknown/promptVersion=v0because thegenerateObjectcall intaskLifecycleonly setmetadata.trigger = 'task-handoff', and that trigger isn't inTRACING_SCENARIO_REGISTRY— so the hook fell through to theUNKNOWN_SCENARIOsentinel.Fix mirrors the
followUpActionpattern:TaskHandoff: 'task_handoff'toTRACING_SCENARIOS(the canonical scenario directory).TASK_TOPIC_HANDOFF_PROMPT_VERSION = 'v1.0'andTASK_TOPIC_HANDOFF_SCHEMA_NAME = 'task_topic_handoff'frompackages/prompts/src/chains/taskTopicHandoff.ts(per the registry comment: "version lives next to the prompt").tracing: { promptVersion, scenario, schemaName } satisfies TracingOptionsat the call site alongside the existingmetadata.trigger, and dedup the schema-name string literal.Also bundles a tiny
💄 style(QueueTray):variant='borderless'on the queued file preview Image.🧪 How to Test
Trigger a task that hands off between topics, then check
dc.lobe.li/llm-generation/task_handoff/...— row's Scenario should now readtask_handoffand prompt should readv1.0-<hash>instead ofunknown/v0.📸 Screenshots / Videos
N/A — backend metadata only (plus a borderless tweak on QueueTray preview).
📝 Additional Information
No schema migrations. The
scenariocolumn accepts any text — the new'task_handoff'value just starts appearing on new rows. Existingunknownrows for task handoff are not back-filled.🤖 Generated with Claude Code