Skip to content

💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version#15191

Merged
arvinxx merged 2 commits into
feat/lobe-tracing-feedback-prepfrom
feat/task-handoff-tracing-metadata
May 25, 2026
Merged

💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version#15191
arvinxx merged 2 commits into
feat/lobe-tracing-feedback-prepfrom
feat/task-handoff-tracing-metadata

Conversation

@arvinxx

@arvinxx arvinxx commented May 25, 2026

Copy link
Copy Markdown
Member

💻 Change Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 👷 build
  • ⚡️ perf
  • ✅ test
  • 📝 docs
  • 🔨 chore

🔗 Related Issue

Stacked on top of #15146.

🔀 Description of Change

Task topic handoff rows were tracing as scenario=unknown / promptVersion=v0 because the generateObject call in taskLifecycle only set metadata.trigger = 'task-handoff', and that trigger isn't in TRACING_SCENARIO_REGISTRY — so the hook fell through to the UNKNOWN_SCENARIO sentinel.

Fix mirrors the followUpAction pattern:

  • Added TaskHandoff: 'task_handoff' to TRACING_SCENARIOS (the canonical scenario directory).
  • Versioned the prompt next to its definition: exported TASK_TOPIC_HANDOFF_PROMPT_VERSION = 'v1.0' and TASK_TOPIC_HANDOFF_SCHEMA_NAME = 'task_topic_handoff' from packages/prompts/src/chains/taskTopicHandoff.ts (per the registry comment: "version lives next to the prompt").
  • Pass tracing: { promptVersion, scenario, schemaName } satisfies TracingOptions at the call site alongside the existing metadata.trigger, and dedup the schema-name string literal.

Also bundles a tiny 💄 style(QueueTray): variant='borderless' on the queued file preview Image.

🧪 How to Test

  • Tested locally
  • Added/updated tests
  • No tests needed

Trigger a task that hands off between topics, then check dc.lobe.li/llm-generation/task_handoff/... — row's Scenario should now read task_handoff and prompt should read v1.0-<hash> instead of unknown / v0.

📸 Screenshots / Videos

N/A — backend metadata only (plus a borderless tweak on QueueTray preview).

📝 Additional Information

No schema migrations. The scenario column accepts any text — the new 'task_handoff' value just starts appearing on new rows. Existing unknown rows for task handoff are not back-filled.

🤖 Generated with Claude Code

arvinxx and others added 2 commits May 25, 2026 12:07
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rsion

Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the
generateObject call only set metadata.trigger and that trigger isn't in the
registry. Add a TaskHandoff scenario const, version the prompt next to its
definition, and pass tracing options explicitly at the call site (mirroring
followUpAction).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label May 25, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @arvinxx, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@vercel

vercel Bot commented May 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
lobehub Ready Ready Preview, Comment May 25, 2026 4:43am

Request Review

@arvinxx arvinxx changed the title ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version 💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version May 25, 2026
@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.88%. Comparing base (1a6fffc) to head (a69d041).

Additional details and impacted files
@@                         Coverage Diff                         @@
##           feat/lobe-tracing-feedback-prep   #15191      +/-   ##
===================================================================
- Coverage                            70.88%   70.88%   -0.01%     
===================================================================
  Files                                 3145     3145              
  Lines                               313097   313107      +10     
  Branches                             27486    33159    +5673     
===================================================================
+ Hits                                221948   221951       +3     
- Misses                               90983    90990       +7     
  Partials                               166      166              
Flag Coverage Δ
app 61.71% <10.00%> (-0.01%) ⬇️
database 92.20% <ø> (ø)
packages/agent-runtime 80.48% <ø> (ø)
packages/builtin-tool-lobe-agent 19.87% <ø> (ø)
packages/context-engine 84.13% <ø> (ø)
packages/conversation-flow 91.28% <ø> (ø)
packages/file-loaders 87.89% <ø> (ø)
packages/memory-user-memory 74.99% <ø> (ø)
packages/model-bank 99.99% <ø> (ø)
packages/model-runtime 83.79% <ø> (ø)
packages/prompts 72.56% <100.00%> (+0.01%) ⬆️
packages/python-interpreter 92.90% <ø> (ø)
packages/ssrf-safe-fetch 0.00% <ø> (ø)
packages/types 35.09% <ø> (ø)
packages/utils 88.20% <ø> (ø)
packages/web-crawler 88.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Store 67.94% <ø> (ø)
Services 54.65% <ø> (ø)
Server 72.18% <10.00%> (-0.01%) ⬇️
Libs 56.42% <ø> (ø)
Utils 85.96% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@arvinxx arvinxx merged commit a38dddf into feat/lobe-tracing-feedback-prep May 25, 2026
36 checks passed
@arvinxx arvinxx deleted the feat/task-handoff-tracing-metadata branch May 25, 2026 04:43
arvinxx added a commit that referenced this pull request May 25, 2026
…ersion (#15191)

* 💄 style(QueueTray): use borderless variant for queued file preview

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version

Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the
generateObject call only set metadata.trigger and that trigger isn't in the
registry. Add a TaskHandoff scenario const, version the prompt next to its
definition, and pass tracing options explicitly at the call site (mirroring
followUpAction).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
arvinxx added a commit that referenced this pull request May 25, 2026
…k router (#15146)

* ✨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router

Wire up the per-call feedback loop foundation.

1. **Pre-allocate tracingId (plan A2)**
   - `TracingOptions.tracingId?: string` — optional caller-supplied UUID.
   - `LLMGenerationTracingService.record` generates one via `randomUUID()`
     when the caller doesn't supply one, so the id is always known
     before DB insert.
   - `LlmGenerationTracingModel.record` accepts an optional `id` and
     forwards it to the insert (Drizzle still autogens when omitted).
   - `aiChat.outputJSON` allocates the id up-front, threads it through
     `tracing.tracingId`, and returns `{ data, tracingId }` so the
     client can wire feedback against the id even though
     `service.record` runs inside Next's `after()`.
   - `aiChatService.generateJSON` consumers (InputEditor, supervisor)
     unwrap the envelope.

2. **New `llmGenerationTracingRouter.recordFeedback`**
   - Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`.
   - Validates `{ tracingId (uuid), signal (positive|negative|neutral),
     source, score?, data? }` and forwards to
     `LLMGenerationTracingService.recordFeedback`.

Follow-up issues already filed:
- LOBE-9488 — `@lobehub/editor` AutoCompletePlugin needs
  `onAccept`/`onReject`/`onCancel` callbacks before the client side can
  capture Tab/Esc/keep-typing signals against the returned tracingId.
- LOBE-9489 — session-level signal modeling (multi-suggestion typing
  sessions) — deferred until per-row feedback data lands.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* 🐛 fix(llm-generation-tracing): surface feedback write failures instead of silent ok

The recordFeedback mutation used to always return `{ ok: true }` even when
the underlying write was silently dropped — `LLMGenerationTracingService`
swallowed both DB-init/update throws and the no-op case where the WHERE
clause (id + userId) matched zero rows. Callers couldn't tell
"persisted" from "lost", which would skew tracing-feedback metrics and
prevent reasoned retry/error handling.

Fix:

- `LlmGenerationTracingModel.updateFeedback` now returns
  `{ updated: boolean }` (via `.returning({ id })`), so the caller knows
  whether the WHERE clause actually matched a row.
- `LLMGenerationTracingService.recordFeedback` throws a typed
  `LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'`
  instead of swallowing — stops logging-only behaviour for DB errors and
  promotes the 0-rows case to an explicit signal.
- `llmGenerationTracingRouter.recordFeedback` catches that error and
  translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and
  `INTERNAL_SERVER_ERROR` for DB outages — `{ ok: true }` only flows
  back when a row was actually patched.

Tests:
- Model: assert `{ updated: true/false }` for happy / cross-user / missing-id
- Service: assert throws on both not_found scenarios
- Router: assert TRPCError code translation for both error kinds

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ✨ feat(input-completion): wire Tab/Esc/typing feedback to recordFeedback

- bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected}
- add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback
- InputEditor: map suggestionId→tracingId, fire positive on accept, negative on
  esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as
  neutral/autocomplete_ime so CJK input doesn't poison the signal

Closes LOBE-9488

* ♻️ refactor(input-completion): fold recordTracingFeedback into aiChatService

Single trpc mutation didn't warrant a dedicated service file; aiChatService
already owns the paired `outputJSON` call that mints the tracingId, so
recordTracingFeedback belongs alongside it.

* 💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version (#15191)

* 💄 style(QueueTray): use borderless variant for queued file preview

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version

Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the
generateObject call only set metadata.trigger and that trigger isn't in the
registry. Add a TaskHandoff scenario const, version the prompt next to its
definition, and pass tracing options explicitly at the call site (mirroring
followUpAction).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(llm-generation-tracing): validate caller-supplied tracingId as UUID

The `outputJSON` route echoed `tracing.tracingId` back to clients without
checking the shape. Because the surrounding `tracing` record is free-form,
a malformed value passed request validation, then failed DB insertion on
the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`),
so callers could receive a tracingId unusable for the feedback flow.

Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)`
so the validation happens at the request boundary; the route can then drop
the redundant `typeof === 'string'` guard.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
hardy-one pushed a commit to hardy-one/lobehub that referenced this pull request May 27, 2026
…k router (lobehub#15146)

* ✨ feat(llm-generation-tracing): pre-allocate tracingId + recordFeedback router

Wire up the per-call feedback loop foundation.

1. **Pre-allocate tracingId (plan A2)**
   - `TracingOptions.tracingId?: string` — optional caller-supplied UUID.
   - `LLMGenerationTracingService.record` generates one via `randomUUID()`
     when the caller doesn't supply one, so the id is always known
     before DB insert.
   - `LlmGenerationTracingModel.record` accepts an optional `id` and
     forwards it to the insert (Drizzle still autogens when omitted).
   - `aiChat.outputJSON` allocates the id up-front, threads it through
     `tracing.tracingId`, and returns `{ data, tracingId }` so the
     client can wire feedback against the id even though
     `service.record` runs inside Next's `after()`.
   - `aiChatService.generateJSON` consumers (InputEditor, supervisor)
     unwrap the envelope.

2. **New `llmGenerationTracingRouter.recordFeedback`**
   - Scenario-agnostic feedback endpoint at `lambda.llmGenerationTracing`.
   - Validates `{ tracingId (uuid), signal (positive|negative|neutral),
     source, score?, data? }` and forwards to
     `LLMGenerationTracingService.recordFeedback`.

Follow-up issues already filed:
- LOBE-9488 — `@lobehub/editor` AutoCompletePlugin needs
  `onAccept`/`onReject`/`onCancel` callbacks before the client side can
  capture Tab/Esc/keep-typing signals against the returned tracingId.
- LOBE-9489 — session-level signal modeling (multi-suggestion typing
  sessions) — deferred until per-row feedback data lands.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* 🐛 fix(llm-generation-tracing): surface feedback write failures instead of silent ok

The recordFeedback mutation used to always return `{ ok: true }` even when
the underlying write was silently dropped — `LLMGenerationTracingService`
swallowed both DB-init/update throws and the no-op case where the WHERE
clause (id + userId) matched zero rows. Callers couldn't tell
"persisted" from "lost", which would skew tracing-feedback metrics and
prevent reasoned retry/error handling.

Fix:

- `LlmGenerationTracingModel.updateFeedback` now returns
  `{ updated: boolean }` (via `.returning({ id })`), so the caller knows
  whether the WHERE clause actually matched a row.
- `LLMGenerationTracingService.recordFeedback` throws a typed
  `LLMGenerationFeedbackError` with `kind: 'not_found' | 'db_failure'`
  instead of swallowing — stops logging-only behaviour for DB errors and
  promotes the 0-rows case to an explicit signal.
- `llmGenerationTracingRouter.recordFeedback` catches that error and
  translates to `TRPCError({ code: 'NOT_FOUND' })` for stale-id and
  `INTERNAL_SERVER_ERROR` for DB outages — `{ ok: true }` only flows
  back when a row was actually patched.

Tests:
- Model: assert `{ updated: true/false }` for happy / cross-user / missing-id
- Service: assert throws on both not_found scenarios
- Router: assert TRPCError code translation for both error kinds

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ✨ feat(input-completion): wire Tab/Esc/typing feedback to recordFeedback

- bump @lobehub/editor to ^4.12.0 for AutoComplete onSuggestion{Accepted,Rejected}
- add llmGenerationTracingService wrapping lambda.llmGenerationTracing.recordFeedback
- InputEditor: map suggestionId→tracingId, fire positive on accept, negative on
  esc, neutral on typing/cursor-move/blur/other; recode IME-driven escape as
  neutral/autocomplete_ime so CJK input doesn't poison the signal

Closes LOBE-9488

* ♻️ refactor(input-completion): fold recordTracingFeedback into aiChatService

Single trpc mutation didn't warrant a dedicated service file; aiChatService
already owns the paired `outputJSON` call that mints the tracingId, so
recordTracingFeedback belongs alongside it.

* 💄 style(llm-generation-tracing): tag task-handoff scenario + prompt version (lobehub#15191)

* 💄 style(QueueTray): use borderless variant for queued file preview

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ✨ feat(llm-generation-tracing): tag task-handoff scenario + prompt version

Task topic handoff was tracing as scenario=unknown / promptVersion=v0 because the
generateObject call only set metadata.trigger and that trigger isn't in the
registry. Add a TaskHandoff scenario const, version the prompt next to its
definition, and pass tracing options explicitly at the call site (mirroring
followUpAction).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(llm-generation-tracing): validate caller-supplied tracingId as UUID

The `outputJSON` route echoed `tracing.tracingId` back to clients without
checking the shape. Because the surrounding `tracing` record is free-form,
a malformed value passed request validation, then failed DB insertion on
the uuid PK and was later rejected by `recordFeedback` (`z.string().uuid()`),
so callers could receive a tracingId unusable for the feedback flow.

Tighten `StructureOutputSchema.tracing` to a `z.object({ tracingId: uuid }).catchall(unknown)`
so the validation happens at the request boundary; the route can then drop
the redundant `typeof === 'string'` guard.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant