✨ feat(database): maintain denormalized topic usage/cost rollup from messages#15365
Merged
Conversation
…messages
Topics carry usage/cost aggregate columns (total_input_tokens /
total_output_tokens / total_tokens / total_cost / usage / cost / model /
provider) mirroring agent_operations, but nothing populated them. Add a
canonical derived-projection rollup maintained live from the topic's messages.
- `recomputeTopicUsage(trx, userId, topicId)` (new `models/topicUsage.ts`):
sums the topic's `role='assistant'` messages (thread messages included — they
carry topic_id too) over the canonical `metadata.usage`, grouped by
(provider, model). Writes the same shape as agent_operations: scalar totals,
a flat `usage` jsonb ({ llm:{ apiCalls, processingTimeMs, tokens }, tools,
humanInteraction }), and a `cost` jsonb ({ total, currency, llm:{ byModel[] },
tools }) — NULL when no model reported cost. `model`/`provider` = the
dominant model by total tokens. Pure derived & idempotent: resets to NULL
when no measurable usage remains, so deletes/regenerations are reflected.
- Hook it into MessageModel at the shared chokepoints, inside the existing
transactions: `update()` (only when the incoming payload carries
`metadata.usage`, i.e. assistant finalize / hetero step — streaming
content-only updates don't trigger it) and `deleteMessage()` /
`deleteMessages()` (recompute affected topics). This covers all LLM-call
write paths since they funnel through MessageModel.update.
- `TopicModel.recomputeUsage(id)` wraps the canonical fn in a transaction for
external callers (e.g. the historical backfill).
Tool/human-interaction sub-totals are left as a zero skeleton (not
reconstructable from assistant messages); the historical backfill will reuse
the same aggregation via raw SQL without bumping updated_at.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19a4bff43c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #15365 +/- ##
==========================================
+ Coverage 71.15% 71.17% +0.01%
==========================================
Files 3200 3201 +1
Lines 319698 319853 +155
Branches 29153 34022 +4869
==========================================
+ Hits 227485 227649 +164
+ Misses 92041 92032 -9
Partials 172 172
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Add tests for the denormalized topic usage rollup: core `recomputeTopicUsage` aggregation (per-model grouping, dominant model, cost-null path, userId scoping, NULL reset), the `TopicModel.recomputeUsage` wrapper, and the `MessageModel` update/delete hooks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
239a596 to
ebb1a24
Compare
Merged
arvinxx
added a commit
that referenced
this pull request
Jun 4, 2026
# 🚀 LobeHub Release (20260604) **Release Date:** June 4, 2026 **Since v2.2.1:** 88 merged PRs · 11 contributors > This week brings Execution Devices out of the lab — run agents and Claude Code on any configured local or remote machine — alongside Claude Opus 4.8, token-usage analytics, and Page sharing. --- ## ✨ Highlights - **Execution Devices** — Pick where an agent runs. Desktop and CLI devices auto-register with a stable machine ID, route through the gateway by channel, and surface a device switcher in the chat input. Run remote Claude Code on a configured device, with a recent-directory picker you can drag to reorder. (#15300, #15315, #15322, #15343, #15351, #15371) - **Claude Opus 4.8** — Day-one support for Anthropic's latest model. (#15314) - **Token-usage analytics** — A new token-usage mode on the activity heatmap, backed by a denormalized topic usage/cost rollup so totals stay accurate without recomputing from messages. (#15365, #15417, #15425) - **Page sharing** — Share a Page through a dedicated document share flow, plus new Workspace and Agent share tables. (#15309, #15439) - **Self-iteration agents** — Agent Signal's execAgent migration lands a server-runtime bridge, async memory writer, and a registered self-iteration tool package, with a CLI trigger command for testing. (#15360, #15364, #15392) - **Knowledge search** — BM25 search now extends to file-backed documents, and the portal ships an editable CodeMirror viewer for local files with document highlighting. (#15247, #15298) --- ## 🏗️ Core Agent & Architecture ### Agent Signal & Runtime - **execAgent migration** — Server-runtime bridge, completion projection, async memory writer, and removal of the legacy `executeSelfIteration` path. (#15392) - Registered the self-iteration builtin tool package and restored the three mode-specific self-iteration agent slugs. (#15202, #15364) - Added a CLI trigger command with a golden-snapshot fixture for Agent Signal. (#15360) - **Skill priority** — Agent Builder now emits a skill-priority instruction with matching server runtime. (#15409) - Retry empty LLM completions instead of silently finishing the turn. (#15355) - Classify topic/agent/session foreign-key violations as `ConversationParentMissing` for clearer recovery. (#15408) - Persist canonical nested usage/performance on assistant messages, and re-link orphan tool messages at the raw bucket write boundary. (#15359, #15438) - Guard `createAgent` against LLM double-encoded array fields. (#15381) --- ## 🖥️ Execution Devices & Gateway - Auto-register desktop and CLI devices with a stable machine ID, and add the `@lobechat/device-identity` package. (#15300, #15321) - New Devices settings page behind the Execution Device Switcher lab, with a device switcher shown for all agents in the chat input. (#15315, #15371) - `connectionId` + channel routing across the gateway client and device list; preset the local device on the first LLM request for the 本机 target. (#15322, #15435) - Run remote Claude Code on a configured device, with drag-to-reorder recent-directory management and client renders for device tool results. (#15343, #15351, #15437) - Preserve content and state across gateway tool calls, and prevent duplicate streaming from stale reconnects. (#15114, #15354) --- ## 🖥️ CLI & Desktop - Preserve content/state for connect local file and shell tools; render the `runCommand` tool result card. (#15441, #15442) - New `lh topic view` command; CLI now auto-registers its device on login, matching desktop. (#15340, #15377) - Resolve CLI tools from the shell `PATH`, and clarify local command session handling. (#15368, #15389) - Relocate visual-ref helpers to `@lobechat/const` to fix a renderer crash; upload `.blockmap` files to S3 for differential updates. (#15326, #15369) - Fix a market OAuth expiry that triggered the wrong re-login modal, and kill dev child processes on parent shutdown. (#15246, #15290) --- ## 🗂️ Pages, Library & Knowledge - Document share flow with business slot stubs, plus Workspace and Agent share tables. (#15309, #15439) - Export Agent profiles as Markdown, preserving an empty agent prompt on export. (#15312, #15316) - Editable CodeMirror viewer for local files with document highlighting; BM25 search extended to file-backed documents. (#15247, #15298) - Default new Agent-doc files to `.md` and preserve IME composition; refresh folder data on slug switch and dedupe breadcrumb fetches. (#15335, #15427) --- ## 💬 Chat & User Experience - Group-by-status mode for the Topic sidebar; dropped the legacy session→agentId compatibility path from Topic queries. (#15366, #15378) - Restore editor focus after the file picker closes, and close the skill dropdown before navigating to settings. (#15391, #15394) - Strip markdown tokens from fallback Topic titles; keep an open ActionBar popup when hovering another message. (#15303, #15372) - Stabilize home starter loading and stop transliterating model names in the home starter; show artifact source while streaming. (#15310, #15324, #15386) - Group the sidebar spacer with recents and agents. (#15373) --- ## 📊 Analytics, Tasks & Notifications - Token-usage mode on the activity heatmap, backed by a denormalized topic usage/cost rollup. (#15365, #15417, #15425) - Push: new `PushChannel`, receipt cron, and `pushToken` tRPC API. (#15233) - Tasks now support file and image attachments. (#15141) --- ## 🧩 Models & Providers - Support Claude Opus 4.8 and configurable model routing with starters. (#15314, #15384) - MiniMax M3: new model entry and an Anthropic video runtime. (#15380, #15403) - Add `intern-s2-preview` with `thinking_mode`, and `step-3.7-flash` support. (#15308, #15317) - Block disabling the official provider; fix default provider setup in business mode. (#15379, #15382) --- ## 🎨 UI & Modals - Migrate modals to `@lobehub/ui/base-ui` (LOBE-9711 + eval batch), including the create-custom-model and feedback/changelog modals. (#15401, #15416) - Restructure confirmModal title and content across deletion flows; polish the service-model form and migrate its Switch to base-ui. (#15426, #15440) - Wrap the BlueBubbles bridge config into a connection card; update `@lobehub/ui` to v5.15.5. (#15325, #15342) --- ## 🔒 Reliability - Replace hardcoded `session_context` values with template variables in credentials. (#15352) - Point `CHANGELOG_URL` to `/changelog`. (#15428) --- ## 👥 Contributors Huge thanks to **11 contributors** who shipped **88 merged PRs** this cycle. @hezhijie0327 · @qybaihe · @sxjeru · @arvinxx · @Innei · @tjx666 · @lijian · @sudongyuer · @cy948 · @rivertwilight · @AmAzing129 Plus @lobehubbot and renovate[bot] for maintenance. --- **Full Changelog**: v2.2.1...release/weekly-20260604
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💻 Change Type
🔀 Description of Change
The
topicstable carries usage/cost aggregate columns (total_input_tokens/total_output_tokens/total_tokens/total_cost/usagejsonb /costjsonb /model/provider) mirroringagent_operations, but nothing populated them. This adds a canonical derived-projection rollup, maintained live from the topic's messages.recomputeTopicUsage(trx, userId, topicId)(newmodels/topicUsage.ts):role='assistant'messages (thread messages included — they also carrytopic_id) over the canonicalmetadata.usage, grouped by(provider, model).agent_operations:usagejsonb — flat aggregate{ llm: { apiCalls, processingTimeMs, tokens }, tools, humanInteraction };costjsonb —{ total, currency, llm: { byModel[] }, tools }, NULL when no model reported cost;byModelkeyedprovider/modelwith merged ModelUsage.model/provider= dominant model by total tokens.Hooks in
MessageModel(inside the existing transactions, at the shared chokepoint all LLM-call write paths funnel through):update()— only when the incoming payload carriesmetadata.usage(assistant finalize / hetero step; streaming content-only updates don't trigger it).deleteMessage()/deleteMessages()— recompute affected topics.TopicModel.recomputeUsage(id)— transaction wrapper for external callers (the historical backfill will reuse the same aggregation via raw SQL, without bumpingupdated_at).Tool / human-interaction sub-totals are left as a zero skeleton (not reconstructable from assistant messages alone); the core
llm.tokensand per-modelcost.llm.byModelare exact.No schema change (columns already exist). Thread-level rollup is intentionally out of scope for this PR.
🧪 How to Test
Run an assistant turn in a topic and inspect the topic row:
total_tokens/total_costand theusage/costjsonb should populate and match the sum over the topic's assistant messages. Delete an assistant message → totals recompute downward.type-checkpasses for the changed files.📝 Additional Information
This is the going-forward write path; a follow-up will backfill historical topics (
updated_atolder than 1 day) reusing the same aggregation.