Skip to content

🔨 feat(db): batch topic usage stats, push tokens, tasks editor_data & document shares migrations#15280

Merged
arvinxx merged 10 commits into
canaryfrom
db/schema-migration-batch
May 28, 2026
Merged

🔨 feat(db): batch topic usage stats, push tokens, tasks editor_data & document shares migrations#15280
arvinxx merged 10 commits into
canaryfrom
db/schema-migration-batch

Conversation

@arvinxx

@arvinxx arvinxx commented May 28, 2026

Copy link
Copy Markdown
Member

💻 Change Type

  • ✨ feat

🔗 Related Issue

Carries the schema-only portion of:

Plus a new topic usage/cost aggregation schema.

🔀 Description of Change

Dedicated migration branch bundling four independent schema changes into a single 0104 migration so the DB structure can land ahead of the feature PRs:

Also widens notification_deliveries.channel $type to include 'push' (TS-only, no DDL).

Scope note: schema definitions + the single migration only. The corresponding models, types, routers, UI and locales remain in the source feature PRs (#15186 / #15059).

The migration uses idempotent clauses (IF NOT EXISTS, DROP CONSTRAINT IF EXISTS + ADD).

🧪 How to Test

  • Tested locally

  • Added/updated tests

  • No tests needed

  • bun run db:generate reports "No schema changes" → schema and snapshot are consistent.

  • bun run type-check clean for all touched schema files.

📝 Additional Information

Branch is based on canary and targets canary. Per repo convention DB-schema PRs may instead go to main first — retarget if that workflow is preferred (would require rebasing onto main).

… document shares

Bundle four independent schema changes onto one migration branch:

- 0104 topics: add usage/cost aggregate columns (total_cost, token totals,
  cost/usage jsonb, model, provider) + model/provider indexes
- 0105 push_tokens: new table for Expo push notification tokens
- 0106 tasks: add editor_data jsonb column
- 0107 document_shares: new table for document share flow

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@arvinxx arvinxx requested review from nekomeowww and tjx666 as code owners May 28, 2026 04:29
@vercel

vercel Bot commented May 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
lobehub Ready Ready Preview, Comment May 28, 2026 11:28am

Request Review

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. feature:api API endpoint and backend issues labels May 28, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've reviewed this pull request using the Sourcery rules engine

Squash the four sequential migrations (0104-0107) into one 0104 SQL file
containing all DDL: topic usage/cost columns, push_tokens table,
tasks.editor_data column, and document_shares table.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.02%. Comparing base (bcf97d9) to head (a5120c0).
⚠️ Report is 10 commits behind head on canary.

Additional details and impacted files
@@           Coverage Diff            @@
##           canary   #15280    +/-   ##
========================================
  Coverage   71.02%   71.02%            
========================================
  Files        3172     3172            
  Lines      317388   317388            
  Branches    27919    28837   +918     
========================================
+ Hits       225437   225439     +2     
+ Misses      91782    91780     -2     
  Partials      169      169            
Flag Coverage Δ
app 61.82% <ø> (+<0.01%) ⬆️
database 92.22% <ø> (ø)
packages/agent-runtime 80.48% <ø> (ø)
packages/builtin-tool-lobe-agent 19.87% <ø> (ø)
packages/context-engine 84.13% <ø> (ø)
packages/conversation-flow 91.28% <ø> (ø)
packages/file-loaders 87.89% <ø> (ø)
packages/memory-user-memory 74.99% <ø> (ø)
packages/model-bank 99.99% <ø> (ø)
packages/model-runtime 84.55% <ø> (ø)
packages/prompts 72.67% <ø> (ø)
packages/python-interpreter 92.90% <ø> (ø)
packages/ssrf-safe-fetch 0.00% <ø> (ø)
packages/types 35.64% <ø> (ø)
packages/utils 88.47% <ø> (ø)
packages/web-crawler 88.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Store 67.70% <ø> (ø)
Services 54.64% <ø> (ø)
Server 72.30% <ø> (+<0.01%) ⬆️
Libs 56.97% <ø> (ø)
Utils 81.43% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

export const documentShares = pgTable(
'document_shares',
{
id: text('id')

@arvinxx arvinxx May 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Innei Should we use uuid for this?


This comment was translated by Claude.

Original Content

@Innei 这个要不要用 uuid ?

arvinxx and others added 2 commits May 28, 2026 12:51
Drop the userId prefix from the push_tokens unique index — one row per
device, reassigned to the new user on switch (upsert by deviceId).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ONLY-yours ONLY-yours force-pushed the db/schema-migration-batch branch from 26a57b5 to d0dfe51 Compare May 28, 2026 05:23
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels May 28, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
arvinxx and others added 2 commits May 28, 2026 17:47
…eId)

This reverts commit addf14c (device-only unique index).

The device-only index conflicts with #15186's pushToken upsert, whose
onConflict target is (userId, deviceId). Restore the composite unique
index so the upsert lands consistently with both PRs.

Also re-point 0105 snapshot prevId to the restored 0104 id and carry the
(userId, deviceId) index forward so the migration chain stays consistent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the `devices` identity anchor (surrogate uuid PK + unique(userId, deviceId))
as the stable, reinstall-proof base for binding agent runtime instances per
machine. Fold the prior 0104/0105 migrations and the new table into a single
idempotent 0104 migration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
arvinxx and others added 2 commits May 28, 2026 19:02
The batch added 8 nullable topic columns (totalCost/usage/model/...) but
topic.create.test.ts still asserted the pre-batch 19-field shape via toEqual.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Align document_shares.id with the other new batch tables (uuid defaultRandom);
table has no consumers yet so no compat impact. Regenerated 0104 + snapshot.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@arvinxx arvinxx merged commit caa7905 into canary May 28, 2026
35 checks passed
@arvinxx arvinxx deleted the db/schema-migration-batch branch May 28, 2026 11:36
@arvinxx arvinxx mentioned this pull request May 28, 2026
arvinxx added a commit that referenced this pull request May 29, 2026
# 🚀 LobeHub Release (20260528)

**Release Date:** May 28, 2026  
**Since v2.2.0:** 220 merged PRs · 15 contributors

> This cycle brings heterogeneous "platform agents" you can dispatch to
local or remote devices, a rebuilt onboarding flow, document-centric
chat, and a unified model-runtime error model — with new DeepSeek V4 and
Gemini 3.5 Flash support along the way.

---

## ✨ Highlights

- **More Hetero Agents (OpenClaw / Hermes)** — Create heterogeneous
agents and dispatch them to local or remote devices through the device
gateway, with an execution-target switcher in the composer and
persistent CLI sessions. (#15065, #15179, #15022)
- **iMessage on Desktop** — New iMessage setup and bridge on desktop,
plus bot attachments across every platform. (#15228, #15227, #15029)
- **Skills in the Composer** — Drag skill chips into chat, trigger
installed skills from the slash menu mid-line, and surface project-level
skills in the homogeneous agent runtime. (#15095, #15061, #15110)
- **New Models** — DeepSeek V4 Flash/Pro and Gemini 3.5 Flash across
providers, with thinking params for structured output and chat cost
estimates. (#15031, #15001, #15051, #14876)
- **Agent Runtime Observability** — OpenTelemetry GenAI semantic
conventions plus per-call generation tracing. (#15123, #15124)

---

## 🤖 Agents & Heterogeneous Runtime

- **Platform agent creation** — OpenClaw/Hermes creation UI, device
guard, and remote dispatch backend. (#15065)
- **Execution-target switcher** — Pick local vs remote execution
directly in the composer; device-selection UX with actionable guidance.
(#15179, #15111)
- **CLI hetero dispatch** — OpenClaw/Hermes dispatch with persistent
sessions and a notify protocol. (#15022)
- **Gateway snapshot as source of truth** — Consume the gateway
`uiMessages` snapshot at step boundaries to keep chat state consistent.
(#15153, #15152)
- **Client sub-agent as a normal tool call** — Simplifies the sub-agent
execution path. (#15281)
- **Hermes agent chain** — Implements the Hermes agent chain logic.
(#15189)
- **Device registry** — TRPC endpoints to register, list, update, and
remove devices. (#15299)
- **Desktop device routing** — Route gateway agent runs through `lh
hetero exec`; restore `userId` in gateway dispatch and gate local-system
by execution target. (#15132, #15232)
- **Agent signals** — Anchor agent-signal receipts to messages and
isolate memory-agent messages into a child thread. (#14969, #14921)

---

## 🚀 Onboarding

- **Simplified first screen** — Defer topic creation to first send.
(#15090)
- **Market Agent Picker** — Added as a classic onboarding step, with
template prefetch. (#14980, #15041)
- **Welcome guidance** — Show agent welcome guidance on first run.
(#15098)
- **Mobile** — Adapt agent onboarding UI and restore Classic-step
padding on mobile. (#15019, #15032)
- **Discovery** — Streamline discovery to a single profession question.
(#14987)
- **Analytics** — Track onboarding step events and create-agent modal
source. (#15133, #15028)

---

## 📄 Documents, Pages & Knowledge

- **Thread chat in preview** — Embed thread chat in the document preview
portal. (#15216)
- **Non-markdown rendering** — Render non-markdown docs as a read-only
highlight. (#15272)
- **Multi-select** — Multi-select delete in the document tree. (#15125)
- **Page-agent streaming** — Preview `initPage` streaming arguments.
(#15039)
- **Per-agent topics** — Per-agent topic management page. (#15207)
- **Server-side category** — Derive document category server-side and
drop frontend predicates. (#15076)

---

## 🧩 Skills & Tools

- **Drag skill chips** — Drag skills into chat input and register
agent-document skills. (#15095)
- **Slash menu** — Installed skills appear in the slash menu with a
mid-line trigger. (#15061)
- **Project skills** — Recognize project-level skills in the homogeneous
agent runtime and surface them regardless of active device. (#15110,
#15177)
- **VFS archiving** — Archive oversized tool results to VFS instead of
truncating. (#15074)
- **@localfile mentions** — Drag folders into chat input as `@localFile`
mentions on desktop. (#15071)

---

## 🧠 Model Runtime & Providers

- **Error spec registry** — Unify error codes into a spec + pattern
registry, split `ProviderBizError` into finer codes, classify Cloud-only
codes via a tier digit, and add `DatabasePersistError`. (#15262, #15286,
#15278, #15279)
- **New models** — DeepSeek V4 Flash/Pro (opencode-go) and Gemini 3.5
Flash; DeepSeek V4 Pro on SiliconCloud. (#15031, #15001, #15017, #15267)
- **Structured output** — Thinking params for structured output, Bedrock
structured generation, and DeepSeek `generateObject` tool choice.
(#15051, #15174, #15054)
- **Cost** — Chat cost estimate support; preserve usage cost in custom
streams. (#14876, #15218)

---

## 💬 Chat & User Experience

- **Follow-up chips** — Extend follow-up chip suggestions to general
chat with scene-specific model config. (#15101, #14797)
- **Input drafts** — Persist unsent input drafts across tab switches and
prevent repeated draft restore. (#14992, #15024)
- **Command menu** — Order topic/message search by recency and promote
inline type filters. (#15094, #14986)
- **Zoom HUD** — Show a zoom-level HUD on Cmd +/− and Cmd 0. (#15294)
- **Copy** — Unescape markdown escapes when copying user messages.
(#15253)

---

## 🖥️ Desktop

- **App Nap fix** — Prevent App Nap from dropping the gateway WebSocket
during display sleep. (#14994)
- **File preview** — Preview `.cjs`/`.mjs`/no-extension files instead of
binary fallback and expand `~` when opening local files. (#15168,
#15284)
- **Cross-platform settings** — Open settings via main-window navigation
on Windows/Linux and restore the route after an update restart. (#15036,
#14922)
- **Token refresh** — Prevent frequent logout from token-refresh
retries. (#14928)

---

## 📊 Observability

- **OTel GenAI** — Instrument Agent Runtime with OpenTelemetry GenAI
semantic conventions. (#15123)
- **Generation tracing** — Per-call `llm_generation_tracing` with a
pre-allocated tracingId and recordFeedback router. (#15124, #15146)
- **Error classification** — Persist `ERROR_CODE_SPECS` classification
on operation errors. (#15273)

---

## 🗃️ Database Migrations

- **Batch migrations** — Topic usage stats, push tokens,
`tasks.editor_data`, and document shares. (#15280)
- **Tracing & eval tables** — Add `llm_generation_tracing` and agent
eval experiment tables. (#15126)

> Self-hosted operators should run the database migration (`pnpm
db:migrate`, or restart with auto-migrate enabled) after upgrading. The
changes are additive and backwards-compatible.

---

## 🔒 Security & Reliability

- **Security:** Remove the `getPlaintextCred` tool to prevent plaintext
credential exposure. (#14998)
- **Security:** Prompt account selection for Google OAuth and add
`prompt=consent` to the OIDC authorization URL to fix missing refresh
tokens. (#15234, #15010)
- **Reliability:** Preserve streamed content across a mid-stream cancel.
(#15173)
- **Reliability:** Bound the Redis command timeout and configure the
Anthropic client timeout. (#15091, #15042)
- **Reliability:** Prevent infinite recursion in the assistant chain.
(#15288)

---

## 👥 Contributors

Huge thanks to **15 contributors** who shipped **220 merged PRs** this
cycle.

@AnotiaWang · @sxjeru · @algojogacor · @hardy-one · @arvinxx · @Innei ·
@tjx666 · @lijian · @AmAzing129 · @rdmclin2 · @neko · @cy948 ·
@CanisMinor · @sudongyuer · @rivertwilight

Plus @lobehubbot and renovate[bot] for maintenance.

---

**Full Changelog**: v2.2.0...release/weekly-20260528
sudongyuer added a commit that referenced this pull request May 29, 2026
Adds attachment / image upload to all four Task input surfaces (Create
Modal, Inline Entry, Task Instruction, Comment Input, Feedback Input)
plus comment edit. Attachments persist in `tasks.editor_data` /
`task_comments.editor_data` as part of the Lexical JSON state and flow
into agent runs via `execAgent.fileIds` — images as multimodal vision
content, documents through `documentService.parseFile` for text
extraction.

Server-side fileId resolution rides on the editor's
`extractMediaFromEditorState` (`@lobehub/editor/headless` 4.15.1), so
no junction tables are needed — editor_data is the single source of
truth. The /f/{fileId} proxy URL contract from the file router stays
the bridge between editor URLs and backend file lookup.

Five UI surfaces share `EditorCanvas` + `editorAttachments` for inline
attachment insertion. Comment display renders the Lexical state via
`@lobehub/editor/renderer`'s `LexicalRenderer` so image sizes round-
trip without the EditorCanvas hydration flash.

DB schema (`tasks.editor_data jsonb` column) landed separately via
#15280.

Fixes LOBE-8967

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sudongyuer added a commit that referenced this pull request May 31, 2026
* ✨ feat(task): support file & image attachments (LOBE-8967)

Adds attachment / image upload to all four Task input surfaces (Create
Modal, Inline Entry, Task Instruction, Comment Input, Feedback Input)
plus comment edit. Attachments persist in `tasks.editor_data` /
`task_comments.editor_data` as part of the Lexical JSON state and flow
into agent runs via `execAgent.fileIds` — images as multimodal vision
content, documents through `documentService.parseFile` for text
extraction.

Server-side fileId resolution rides on the editor's
`extractMediaFromEditorState` (`@lobehub/editor/headless` 4.15.1), so
no junction tables are needed — editor_data is the single source of
truth. The /f/{fileId} proxy URL contract from the file router stays
the bridge between editor URLs and backend file lookup.

Five UI surfaces share `EditorCanvas` + `editorAttachments` for inline
attachment insertion. Comment display renders the Lexical state via
`@lobehub/editor/renderer`'s `LexicalRenderer` so image sizes round-
trip without the EditorCanvas hydration flash.

DB schema (`tasks.editor_data jsonb` column) landed separately via
#15280.

Fixes LOBE-8967

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(task): correct fileId prefix + accept nodes without status

Real-world editor_data exposed two bugs in the regex-based extract:

1. `fileId` prefix was wrong — the regex looked for `fle_…` but
   `idGenerator('files')` actually produces `file_…`, so every proxy
   URL `/f/file_…` silently failed to match.
2. `@lobehub/editor`'s `extractMediaFromEditorState` requires
   `status === 'uploaded'` strictly. Editor data from the cloud upload
   path and from historical inserts omits the `status` field entirely,
   so the upstream helper silently dropped everything. Walk the tree
   ourselves and treat a missing `status` as uploaded.

Verified against real `tasks.editor_data` rows: T-6 (proxy URL form)
now extracts `file_…` correctly. T-8 (cloud R2 signed URL form) still
returns `[]` — that requires either aligning cloud's `createFile` to
return the proxy URL or adding a DB-fallback resolver, tracked as a
follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(task): resolve fileIds from pre-signed editor URLs via files.url lookup

Root cause: `fileService.getFileAccessUrl()` returns different URL forms
depending on the environment:

- prod / non-dev → `getFileProxyUrl(fileId)` = `${APP_URL}/f/{fileId}`
- dev → `getFullFileUrl(file.url)` = a pre-signed R2/S3 URL

The dev branch is intentional so remote model providers can fetch the
file directly (proxy URLs point to localhost and aren't reachable). But
the pre-signed URL doesn't contain the fileId anywhere, so our regex
extract silently returned [] for every local upload — agent never saw
any attached image.

Same shape happens for historical cloud data where the editor stored
pre-signed URLs.

Fix: make `extractFileIdsFromEditorData` async and take a `{ db, userId }`
context. Fast path stays the proxy-URL regex; URLs that don't match fall
back to a single batched `SELECT id FROM files WHERE user_id = ? AND url
IN (…)` keyed on the storage path extracted from each URL's pathname.

Verified against real local data:

  T-6 (proxy URL form)         → file_2vFD2sdzW9VO   (regex fast path)
  T-8 (pre-signed R2 URL)      → file_cAQ4naT8G8r5   (DB fallback)
  T-9 (pre-signed R2 URL × 2)  → file_…, file_…      (DB fallback)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(task): dedupe fileIds by storage key in DB fallback

Same bytes re-uploaded by the same user produce multiple `files` rows
with identical `url` + `file_hash`. The DB fallback in
`extractFileIdsFromEditorData` was returning every matching row, so a
task with one inline image but three historical upload attempts fed
the agent three copies of the same image — wasteful multimodal tokens
and noisy provider input.

Group results by `files.url` and keep the first row per key. Verified
against real local data:

  T-6  (1 img, 1 upload)              → 1 fileId
  T-8  (1 img, 1 upload)              → 1 fileId
  T-9  (1 img, 2 dup uploads)         → 1 fileId (was 2)
  T-10 (1 img, 3 dup uploads)         → 1 fileId (was 3)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 💄 style(editor): render inline file nodes as block-level cards

The default @lobehub/editor `ReactFile` decorator paints file attachments
as a tiny inline pill (icon + filename in monospace, inline-block with
0.4em padding), so a single PDF on its own line looked cramped and
hugged the surrounding text.

Override the upstream styling via the `className` prop the plugin
already exposes: full-width flex row, 10px gap, 14px padding,
`borderRadiusLG` corner, subtle hover, primary tint on `.selected`.
Aligns the editor's file attachment row with the Linear attachment
card look — and with the LexicalRenderer card the comment thread
already uses, so the same file looks consistent across surfaces.

The upstream component still only renders icon + name (no size), but
the layout change is the main UX win.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 💄 style(editor): Linear-style file card with hover download

Replace the upstream inline pill FileNode UI with a full-width card
(icon + name + size + hover-revealed download button) wired in both the
live editor and the read-only LexicalRenderer for saved comments.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* 🐛 fix(editor): use existing editor:file.* keys for file card states

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature:api API endpoint and backend issues size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants