Skip to content

fix(controller): replace better-sqlite3 with sql.js for desktop ABI compatibility#298

Merged
alchemistklk merged 21 commits intomainfrom
fix/sqljs-skill-db
Mar 20, 2026
Merged

fix(controller): replace better-sqlite3 with sql.js for desktop ABI compatibility#298
alchemistklk merged 21 commits intomainfrom
fix/sqljs-skill-db

Conversation

@alchemistklk
Copy link
Copy Markdown
Contributor

Summary

  • Root cause: better-sqlite3 is a native addon compiled for system Node.js ABI 127, but the packaged controller sidecar runs under Electron's Node.js ABI 136. ELECTRON_RUN_AS_NODE=1 doesn't change the ABI. The try/catch silently swallowed the failure, making SkillDb undefined and skipping all curated skill installation.
  • Fix: Replace better-sqlite3 with sql.js (SQLite compiled to WASM) — zero native code, no ABI issues, full SQL capability for ~12K catalog records.
  • SkillDb constructor → async factory (SkillDb.create()) with manual persistence via db.export() + atomic temp+rename write
  • SkillDb is now required (not optional) in CatalogManager — no more silent degradation
  • getCatalog() driven by DB records instead of file-system scanning
  • Boot-time reconciliation (reconcileDbWithDisk) syncs DB ↔ disk state bidirectionally
  • On-disk curated skills are recorded eagerly during installCuratedSkills() (not deferred to reconciliation on a later restart)
  • Fix nano-banana-one-shop display name in SKILL.md frontmatter

Test plan

  • pnpm typecheck passes
  • pnpm test -- tests/desktop/skill-db.test.ts — all 13 tests pass (async init, persistence, bulk install, reconciliation)
  • Fresh install (rm skill-ledger.db + bundled-skills): all 20 curated skills installed with same timestamp
  • DB-only reset (skills on disk): all on-disk skills recorded immediately via Step 1b
  • User uninstall persists across restart (weather uninstalled → restart → stays gone)
  • Reconciliation auto-corrects: deleted skill dir → marked uninstalled in DB
  • pnpm desktop:dist:mac:unsigned builds successfully
  • sql-wasm.wasm present in sidecar (Nexu.app/Contents/Resources/runtime/controller/node_modules/sql.js/dist/sql-wasm.wasm)
  • No better-sqlite3 remnants in packaged app
  • Install built .app → verify controller log has no ABI error, skills page shows all curated skills

🤖 Generated with Claude Code

lefarcen and others added 18 commits March 19, 2026 20:25
- Add macOS-style Switch component (3 sizes + loading state)
- Sync CSS design tokens from design-system 2.0
- Rewrite Home page with first-run (Idle) and operational states
- Add Enabled/Providers grouping to Settings provider panel
- Integrate xs Switch for Skills install/uninstall toggle
- Add breathe animation for recommended channel cards
- Add InlineModelSelector component for Hero status bar

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update sidebar nav to use nav-item/nav-item-active classes
- Add Conversations nav item to main navigation
- Simplify sidebar session items with proper styling
- Refactor home running state channels panel with Chat button
- Update settings page header with heading-page class

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace Great Vibes with Caveat font to match design-system spec
for the "nexu alpha" heading on home page.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Visual design restoration from design-system PR #85:
- macOS-style Switch component (3 sizes + loading)
- Synced CSS design tokens
- Rewritten Home page (first-run + operational states)
- Settings provider panel with Enabled/Providers grouping
- Skills card Switch integration
- Sidebar nav refinements
- Caveat font for nexu alpha heading

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove Conversations from main nav (keep as section label only)
- Remove skill count from Skills nav item
- Add fixed sidebar toggle button next to traffic lights (desktop)
- Use logo image instead of text brand for desktop client
- Adjust spacing for desktop client header area

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update Channels panel: connected channels show horizontal rows with
  icon + name + green dot + "Connected" button + "Chat ↗" link
- Not-connected channels display as dashed-border small cards with Cable icon
- Remove bottom quick action cards (conversations, skills, github star)
- Clean up unused imports (navigate, MessageSquare, Sparkles, GITHUB_SVG)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change Hero from vertical-centered to horizontal layout (flex items-center gap-4)
- Use var(--color-success) for Running badge color consistency
- Reduce video size from w-32 to w-28 (matching prototype)
- Place model selector and message stats side-by-side in Hero right area
- Remove standalone CTA button (Chat in Feishu) - channels panel has Chat link
- Clean up unused code (getChannelShortNames, FeishuIconChat, firstChannel vars)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Click Connected button to disconnect channel (calls deleteApiV1ChannelsByChannelId)
- Add hover:text-danger and hover:bg-surface-3 styles matching prototype
- Auto-refresh channels list after disconnect

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add border, background, padding (px-2.5 py-1 rounded-lg)
- Change text color from text-muted to text-primary
- Increase font size from 11px to 12px, add font-medium
- Increase icon sizes (14/13/10) and gap (1.5)
- Wrap provider logo in 4x4 centered container

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: restore desktop-managed model catalog

* fix: allow selecting desktop cloud models

* chore: remove API smoke workflows
- Copy logo-black-1.svg to public/brand/
- Add Icon row with Help menu and GitHub link at sidebar bottom
- Add help menu with Documentation, Contact, Changelog links
- Add i18n keys for help menu items

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove General tab, directly show Providers panel
- Remove Current Model Selector from Settings page
- Update page header layout to match prototype (flex justify-between, mb-10)
- Simplify page structure

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move Save/Remove buttons before model list for better visibility
- User doesn't need to scroll to find action buttons
- Change margin from mt-6 to mb-6 for proper spacing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add WebkitAppRegion: no-drag to prevent button from being draggable
- Adjust position from top-[6px] to top-[8px] to align with traffic lights
- Adjust left from 80px to 76px for better positioning

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(desktop): replace JSON skill ledger with SQLite and route ops through IPC

Replace the fragile .curated-state.json ledger with a SQLite database
(better-sqlite3) in the Electron main process. This eliminates race
conditions and state loss when the bundled-skills directory is wiped.

- Add SkillDb module with schema, install/uninstall tracking, bulk ops,
  and automatic migration from legacy .curated-state.json
- Refactor CatalogManager and curated-skills to use SkillDb instead of
  JSON file state
- Add IPC wrappers (skillhub:*) and switch web hooks to IPC in Electron
  with HTTP fallback for cloud
- Remove duplicate curated state ledger code from API sidecar routes
- Add controller skillhub route documentation for desktop vs cloud split
- Exclude sharp/@img and sqlite build artifacts from electron-builder
  asar to reduce package size (~130MB saved)
- Skip catalog sync and curated skill install in CI to avoid ClawHub
  rate-limit failures

* fix(desktop): make SkillDb init resilient and skip skillhub in CI

- Wrap SkillDb construction in try-catch so ABI mismatch between system
  Node.js and dev Electron doesn't crash app startup. CatalogManager
  falls back to no-db mode (skills still work, just no cross-restart
  uninstall persistence).
- Skip catalog sync and curated skill install when CI env var is set
  to avoid ClawHub rate-limit failures on shared runners.
- Add desktop vs cloud architecture documentation to controller
  skillhub routes.

* refactor: move skillhub from desktop IPC to controller HTTP

Move CatalogManager, SkillDb, and curated-skills from the Electron main
process into the controller so the web app always uses HTTP for skill
catalog/install/uninstall operations. This eliminates the webview IPC
bridge for skills and makes the controller self-contained for both
desktop and cloud deployments.

- Move skillhub modules to apps/controller/src/services/skillhub/
- Add SkillhubService wrapper with lifecycle management
- Replace stub skillhub routes with real CatalogManager-backed handlers
- Simplify web hooks to HTTP-only (remove IPC branching)
- Remove all skillhub IPC from desktop (index.ts, ipc.ts, host.ts)
- Pass SKILLHUB_STATIC_SKILLS_DIR and OPENCLAW_CURATED_SKILLS_DIR
  to controller sidecar via manifests.ts
- Remove better-sqlite3, clawhub, npm, @electron/rebuild from desktop
- Use PATH-based npm in controller instead of bundled npm dep
- Delete obsolete legacy skillhub-routes test

* fix(web): replace role=button div with semantic button element

Fixes biome a11y/useSemanticElements lint error on the channel card
in home.tsx. The outer div with role="button" is now a native <button>,
which handles keyboard events and accessibility automatically.
- Add hard rules to AGENTS.md for controller dep size and native addon
  placement (controller, not Electron main process)
- Add controller sidecar packaging section to desktop-runtime-guide.md
  with size check workflow and native module guidance
- Update "Where to look" table: skill catalog path now points to
  apps/controller/src/services/skillhub/
- Add SkillHub HTTP architecture note to desktop local dev section
…g-rules

# Conflicts:
#	apps/controller/src/app/container.ts
#	apps/controller/src/routes/desktop-compat-routes.ts
#	apps/controller/src/store/nexu-config-store.ts
#	apps/desktop/dev.sh
#	apps/desktop/main/index.ts
#	apps/web/src/components/inline-model-selector.tsx
#	apps/web/src/components/skills/community-skill-card.tsx
#	apps/web/src/index.css
#	apps/web/src/layouts/workspace-layout.tsx
#	apps/web/src/pages/home.tsx
#	apps/web/src/pages/models.tsx
#	pnpm-lock.yaml
…ompatibility

better-sqlite3 is a native addon that fails in the packaged desktop app
due to ABI mismatch (CI compiles for system Node ABI 127, but the
controller sidecar runs under Electron's Node ABI 136). This caused
SkillDb to silently become undefined, skipping curated skill installation
entirely.

Replace with sql.js (SQLite compiled to WASM) — no native code, no ABI
issues. Key changes:

- SkillDb: sync constructor → async factory (SkillDb.create())
- Manual persistence via db.export() + atomic file write
- SkillDb is now required (not optional) in CatalogManager
- getCatalog() driven by DB records instead of file scanning
- Boot-time reconciliation syncs DB ↔ disk state
- On-disk skills recorded eagerly during installCuratedSkills()
- Fix nano-banana-one-shop display name in SKILL.md frontmatter
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Mar 20, 2026

Deploying nexu-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: ee6b55d
Status: ✅  Deploy successful!
Preview URL: https://55c99354.nexu-docs.pages.dev
Branch Preview URL: https://fix-sqljs-skill-db.nexu-docs.pages.dev

View logs

- getCatalog() returns cached JSON fallback when init is in-flight
- Mutating routes (install/uninstall/refresh) await init completion
- initVersion counter prevents stale init from running after dispose
- Routes call SkillhubService methods directly instead of .catalog
@alchemistklk
Copy link
Copy Markdown
Contributor Author

/cr

@slack-code-review-channel
Copy link
Copy Markdown

✅ CR topic created in Feishu topic group Refly CR.

lefarcen
lefarcen previously approved these changes Mar 20, 2026
The captureLogs step copies the electron logs directory recursively.
Temp files like desktop-diagnostics.json.tmp can vanish between the
directory existence check and the recursive cp, causing ENOENT. Catch
and ignore ENOENT during copy so transient files don't fail the CI run.
@alchemistklk alchemistklk merged commit a6f29c1 into main Mar 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants