🔥 chore(web-crawler): remove WeChat URL rules#14633
Conversation
Drop the `weixin.sogou.com` and `mp.weixin.qq.com` rules from the crawler URL ruleset since they are no longer needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a79653fac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| impls: ['search1api', 'jina'], | ||
| urlPattern: 'https://mp.weixin.qq.com(.*)', | ||
| }, | ||
| // GitHub source code parsing |
There was a problem hiding this comment.
Preserve WeChat-specific crawler ordering
When self-hosters configure CRAWLER_IMPLS, this deletion makes mp.weixin.qq.com use that configured order instead of the previous search1api → jina override: SearchService.crawlPages passes the env list into new Crawler, and Crawler.crawl only uses rule impls when applyUrlRules finds a match, otherwise it falls back to this.impls. With the documented example CRAWLER_IMPLS="naive,search1api", protected WeChat Official Account links are now tried by naive before Search1API, so a long block/redirect page can be accepted as a successful crawl and Search1API is never reached. Please keep a WeChat rule, or otherwise force the Search1API/Jina ordering for these URLs if WeChat links remain supported.
Useful? React with 👍 / 👎.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #14633 +/- ##
==========================================
- Coverage 65.93% 65.93% -0.01%
==========================================
Files 2887 2887
Lines 250248 250240 -8
Branches 24242 25096 +854
==========================================
- Hits 165004 164994 -10
- Misses 85092 85094 +2
Partials 152 152
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Drop the `weixin.sogou.com` and `mp.weixin.qq.com` rules from the crawler URL ruleset since they are no longer needed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the `weixin.sogou.com` and `mp.weixin.qq.com` rules from the crawler URL ruleset since they are no longer needed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the `weixin.sogou.com` and `mp.weixin.qq.com` rules from the crawler URL ruleset since they are no longer needed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the `weixin.sogou.com` and `mp.weixin.qq.com` rules from the crawler URL ruleset since they are no longer needed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# 🚀 LobeHub Release (20260518) **Release Date:** May 18, 2026 **Since v2.1.58:** 208 merged PRs · 209 commits · 16 contributors > v2.2.0 introduces the **Chief Agent Operator** — an agent that runs itself end-to-end. It self-iterates against its own output, assembles sub-agent teams on demand through the heterogeneous runtime, and drives a unified task system that knows when to pause for a human. Self-review, AssistantGroup, and tasks/scheduling all converge into one operator surface. --- ## ✨ Highlights ### 🎩 Chief Agent Operator - **Self-iteration exits Lab** — Agent Signal's self-review pipeline ships proposal actions straight into briefs and auto-executes the approved follow-ups, with prompts hardened against eval. The operator now critiques and re-runs its own work without a human in the loop. (#14769, #14583, #14647, #14882) - **Auto-formed agent teams** — Heterogeneous AssistantGroup gains Monitor-style signal callbacks, read-only SubAgent threads with breadcrumb headers, and a thread switcher. The operator dispatches sub-agents and you can step into any branch to see what the team is doing. (#14859, #14658, #14845, #14715) - **Task system as the operator's runway** — Claude Code surfaces task tools, AskUserQuestion freeform notes, and a dedicated `waitingForHuman` topic status; `lobe-task` exposes `setTaskSchedule`; the scheduler is hardened (maxExecutions cap, sub-10min heartbeat block, race-free SchedulerForm). Long-running operator runs no longer go silent and stop themselves when human input is needed. (#14870, #14639, #14713, #14865, #14853) ### 🚀 Cloud & runtime - **Cloud Claude Code V3** — Repo picker, GitHub token flow, and sandbox-aware context bring cloud-hosted Claude Code to feature parity with local; cloud sandbox completion now triggers the task lifecycle end-to-end. (#14568, #14822, #14681) - **Heterogeneous agent multi-replica safety** — Subagent threads, ingest refresh, and parallel-tool counts now survive replica swaps without losing parent_id or rolling back tool state. (#14897, #14631, #14806, #14838) - **Built-in tool lifecycle hooks** — `onBeforeCall` / `onAfterCall` land on the built-in tool runtime; sub-agent dispatch moves to `lobe-agent`; self-iteration aligns with the shared inspector pattern. (#14719, #14715, #14827) - **Knowledge base RAG unified** — Client and server share one `KnowledgeBaseSearchService`; KB files preserved on `NoSuchKey` instead of silently lost. (#14673, #14501) ### 💬 Workspace experience - **Home daily brief + recommendations** — The home screen opens with a linkable welcome, paired input hint, and a recommendations module sourced from the operator's hetero action library. (#14589, #14645, #14770) - **Chat mode + redesigned action bar** — The chat input gains a Chat/Agent mode toggle and a re-pitched action bar with icon-and-color action tag chips. (#14774, #14903, #14846) - **Documents tree, optimistic** — Document tree creates, deletes, and inline renames now apply optimistically; the agent-documents index hides web crawls and switches to a table layout. (#14714, #14292) - **Branded MCP inspectors** — Linear MCP tool calls render with the same branded inspector as the built-in Linear skill; CC MCP and built-in skills now share inspector code. (#14864, #14884) - **Bot identity gating** — Device tools are gated by sender identity, the activator bypass is closed, and Slack mpim plus Discord DM regressions are fixed. (#14634, #14664, #14733) --- ## 🏗️ Core Agent & Signal Pipeline ### Self-iteration & Agent Signal - Self-iteration graduates out of Lab, with service, tool, name, and concept structure unified across `agent-signal`, `prompts`, `database`, and `builtin-tool-self-iteration`. (#14699, #14769) - Self-review now proposes actions to briefs and auto-executes the approved set, with eval-verified prompt hardening. (#14583, #14657, #14647) - Self-iteration built-in tool aligns with the shared runtime + inspector patterns. (#14827) - Agent Signal prompts adapt their response language and avoid blocking agent execution. (#14890, #14775, #14882) - Receipt descriptions now carry an Agent Signal marker, and self-review hinted skill documents route correctly. (#14764, #14895) ### Heterogeneous agent runtime - Subagent threads render read-only with a breadcrumb header and thread switcher; SUBAGENT badge dropped, indentation tightened. (#14658, #14845, #14783) - Multi-replica safety: ingest refresh restores tools/model from DB to fix parent_id breaks; new-step assistants sync across replicas; subagent-tagged events no longer leak into the main gateway handler. (#14897, #14631, #14838) - Fetch-triggering events are deferred to keep parallel tool counts from rolling back. (#14806) - AskUserQuestion is wired for Claude Code, with auto-decline disabled and a freeform note input on the cloud side; `waitingForHuman` is a first-class topic status. (#14639, #14629, #14870) - AssistantGroup gains Monitor-style signal callbacks; project skills surface in the working sidebar and markdown preview. (#14859, #14896) - Cloud Claude Code V3 — repo picker, GitHub token, sandbox context; credentials alert and disabled input when not configured. (#14568, #14822) - Cloud sandbox completion now triggers the task lifecycle end-to-end. (#14681) ### Agent runtime & context engine - Built-in tool runtime gets `onBeforeCall` / `onAfterCall` lifecycle hooks. (#14719) - `CompletionLifecycle`, `HumanInterventionHandler`, and `stepPresentation` are extracted from the runtime monolith. (#14441) - Per-tool timeout is honored end-to-end for client tool dispatch. (#14817) - Compression budget accounts for `tool_calls`, reasoning content, and tool defs; `call_llm` forwards tools into the budget. (#14813, #14837) - Pre-flight context check now fails fast for OpenAI-compatible providers. (#14824) - Malformed `tool_call` names are recovered instead of finishing the step silently. (#14577) - Sub-agent dispatch moves from `lobe-gtd` to `lobe-agent`. (#14715) - Hidden built-in tools now appear in the system prompt @-mention list. (#14823) ### Agent tracing & operations - New `agent_operations` table and runtime persistence for every hetero-agent operation. (#14416, #14736) - `signOperationJwt` issues 4-hour signed operation tokens. (#14586) - S3 trace snapshots are zstd-compressed; DB `trace_s3_key` aligns with the `.json.zst` suffix; legacy `.json` fallback preserved on fetch. (#14807, #14860, #14826) --- ## 📱 Platform & Integrations ### Bot / Channels - Device tools are gated by sender identity. (#14634) - Activator bypass closed and device-access checks converged. (#14664) - Slack mpim supported; Discord DM regression fixed; Slack connect + slash commands repaired. (#14733, #14591) - Bot channels, bot watch, bot callback service, and system bot reliability fixes. (#14847, #14796, #14570, #14784, #14649) - Online Messager scaffolding. (#14755) ### Onboarding - Home daily brief with linkable welcome and paired input hint. (#14589) - Recommendations module sourced from the hetero agent action library. (#14645) - Chat onboarding passes request triggers via metadata and preserves the resume request. (#14770, #14798) - Discovery turn progress gated by phase, with a reminder on stalled discovery. (#14842, #14833) - FullNameStep back button rejoins the shared prefix; ModeSwitch hidden in production. (#14898, #14760) - Agent marketplace folds into the web onboarding tool. (#14578, #14672) - Onboarding interests stored as keys instead of free text; early-exit skips marketplace and drops CJK prompts. (#14624, #14598) ### Model providers - Gemini 3.1 Flash-Lite cards; Gemini schema sanitizer drops non-compliant `enum` / `required`; zero `cachedContentTokenCount` handled in usage conversion. (#14604, #14740, #14567) - DeepSeek-V4 model cards and pricing restored to official rates. (#14110, #14911) - ernie-5.1 and spark-x2-flash support; Grok 4.3 `reasoning_effort` support. (#14643, #14731, #14642) - SiliconCloud catalog synced with API; duplicates removed; reasoning params adjusted. (#14464) - Minimax derives `max_tokens` from context window to avoid `ExceededContextWindow`. (#14814) - aihubmix uses the full models endpoint for a complete list; stale empty-apiKey test dropped. (#14511, #14669) - Stream parse errors are enriched with provider + model context. (#14636) - Visual content parts are consumed in the server runtime; video image references move to a JSON object. (#14637, #14900) - Google function call magic `thoughtSignature` now attaches to every part, not just the last turn. (#14904) - Service model assignments settings added; model extend-param options removed. (#14712, #14607) ### Built-in tools & knowledge base - `lobe-task` exposes `setTaskSchedule`; task scheduler hardened (maxExecutions cap, sub-10min heartbeat blocked, SchedulerForm race fix, rapid automation-mode toggle stabilized). (#14713, #14865, #14853, #14801) - KnowledgeBaseSearchService shares RAG runtime across client and server. (#14673) - KB files preserved on `NoSuchKey` and orphan documents/tasks cleaned. (#14501) - Document tree gets optimistic create/delete + inline rename. (#14714) - agent-documents index hides web crawls and switches to a table layout. (#14292) - `lobe-clarify` and SKILL.md frontmatter parsing/edit validation are unified. (#14566) - AnalyzeVisualMedia inspector + Portal HTML preview refactor; HTML preview restored for AssistantGroup messages. (#14777, #14811) - Branded inspector shared between CC MCP and built-in Linear skill. (#14884, #14864) --- ## 🖥️ CLI & User Experience ### Chat & Conversation - Chat mode toggle and redesigned chat input action bar. (#14774) - Action tag chips switch to icon + colored label; ActionDropdown closes on sibling-open and focus-out; submenu uses native header/footer slots. (#14903, #14802, #14901) - Action bar padding equalized around the send button; skeleton shows in action bar while config loads. (#14846, #14656) - `useCmdEnterToSend` is respected in thread & task inputs; send button enables after pasting into thread/comment input. (#14850, #14816) - TopicChatDrawer state preserved during close animation. (#14803) - Only the last assistant block animates during markdown streaming. (#14906) - Right working panel no longer auto-collapses on chat mount; home agent config fetched so knowledge toggles reflect in UI. (#14883, #14834) ### Tasks - Task scheduler, hotkey, comment, and TodoList polish. (#14707) - Add Subtask button & card baseline aligned; activity card stop run; task agent manager polish. (#14848, #14559, #14569) - Task template skeleton CLS reduced; task page placeholder copy refreshed. (#14788, #14704) - Task agent model snapshotted into `task.config` at create time. (#14670) - User-feedback card, task card polish, and Run-now context menu in markdown. (#14727) - Inline skill auth in recommended task templates. (#14676) ### Navigation & Layout - Tab bar gains a Chrome-style divider between inactive tabs. (#14892) - SideBarDrawer & header layout polish; nav ActionIcon sizing unified; TodoList encapsulation improved. (#14762, #14692) - Desktop header icons, sidebar density, and task menus polished. (#14724) - Standardized header action icon sizes. (#14717) - Chat topic title length increased; copy session ID added to topic dropdown menu. (#14659, #14595) - Heterogeneous agent topic rows regain indentation. (#14783) ### Other polish - Usage token details shortened; tool execution time formatted as `Xmin Ys`. (#14849, #14641) - Tool arguments display gets word-wrap toggle; long tool-call params wrap instead of truncate. (#14706, #14640) - Editor stops showing per-line placeholder once content is present. (#14852) - Visible divider between queued messages; intervention confirmation bar polished. (#14593, #14587) - Credit top-up copy refreshed; auth captcha retry copy refreshed; brief recommendations layout polished. (#14821, #14561, #14871) --- ## 🔧 Tooling & Developer Experience - Dev-only feature flag override panel. (#14565) - `__DEV__` define replaces `process.env.NODE_ENV` in the SPA. (#14696) - Agent-settings drops Meta/Documents tabs and restores `inputTemplate`. (#14874) - `local-system` forwards all `grepContent` params and moves the executor to `/client`. (#14888) - `lobe-task` and `setTaskSchedule` exposed. (#14713) - Memory user-memory benchmark agent config and source-id extraction schemas. (#14779, #14778) - CLI man page drops stale cron entry; `clearMessages` hotkey removed. (#14709, #14906) - Skill docs simplified; cloud heteroContext gains sandbox TTL + public-repo fork push guide. (#14785, #14761) --- ## 🔒 Security & Reliability - **Security:** Sensitive comments and examples sanitized from the production JS bundle. (#14557) - **Security:** Inactive OIDC access rejected. (#14674) - **Security:** CASC `new Function()` template replaced with safe string builders. (#14751) - **Security:** Sign-in captcha flow removed in favor of safer flow. (#14573) - **Security:** Desktop local file previews restricted to safe roots. (#14789) - **Security:** Image binary capped at 3.75 MB so base64 payload stays under the Anthropic 5 MB limit. (#14711) - **Reliability:** Neon/Node pools get error listeners to prevent Lambda crashes. (#14606) - **Reliability:** `paradedb.match(...)` replaces hardcoded normalizer in memory search. (#14590) - **Reliability:** `PlaceholderVariablesProcessor` errors carry diagnostic context. (#14741) - **Reliability:** File storage upload checks are serialized; multiple account link bug fixed. (#14829, #14562) - **Reliability:** `ScrollShadow` replaced with `ScrollArea` to fix a React infinite render loop (error code 185). (#14689) - **Reliability:** Embedding token cap enforced — long memory queries are limited and truncated before search. (#14757) - **Reliability:** Embed binary blob guard + oversized output cap in `local-system.readFile`. (#14602) - **Reliability:** Windows npm CLI shims resolved before spawning agents. (#14772, #14720) - **Reliability:** Vite pinned to 8.0.12 to avoid the rolldown 1.0.1 preload regression; desktop runtime externals split from native deps. (#14804, #14776) - **Reliability:** Old lobehub cron job removed; WeChat URL rules dropped from web crawler. (#14630, #14633) --- ## 👥 Contributors Huge thanks to **16 contributors** who shipped **208 merged PRs** this cycle. @hezhijie0327 · @sxjeru · @hardy-one · @Bianzinan · @brone1323 · @YuSaZh · @Wxh16144 · @arvinxx · @Innei · @tjx666 · @neko · @lijian · @rdmclin2 · @sudongyuer · @AmAzing129 · @rivertwilight Plus @lobehubbot for maintenance translations. --- **Full Changelog**: v2.1.58...v2.2.0
Summary
weixin.sogou.com(Sogou WeChat search redirect) andmp.weixin.qq.com(WeChat official accounts) entries fromcrawUrlRules.urlRules.test.tsand the WeChat assertion in the "Chinese platforms" coverage check.These rules previously routed WeChat URLs to
search1api/jina; both impls handle these URLs out of the box now and the dedicated rules are no longer needed.Test plan
bunx vitest run --silent='passed-only' packages/web-crawler/src/__tests__/urlRules.test.ts(22/22 pass)🤖 Generated with Claude Code