feat(prompt): prefer MCP search tools over built-in defaults#2066
Conversation
Add a `# Tool Selection` section to the default system prompt that instructs the model to prefer MCP-provided tools (e.g. Exa) over the built-in web_search/web_fetch. MCP tools typically offer higher quality — semantic search over keyword matching, date and domain filtering. The instruction includes a fallback to built-in tools when MCP tools are unavailable. This is a low-risk change: one paragraph of prompt text, no runtime overhead, and fully backward-compatible.
esengine
left a comment
There was a problem hiding this comment.
Thanks for the contribution — first-time PRs are very welcome here. A few things to discuss before this can merge:
-
Wrong target prompt.
defaultSystemPromptinsrc/cli/index.tsis only used by thechatcommand (a thin generic chat surface). The actual coding agent — what 99% of users run — usescodeSystemPromptinsrc/code/prompt.ts. So as written, this change won't affect the path you're trying to improve. -
Trailing newline removed. The diff strips the final newline from
src/cli/index.ts(\ No newline at end of file). That'll fail biome onnpm run verify. Please re-add it. -
Vendor name in the prompt. Hard-coding "Exa" by name in the system prompt reads as promotional and ages poorly (today's preferred MCP server isn't tomorrow's). If we add this guidance, it should be vendor-neutral — something like "prefer installed MCP search tools when they cover the same purpose".
-
Open question on the premise. Built-in
web_searchis in-house and predictable; MCP tools can be slow, rate-limited, or misconfigured. A blanket "prefer MCP" instruction may regress sessions where the user installed an MCP server casually. I'd want to see this guarded — e.g., only when the model can see the MCP tool is healthy — before instructing the model to prefer it.
If you want to retarget this at codeSystemPrompt with the vendor name dropped, I'm open to discussing it further; otherwise feel free to close and we can revisit when there's a clearer signal that MCP tools are reliably better.
|
All four points are spot-on, thanks for the careful review. Will update:
Pushing the fix shortly. |
|
esengine
left a comment
There was a problem hiding this comment.
Re-reviewed — all four points from my original review are addressed: (1) retargeted to codeSystemPrompt in src/code/prompt.ts (the path 99% of users hit), not the chat-only defaultSystemPrompt; (2) the cli/index.ts trailing-newline issue is gone with the retarget (biome's green); (3) vendor-neutral now — 'prefer installed MCP-provided tools', no 'Exa'; (4) the blanket-prefer premise I worried about is guarded by the explicit fallback ('if an MCP tool fails or times out, fall back to the built-in'), which bounds the downside when someone has a flaky MCP server installed. CI green. Good — merging.
What
Adds a
# Tool Selectionsection to the default system prompt so the model prefers MCP-provided tools (e.g. Exa) over the built-inweb_search/web_fetchwhen both serve the same purpose.Why
The built-in
web_searchdoes keyword matching only — no semantic search, no date/domain filtering. MCP tools like Exa (27k uses on Smithery) offer semantic search, date filtering, domain scoping, and highlighted excerpts. Without explicit guidance, the model defaults to the familiar built-in tools, missing the quality gain from installed MCP servers.Change
One paragraph added to
defaultSystemPrompt()insrc/cli/index.ts:Impact