Foundry IQ: Improve recall by up to 54% with knowledge bases

What’s driving the improvements

• Improved knowledge source retrieval. A static retrieval orchestration workflow is replaced with a dynamic agentic retrieval loop. Now the agent batches requests and customizes queries for each knowledge source. The agent can review the retrieved content and determine if follow up queries are required.

• Enhanced semantic ranker. We retrained our reranking model to surface more relevant passages from large document collections. This returns higher-quality context with fewer irrelevant passages from knowledge sources.
• Improved answer synthesis. Knowledge base answer synthesis output mode now returns better and more complete answers to a wider variety of questions. We improved answer-generation guidelines so the model synthesizes retrieved content into better-structured responses, while staying grounded on the retrieved evidence.
• Token caching and prompt efficiency. We reduced token cost by making LLM calls cache friendly to benefit from model provider caching. This reduces token processing without sacrificing answer quality.
• Schema specialization for MCP calls. The MCP tool description for KB automatically adapts based on the model and configuration being used.
  • Model-size variants adapt the schema to nano, mini, and full-size models. Smaller models receive tighter guidance to avoid redundant queries and preserve key terms. Larger models have more flexibility while still being guided away from excessive query splitting and unnecessary tool calls.
  • Retrieval-mode variants adapt the schema for how retrieval is performed. At minimal retrieval effort, the calling model is directed to generate keyword preserving queries. At low and medium effort, the schema shifts to require full task delegation including all constraints.

Metrics

In our previous work, we relied on LLM-as-a-judge relevance as the primary answer and content quality metric. We improve upon this with ground-truth-based metrics, which take inspiration from a Evaluations Report for the TREC 2024 RAG Track and define ground-truth nuggets as pieces of information that a perfect answer to a query should contain. This unlocks a range of questions we couldn't previously answer: how good the result is relative to the gold answer, what is missing for it to be perfect, or what share of the ground truth is covered. Throughout this blog post, we use Evidence Recall, which is the share of ground-truth nuggets that are covered by the documents the agent retrieved.

For agentic evaluation, our system simulates an answer-blinded user with a specific information need. The simulated user requests information from a search agent connected to a search index or knowledge base. An independent judge observes the interaction and generates key metrics such as evidence recall and answer correctness. We’ve found that a simulated user without access to the answer is critical to replicating user interactions where a user is truly attempting to achieve a search task.

Knowledge bases offer superior recall and cost tradeoffs

We expect developers will have different tradeoffs for how much incremental token cost they are willing to spend on incremental recall (measured as evidence recall). We can compare retrieval systems based on how efficiently they make the tradeoff. The best system, at any given operating point, will operate on the evidence recall and cost frontier. We ran extensive tests of both knowledge bases and standalone search indexes with different model and parameter settings to demonstrate that knowledge bases operate on the frontier compared to standalone retrieval tools:

• Lowest cost: A minimal knowledge base with gpt-5.4-mini for agent orchestration (low reasoning) is cheaper and higher quality than any standalone hybrid search tool.
• Highest Evidence Recall: A minimal knowledge base with gpt-5.4 for agent orchestration (medium reasoning) offers the most evidence recall of any solution we tested.
• Balanced: A medium knowledge base that uses gpt-5.4 for agentic retrieval combined with gpt-5.4-mini (low reasoning) for agent orchestration provides a balance between the lowest cost and highest evidence retrieval solutions.

The graph below shows this tradeoff at different points.

Figure 3. Pareto frontier diagram showing cost (tokens) vs evidence coverage across agent orchestrated retrieval using either knowledge base or a standalone search index. Knowledge bases (green) clearly sit on the efficiency frontier and various stand-alone tool options (red) lag significantly. Dataset is BrowseComp-Plus.

Improved MCP schemas for Agentic Retrieval

Knowledge bases can be queried over a REST API or as an MCP tool.  Our latest preview has improved how the knowledge base is presented to agents over MCP. Instead of serving a single fixed schema, the knowledge base dynamically specializes the schema for the orchestrating model’s size and retrieval settings. This improves how efficiently MCP clients can use the knowledge base:

• Model-size variants adapt the schema to nano, mini, and full-size models. Smaller models receive tighter guidance to avoid redundant queries and preserve key terms. Larger models have more flexibility while still being guided away from excessive query splitting and unnecessary tool calls.
• Retrieval-mode variants adapt the schema for how retrieval is performed. At minimal retrieval effort, the calling model is directed to generate keyword preserving queries. At low and medium effort, the schema shifts to require full task delegation including all constraints. Because even smaller models can delegate retrieval tasks well, developers can choose a more responsive user-facing model while still accessing stronger agentic search capabilities through the knowledge base.

Model Query Generation Can Distort Retrieval Intent

Consider this accounting-focused user question:

“What are swaptions used for in the context of undesignated derivatives and derivative instruments?”

The outer model generates the following search query:

“swaptions used to achieve a targeted mix of fixed and variable rate debt”

The missing terminology “undesignated derivatives” and additional term “debt” may look like a minor substitution, but it changes the retrieval target. The model has injected a debt-related answer hypothesis into the query. Instead of targeting documents about the accounting treatment of derivatives, retrieval pulls toward interest rates instead. To the user, this looks like poor retrieval. But the hidden failure happened when the outer model distorted the retrieval task based on its assumptions. This is surprisingly common, particularly when the model believes it can answer the user’s question directly. It’s a core behavior that this update aims to improve.

Measuring Subagent Delegation Accuracy

To measure delegation accuracy, we labeled a dataset of user questions and follow-ups with the retrieval-critical semantics they contained. Examples include entities, keywords, units of measure, alphanumeric tokens, filters, and structural requests. For follow-up questions, we resolved the full conversational context to produce a specific retrieval request. If the initial user question was “What are the copays for the high-deductible plan?” with follow-up “what about PPO?”, the resolved question would be “What are the copays for the PPO plan?”.

We calculate delegation accuracy as function of retrieval-critical constraints preserved in the delegated task. These constraints are conditions in the query that change the scope of what context is relevant (e.g. “Financial reports from 2026”, “Part number XYZ2B manual”). Handoff from the orchestrating agent to the retriever must maintain these constraints. We penalize both dropped constraints and unsupported model-added constraints. Semantic equivalents are not penalized when they represent the same retrieval constraint, for example “grouped by” vs “group by”.

Results: Knowledge Base delegation improves retrieval across datasets

To measure impact, we compared medium knowledge bases with an agent using a standalone search index. The evaluation uses a two-agent setup, with a simulated user agent presenting questions to either the knowledge base or the search agent. We measured constraint-preservation rates to determine if preservation correlates with higher evidence recall and better answer quality:

Figure 4: Delegation accuracy by conversational turn. Knowledge bases preserves more retrieval-critical semantics than search agents across all six turns, with an average gap of +0.370

Table 2: Comparing a search agent and knowledge base across four datasets, 4,100 scenarios, 3x (gpt-5.4-mini using a medium knowledge base with gpt-5.4)

Across four datasets and more than 4,000 scenarios, Knowledge Base consistently improves evidence recall and answer match against a Search Agent while also reducing the number of tool calls. The largest answer-quality gains and tool call savings appear on more demanding tasks such as those presented by BrowseComp-Plus.