fix(search): clamp inf/nan scores from vector search to prevent JSON serialization failure#824
Conversation
…serialization failure When local vector search returns inf scores (e.g., zero vectors or embedding overflow), the hierarchical retriever passes them through to the API response. FastAPI/Starlette's JSON encoder rejects inf/nan with: ValueError: Out of range float values are not JSON compliant: inf Fix: 1. hierarchical_retriever.py: clamp semantic_score and final_score to 0.0 when math.isfinite() returns False 2. search.py: add _sanitize_floats() as a defense-in-depth layer on the find and search endpoints Closes #inf-score
|
Failed to generate code suggestions for PR |
|
Thanks for this PR! The fix itself looks reasonable, but we would like to dig deeper into the root cause to determine whether additional safeguards are needed at the embedding pipeline level. A few questions:
This context will help us understand whether the inf originates from the embedding model itself, a dimension mismatch after switching models, or a bug in the vector ingestion pipeline. |
|
Thanks for the review! Here are answers to your questions: 1. Discovery: 2. Embedding model: 3. Source of inf: 4. Reproduction scope: Additional context: Would it help if I tried to capture a specific inf-producing query and its associated vector data from sqlite-vec? |
|
Thanks for the detailed answers! This is very helpful. Since the model was never switched and the issue is sporadic with short text content, the most likely root cause is near-zero embedding vectors leading to inf during cosine similarity computation. Your two-layer defense approach (clamping at retriever + sanitizing at API) is a solid fix for the immediate problem. Let's go ahead and merge this. If you're able to reproduce the issue and capture the specific query along with its associated vector data from sqlite-vec, that would be great for a follow-up investigation into the embedding pipeline — but it's not a blocker for this PR. Thanks for the contribution! |
…serialization failure (volcengine#824) When local vector search returns inf scores (e.g., zero vectors or embedding overflow), the hierarchical retriever passes them through to the API response. FastAPI/Starlette's JSON encoder rejects inf/nan with: ValueError: Out of range float values are not JSON compliant: inf Fix: 1. hierarchical_retriever.py: clamp semantic_score and final_score to 0.0 when math.isfinite() returns False 2. search.py: add _sanitize_floats() as a defense-in-depth layer on the find and search endpoints Closes #inf-score Co-authored-by: a1461750564 <a1461750564@users.noreply.github.com>
Problem
/api/v1/search/findand/api/v1/search/searchreturn500 INTERNALwith:This happens when local vector search returns
infscores (e.g., zero vectors, embedding overflow), which get passed through to the API response. Starlette'sjson.dumps(allow_nan=False)rejectsinf/nan.Root Cause
In
hierarchical_retriever.py,_convert_to_matched_contexts()reads_scorefrom vector search results and blends it with hotness score. When the source score isinf, the blendedfinal_scoreis alsoinf, causing JSON serialization to fail.Fix
Two changes:
openviking/retrieve/hierarchical_retriever.py: Clampsemantic_scoreandfinal_scoreto0.0whenmath.isfinite()returnsFalseopenviking/server/routers/search.py: Add_sanitize_floats()recursive sanitizer as defense-in-depth onfindandsearchendpointsEnvironment
qwen3-embedding:0.6b(1024d)Reproduction