[Bug]: add-resource sends oversized input to OpenAI embeddings API during repo import #616
Description
Bug Description
Importing a GitHub repository with ov add-resource ... --wait succeeds at the filesystem level, but semantic processing fails during embedding when OpenViking sends an input larger than the embedding model's token limit.
In my case, the resource tree is created under viking://resources/..., but generated semantic artifacts such as .overview.md and .abstract.md remain default/empty, and the server logs show an OpenAI embeddings request failing because the input was too large.
Steps to Reproduce
Ran the quickstart.
-
Start the OpenViking server.
-
Run:
ov add-resource https://github.com/volcengine/OpenViking --wait
-
Check the imported tree:
ov ls viking://resources/ -l 256 -n 256 ov tree viking://resources/volcengine -L 2
-
Inspect generated semantic files:
ov cat viking://resources/volcengine/.overview.md ov cat viking://resources/volcengine/.abstract.md
Expected Behavior
OpenViking should chunk, truncate, or otherwise bound text before sending it to the embedding provider, so that importing a repository does not fail when one file or generated intermediate artifact exceeds the embedding model's input limit.
At minimum, it would help if:
- oversized content were split before embedding,
- the failing file/resource were identified in logs,
- semantic artifact generation degraded gracefully instead of leaving empty/default
.overview.md/.abstract.md.
Actual Behavior
Chunks are too big and openAI rejects them.
Minimal Reproducible Example
Error Logs
## Server traceback
Traceback (most recent call last):
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openviking\models\embedder\openai_embedders.py", line 99, in embed
response = self.client.embeddings.create(**kwargs)
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openai\resources\embeddings.py", line 136, in create
return self._post(
~~~~~~~~~~^
"/embeddings",
^^^^^^^^^^^^^^
...<8 lines>...
cast_to=CreateEmbeddingResponse,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openai\_base_client.py", line 1297, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openai\_base_client.py", line 1070, in request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 13327 tokens (13327 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openviking\storage\collection_schemas.py", line 196, in on_dequeue
result: EmbedResult = await asyncio.to_thread(
^^^^^^^^^^^^^^^^^^^^^^^^
self._embedder.embed, embedding_msg.message
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\Users\Utilisateur\AppData\Local\Programs\Python\Python313\Lib\asyncio\threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Utilisateur\AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Code Repository\sandbox-vicking\.venv\Lib\site-packages\openviking\models\embedder\openai_embedders.py", line 104, in embed
raise RuntimeError(f"OpenAI API error: {e.message}") from e
RuntimeError: OpenAI API error: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 13327 tokens (13327 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}OpenViking Version
0.2.6
Python Version
3.13.7
Operating System
Windows
Model Backend
OpenAI
Additional Context
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status