Do you need to file an issue?
Describe the bug
When I try to add some documents to my knowledge base, i get an error telling me that the max input length is 8192 tokens. It doesn't happen with every document and I cannot identify why with certain documents happen (doesn't seem to be a size problem as I have ingested heavier documents correctly)
It has happened with the next models:
- openai/text-embedding-3-small
- openai/text-embedding-3-large
- qwen/qwen3-embedding-8b
Maybe the chunking isn't working as expected?
Steps to reproduce
Using OpenAI embeddings:
- Create a new Knowledge base
- Try to ingest this pdf file
- You will get the error I'm talking about
Using Qwen embeddings:
- Create a new Knowledge base
- Try to ingest this pdf file
- You will get the error I'm talking about
Expected Behavior
The document should have been ingested without any problem
Related Module
Knowledge Base Management
Configuration Used
I don't see any chunking settings to apply in the Embedding settings tab. I have used the 3 models stated on the despcription, all with auto dimensions:
Logs and screenshots
Log with openai/text-embedding-3-large and - openai/text-embedding-3-small
Processing 1 file(s) for KB 'master'
[kb_upload_20260525_204139_ec9dd22b] Processing 1 file(s) for KB 'master'
[kb_upload_20260525_204139_ec9dd22b] Processing 1 files... (0/1, 0%)
Validating documents for 'master'...
Recovering staged file: 2025_Dritsas_database_systems_in_the_big_data_era.pdf
Staged 1 new file(s)
[kb_upload_20260525_204139_ec9dd22b] Staged 1 new file(s)
[kb_upload_20260525_204139_ec9dd22b] Indexing (LlamaIndex) 2025_Dritsas_database_systems_in_the_big_data_era.pdf (1/1, 100%)
Adding 1 document(s) to KB 'master'
Initialized embedding client with openrouter adapter (model: openai/text-embedding-3-large, dimensions: 0)
Adding 1 documents to KB 'master' using LlamaIndex
Verifying embedding API connectivity...
HTTP Request: POST https://openrouter.ai/api/v1/embeddings "HTTP/1.1 200 OK"
Successfully generated 1 embeddings (model: text-embedding-3-large, dimensions: 3072)
Embedding API OK (returned 3072-dim vector)
Parsing document: 2025_Dritsas_database_systems_in_the_big_data_era.pdf
Loaded: 2025_Dritsas_database_systems_in_the_big_data_era.pdf (83145 chars)
Loading existing index from /app/data/knowledge_bases/master/version-1...
Failed to add documents: Embedding provider returned error payload: message=HTTP 400: {
"error": {
"message": "Invalid 'input[0]': maximum input length is 8192 tokens.",
"type": "invalid_request_error",
"param": null,
"code": null
}
}, code=400, type=None
Traceback (most recent call last):
File "/app/deeptutor/services/rag/pipelines/llamaindex/pipeline.py", line 247, in add_documents
num_added = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/pipeline.py", line 249, in <lambda>
lambda: storage.insert_documents(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/storage.py", line 82, in insert_documents
count = ingestion.insert_documents_into_index(index, documents, show_progress=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/ingestion.py", line 71, in insert_documents_into_index
index.insert_nodes(nodes)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 354, in insert_nodes
self._insert(nodes, **insert_kwargs)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 313, in _insert
self._add_nodes_to_index(self._index_struct, nodes, **insert_kwargs)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 231, in _add_nodes_to_index
nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 138, in _get_node_with_embedding
id_to_embed_map = embed_nodes(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/utils.py", line 196, in embed_nodes
new_embeddings = embed_model.get_text_embedding_batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 413, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 508, in get_text_embedding_batch
embeddings = self._get_text_embeddings(cur_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 126, in _get_text_embeddings
result = self._run_in_new_loop(self._aget_text_embeddings(texts))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 84, in _run_in_new_loop
return loop.run_until_complete(coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 110, in _aget_text_embeddings
embeddings = await client.embed(texts, progress_callback=self._progress_callback)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/client.py", line 97, in embed
response = await self.adapter.embed(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/adapters/openai_compatible.py", line 293, in embed
embeddings = self._extract_embeddings_from_response(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/adapters/openai_compatible.py", line 65, in _extract_embeddings_from_response
raise ValueError(
ValueError: Embedding provider returned error payload: message=HTTP 400: {
"error": {
"message": "Invalid 'input[0]': maximum input length is 8192 tokens.",
"type": "invalid_request_error",
"param": null,
"code": null
}
}, code=400, type=None
Failed 2025_Dritsas_database_systems_in_the_big_data_era.pdf: Embedding provider returned error payload: message=HTTP 400: {
"error": {
"message": "Invalid 'input[0]': maximum input length is 8192 tokens.",
"type": "invalid_request_error",
"param": null,
"code": null
}
}, code=400, type=None
Indexed 0 file(s)
[kb_upload_20260525_204139_ec9dd22b] Indexed 0 file(s)
[kb_upload_20260525_204139_ec9dd22b] Successfully processed 0 files!
Processed 0 file(s) for 'master'
[kb_upload_20260525_204139_ec9dd22b] Processed 0 file(s) for 'master'
Log with qwen/qwen3-embedding-8b:
Processing 1 file(s) for KB 'master'
[kb_upload_20260525_204717_04f120b0] Processing 1 file(s) for KB 'master'
[kb_upload_20260525_204717_04f120b0] Processing 1 files... (0/1, 0%)
Validating documents for 'master'...
Recovering staged file: 2015_Upegui_entendimiento_de_fenomenos_ambientales_mediante_analisis_de_datos.pdf
Staged 1 new file(s)
[kb_upload_20260525_204717_04f120b0] Staged 1 new file(s)
[kb_upload_20260525_204717_04f120b0] Indexing (LlamaIndex) 2015_Upegui_entendimiento_de_fenomenos_ambientales_mediante_analisis_de_datos.pdf (1/1, 100%)
Adding 1 document(s) to KB 'master'
Adding 1 documents to KB 'master' using LlamaIndex
Verifying embedding API connectivity...
HTTP Request: POST https://openrouter.ai/api/v1/embeddings "HTTP/1.1 200 OK"
Successfully generated 1 embeddings (model: Qwen/Qwen3-Embedding-8B, dimensions: 3072)
Embedding API OK (returned 3072-dim vector)
Parsing document: 2015_Upegui_entendimiento_de_fenomenos_ambientales_mediante_analisis_de_datos.pdf
Loaded: 2015_Upegui_entendimiento_de_fenomenos_ambientales_mediante_analisis_de_datos.pdf (362857 chars)
Loading existing index from /app/data/knowledge_bases/master/version-1...
Failed to add documents: Embedding provider returned error payload: message=HTTP 400: {"code":20015,"message":"The parameter is invalid. Please check again.","data":null}, code=400, type=None
Traceback (most recent call last):
File "/app/deeptutor/services/rag/pipelines/llamaindex/pipeline.py", line 247, in add_documents
num_added = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/pipeline.py", line 249, in <lambda>
lambda: storage.insert_documents(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/storage.py", line 82, in insert_documents
count = ingestion.insert_documents_into_index(index, documents, show_progress=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/ingestion.py", line 71, in insert_documents_into_index
index.insert_nodes(nodes)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 354, in insert_nodes
self._insert(nodes, **insert_kwargs)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 313, in _insert
self._add_nodes_to_index(self._index_struct, nodes, **insert_kwargs)
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 231, in _add_nodes_to_index
nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 138, in _get_node_with_embedding
id_to_embed_map = embed_nodes(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/indices/utils.py", line 196, in embed_nodes
new_embeddings = embed_model.get_text_embedding_batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 413, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 508, in get_text_embedding_batch
embeddings = self._get_text_embeddings(cur_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 126, in _get_text_embeddings
result = self._run_in_new_loop(self._aget_text_embeddings(texts))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 84, in _run_in_new_loop
return loop.run_until_complete(coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/app/deeptutor/services/rag/pipelines/llamaindex/embedding_adapter.py", line 110, in _aget_text_embeddings
embeddings = await client.embed(texts, progress_callback=self._progress_callback)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/client.py", line 97, in embed
response = await self.adapter.embed(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/adapters/openai_compatible.py", line 293, in embed
embeddings = self._extract_embeddings_from_response(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/deeptutor/services/embedding/adapters/openai_compatible.py", line 65, in _extract_embeddings_from_response
raise ValueError(
ValueError: Embedding provider returned error payload: message=HTTP 400: {"code":20015,"message":"The parameter is invalid. Please check again.","data":null}, code=400, type=None
Failed 2015_Upegui_entendimiento_de_fenomenos_ambientales_mediante_analisis_de_datos.pdf: Embedding provider returned error payload: message=HTTP 400: {"code":20015,"message":"The parameter is invalid. Please check again.","data":null}, code=400, type=None
Indexed 0 file(s)
[kb_upload_20260525_204717_04f120b0] Indexed 0 file(s)
[kb_upload_20260525_204717_04f120b0] Successfully processed 0 files!
Processed 0 file(s) for 'master'
[kb_upload_20260525_204717_04f120b0] Processed 0 file(s) for 'master'
Additional Information
- DeepTutor Version: 1.4.0 Docker Image
- Operating System: macOS 15.7.5
Do you need to file an issue?
Describe the bug
When I try to add some documents to my knowledge base, i get an error telling me that the max input length is 8192 tokens. It doesn't happen with every document and I cannot identify why with certain documents happen (doesn't seem to be a size problem as I have ingested heavier documents correctly)
It has happened with the next models:
Maybe the chunking isn't working as expected?
Steps to reproduce
Using OpenAI embeddings:
Using Qwen embeddings:
Expected Behavior
The document should have been ingested without any problem
Related Module
Knowledge Base Management
Configuration Used
I don't see any chunking settings to apply in the Embedding settings tab. I have used the 3 models stated on the despcription, all with auto dimensions:
Logs and screenshots
Log with openai/text-embedding-3-large and - openai/text-embedding-3-small
Log with qwen/qwen3-embedding-8b:
Additional Information