fix: add missing 'json' keyword to graph memory prompts (fixes #4248)#4249
Conversation
|
|
xkonjin
left a comment
There was a problem hiding this comment.
Quick review pass:
- Main risk area here is data integrity, transaction boundaries, and backward-compatible persistence.
- I didn’t see targeted regression coverage in the diff; please add or point CI at a focused test for the changed path in utils.ts, graph_memory.ts.
- Before merge, I’d smoke-test the behavior touched by utils.ts, graph_memory.ts with malformed input / retry / rollback cases, since that’s where this class of change usually breaks.
2bd33cc to
01968b2
Compare
|
@xkonjin Thanks for the review! To clarify:
Happy to add a targeted test if the maintainers want one, but the failure mode here is a hard 400 from OpenAI (not a silent data integrity issue). |
|
I have read the CLA Document and I hereby sign the CLA |
3f8ff3b to
597d6c6
Compare
|
@xkonjin Added regression tests as requested — The entity extraction prompt in Thanks for the review! |
|
@xkonjin Thanks for the review! You're right — I'll add focused regression tests for the changed paths in |
0d87c72 to
32a1c68
Compare
|
Thanks for the review @xkonjin! Rebased onto latest Regarding the regression coverage you asked about — the branch already includes two dedicated test suites (48 tests total, all passing):
|
32a1c68 to
4e9f954
Compare
|
Friendly ping — CI is green, tests pass, rebased on latest. Ready for review whenever convenient. Happy to address any feedback. 🙏 |
…n_object response format
When using OpenAI with response_format: { type: 'json_object' }, the API
requires at least one message to contain the word 'json'. The
DELETE_RELATIONS_SYSTEM_PROMPT and the entity extraction prompt in
_retrieveNodesFromData() were missing this keyword, causing a 400 error
on every graph memory write.
The EXTRACT_RELATIONS_PROMPT already had 'Please provide your response
in JSON format.' appended — this commit applies the same fix to the two
remaining prompts.
Fixes #4248
Verifies that DELETE_RELATIONS_SYSTEM_PROMPT and getDeleteMessages() output contain the "json" keyword required by OpenAI's json_object response format. Without it, the API returns a 400 error. Signed-off-by: Giulio Leone <6887247+giulio-leone@users.noreply.github.com>
Address reviewer feedback (xkonjin) requesting targeted regression coverage for the changed paths in utils.ts and graph_memory.ts. New tests (21 total, up from 2): - JSON keyword requirement: verify all three json_object call sites (DELETE_RELATIONS_SYSTEM_PROMPT, EXTRACT_RELATIONS_PROMPT + suffix, inline entity extraction prompt) contain 'json' - getDeleteMessages: USER_ID substitution, return shape, content inclusion, plus malformed/edge-case inputs (empty strings, special characters, unicode, very long strings) - formatEntities: single/multiple entities, empty array, special chars - Prompt structural invariants: placeholder presence, non-empty guards
Address reviewer feedback (xkonjin) requesting smoke-tests for the changed code paths in graph_memory.ts with malformed input and edge cases. New test file: graph-memory-parsing.test.ts (27 tests) Mocks neo4j-driver, LLMFactory, and EmbedderFactory to unit-test the three private methods that use json_object response format: _retrieveNodesFromData: - well-formed tool calls, string responses, undefined/empty toolCalls - malformed JSON in arguments, missing entities key - unrelated tool call names, entity normalisation - json_object format and JSON keyword in system prompt _establishNodesRelationsFromData: - well-formed tool calls, string/empty responses - missing entities key in arguments - malformed JSON throws (no try/catch in source) - prompt suffix with/without custom prompt - JSON keyword present in system message _getDeleteEntitiesFromSearchOutput: - well-formed delete calls, string/empty responses - filtering by tool call name, multiple deletes - empty search output, JSON keyword in prompt Cross-cutting: - JSON keyword present for varied userId values (empty, unicode, special chars) - Entity normalisation (spaces→underscores, lowercasing)
4e9f954 to
909cf66
Compare
Summary
Fixes #4248 — Graph memory writes fail with OpenAI 400 error because
DELETE_RELATIONS_SYSTEM_PROMPTand the entity extraction prompt are missing the required "json" keyword when used withresponse_format: { type: 'json_object' }.Problem
OpenAI's API enforces that when
response_format: { type: 'json_object' }is set, at least one message in the conversation must contain the word "json". Two prompts in the graph memory pipeline violated this:DELETE_RELATIONS_SYSTEM_PROMPTinutils.ts— used by_getDeleteEntitiesFromSearchOutput()graph_memory.ts— used by_retrieveNodesFromData()This caused every
memory.add()call withenableGraph: trueand OpenAI as the LLM provider to fail with:Fix
Added "Respond in JSON format." to both prompts, consistent with the existing fix in
EXTRACT_RELATIONS_PROMPTwhich already had"\nPlease provide your response in JSON format."appended.Changes
mem0-ts/src/oss/src/graphs/utils.ts: Added "Respond in JSON format." toDELETE_RELATIONS_SYSTEM_PROMPTmem0-ts/src/oss/src/memory/graph_memory.ts: Added "Respond in JSON format." to the inline entity extraction system prompt in_retrieveNodesFromData()Testing
Verified that all three
json_objectresponse format calls now have the required keyword:_getDeleteEntitiesFromSearchOutput()— ✅ fixed_retrieveNodesFromData()— ✅ fixed_getRelatedEntities()— ✅ already had it (EXTRACT_RELATIONS_PROMPT)