Symptom from sandbox v0.18.4 field test
After v0.18.4's #124 fix landed, mistral's tool_call IS now correctly parsed by hermes (no more empty-retry loop). But the actual side-effect doesn't fire:
Most likely root cause
write_file not in agent.valid_tool_names for the sandbox session. The validation at conversation_loop.py:3207-3210 puts it in invalid_tool_calls; line 3236 appends an error tool message; the model on its NEXT turn sees the error AND hallucinates success in its narration.
The verbose-only print at line 3219 (agent._vprint(...)) is invisible in default runs. Operators see only the model's hallucinated narration; the underlying mismatch is invisible.
Fix
Upgrade the invalid_tool_call diagnostic from verbose-only print to:
logger.warning with invalid name + count + sample of registered names + model + provider — operators see it in agent.log regardless of verbose
agent._emit_status with the mismatch — visible in user-facing status stream
Operator can immediately tell:
- WHICH name the model invented (or which name our sandbox doesn't register)
- HOW MANY tools the sandbox actually has registered
- WHICH tools the sandbox DOES have (sample)
This is observability, not behavior change — the existing invalid_tool_call retry path remains. But the mismatch becomes visible at boot/dispatch time instead of being hidden by model hallucination on the next turn.
Acceptance
- Sandbox session that calls
write_file (or any unregistered tool) surfaces a clear WARN + status line naming the tool + the registered tool sample.
- Operator can correlate hermes log entries with devagentic-side dispatch logs via model + provider in the WARN.
Composition
Same observability family as PR #95 / #96 (env-var probes). Catches sandbox-config bugs that look like recovery failures.
Symptom from sandbox v0.18.4 field test
After v0.18.4's #124 fix landed, mistral's tool_call IS now correctly parsed by hermes (no more empty-retry loop). But the actual side-effect doesn't fire:
tool_call(name="write_file", args={"path":"/tmp/random_test.py","content":"..."})/tmp/random_test.py has been created with the following content: import random; print(random.randint(1, 100)); Now executing it:/tmp/random_test.pydoesn't exist on the containerMost likely root cause
write_filenot inagent.valid_tool_namesfor the sandbox session. The validation at conversation_loop.py:3207-3210 puts it ininvalid_tool_calls; line 3236 appends an error tool message; the model on its NEXT turn sees the error AND hallucinates success in its narration.The verbose-only print at line 3219 (
agent._vprint(...)) is invisible in default runs. Operators see only the model's hallucinated narration; the underlying mismatch is invisible.Fix
Upgrade the invalid_tool_call diagnostic from verbose-only print to:
logger.warningwith invalid name + count + sample of registered names + model + provider — operators see it in agent.log regardless of verboseagent._emit_statuswith the mismatch — visible in user-facing status streamOperator can immediately tell:
This is observability, not behavior change — the existing invalid_tool_call retry path remains. But the mismatch becomes visible at boot/dispatch time instead of being hidden by model hallucination on the next turn.
Acceptance
write_file(or any unregistered tool) surfaces a clear WARN + status line naming the tool + the registered tool sample.Composition
Same observability family as PR #95 / #96 (env-var probes). Catches sandbox-config bugs that look like recovery failures.