Skip to content

feat: add Hermes tool call parser#1596

Merged
jundot merged 6 commits into
jundot:mainfrom
scubamount:feat/hermes-tool-call-parser
Jun 6, 2026
Merged

feat: add Hermes tool call parser#1596
jundot merged 6 commits into
jundot:mainfrom
scubamount:feat/hermes-tool-call-parser

Conversation

@scubamount

@scubamount scubamount commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds fallback support for the Hermes-style/Liquid tool call wire format:

<|tool_call_start|>[function_name(arg=value)]<|tool_call_end|>

Some clients/agents emit tool calls using these markers instead of oMLX's currently supported XML/prose bracket formats. Without this fallback, oMLX can surface raw marker text to users instead of returning structured tool calls.

This PR:

  • parses Hermes-style JSON payloads inside <|tool_call_start|>...<|tool_call_end|>
  • parses Python-expression-like bracket payloads such as [read_file(path='/tmp/a.log')]
  • supports multiple calls in one Hermes block via AST parsing, e.g. [tool_a(...), tool_b(...)]
  • suppresses Hermes start/end markers during streaming even when tokenizer metadata is absent
  • suppresses orphan/split Hermes closing markers so <|tool_call_end|> cannot leak if it arrives without a visible open marker
  • strips any remaining complete Hermes envelopes from final assistant text if parsing fails
  • classifies LFM2 text checkpoints as LLMs instead of embeddings by treating lfm2 as ambiguous and relying on architecture/name checks

Verification

/opt/homebrew/opt/python@3.12/libexec/bin/python3 -m pytest tests/test_tool_calling.py -q
161 passed in 0.16s

/opt/homebrew/opt/python@3.12/libexec/bin/python3 -m pytest tests/test_model_discovery.py -q
119 passed in 0.16s

scubamount added 6 commits June 1, 2026 20:50
@scubamount
feat: add Hermes tool call parser
aeba69d 
Add support for Hermes-style tool call format:
<|tool_call_start|>[func_name(arg1=val1)]<|tool_call_end|>

Supports both bracket format (Python-style keyword args) and JSON format.
Adds to the fallback chain in parse_tool_calls() and strips markers
when parsing fails to prevent raw markup leaking to API responses.
@scubamount
fix: handle Hermes streaming tool calls
3af010a
@scubamount
fix: suppress orphan Hermes end markers
4b906dc
@scubamount
docs: clarify Hermes tool-call format
811cd56
@scubamount
docs: avoid Hermes model wording
6c8e05d
@scubamount
fix: classify LFM text models as LLMs
52aeefd
@scubamount

scubamount commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Cross-ref: this PR is the server-side counterpart to NousResearch/hermes-agent#37229 (fix: normalize local tool JSON fallbacks).

These two PRs are co-dependent:

  • omlx#1596 (this PR): parses Hermes-style <|tool_call_start|>...<|tool_call_end|> markers at the inference server layer, so OMLX returns structured tool calls instead of raw marker text.
  • hermes-agent#37229: normalizes local-model assistant JSON with commands/tools_used into OpenAI-style tool calls on the hermes-agent client side.

Without both, local model tool calling is broken:

  • Without omlx#1596 → OMLX leaks raw Hermes markers to hermes-agent
  • Without hermes-agent#37229 → hermes-agent cannot normalize the mixed JSON/bracket payloads that local models emit

Both should merge together for a complete fix.

@scubamount scubamount mentioned this pull request Jun 2, 2026
Open
@jundot

jundot commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Thanks for the PR. I checked the Hermes fallback path, streaming marker suppression, and the LFM2 discovery change against the existing parser/model-discovery call paths.

The AST-based bracket parser covers the multi-call case without the old comma-splitting risk, and the streaming filter keeps the Hermes markers out of visible deltas when tools are enabled. The focused tool-calling, model discovery, tokenizer/audio discovery, and adapter tests all pass on the PR branch.

This looks good to me, and I'm going to merge it.

@jundot jundot merged commit a9c357e into jundot:main Jun 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@scubamount @jundot