Skip to content

Bbednarski/nmf 41 hermes middleware nemoflow optional#29551

Closed
bbednarski9 wants to merge 3 commits into
NousResearch:mainfrom
bbednarski9:bbednarski/nmf-41-hermes-middleware-nemoflow-optional
Closed

Bbednarski/nmf 41 hermes middleware nemoflow optional#29551
bbednarski9 wants to merge 3 commits into
NousResearch:mainfrom
bbednarski9:bbednarski/nmf-41-hermes-middleware-nemoflow-optional

Conversation

@bbednarski9

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR extends Hermes' native middleware hooks into an observer-grade telemetry surface and adds an optional NeMo-Flow observability plugin that consumes those hooks without making NeMo-Flow a required Hermes dependency.

The main problem this solves is telemetry fidelity. Before this change, observer plugins could see some LLM/tool activity, but several important lifecycle edges were lossy or hard to correlate: provider request IDs, turn IDs, tool-call IDs, approval decisions, blocked tools, Hermes-owned built-in tools, subagent boundaries, session finalization, and interrupted tool execution. That made it difficult for ATIF/ATOF exporters or other observability backends to reconstruct complete trajectories from public Hermes middleware alone.

This approach keeps the core Hermes contract backend-neutral. The middleware additions are additive fields on existing hooks, while the NeMo-Flow integration lives as a peer observability plugin. Existing plugins can ignore the new fields, and users only opt into NeMo-Flow if the optional package is installed and the plugin is enabled.

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Added telemetry_schema_version = "hermes.observer.v1" and additive correlation fields across Hermes middleware hooks in hermes_cli/plugins.py, model_tools.py, agent/conversation_loop.py, agent/tool_executor.py, agent/agent_runtime_helpers.py, tools/approval.py, tools/delegate_tool.py, cli.py, and gateway/run.py.
  • Added stable turn/API/tool correlation fields including turn_id, api_request_id, tool_call_id, request/response payloads, timestamps, duration, status, error type/message, approval metadata, session boundary reason, and subagent parent/child lineage.
  • Added blocked-tool terminal hook emission with status="blocked" so observers can close tool spans even when execution is skipped by guardrails, approval denial, or plugin policy.
  • Added post-tool hook emission for Hermes-owned tools that bypass model_tools.handle_function_call, including built-in agent-loop tools, context-engine tools, memory-provider tools, and delegate paths.
  • Added cancellation telemetry for KeyboardInterrupt during tool execution so interrupted terminal spans close with status="cancelled" instead of leaving incomplete ATOF/ATIF trajectories.
  • Added quiet single-query interrupted session-end emission so hermes.session.end records interrupted=true and reason="keyboard_interrupt" before shutdown finalization.
  • Added optional NeMo-Flow observability plugin files:
    • plugins/observability/nemo_flow/plugin.yaml
    • plugins/observability/nemo_flow/__init__.py
    • plugins/observability/nemo_flow/README.md
  • Added unit coverage for observer hook shape, NeMo-Flow plugin behavior, session boundary hooks, blocked/cancelled tool hooks, Hermes-owned tool completion hooks, approval correlation, and model-tool hook behavior.

How to Test

  1. Run the focused unit tests for the middleware/plugin paths:

    scripts/run_tests.sh \
      tests/plugins/test_nemo_flow_plugin.py \
      tests/test_model_tools.py \
      tests/run_agent/test_run_agent.py::TestExecuteToolCalls \
      tests/cli/test_session_boundary_hooks.py \
      tests/gateway/test_session_boundary_hooks.py \
      tests/hermes_cli/test_plugins.py
  2. Verify the latest cancellation regression path:

    scripts/run_tests.sh \
      tests/run_agent/test_run_agent.py::TestExecuteToolCalls \
      tests/cli/test_session_boundary_hooks.py

    Local result: 15 passed.

  3. Run an end-to-end Hermes + NeMo-Flow smoke with local Ollama and inspect generated ATOF/ATIF:

    • ATOF output: /private/tmp/hermes-nemo-flow-e2e-dkhpmg/nemo-flow/atof/interrupted-sleep-fixed-atof.jsonl
    • ATIF output: /private/tmp/hermes-nemo-flow-e2e-dkhpmg/nemo-flow/atif/interrupted-sleep-fixed-atif-20260520_171847_6b77ed.json
    • Expected ATOF sequence includes:
      • terminal scope start
      • terminal scope end with status="cancelled"
      • hermes.session.end mark with interrupted=true and reason="keyboard_interrupt"

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS, local Ollama qwen3.6:35b, Phoenix/ATIF visualization smoke

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…vents, so NeMo-Flow tool spans can close for built-ins that bypass model_tools.handle_function_call.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins comp/agent Core agent loop, run_agent.py, prompt builder telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge labels May 21, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Follows up on #28611 (same author, draft NeMo-Flow PR). This is a cleaner, non-draft version that adds the NeMo-Flow observability plugin as an optional in-tree plugin. Note: adds new files under plugins/observability/nemo_flow/. Related to #6642 (unified telemetry tracking issue) and #26696 (token telemetry plugin request).

@bbednarski9

bbednarski9 commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

Closing out this PR because the work has shifted to the scope of #29722 and #29724

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants