Skip to content

[Bug]: observability/langfuse: GENERATION spans created but missing usage/token counts/costs (no post-LLM hook activity logged) #42306

@nathanpt

Description

@nathanpt

Problem

The Langfuse plugin initializes and creates traces/GENERATION spans with input/output/metadata, but usage details, token counts, and costs are never attached. No post-LLM hook activity is logged even with debug enabled.

Steps to Reproduce

  1. Enable observability/langfuse.
  2. Run hermes -p "profile" (in my case researcher)
  3. Send simple message: test.
  4. Check Langfuse trace and agent logs.

Expected Behavior

Full usage details should be attached to GENERATION spans from Hermes' internal agent.usage_pricing (which correctly logs e.g. in=11154 out=241 on every API call).

Actual Behavior

  • Langfuse startup succeeds and starts a trace:
INFO langfuse: Startup: Langfuse tracer successfully initialized | public_key=... 
INFO hermes_plugins.observability__langfuse: Langfuse tracing: started trace ... 
  • GENERATION span is created in Langfuse with input/output, finish_reason, api_duration, etc.
  • No usage/usage_details/tokens/costs attached.
  • No further Langfuse logs during/after LLM turns (no post_llm_call, pre_llm_call, generation, usage_pricing attachment, observation, or hook mentions) even with HERMES_LANGFUSE_DEBUG=true, LANGFUSE_DEBUG=true, and HERMES_LOG_LEVEL=DEBUG.
  • See langfuse's trace log for one example: json_log_view.json

Affected Component

Other

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

Report       https://paste.rs/8Nuhl
  agent.log    https://paste.rs/hHkC6
  gateway.log  https://paste.rs/0Docy

Operating System

Ubuntu 26.04 (Linux 7.0.0-22-generic x86_64)

Python Version

Python: 3.11.15

Hermes Version

Hermes version: 0.16.0 (2026.6.5) [74239b4]

Additional Logs / Traceback (optional)

# Additional Details
- Works correctly on the default profile (full usage captured in Langfuse).
- Hermes internal logs show complete token usage on every call.
- /usage inside sessions shows correct numbers.
- Traces contain rich metadata (provider, base_url, api_mode, finish_reason, etc.) but usage attachment is missing.
- No errors thrown; plugin is fail-open / silent on the attachment step.

# Sample Langfuse trace structure (from earlier):
- CHAIN "Hermes turn" with input/output
- GENERATION "LLM call 1" with input/output/metadata
- No usage block

# Relevant log behavior (with debug):
- Startup trace creation logged.
- No additional Langfuse-related output during LLM response processing.

Workarounds Tried
- Fresh sessions via /new
- Different models/providers
- Pure LLM vs tool turns
- Debug flags enabled
- .env verified correct and matching working default profile
- Rebooted server (turn it off and back on)

Root Cause Analysis (optional)

No response

Proposed Fix (optional)

No response

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low — cosmetic, nice to havecomp/pluginsPlugin system and bundled pluginstelemetryTouches outbound telemetry, usage attribution, or analytics — needs opt-in gating before mergetype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions