feat(plugins): add bundled observability/langfuse plugin (salvage of #16845)#16917
Merged
Conversation
Opt-in Langfuse tracing for Hermes conversations — LLM calls, tool usage, usage/cost breakdown per span. Hooks into pre/post_api_request, pre/post_llm_call, pre/post_tool_call. SDK is optional; missing SDK or credentials renders the plugin inert. Salvaged from PR #16845 by @kshitijk4poor, who wrote the plugin (~875 LOC, 6 hooks, Langfuse usage-details/cost-details normalization, read_file payload summarization). Salvage scope (why this isn't PR #16845 as-authored): - Lives at plugins/observability/langfuse/ (standalone kind, opt-in via plugins.enabled) instead of a new parallel optional-plugins/ directory. Standalone bundled plugins are already opt-in — only their plugin.yaml is scanned at startup; the Python module is not imported unless the user enables it. The premise of optional-plugins/ (avoid import cost for users who don't want it) is already solved by the existing plugin system. - Dropped the triple activation gate (plugins.enabled + plugins.langfuse.enabled + HERMES_LANGFUSE_ENABLED). The Hermes plugin system's own enable/disable is authoritative; runtime credentials gate whether the hook actually traces. - Rewrote _is_enabled() → cached _get_langfuse() with an _INIT_FAILED sentinel. The original called hermes_cli.config.load_config() from every hook invocation (full yaml parse + deep merge + env expansion on every pre/post_tool_call, potentially 100+ times per turn). The cached version reads env once and returns the cached client or None on every subsequent call with zero further work. - hermes tools → Langfuse Observability post-setup adds observability/langfuse to plugins.enabled directly (via _save_enabled_set) instead of going through an install-copy flow. Enable: hermes tools # interactive hermes plugins enable observability/langfuse # manual Required env (set by `hermes tools` or in ~/.hermes/.env): HERMES_LANGFUSE_PUBLIC_KEY HERMES_LANGFUSE_SECRET_KEY HERMES_LANGFUSE_BASE_URL # optional Co-authored-by: kshitijk4poor <kshitijk4poor@gmail.com>
teknium1
added a commit
that referenced
this pull request
Apr 28, 2026
…ence (#16929) Documents the langfuse plugin shipped in #16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
ulasbilgen
pushed a commit
to ulasbilgen/hermes-adhd-agent
that referenced
this pull request
May 1, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
donald131
pushed a commit
to donald131/hermes-agent
that referenced
this pull request
May 2, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
4 tasks
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
dannyJ848
pushed a commit
to dannyJ848/hermes-agent
that referenced
this pull request
May 17, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
Egavasyug
pushed a commit
to Egavasyug/hermes-agent
that referenced
this pull request
Jun 10, 2026
…ence (NousResearch#16929) Documents the langfuse plugin shipped in NousResearch#16917: - website/docs/user-guide/features/built-in-plugins.md: new observability/langfuse section (setup wizard vs manual, hook-by-hook behaviour, verify / optional tuning / disable) - website/docs/reference/environment-variables.md: Langfuse Observability subsection under Tool APIs listing the 3 required + 5 optional env vars, with a back-link to the built-in-plugins page Validated: ascii-guard clean, npm run build succeeds, #observabilitylangfuse anchor resolves. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Langfuse tracing ships as a bundled opt-in Hermes plugin.
Salvaged from #16845 (@kshitijk4poor). Plugin implementation is their
work; the activation model was reshaped to fit the existing plugin
system instead of introducing a parallel
optional-plugins/directory.Changes
plugins/observability/langfuse/__init__.py(6 hooks, ~870 LOC),plugin.yaml,README.mdhermes_cli/config.pyOPTIONAL_ENV_VARSentries for the 3 user-facing Langfuse keys; optional tuning keys in_EXTRA_ENV_KEYShermes_cli/tools_config.pyTOOL_CATEGORIES["langfuse"](Cloud + Self-hosted) +_run_post_setup("langfuse")tests/plugins/test_langfuse_plugin.pyWhy this isn't #16845 as-authored
plugins/observability/langfuse/, notoptional-plugins/observability/langfuse/. Standalone bundled plugins are already opt-in — discovery only parses eachplugin.yaml; the Python module isn't imported unless the plugin is inplugins.enabled. The new parallel directory solved a problem the existing system already handles.plugins.enabled+plugins.langfuse.enabled+HERMES_LANGFUSE_ENABLEDin series. Now: the plugin system's own enable/disable decides whether the plugin loads; credentials decide whether hooks trace._get_langfuse()caches with an_INIT_FAILEDsentinel. The original calledhermes_cli.config.load_config()inside_is_enabled()on every hook — full YAML parse + deep merge + env expansion, up to hundreds of times per turn on long tool loops. The rewrite reads env once, caches success or failure, and short-circuits all subsequent calls.hermes tools → Langfusepost-setup opts the bundled plugin intoplugins.enableddirectly (via_save_enabled_set), instead of going through a separate copy-into-user-plugins install flow.Activation
Validation
scripts/run_tests.sh tests/plugins/test_langfuse_plugin.py tests/hermes_cli/test_plugins.py tests/hermes_cli/test_plugins_cmd.py tests/hermes_cli/test_tools_config.py→ 171 passedHERMES_HOME:observability/langfuse,kind=standalone,enabled=Falseby default_get_langfuse()returnsNonewithout credentials_get_langfuse()calls perform 0 env reads (cache verified)hermes_cli.confignever imported by the pluginCloses #16845.