fix(tests): attach caplog to specific logger in 3 order-dependent tests#11453
Merged
Conversation
Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit 436a735 (PR #11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
konsisumer
added a commit
to konsisumer/hermes-agent
that referenced
this pull request
Apr 17, 2026
…-deprecation-github-models-413 Resolves conflict in tests/hermes_cli/test_plugins.py and tests/test_plugin_skills.py by adopting main's caplog-with-logger-name approach (PR NousResearch#11453) over the monkeypatch-setattr approach from this branch. Both fix the same xdist flakiness; main's is simpler and now the canonical project fix.
teknium1
added a commit
that referenced
this pull request
Apr 17, 2026
Fixes the recurring 'works locally, fails in CI' (and vice versa) class
of flakes by making tests hermetic and providing a canonical local runner
that matches CI's environment.
## Layer 1 — hermetic conftest.py (tests/conftest.py)
Autouse fixture now unsets every credential-shaped env var before every
test, so developer-local API keys can't leak into tests that assert
'auto-detect provider when key present'.
Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD,
_CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of
credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID,
FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that
change auto-detect behavior.
Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET,
HERMES_SESSION_*, etc.) that mutate agent behavior.
Also:
- Redirects HOME to a per-test tempdir (not just HERMES_HOME), so
code reading ~/.hermes/* directly can't touch the real dir.
- Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to
match CI's deterministic runtime.
The old _isolate_hermes_home fixture name is preserved as an alias so
any test that yields it explicitly still works.
## Layer 2 — scripts/run_tests.sh canonical runner
'Always use scripts/run_tests.sh, never call pytest directly' is the
new rule (documented in AGENTS.md). The script:
- Unsets all credential env vars (belt-and-suspenders for callers
who bypass conftest — e.g. IDE integrations)
- Pins TZ/LANG/PYTHONHASHSEED
- Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on
a 20-core workstation surfaces test-ordering flakes CI will never
see, causing the infamous 'passes in CI, fails locally' drift)
- Finds the venv in .venv, venv, or main checkout's venv
- Passes through arbitrary pytest args
Installs pytest-split on demand so the script can also be used to run
matrix-split subsets locally for debugging.
## Remove 3 module-level dotenv stubs that broke test isolation
tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a
module-level:
if 'dotenv' not in sys.modules:
fake_dotenv = types.ModuleType('dotenv')
fake_dotenv.load_dotenv = lambda *a, **kw: None
sys.modules['dotenv'] = fake_dotenv
This patches sys.modules['dotenv'] to a fake at import time with no
teardown. Under pytest-xdist LoadScheduling, whichever worker collected
one of these files first poisoned its sys.modules; subsequent tests in
the same worker that imported load_dotenv transitively (e.g.
test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and
saw their assertions fail.
dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml),
so the defensive stub was never needed. Removed.
## Validation
- tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4
failures in test_env_loader.py before this fix)
- tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py,
tests/test_hermes_logging.py combined: 123 passed (the caplog
regression tests from PR #11453 still pass)
- Local full run shows no F/E clusters in the 0-55% range that were
previously present before the conftest hardening
## Background
See AGENTS.md 'Testing' section for the full list of drift sources
this closes. Matrix split (closed as #11566) will be re-attempted
once this foundation lands — cross-test pollution was the root cause
of the shard-3 hang in that PR.
teknium1
added a commit
that referenced
this pull request
Apr 17, 2026
….sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
teknium1
added a commit
that referenced
this pull request
Apr 17, 2026
Target: <2min CI test wall time. Runs the Tests workflow as a 4-way matrix instead of one job. Each shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with -n auto xdist inside. Total effective parallelism: 16 workers across 4 machines (vs 4 workers on 1 machine today). Was previously tried in #11566 and closed — shard 3 hung at 97% complete for 100+ seconds with dozens of E/F markers. Root cause was cross-test pollution exposed by splitting test files across shards (e.g. the three test files that mutated sys.modules['dotenv'] at import time poisoned whichever shard they landed in). That's now fixed by #11453 and #11577: conftest is hermetic, the dotenv stub bombs are removed, and tests no longer depend on each other's env-var side effects. Changes: - pyproject.toml: add pytest-split>=0.9,<1 to dev extras - .github/workflows/tests.yml: 'test' job becomes matrix-split into 4 groups with fail-fast: false. Runs 'pytest --splits 4 --group N'. pytest-split composes with -n auto from pyproject addopts. e2e job is unchanged (already small, 20s). Expected timing: Before: ~4m total (243s test step + ~25s setup) After: ~90-115s total (shard wall time ~60-90s + ~25s setup) Hash-based split is deterministic; no .test_durations file needed yet. Can add one later via --store-durations for better shard balance.
teknium1
added a commit
that referenced
this pull request
Apr 17, 2026
Target: <2min CI test wall time. Runs the Tests workflow as a 4-way matrix instead of one job. Each shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with -n auto xdist inside. Total effective parallelism: 16 workers across 4 machines (vs 4 workers on 1 machine today). Was previously tried in #11566 and closed — shard 3 hung at 97% complete for 100+ seconds with dozens of E/F markers. Root cause was cross-test pollution exposed by splitting test files across shards (e.g. the three test files that mutated sys.modules['dotenv'] at import time poisoned whichever shard they landed in). That's now fixed by #11453 and #11577: conftest is hermetic, the dotenv stub bombs are removed, and tests no longer depend on each other's env-var side effects. Changes: - pyproject.toml: add pytest-split>=0.9,<1 to dev extras - .github/workflows/tests.yml: 'test' job becomes matrix-split into 4 groups with fail-fast: false. Runs 'pytest --splits 4 --group N'. pytest-split composes with -n auto from pyproject addopts. e2e job is unchanged (already small, 20s). Expected timing: Before: ~4m total (243s test step + ~25s setup) After: ~90-115s total (shard wall time ~60-90s + ~25s setup) Hash-based split is deterministic; no .test_durations file needed yet. Can add one later via --store-durations for better shard balance.
ulasbilgen
pushed a commit
to ulasbilgen/hermes-adhd-agent
that referenced
this pull request
May 1, 2026
…ts (NousResearch#11453) Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit f24e952 (PR NousResearch#11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
ulasbilgen
pushed a commit
to ulasbilgen/hermes-adhd-agent
that referenced
this pull request
May 1, 2026
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
…ts (NousResearch#11453) Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit ebb8b9b (PR NousResearch#11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…ts (NousResearch#11453) Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit 436a735 (PR NousResearch#11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…ts (NousResearch#11453) Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit 436a735 (PR NousResearch#11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
Egavasyug
pushed a commit
to Egavasyug/hermes-agent
that referenced
this pull request
Jun 10, 2026
…ts (NousResearch#11453) Three tests in tests/test_plugin_skills.py and tests/hermes_cli/test_plugins.py used caplog.at_level(logging.WARNING) without specifying a logger. When another test earlier in the same xdist worker touched propagation on tools.skills_tool or hermes_cli.plugins, caplog would miss the warning and the assertion would fail intermittently in CI. These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each), including the recent main failure on commit 4ec914f (PR NousResearch#11398). Fix: pass logger="tools.skills_tool" / logger="hermes_cli.plugins" to caplog.at_level() so the handler attaches directly to the logger under test and capture is independent of global propagation state. Affected tests: - tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_served - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejected - tests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejected No production code change. Verified passing under xdist (-n 4) alongside test_hermes_logging.py (the test most likely to poison the logger state).
Egavasyug
pushed a commit
to Egavasyug/hermes-agent
that referenced
this pull request
Jun 10, 2026
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three tests use
caplog.at_level(logging.WARNING)without specifying a logger. When another test earlier in the same pytest-xdist worker touches propagation ontools.skills_toolorhermes_cli.plugins, caplog misses the warning and the assertion fails intermittently in CI.These three tests accounted for 15 of the last ~30 Tests workflow failures (5 each) — they were the dominant cause of flaky CI including the latest main regression on #11398.
Affected tests
tests/test_plugin_skills.py::TestSkillViewPluginGuards::test_injection_logged_but_servedtests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_empty_name_rejectedtests/hermes_cli/test_plugins.py::TestPluginCommands::test_register_command_builtin_conflict_rejectedFix
Pass
logger="tools.skills_tool"/logger="hermes_cli.plugins"tocaplog.at_level()so the handler attaches directly to the logger under test. Capture now bypasses propagation entirely.No production code change.
Test plan
pytest -n 4alongsidetests/test_hermes_logging.py(the test most likely to poison the logger state — it explicitly resets.propagateand has a comment calling out 'cross-test pollution defense')pytest tests/test_hermes_logging.py tests/test_plugin_skills.py tests/hermes_cli/test_plugins.py -n 4