ci: split Tests workflow into 4 parallel shards (target <2min total)#11566
ci: split Tests workflow into 4 parallel shards (target <2min total)#11566teknium1 wants to merge 1 commit into
Conversation
Reduces CI wall time by running the test suite as 4 parallel matrix
jobs instead of a single job. Each shard runs ~3,000 tests in
parallel, so total wall time drops from ~4min to ~60-90s.
Changes:
- Add pytest-split to dev extras (deterministic test splitting,
composes with pytest-xdist's -n auto inside each shard).
- Matrix-split tests.yml 'test' job into 4 groups. Each shard runs
'pytest ... --splits 4 --group N' and parallelizes inside with
the -n auto already in pyproject.toml's addopts.
- fail-fast: false so all shards finish even if one fails
(consistent with current behavior when there's no matrix).
Expected CI timing:
Before: 243s single-job (4m03s)
After: ~60-90s per shard in parallel + ~25s install overhead
\u2192 total CI ~90-115s
No test-file changes. Deterministic hash-based distribution (no
.test_durations file yet; can add one later for better balance).
The e2e job is unchanged — it's already small (20s) and runs
separately.
|
|
Closing — matrix split surfaced pre-existing cross-test pollution that caused shard 3 to hang for 100s+ on GHA. The root cause (local-vs-CI env divergence + orphan processes) needs to be fixed first. Reopening once hermetic test infra lands. |
Fixes the recurring 'works locally, fails in CI' (and vice versa) class
of flakes by making tests hermetic and providing a canonical local runner
that matches CI's environment.
## Layer 1 — hermetic conftest.py (tests/conftest.py)
Autouse fixture now unsets every credential-shaped env var before every
test, so developer-local API keys can't leak into tests that assert
'auto-detect provider when key present'.
Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD,
_CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of
credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID,
FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that
change auto-detect behavior.
Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET,
HERMES_SESSION_*, etc.) that mutate agent behavior.
Also:
- Redirects HOME to a per-test tempdir (not just HERMES_HOME), so
code reading ~/.hermes/* directly can't touch the real dir.
- Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to
match CI's deterministic runtime.
The old _isolate_hermes_home fixture name is preserved as an alias so
any test that yields it explicitly still works.
## Layer 2 — scripts/run_tests.sh canonical runner
'Always use scripts/run_tests.sh, never call pytest directly' is the
new rule (documented in AGENTS.md). The script:
- Unsets all credential env vars (belt-and-suspenders for callers
who bypass conftest — e.g. IDE integrations)
- Pins TZ/LANG/PYTHONHASHSEED
- Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on
a 20-core workstation surfaces test-ordering flakes CI will never
see, causing the infamous 'passes in CI, fails locally' drift)
- Finds the venv in .venv, venv, or main checkout's venv
- Passes through arbitrary pytest args
Installs pytest-split on demand so the script can also be used to run
matrix-split subsets locally for debugging.
## Remove 3 module-level dotenv stubs that broke test isolation
tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a
module-level:
if 'dotenv' not in sys.modules:
fake_dotenv = types.ModuleType('dotenv')
fake_dotenv.load_dotenv = lambda *a, **kw: None
sys.modules['dotenv'] = fake_dotenv
This patches sys.modules['dotenv'] to a fake at import time with no
teardown. Under pytest-xdist LoadScheduling, whichever worker collected
one of these files first poisoned its sys.modules; subsequent tests in
the same worker that imported load_dotenv transitively (e.g.
test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and
saw their assertions fail.
dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml),
so the defensive stub was never needed. Removed.
## Validation
- tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4
failures in test_env_loader.py before this fix)
- tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py,
tests/test_hermes_logging.py combined: 123 passed (the caplog
regression tests from PR #11453 still pass)
- Local full run shows no F/E clusters in the 0-55% range that were
previously present before the conftest hardening
## Background
See AGENTS.md 'Testing' section for the full list of drift sources
this closes. Matrix split (closed as #11566) will be re-attempted
once this foundation lands — cross-test pollution was the root cause
of the shard-3 hang in that PR.
….sh (#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR #11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as #11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR #11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
Target: <2min CI test wall time. Runs the Tests workflow as a 4-way matrix instead of one job. Each shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with -n auto xdist inside. Total effective parallelism: 16 workers across 4 machines (vs 4 workers on 1 machine today). Was previously tried in #11566 and closed — shard 3 hung at 97% complete for 100+ seconds with dozens of E/F markers. Root cause was cross-test pollution exposed by splitting test files across shards (e.g. the three test files that mutated sys.modules['dotenv'] at import time poisoned whichever shard they landed in). That's now fixed by #11453 and #11577: conftest is hermetic, the dotenv stub bombs are removed, and tests no longer depend on each other's env-var side effects. Changes: - pyproject.toml: add pytest-split>=0.9,<1 to dev extras - .github/workflows/tests.yml: 'test' job becomes matrix-split into 4 groups with fail-fast: false. Runs 'pytest --splits 4 --group N'. pytest-split composes with -n auto from pyproject addopts. e2e job is unchanged (already small, 20s). Expected timing: Before: ~4m total (243s test step + ~25s setup) After: ~90-115s total (shard wall time ~60-90s + ~25s setup) Hash-based split is deterministic; no .test_durations file needed yet. Can add one later via --store-durations for better shard balance.
Target: <2min CI test wall time. Runs the Tests workflow as a 4-way matrix instead of one job. Each shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with -n auto xdist inside. Total effective parallelism: 16 workers across 4 machines (vs 4 workers on 1 machine today). Was previously tried in #11566 and closed — shard 3 hung at 97% complete for 100+ seconds with dozens of E/F markers. Root cause was cross-test pollution exposed by splitting test files across shards (e.g. the three test files that mutated sys.modules['dotenv'] at import time poisoned whichever shard they landed in). That's now fixed by #11453 and #11577: conftest is hermetic, the dotenv stub bombs are removed, and tests no longer depend on each other's env-var side effects. Changes: - pyproject.toml: add pytest-split>=0.9,<1 to dev extras - .github/workflows/tests.yml: 'test' job becomes matrix-split into 4 groups with fail-fast: false. Runs 'pytest --splits 4 --group N'. pytest-split composes with -n auto from pyproject addopts. e2e job is unchanged (already small, 20s). Expected timing: Before: ~4m total (243s test step + ~25s setup) After: ~90-115s total (shard wall time ~60-90s + ~25s setup) Hash-based split is deterministic; no .test_durations file needed yet. Can add one later via --store-durations for better shard balance.
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
….sh (NousResearch#11577) * test: make test env hermetic; enforce CI parity via scripts/run_tests.sh Fixes the recurring 'works locally, fails in CI' (and vice versa) class of flakes by making tests hermetic and providing a canonical local runner that matches CI's environment. ## Layer 1 — hermetic conftest.py (tests/conftest.py) Autouse fixture now unsets every credential-shaped env var before every test, so developer-local API keys can't leak into tests that assert 'auto-detect provider when key present'. Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID, FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that change auto-detect behavior. Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET, HERMES_SESSION_*, etc.) that mutate agent behavior. Also: - Redirects HOME to a per-test tempdir (not just HERMES_HOME), so code reading ~/.hermes/* directly can't touch the real dir. - Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to match CI's deterministic runtime. The old _isolate_hermes_home fixture name is preserved as an alias so any test that yields it explicitly still works. ## Layer 2 — scripts/run_tests.sh canonical runner 'Always use scripts/run_tests.sh, never call pytest directly' is the new rule (documented in AGENTS.md). The script: - Unsets all credential env vars (belt-and-suspenders for callers who bypass conftest — e.g. IDE integrations) - Pins TZ/LANG/PYTHONHASHSEED - Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on a 20-core workstation surfaces test-ordering flakes CI will never see, causing the infamous 'passes in CI, fails locally' drift) - Finds the venv in .venv, venv, or main checkout's venv - Passes through arbitrary pytest args Installs pytest-split on demand so the script can also be used to run matrix-split subsets locally for debugging. ## Remove 3 module-level dotenv stubs that broke test isolation tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a module-level: if 'dotenv' not in sys.modules: fake_dotenv = types.ModuleType('dotenv') fake_dotenv.load_dotenv = lambda *a, **kw: None sys.modules['dotenv'] = fake_dotenv This patches sys.modules['dotenv'] to a fake at import time with no teardown. Under pytest-xdist LoadScheduling, whichever worker collected one of these files first poisoned its sys.modules; subsequent tests in the same worker that imported load_dotenv transitively (e.g. test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and saw their assertions fail. dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml), so the defensive stub was never needed. Removed. ## Validation - tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4 failures in test_env_loader.py before this fix) - tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py, tests/test_hermes_logging.py combined: 123 passed (the caplog regression tests from PR NousResearch#11453 still pass) - Local full run shows no F/E clusters in the 0-55% range that were previously present before the conftest hardening ## Background See AGENTS.md 'Testing' section for the full list of drift sources this closes. Matrix split (closed as NousResearch#11566) will be re-attempted once this foundation lands — cross-test pollution was the root cause of the shard-3 hang in that PR. * fix(conftest): don't redirect HOME — it broke CI subprocesses PR NousResearch#11577's autouse fixture was setting HOME to a per-test tempdir. CI started timing out at 97% complete with dozens of E/F markers and orphan python processes at cleanup — tests (or transitive deps) spawn subprocesses that expect a stable HOME, and the redirect broke them in non-obvious ways. Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift fixes) are unchanged and still in place. HERMES_HOME redirection is also unchanged — that's the canonical way to isolate tests from ~/.hermes/, not HOME. Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"` instead of `get_hermes_home()` is a bug to fix at the callsite, not something to paper over in conftest.
Summary
Splits the main
Testsworkflow into 4 parallel matrix jobs via pytest-split. Each shard runs ~3,000 tests in parallel instead of one job running all 12,098.Expected impact: CI wall time drops from ~4min to ~90-115s total. Hits the <2min target Teknium asked for.
Why this approach
Per-platform test dedup turned out to be a dead end \u2014 each platform's SDK is too different (slack-bolt / python-telegram-bot / mautrix / discord.py / Baileys JS bridge). Tests look similar in shape but use incompatible mock setups.
The biggest lever for <2min is CI parallelism, not test deletion. Matrix split gives us ~4x parallelism for free:
-n auto(already in pyproject.toml addopts) parallelizes with xdist on the 4 coresChanges
pyproject.toml: addpytest-split>=0.9,<1to dev extras.github/workflows/tests.yml:testjob becomes matrix-split into 4 shards. Each shard runspytest --splits 4 --group Nfail-fast: false\u2014 all shards finish even if one fails, matching current behaviorLocal verification
Ran shard 1 locally on the current main + this change:
37s wall timefor 3,025 tests on my 16-core box. On ubuntu-latest with 4 cores, expected ~60-90s per shard. All 4 shards in parallel \u2248 max of slowest \u2248 60-90s.3 local-only failures during the run (cross-test pollution from other test ordering) \u2014 same pattern as CI-vs-local environmental flakes. CI is the source of truth.
Timing math
-n auto= 4-n auto= 4Follow-ups
.test_durationsfrom--store-durationsfor better shard balance (optional; current hash split is already within ~0.1% of equal).