Rebased + CI triage.

Rebased onto current origin/main (32899279a) via clean cherry-pick — the branch had drifted to 284 files / -16590 lines of staleness vs main; it's now back to the focused 6-file change. 80/80 local unit tests pass (tests/hermes_cli/test_service_manager.py + test_container_boot.py).

CI triage of the prior build-amd64 failure — one real failure: tests/docker/test_gateway_run_supervised.py::test_supervised_gateway_stdout_reaches_docker_logs. The ⚕ Hermes Gateway Starting banner did not reach docker logs after the 8s wait, though the gateway booted ("gateway is now running under s6 supervision" was present).

Root-cause notes for reviewers:

• The exec s6-setuidgid hermes s6-log 1 n10 s1000000 T "$log_dir" line — incl. the 1 stdout-tee directive — is byte-identical to main; this PR only changes $log_dir to a per-container nested path and adds a mkdir -p/chown for the deeper path.
• The #34480 log/down marker is correctly gated behind if not start:, so it does not affect the owned/running profile the test exercises.
• Most plausible regression: a startup race — the extra mkdir -p "$base_dir/$container_id" + chown -R on the deeper nested path delays s6-log's tee just enough that the gateway's banner is written before the stdout pipe has a consumer, so it lands in current but not docker logs.

I've re-triggered CI on the clean commit to confirm whether the failure reproduces post-rebase before proposing a fix (candidate: pre-create + chown the nested $log_dir at register time in container_boot/_render_log_run, so s6-log doesn't mkdir on the hot path). Will follow up with the verified fix rather than a blind patch to s6 internals.

Pushed 7b0fa35f4: the clean rebase confirmed build-amd64 still failed on test_supervised_gateway_stdout_reaches_docker_logs — so this was not staleness, it's the real per-container startup race.

Root cause: the log/run script from _render_log_run runs mkdir -p + recursive chown on the hot path every time s6-supervise (re)starts the logger. On a cold boot the gateway writes its rich-console banner to stdout before the logger pipeline finishes that mkdir/chown and takes over the fd, so the banner is lost and never reaches docker logs.

Fix: _register_service (cont-init.d, runs as root before s6-svscan starts the loggers) now pre-creates + chowns the per-container nested log dir up front via the new _precreate_log_dir / _resolve_container_log_dir helpers. The resolver mirrors _render_log_run's runtime resolution exactly (HERMES_CONTAINER_ID → $HOSTNAME → hostname → /etc/hostname, gated by HERMES_GATEWAY_LOG_PER_CONTAINER), so the pre-created dir is the same path s6-log locks. With the dir already present and hermes-owned, the script's mkdir -p is a no-op and s6-log forwards stdout immediately.

Pre-creation is best-effort; the script's own mkdir/chown remain as a fallback so behavior never regresses off-container. Added 5 unit tests (resolution: nested / hostname fallback / flat opt-out; pre-creation: direct + end-to-end via reconcile). Local suite: 31 passed, ruff clean. Waiting on build-amd64 to confirm green.

Found the real root cause by reading the CI docker logs (491662877):

The CI docker logs showed only stderr reached the stream — skill-sync output, s6-rc service messages, the → gateway is now running under s6 supervision line, and Python-logging WARNINGs. The ⚕ Hermes Gateway Starting... banner was absent from both docker logs and the rotated current file (which was empty).

Root cause: under s6 supervision the gateway child's stdout is a pipe to s6-log, not a TTY, so Python block-buffers it. The banner is printed but sits in Python's stdout buffer and is never handed to s6-log's 1 stdout-tee within the window — in practice until unrelated later output happens to fill the ~4-8 KiB buffer. Python's logging defaults to stderr (which is why errors/WARNINGs showed but the stdout banner didn't), confirming the buffering diagnosis.

Fix: sys.stdout.flush() immediately after the banner print()s in gateway.py, so the banner is pushed to s6-log at startup and tee'd to /init's stdout = docker logs.

I also kept the earlier container_boot change (pre-create + chown the per-container log dir at register time) as defense-in-depth — it removes the mkdir/chown from the s6-log hot path — but the flush is the load-bearing fix. Waiting on build-amd64 to confirm green; not claiming fixed until CI says so.

The underlying problem is weird, as this two-container setup is default in the docker-compose provided by the project:

• https://github.com/NousResearch/hermes-agent/blob/1fc7bdc5e64e052bc61d3ddb9e6f96cf6c7461dc/docker-compose.yml

The two-container setup in the default docker-compose is a bug (see #36235). 5671059 deprecated the two-container setup in favor of the HERMES_DASHBOARD=1 flag, but failed to update the example docker-compose.yml. https://hermes-agent.nousresearch.com/docs/user-guide/docker explicitly states "Running the dashboard as a separate container is not supported: its gateway-liveness detection requires a shared PID namespace with the gateway process." and "Never run two Hermes gateway containers against the same data directory simultaneously — session files and memory stores are not designed for concurrent write access.".

Rebased clean + the deprecation question, head-on

Rebase: Rebased onto current origin/main via clean cherry-pick of the three focused commits — the branch had ballooned to ~1,300 files of staleness against main; it's now back to the intended 7 files / +390 / −7. mergeable is no longer CONFLICTING. Local unit suite green: tests/hermes_cli/test_service_manager.py + tests/hermes_cli/test_container_boot.py → 90 passed.

The bigger question @glensc / @hulet raised — and I think you're right. Thank you both for the pointer; it changes my read of this PR.

The two structural changes here (per-container log-dir nesting for #34457, and the log/down marker for #34480) exist to make a gateway + dashboard two-container stack sharing one /opt/data volume stop crash-looping on the s6-log lock. But current main now documents that exact topology as unsupported, in three places in website/docs/user-guide/docker.md:

• L135: "Running the dashboard as a separate container is not supported: its gateway-liveness detection requires a shared PID namespace with the gateway process."
• L174: "Never run two Hermes gateway containers against the same data directory simultaneously — session files and memory stores are not designed for concurrent write access."
• L257: "never point two containers at the same ~/.hermes directory simultaneously … cross-container sharing of a data volume corrupts session files and memory stores."

Commit 5671059 ("feat(docker): launch dashboard as side-process via HERMES_DASHBOARD=1") deprecated the side-car dashboard container and collapsed the two-service Compose example into a single service. Issue #34457's reporter hit the lock loop precisely because they ran the now-deprecated gateway+dashboard-sidecar-on-shared-volume stack.

So I no longer think the structural parts of this PR should land as-is. Making the shared-volume topology log "more safely" works against the project's documented position that the topology corrupts data — it risks signaling that shared volumes are supported when the maintainers have deliberately moved away from them. The clean fix for #34457 is the single-container HERMES_DASHBOARD=1 path that already exists on main, plus (optionally) a doc/Compose-example update so users aren't pointed at the deprecated layout (the example docker-compose.yml still shows two services — that's arguably the real remaining bug, and it's tracked separately).

The one piece that is topology-independent is the sys.stdout.flush() in gateway.py (the startup banner reaching docker logs under s6). In full honesty: CI shows that flush alone does not fix test_supervised_gateway_stdout_reaches_docker_logs — in the latest build-amd64 run the banner reached neither docker logs nor the rotated current file, so there's a deeper buffering/hand-off issue in the supervised gateway run path that this PR does not fully solve. I don't want to claim a fix CI contradicts.

My recommendation: close this PR in favor of the supported single-container HERMES_DASHBOARD=1 path, and (if useful) let me open a small, separate, topology-independent PR for just the docker logs banner issue once I've root-caused why the flush isn't sufficient — plus possibly a docs/Compose-example PR to remove the deprecated two-container example so the next person doesn't fall into #34457.

I've left the branch in a clean, rebased, scoped state so it's easy to review if you'd rather salvage any part — but I'll defer to you on disposition and won't close it myself. Does closing in favor of HERMES_DASHBOARD=1 (and a follow-up docs/Compose cleanup) match how you'd like #34457 / #34480 resolved?

After rebasing this clean onto current main and digging into the CI failure, I'm recommending closure of this PR — and want to lay out why transparently for the maintainers.

The build-amd64 failure (test_supervised_gateway_stdout_reaches_docker_logs) is not fixable at this layer. The startup banner reaches neither docker logs nor the log file even with the flush fix, which points to a deeper s6 hand-off issue rather than the per-container log lock this PR targets.

More fundamentally: the structural parts of this PR (#34457 per-container log nesting + #34480 logger-down marker) are designed around the gateway + dashboard two-container shared-volume topology — which commit 5671059 deprecated in favor of the single-container HERMES_DASHBOARD=1 flag (see docs L135/174/257 and #36235). The setup this PR hardens is on its way out, so investing further here is likely wasted effort.

The topology-independent flush fix is preserved in the diff if any of it is useful elsewhere, but on its own it's CI-proven insufficient.

Recommendation: close this in favor of the single-container path + a separate docs/compose cleanup to remove the deprecated two-container example. Happy to open that follow-up if it'd help. Deferring the actual close to you all rather than doing it unilaterally. 🙏