Skip to content

Linux installer hardening: discover & gate every service that could bypass the proxy#113

Merged
bglusman merged 2 commits intomainfrom
install/linux-hardening
May 3, 2026
Merged

Linux installer hardening: discover & gate every service that could bypass the proxy#113
bglusman merged 2 commits intomainfrom
install/linux-hardening

Conversation

@bglusman
Copy link
Copy Markdown
Owner

@bglusman bglusman commented May 2, 2026

Why

Today's installer writes a systemd drop-in for openclaw-gateway.service only. We discovered (debugging Librarian on 231) that this leaves a wide gap on Linux:

  • Chrome runs under a separate chrome-cdp.service whose ExecStart= hardcodes /opt/google/chrome/chrome with no --proxy-server flag. Chrome on Linux headless does not honor HTTPS_PROXY env reliably, so the existing env-only drop-in pattern doesn't reach it.
  • OpenClaw's chrome-launcher actively passes --no-proxy-server if no proxy arg is set, defeating env-var detection even when it might otherwise work.
  • Linux had no equivalent of the macOS keychain-CA install added in PR Automate managed OpenClaw local install #107, so even if Chrome did go through the proxy it'd hit TLS errors.

Net effect on Linux: every Calciforge install was silently failing to gate Chrome's HTTPS traffic — exactly the threat the gateway exists to mitigate. This PR closes that gap.

What it does

New module crates/calciforge/src/install/linux_hardening.rs (pure logic, unit-tested) plus a side-effecting orchestrator run_linux_hardening_pass() in executor.rs:

  1. Service discovery — runs systemctl list-units --type=service --state=running on the target and classifies each match as BrowserExecStart (needs --proxy-server injected) vs EnvOnly (env drop-in suffices). Heuristics: browser binaries (chrome/chromium/headless in ExecStart), Node-launched agents with OPENCLAW_* env hints, anything matching *claw* in Description, plus a per-claw user-supplied extras list. Nothing hardcoded; a previously-running service named nonzeroclaw would not be picked up unless it actually matches.

  2. Two drop-in shapes

    • Env-only: existing pattern, used for openclaw-gateway, cdp-guard, agent orchestrators
    • ExecStart override (NEW): parses the current ExecStart= (handles line continuations, comments), idempotently injects --proxy-server=http://127.0.0.1:<port> after the binary path, writes a drop-in with ExecStart= (empty clear) followed by the rewritten command. Re-running the install produces the same content; double-injection guarded.
  3. Linux CA install — parity with macOS PR Automate managed OpenClaw local install #107:

    • System bundle: /usr/local/share/ca-certificates/calciforge-ca.crt + update-ca-certificates
    • Chrome NSS DB(s): auto-installs libnss3-tools (apt/dnf/yum/pacman detected), then certutil -d sql:<path> -A -t "C,," -n "Calciforge MITM CA" -i ca.pem. Discovers NSS DB paths from --user-data-dir=... in discovered Chrome services + the standard ~/.pki/nssdb.
  4. Dependency-ordered restart with health checks — browsers first, orchestrators second, gateways third, others last. After each: systemctl is-active. On failure, captures journalctl -u <service> -n 20 and bails with the context.

  5. Post-install verify-or-fail (CRITICAL) — fetches a configurable test URL through the proxy with --cacert of the system bundle. Asserts both the block-page body marker AND X-Calciforge-Blocked: true header. If the proxy didn't actually gate the request, install fails loud. The whole point of this PR.

  6. Bypass audit (warning, not error) — ss -tnp state established '( dport = :443 )' to flag any agent process talking direct upstream after install. Catches "we missed a service" cases.

  7. Shared-host warning banner — printed at start of the pass:

    Calciforge will configure agent-related system services to route through the local security proxy. It will NOT touch human user sessions on this host. If you share this machine with humans whose own browsing should also be inspected, they need to opt in separately by running calciforge-trust-user <username>.

    That calciforge-trust-user helper is left as a TODO; out of scope for this PR.

Opt-in

The Linux hardening pass is opt-in per claw target via a new linux_hardening: bool field on ClawTarget, defaulted to false. Two reasons:

  • Hardening is destructive (writes drop-ins, installs CAs, restarts services) — safer to require explicit opt-in
  • Keeps all 11 existing apply_remote_config_* mock-SSH tests passing untouched

Two more new fields:

  • linux_hardening_extras: Vec<String> — additional service names to discover beyond the heuristics
  • linux_hardening_verify_url: Option<String> — override the post-install verify URL

CLI parsing in parse_claw_spec updated; wizard updated.

Stats

LOC +1105 / -2 across 6 files
New tests +22 unit tests in install::linux_hardening (ExecStart parsing, idempotent injection, classification heuristics, render shape, package-manager probe)
Build / clippy / fmt clean
Failing tests 1 pre-existing on main (artifacts::tests::detect_mime_type_reports_read_errors — chmod-based, fails when running as root, unrelated to this work)

What's deferred (next-pass work, not blocking)

  1. calciforge-trust-user shell helper (the human-opt-in script the banner promises)
  2. CLI flags --skip-bypass-audit and --include-user-sessions=USER
  3. Doc updates — no docs/installation.md exists yet; macOS CA flow currently lives in install.sh comments. Worth a separate docs PR aligning macOS + Linux.

Verified via

  • 22 new unit tests cover the pure logic (parsing, classification, rendering, idempotency)
  • The drop-in shape was modeled on the manual fix that's currently working on Librarian (231) — same ExecStart= clear + rewrite pattern
  • cargo build -p calciforge, cargo clippy --all-targets -- -D warnings, cargo fmt all clean

Not verified via live linux_hardening = true install against 231 — the pass writes drop-ins for every discovered service and restarts them, which would clobber the manual fix that's currently live on 231 in the middle of debugging. Recommend a controlled test against a fresh host (or 231 after we're done iterating) as the merge gate.

Notes

  • nonzeroclaw.service (you mentioned was running on 231 — turns out it's not a current thing) would NOT be configured by this pass unless you put it in the extras list.
  • API keys still in plaintext in /root/.config/systemd/user/openclaw-gateway.service on 231. Separate problem; recommend rotating.

🤖 Generated with Claude Code

…erride, CA + verify

Linux-side parity for the macOS keychain trust pass that ships in
PR #107. When a claw spec opts in via `linux_hardening=true`, the
installer:

- Discovers all agent-related services on the target via heuristics on
  `systemctl list-units --state=running` (browser binary names,
  `*claw*` description match, `node` + `OPENCLAW_*` env hints,
  operator-supplied extras).
- Writes a systemd drop-in per service. For browser services
  (`chrome-cdp` and friends), uses an ExecStart-override drop-in that
  injects `--proxy-server=...` after the binary path while preserving
  all other args verbatim — Chrome on Linux headless does not honor
  `HTTPS_PROXY` env reliably, so the explicit flag is required. For
  everything else, uses the env-only drop-in shape.
- Installs the Calciforge MITM CA into the system bundle
  (`/usr/local/share/ca-certificates/calciforge-ca.crt` +
  `update-ca-certificates`) and into Chrome's per-user NSS DB at
  `~/.pki/nssdb` via `certutil`. Detects the package manager
  (apt/dnf/yum/pacman) before installing `libnss3-tools` /
  `nss-tools`; bails with a clear error if no supported PM is found.
- Restarts services in dependency order (browsers → orchestrators →
  gateway) and asserts each becomes `active`. On failure, attaches the
  last 20 lines of journalctl output to the bail message.
- Verifies the result by `curl`-ing a known-blocked URL through the
  proxy and asserting the Calciforge block-page marker plus
  `X-Calciforge-Blocked: true` header are returned. Fails loud if any
  service still bypasses — that's the whole point of the pass.
- Audits `ss -tnp` for established :443 connections that aren't going
  to the proxy port and warns (not errors) on hits.
- Prints a prominent banner up front clarifying that Calciforge will
  not touch human-user sessions on shared hosts; opt-in for those is
  deferred to a follow-up `calciforge-trust-user` script (TODO comment
  in code).

The new module `install::linux_hardening` houses the pure logic
(ExecStart parser/rewriter, service classifier, response-block-page
detector, package-manager probe). Side-effecting parts shell out via
`SshClient`. 22 new unit tests exercise the pure functions; existing
`apply_remote_config_*` tests are untouched (the new path is gated on
the new `linux_hardening` field, default false).

Three new fields on `ClawTarget`: `linux_hardening`,
`linux_hardening_extras`, `linux_hardening_verify_url`. CLI parser
accepts `linux_hardening=true,linux_hardening_extras=foo;bar,linux_hardening_verify_url=...`
in `--claw` specs.

The macOS launchd path is untouched. Existing systemd env-only flow
(for `openclaw-gateway` only) is preserved as-is when the new opt-in
flag is not set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 2, 2026 22:22
@qodo-code-review
Copy link
Copy Markdown

ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in Linux/systemd “hardening pass” to the Calciforge installer to ensure all agent-related services (notably headless Chrome) are routed through the local security proxy, installs the MITM CA into system/NSS trust stores, and verifies proxy gating via a known-blocked URL.

Changes:

  • Extend ClawTarget / CLI / wizard to support linux_hardening, linux_hardening_extras, and linux_hardening_verify_url.
  • Introduce install::linux_hardening pure-logic module (classification, ExecStart parsing/rewrite, verification markers, package-manager detection) with unit tests.
  • Add a side-effecting run_linux_hardening_pass() orchestrator in executor.rs to discover services, write drop-ins, install CA trust, restart units, and verify blocking.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
crates/calciforge/src/install/wizard.rs Initializes new linux hardening fields in wizard flows and tests.
crates/calciforge/src/install/model.rs Adds new linux hardening knobs to ClawTarget with docs and updates tests.
crates/calciforge/src/install/mod.rs Exposes the new linux_hardening module.
crates/calciforge/src/install/linux_hardening.rs Implements testable Linux hardening logic (classification, ExecStart rewrite, verify parsing, package-manager detection) + unit tests.
crates/calciforge/src/install/executor.rs Wires opt-in hardening pass into install execution and performs remote system changes via SSH.
crates/calciforge/src/install/cli.rs Parses new linux_hardening* fields from claw specs.

Comment on lines +208 to +223
linux_hardening: kv
.get("linux_hardening")
.map(|v| matches!(v.as_str(), "1" | "true" | "yes"))
.unwrap_or(false),
linux_hardening_extras: kv
.get("linux_hardening_extras")
.map(|value| {
value
.split(';')
.map(str::trim)
.filter(|s| !s.is_empty())
.map(String::from)
.collect()
})
.unwrap_or_default(),
linux_hardening_verify_url: kv.get("linux_hardening_verify_url").cloned(),

// 2) Service discovery via `systemctl list-units` + per-unit cat.
// We use `--no-pager --no-legend --plain` so output is stable for parsing.
let list_cmd = "systemctl list-units --type=service --state=running --no-legend --no-pager --plain || true";
Comment on lines +1118 to +1142
// 2) Service discovery via `systemctl list-units` + per-unit cat.
// We use `--no-pager --no-legend --plain` so output is stable for parsing.
let list_cmd = "systemctl list-units --type=service --state=running --no-legend --no-pager --plain || true";
let list_out = deps.ssh.run(&claw.host, key, list_cmd)?;
if !list_out.success {
bail!(
"failed to list running services on {}: {}",
claw.host,
list_out.stderr.trim()
);
}

let extras_owned: Vec<String> = claw.linux_hardening_extras.clone();
let mut configured: Vec<(String, DropInShape)> = Vec::new();

for line in list_out.stdout.lines() {
let unit = line.split_whitespace().next().unwrap_or("");
if !unit.ends_with(".service") {
continue;
}

// Pull description + ExecStart from `systemctl cat`. May fail for
// generated/transient units; we treat that as "skip silently".
let cat_cmd = format!("systemctl cat {} 2>/dev/null || true", shell_quote(unit));
let cat_out = deps.ssh.run(&claw.host, key, &cat_cmd)?;
Comment on lines +1170 to +1176
let dir = format!("/etc/systemd/system/{unit}.d");
let path = format!("{dir}/10-calciforge-proxy.conf");
let content = render_env_only_dropin(proxy_endpoint, no_proxy)?;
deps.ssh
.run(&claw.host, key, &format!("mkdir -p {}", shell_quote(&dir)))?;
deps.ssh.write_file(&claw.host, key, &path, &content)?;
configured.push((unit.to_string(), shape));
Comment on lines +1235 to +1237
// System CA bundle: copy + refresh.
let install_system_ca = format!(
"sudo -n cp {} /usr/local/share/ca-certificates/calciforge-ca.crt && sudo -n {}",
.as_deref()
.unwrap_or(DEFAULT_VERIFY_URL);
let verify_cmd = format!(
"curl --max-time 10 -sS -i -x {} --cacert /etc/ssl/certs/ca-certificates.crt {} 2>&1 || true",
Comment on lines +313 to +320
[Service]\n\
Environment=\"HTTP_PROXY={proxy}\"\n\
Environment=\"HTTPS_PROXY={proxy}\"\n\
Environment=\"ALL_PROXY={proxy}\"\n\
Environment=\"NO_PROXY={no_proxy}\"\n\
Environment=\"NODE_USE_SYSTEM_CA=1\"\n\
ExecStart=\n\
ExecStart={rewritten}\n",
Comment on lines +146 to +149
// *claw* in description (case-insensitive).
if desc_lc.contains("claw") || name_lc.contains("claw") {
return Some((DropInShape::EnvOnly, "claw description match".into()));
}
Eight fixes from review:

1. Added unit tests for parse_claw_spec linux_hardening fields:
   truthy/falsy values, default-when-absent, extras split-and-trim,
   verify-url passthrough.

2. Dropped the `|| true` from `systemctl list-units`. Real failure
   here should bail loudly, not silently proceed with empty service set.

3. Discovery now uses the SAME systemctl scope (system vs user) as the
   eventual restart. Discovering against system-scope when the OpenClaw
   service is user-scope silently missed the right units.

4. Drop-ins are now written to `~/.config/systemd/user/{unit}.d/`
   when service_mode is user, instead of always `/etc/systemd/system/`.
   The system-scope path required sudo unnecessarily AND wouldn't
   affect `systemctl --user` services at all.

5. PackageManager now exposes `system_ca_anchor_dir()` per distro:
   /usr/local/share/ca-certificates (Debian), /etc/pki/ca-trust/source/anchors
   (RHEL/Fedora), /etc/ca-certificates/trust-source/anchors (Arch).
   Hardcoding Debian's path meant the trust-anchor refresh on RHEL
   never picked up the cert (silently broken on those distros).

6. Verification curl now uses `pm.system_ca_bundle_path()` per distro
   instead of hardcoded `/etc/ssl/certs/ca-certificates.crt`. Fedora's
   bundle lives at /etc/pki/tls/certs/ca-bundle.crt; verification was
   failing even when CA install had succeeded.

7. `render_exec_start_override` now escapes `\` and `"` in the
   Environment="..." values via new `systemd_environment_value` helper.
   A proxy_endpoint or no_proxy list with a quote/backslash in it was
   producing malformed drop-ins (unit fails to start at best, systemd
   directive injection at worst).

8. `*claw*` heuristic now reports which side hit (unit name vs
   description) so the audit log line matches what was actually
   matched. Comment + reason string aligned.
@bglusman bglusman enabled auto-merge (squash) May 3, 2026 01:35
@bglusman bglusman merged commit 4d60dbd into main May 3, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants