Skip to content

Commit 904ea58

Browse files
jyaunchesericksoa
jyaunches
and
ericksoa
authored
ci(e2e): move gateway health guard to regressions (#3411)
## Summary Moves the existing #3111 `gateway-health-honest-e2e` coverage guard out of scheduled `nightly-e2e.yaml` and into the new `regression-e2e.yaml` holding-pen workflow. ## Why Failing-test-first guards and high-signal regression anchors should be easy to dispatch while fixes are in flight, but should not automatically keep scheduled nightly red. The new regression workflow gives us a place to keep these guards and periodically review/promote stable ones into nightly. ## What changed - Removed `gateway-health-honest-e2e` from `nightly-e2e.yaml` job list and reporting dependencies. - Added `regression-e2e.yaml` with `gateway-health-honest-e2e` as a manually dispatchable regression job. - Left `test/e2e/test-gateway-health-honest.sh` unchanged. ## Validation - YAML parse for `nightly-e2e.yaml` - YAML parse for `regression-e2e.yaml` Related: #3111 Related PR: #3362 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Removed the automated nightly E2E run for gateway-health-honest and updated nightly job selection and downstream notifications accordingly * Added a manual regression E2E workflow for gateway-health-honest with on-demand job selection, gating logic, concurrency control, and failure-artifact upload * Updated local testing guidance to recommend using the new regression workflow for gateway-health-honest runs [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3411) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
1 parent 30b2061 commit 904ea58

3 files changed

Lines changed: 92 additions & 69 deletions

File tree

‎.coderabbit.yaml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -501,13 +501,14 @@ reviews:
501501
false-positive (detached-zombie isPidAlive + metadata-only
502502
isGatewayHealthy) regardless of OpenShell packaging choices.
503503
504-
**E2E test recommendation:**
505-
- `gateway-health-honest-e2e` — false-positive health check when
506-
the gateway binary crashes on startup (#3111)
504+
**Regression E2E test recommendation:**
505+
- Run `regression-e2e.yaml` with the jobs input set to
506+
gateway-health-honest-e2e to validate false-positive health
507+
checks when the gateway binary crashes on startup (#3111).
507508
508509
To run selectively:
509510
```
510-
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=gateway-health-honest-e2e
511+
gh workflow run regression-e2e.yaml --ref <branch> -f 'jobs=gateway-health-honest-e2e'
511512
```
512513
513514
- path: ".github/workflows/nightly-e2e.yaml"

‎.github/workflows/nightly-e2e.yaml‎

Lines changed: 1 addition & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,6 @@
2222
# Validates real v0.0.36 curl install upgrade into
2323
# the current supported OpenShell with pre-upgrade backup, restored
2424
# agent state, and the same agent type running.
25-
# gateway-health-honest-e2e
26-
# Coverage guard for #3111: onboard must not log
27-
# "Docker-driver gateway is healthy" when the gateway
28-
# binary crashes on startup (detached-zombie + metadata-
29-
# only health check regression).
3025
# hermes-e2e Hermes Agent E2E — install → onboard --agent hermes → health
3126
# probe → live inference. Validates the multi-agent architecture.
3227
# hermes-inference-switch-e2e
@@ -72,7 +67,7 @@ on:
7267
messaging-compatible-endpoint-e2e,
7368
kimi-inference-compat-e2e,
7469
token-rotation-e2e, sandbox-survival-e2e,
75-
openshell-gateway-upgrade-e2e, gateway-health-honest-e2e,
70+
openshell-gateway-upgrade-e2e,
7671
issue-2478-crash-loop-recovery-e2e, hermes-e2e,
7772
hermes-inference-switch-e2e, hermes-discord-e2e,
7873
hermes-slack-e2e, sandbox-operations-e2e, inference-routing-e2e,
@@ -1299,58 +1294,6 @@ jobs:
12991294
/tmp/nemoclaw-e2e-openshell-gateway-compatible-mock.log
13001295
if-no-files-found: ignore
13011296

1302-
# ── Gateway health-honesty E2E ──────────────────────────────
1303-
# Coverage guard for #3111. Issue #3111 reported that onboard prints
1304-
# "✓ Docker-driver gateway is healthy" on Ubuntu 22.04 even though the
1305-
# shipped openshell-gateway binary (GNU-linked against GLIBC 2.38/2.39)
1306-
# crashes immediately on a 22.04 host (GLIBC 2.35).
1307-
#
1308-
# Root cause is platform-independent: the detached child remains a
1309-
# zombie so isPidAlive() returns true, registerDockerDriverGatewayEndpoint()
1310-
# writes metadata without any TCP probe, and isGatewayHealthy() is a
1311-
# string match on openshell CLI output rather than a real health check.
1312-
# Any scenario where the gateway binary fails before serving connections
1313-
# will surface the same false-positive log on ANY Linux host — not just
1314-
# Ubuntu 22.04.
1315-
#
1316-
# This test sabotages the gateway binary with a shim that matches the
1317-
# #3111 failure mode (immediate exit with GLIBC-style stderr) and asserts
1318-
# that onboard does NOT log "healthy" and exits non-zero.
1319-
gateway-health-honest-e2e:
1320-
if: >-
1321-
github.repository == 'NVIDIA/NemoClaw' &&
1322-
(github.event_name != 'workflow_dispatch' ||
1323-
inputs.jobs == '' ||
1324-
contains(format(',{0},', inputs.jobs), ',gateway-health-honest-e2e,'))
1325-
runs-on: ubuntu-latest
1326-
timeout-minutes: 20
1327-
steps:
1328-
- name: Checkout
1329-
uses: actions/checkout@v6
1330-
1331-
- name: Setup Node
1332-
uses: actions/setup-node@v6
1333-
with:
1334-
node-version: "22"
1335-
1336-
- name: Run gateway health-honesty E2E test
1337-
env:
1338-
GITHUB_TOKEN: ${{ github.token }}
1339-
NEMOCLAW_NON_INTERACTIVE: "1"
1340-
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
1341-
run: bash test/e2e/test-gateway-health-honest.sh
1342-
1343-
- name: Upload gateway health-honesty logs on failure
1344-
if: failure()
1345-
uses: actions/upload-artifact@v4
1346-
with:
1347-
name: gateway-health-honest-logs
1348-
path: |
1349-
/tmp/nemoclaw-e2e-gateway-health-honest.log
1350-
/tmp/nemoclaw-e2e-gateway-health-honest-start.log
1351-
/tmp/nemoclaw-e2e-gateway-health-honest-process.log
1352-
if-no-files-found: ignore
1353-
13541297
# ── Hermes rebuild upgrade E2E ──────────────────────────────
13551298
# Same upgrade scenario as OpenClaw but for Hermes Agent.
13561299
rebuild-hermes-e2e:
@@ -1924,7 +1867,6 @@ jobs:
19241867
rebuild-openclaw-e2e,
19251868
upgrade-stale-sandbox-e2e,
19261869
openshell-gateway-upgrade-e2e,
1927-
gateway-health-honest-e2e,
19281870
rebuild-hermes-e2e,
19291871
rebuild-hermes-stale-base-e2e,
19301872
double-onboard-e2e,
@@ -2016,7 +1958,6 @@ jobs:
20161958
rebuild-openclaw-e2e,
20171959
upgrade-stale-sandbox-e2e,
20181960
openshell-gateway-upgrade-e2e,
2019-
gateway-health-honest-e2e,
20201961
rebuild-hermes-e2e,
20211962
rebuild-hermes-stale-base-e2e,
20221963
double-onboard-e2e,
@@ -2156,7 +2097,6 @@ jobs:
21562097
rebuild-openclaw-e2e,
21572098
upgrade-stale-sandbox-e2e,
21582099
openshell-gateway-upgrade-e2e,
2159-
gateway-health-honest-e2e,
21602100
rebuild-hermes-e2e,
21612101
rebuild-hermes-stale-base-e2e,
21622102
double-onboard-e2e,

‎.github/workflows/regression-e2e.yaml‎

Lines changed: 86 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ on:
2121
jobs:
2222
description: >-
2323
Comma-separated regression job names to run (empty = all).
24-
Valid: dashboard-remote-bind-e2e
24+
Valid: dashboard-remote-bind-e2e,gateway-health-honest-e2e
2525
required: false
2626
type: string
2727
default: ""
@@ -41,12 +41,43 @@ concurrency:
4141
cancel-in-progress: true
4242

4343
jobs:
44+
select_regression_jobs:
45+
runs-on: ubuntu-latest
46+
outputs:
47+
dashboard: ${{ steps.select.outputs.dashboard }}
48+
gateway: ${{ steps.select.outputs.gateway }}
49+
steps:
50+
- id: select
51+
env:
52+
JOBS: ${{ inputs.jobs }}
53+
run: |
54+
set -euo pipefail
55+
normalized="$(printf '%s' "$JOBS" | tr -d '[:space:]')"
56+
57+
includes_job() {
58+
case ",${normalized}," in
59+
*",$1,"*) return 0 ;;
60+
*) return 1 ;;
61+
esac
62+
}
63+
64+
if [ -z "$normalized" ] || includes_job "dashboard-remote-bind-e2e"; then
65+
echo "dashboard=true" >> "$GITHUB_OUTPUT"
66+
else
67+
echo "dashboard=false" >> "$GITHUB_OUTPUT"
68+
fi
69+
70+
if [ -z "$normalized" ] || includes_job "gateway-health-honest-e2e"; then
71+
echo "gateway=true" >> "$GITHUB_OUTPUT"
72+
else
73+
echo "gateway=false" >> "$GITHUB_OUTPUT"
74+
fi
75+
4476
dashboard-remote-bind-e2e:
77+
needs: select_regression_jobs
4578
if: >-
4679
github.repository == 'NVIDIA/NemoClaw' &&
47-
(github.event_name != 'workflow_dispatch' ||
48-
inputs.jobs == '' ||
49-
contains(format(',{0},', inputs.jobs), ',dashboard-remote-bind-e2e,'))
80+
needs.select_regression_jobs.outputs.dashboard == 'true'
5081
uses: ./.github/workflows/e2e-branch-validation.yaml
5182
with:
5283
branch: ${{ github.ref_name }}
@@ -55,3 +86,54 @@ jobs:
5586
use_launchable: true
5687
keep_alive: ${{ inputs.keep_alive }}
5788
secrets: inherit
89+
90+
# ── Gateway health-honesty E2E ──────────────────────────────
91+
# Coverage guard for #3111. Issue #3111 reported that onboard prints
92+
# "✓ Docker-driver gateway is healthy" on Ubuntu 22.04 even though the
93+
# shipped openshell-gateway binary (GNU-linked against GLIBC 2.38/2.39)
94+
# crashes immediately on a 22.04 host (GLIBC 2.35).
95+
#
96+
# Root cause is platform-independent: the detached child remains a
97+
# zombie so isPidAlive() returns true, registerDockerDriverGatewayEndpoint()
98+
# writes metadata without any TCP probe, and isGatewayHealthy() is a
99+
# string match on openshell CLI output rather than a real health check.
100+
# Any scenario where the gateway binary fails before serving connections
101+
# will surface the same false-positive log on ANY Linux host — not just
102+
# Ubuntu 22.04.
103+
#
104+
# This test sabotages the gateway binary with a shim that matches the
105+
# #3111 failure mode (immediate exit with GLIBC-style stderr) and asserts
106+
# that onboard does NOT log "healthy" and exits non-zero.
107+
gateway-health-honest-e2e:
108+
needs: select_regression_jobs
109+
if: >-
110+
github.repository == 'NVIDIA/NemoClaw' &&
111+
needs.select_regression_jobs.outputs.gateway == 'true'
112+
runs-on: ubuntu-latest
113+
timeout-minutes: 20
114+
steps:
115+
- name: Checkout
116+
uses: actions/checkout@v6
117+
118+
- name: Setup Node
119+
uses: actions/setup-node@v6
120+
with:
121+
node-version: "22"
122+
123+
- name: Run gateway health-honesty E2E test
124+
env:
125+
GITHUB_TOKEN: ${{ github.token }}
126+
NEMOCLAW_NON_INTERACTIVE: "1"
127+
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
128+
run: bash test/e2e/test-gateway-health-honest.sh
129+
130+
- name: Upload gateway health-honesty logs on failure
131+
if: failure()
132+
uses: actions/upload-artifact@v4
133+
with:
134+
name: gateway-health-honest-logs
135+
path: |
136+
/tmp/nemoclaw-e2e-gateway-health-honest.log
137+
/tmp/nemoclaw-e2e-gateway-health-honest-start.log
138+
/tmp/nemoclaw-e2e-gateway-health-honest-process.log
139+
if-no-files-found: ignore

0 commit comments

Comments
 (0)