Skip to content

Commit 29d4581

Browse files
SantiagoDePoloniaSantiagoDePolonia
authored
chore: updated release e2e scenarios and added runner (#203)
1 parent cc42d4a commit 29d4581

2 files changed

Lines changed: 516 additions & 37 deletions

File tree

‎tests/e2e/release-e2e-scenarios.md‎

Lines changed: 76 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,54 @@ These scenarios are prepared for execution across these local gateways:
88
- `http://localhost:18082` - MongoDB-backed smoke gateway
99
- `http://localhost:18083` - SQLite-backed guardrail gateway
1010

11+
## Recommended runner
12+
13+
Use the checked-in runner to execute this matrix without manually replaying the
14+
shared setup blocks:
15+
16+
```bash
17+
tests/e2e/run-release-e2e.sh
18+
tests/e2e/run-release-e2e.sh --list
19+
tests/e2e/run-release-e2e.sh --from S54 --to S58
20+
tests/e2e/run-release-e2e.sh --scenario S61,S62,S70 --keep-artifacts
21+
```
22+
23+
The runner treats this markdown file as the source of truth, replays the setup
24+
blocks automatically for each scenario, writes a raw log plus a TSV summary
25+
under `QA_RUN_DIR` (default: `/tmp/gomodel-release-e2e-$QA_SUFFIX`), and
26+
supports partial reruns.
27+
28+
Stateful note:
29+
30+
- `S13`-`S60` mutate shared aliases/files/batches
31+
- `S64`-`S79` mutate managed keys, workflows, and auth artifacts
32+
- For stateful partial reruns, prefer a contiguous range that includes the
33+
prerequisite setup scenarios, or rerun with the same `--qa-suffix` and
34+
`--keep-artifacts`
35+
1136
## Common environment
1237

1338
```bash
39+
export QA_SUFFIX="${QA_SUFFIX:-$(date +%s)-$$}"
40+
export QA_RUN_DIR="${QA_RUN_DIR:-/tmp/gomodel-release-e2e-$QA_SUFFIX}"
41+
export QA_OPENAI_ALIAS="${QA_OPENAI_ALIAS:-qa-gpt-latest-$QA_SUFFIX}"
42+
export QA_ANTHROPIC_ALIAS="${QA_ANTHROPIC_ALIAS:-qa-sonnet-thinking-$QA_SUFFIX}"
43+
44+
mkdir -p "$QA_RUN_DIR"
45+
1446
export BASE_URL=http://localhost:18080
1547
export PG_BASE_URL=http://localhost:18081
1648
export MONGO_BASE_URL=http://localhost:18082
1749
export GR_BASE_URL=http://localhost:18083
1850

19-
cat > /tmp/qa-openai-batch.jsonl <<'EOF'
51+
cat > "$QA_RUN_DIR/qa-openai-batch.jsonl" <<'EOF'
2052
{"custom_id":"qa-batch-1","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"Reply with exactly QA_BATCH_FILE_OK"}],"max_tokens":20}}
2153
EOF
2254

23-
printf 'qa file payload\n' > /tmp/qa-upload.txt
55+
printf 'qa file payload\n' > "$QA_RUN_DIR/qa-upload.txt"
2456

25-
export BATCH_FILE=/tmp/qa-openai-batch.jsonl
26-
export UPLOAD_FILE=/tmp/qa-upload.txt
57+
export BATCH_FILE="$QA_RUN_DIR/qa-openai-batch.jsonl"
58+
export UPLOAD_FILE="$QA_RUN_DIR/qa-upload.txt"
2759
```
2860

2961
## Auth-enabled runtime environment
@@ -42,19 +74,23 @@ set -a
4274
source .env
4375
set +a
4476

77+
export QA_SUFFIX="${QA_SUFFIX:-$(date +%s)-$$}"
78+
export QA_RUN_DIR="${QA_RUN_DIR:-/tmp/gomodel-release-e2e-$QA_SUFFIX}"
79+
80+
mkdir -p "$QA_RUN_DIR"
81+
4582
export AUTH_BASE_URL=http://localhost:8080
4683
export ADMIN_AUTH_HEADER="Authorization: Bearer $GOMODEL_MASTER_KEY"
4784

48-
export QA_SUFFIX="${QA_SUFFIX:-$(date +%s)}"
4985
export QA_AUTH_KEY_NAME="qa-release-auth-key-$QA_SUFFIX"
5086
export QA_WORKFLOW_NAME="qa-release-workflow-$QA_SUFFIX"
5187
export QA_USER_PATH="/team/release/e2e/$QA_SUFFIX"
5288
export QA_CACHE_USER_PATH="/team/cache/e2e/$QA_SUFFIX"
5389

54-
export QA_AUTH_KEY_JSON="/tmp/qa-release-auth-key-$QA_SUFFIX.json"
55-
export QA_AUTH_KEY_VALUE_FILE="/tmp/qa-release-auth-key-$QA_SUFFIX.token"
56-
export QA_WORKFLOW_JSON="/tmp/qa-release-workflow-$QA_SUFFIX.json"
57-
export QA_WORKFLOW_ID_FILE="/tmp/qa-release-workflow-$QA_SUFFIX.id"
90+
export QA_AUTH_KEY_JSON="$QA_RUN_DIR/qa-release-auth-key.json"
91+
export QA_AUTH_KEY_VALUE_FILE="$QA_RUN_DIR/qa-release-auth-key.token"
92+
export QA_WORKFLOW_JSON="$QA_RUN_DIR/qa-release-workflow.json"
93+
export QA_WORKFLOW_ID_FILE="$QA_RUN_DIR/qa-release-workflow.id"
5894

5995
export QA_AUTH_REQ1="qa-auth-cacheoff-$QA_SUFFIX-1"
6096
export QA_AUTH_REQ2="qa-auth-cacheoff-$QA_SUFFIX-2"
@@ -68,7 +104,9 @@ cleanup_release_auth_artifacts() {
68104
}
69105

70106
cleanup_release_auth_artifacts
71-
trap 'cleanup_release_auth_artifacts' EXIT
107+
if [ "${RUN_RELEASE_E2E_PERSIST_STATE:-0}" != "1" ]; then
108+
trap 'cleanup_release_auth_artifacts' EXIT
109+
fi
72110
```
73111

74112
## 1. Infra, discovery, observability
@@ -181,7 +219,7 @@ curl -sS "$BASE_URL/admin/api/v1/aliases" | jq '.'
181219
Creates an alias pointing to the newest cheap OpenAI model.
182220

183221
```bash
184-
curl -sS -X PUT "$BASE_URL/admin/api/v1/aliases/qa-gpt-latest" \
222+
curl -sS -X PUT "$BASE_URL/admin/api/v1/aliases/$QA_OPENAI_ALIAS" \
185223
-H 'Content-Type: application/json' \
186224
-d '{"target_model":"gpt-4.1-nano","target_provider":"openai","description":"QA alias for release e2e"}' \
187225
| jq '.'
@@ -192,7 +230,7 @@ curl -sS -X PUT "$BASE_URL/admin/api/v1/aliases/qa-gpt-latest" \
192230
Creates an alias pointing to `claude-sonnet-4-6`.
193231

194232
```bash
195-
curl -sS -X PUT "$BASE_URL/admin/api/v1/aliases/qa-sonnet-thinking" \
233+
curl -sS -X PUT "$BASE_URL/admin/api/v1/aliases/$QA_ANTHROPIC_ALIAS" \
196234
-H 'Content-Type: application/json' \
197235
-d '{"target_model":"claude-sonnet-4-6","target_provider":"anthropic","description":"QA alias for anthropic reasoning"}' \
198236
| jq '.'
@@ -204,7 +242,7 @@ Checks that aliases are discoverable through the public model list.
204242

205243
```bash
206244
curl -sS "$BASE_URL/v1/models" \
207-
| jq -r '.data[] | select(.id=="qa-gpt-latest" or .id=="qa-sonnet-thinking") | {id,owned_by}'
245+
| jq -r --arg openai_alias "$QA_OPENAI_ALIAS" --arg anthropic_alias "$QA_ANTHROPIC_ALIAS" '.data[] | select(.id==$openai_alias or .id==$anthropic_alias) | {id,owned_by}'
208246
```
209247

210248
## 3. Chat completions
@@ -304,7 +342,7 @@ Checks alias resolution for OpenAI models.
304342
```bash
305343
curl -sS "$BASE_URL/v1/chat/completions" \
306344
-H 'Content-Type: application/json' \
307-
-d '{"model":"qa-gpt-latest","messages":[{"role":"user","content":"Reply with exactly QA_ALIAS_OK"}],"max_tokens":20}' \
345+
-d "{\"model\":\"$QA_OPENAI_ALIAS\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly QA_ALIAS_OK\"}],\"max_tokens\":20}" \
308346
| jq '{model,provider,answer:.choices[0].message.content}'
309347
```
310348

@@ -315,7 +353,7 @@ Checks alias resolution for Anthropic models plus reasoning.
315353
```bash
316354
curl -sS "$BASE_URL/v1/chat/completions" \
317355
-H 'Content-Type: application/json' \
318-
-d '{"model":"qa-sonnet-thinking","messages":[{"role":"user","content":"Reply with exactly QA_ALIAS_SONNET_OK"}],"reasoning":{"effort":"high"},"max_tokens":128}' \
356+
-d "{\"model\":\"$QA_ANTHROPIC_ALIAS\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly QA_ALIAS_SONNET_OK\"}],\"reasoning\":{\"effort\":\"high\"},\"max_tokens\":128}" \
319357
| jq '{model,provider,answer:.choices[0].message.content}'
320358
```
321359

@@ -382,7 +420,7 @@ Checks alias resolution on `/v1/responses`.
382420
```bash
383421
curl -sS "$BASE_URL/v1/responses" \
384422
-H 'Content-Type: application/json' \
385-
-d '{"model":"qa-gpt-latest","input":"Reply with exactly QA_RESP_ALIAS_OK","max_output_tokens":20}' \
423+
-d "{\"model\":\"$QA_OPENAI_ALIAS\",\"input\":\"Reply with exactly QA_RESP_ALIAS_OK\",\"max_output_tokens\":20}" \
386424
| jq '{status,model,provider,output}'
387425
```
388426

@@ -559,10 +597,10 @@ curl -sS "$BASE_URL/v1/batches" \
559597
Checks that a batch provider mismatch is rejected before upstream submission.
560598

561599
```bash
562-
cat > /tmp/qa-mixed-provider-batch.jsonl <<'EOF'
563-
{"custom_id":"qa-mixed-1","method":"POST","url":"/v1/chat/completions","body":{"model":"qa-sonnet-thinking","messages":[{"role":"user","content":"Reply with exactly QA_MIXED_ALIAS_BATCH"}],"max_tokens":32}}
600+
cat > "$QA_RUN_DIR/qa-mixed-provider-batch.jsonl" <<EOF
601+
{"custom_id":"qa-mixed-1","method":"POST","url":"/v1/chat/completions","body":{"model":"$QA_ANTHROPIC_ALIAS","messages":[{"role":"user","content":"Reply with exactly QA_MIXED_ALIAS_BATCH"}],"max_tokens":32}}
564602
EOF
565-
FILE_ID=$(curl -sS "$BASE_URL/v1/files?provider=openai" -F purpose=batch -F file=@/tmp/qa-mixed-provider-batch.jsonl | jq -r '.id')
603+
FILE_ID=$(curl -sS "$BASE_URL/v1/files?provider=openai" -F purpose=batch -F "file=@$QA_RUN_DIR/qa-mixed-provider-batch.jsonl" | jq -r '.id')
566604
curl -sS -i "$BASE_URL/v1/batches" \
567605
-H 'Content-Type: application/json' \
568606
-d "{\"input_file_id\":\"$FILE_ID\",\"endpoint\":\"/v1/chat/completions\",\"completion_window\":\"24h\",\"metadata\":{\"provider\":\"openai\",\"suite\":\"qa-mixed-provider\"}}"
@@ -698,50 +736,50 @@ curl -sS "$GR_BASE_URL/admin/api/v1/usage/summary" | jq '.'
698736

699737
### S59 Delete OpenAI alias
700738

701-
Removes `qa-gpt-latest`.
739+
Removes the per-run OpenAI alias.
702740

703741
```bash
704-
curl -sS -X DELETE -i "$BASE_URL/admin/api/v1/aliases/qa-gpt-latest"
742+
curl -sS -X DELETE -i "$BASE_URL/admin/api/v1/aliases/$QA_OPENAI_ALIAS"
705743
```
706744

707745
### S60 Delete Anthropic alias
708746

709-
Removes `qa-sonnet-thinking`.
747+
Removes the per-run Anthropic alias.
710748

711749
```bash
712-
curl -sS -X DELETE -i "$BASE_URL/admin/api/v1/aliases/qa-sonnet-thinking"
750+
curl -sS -X DELETE -i "$BASE_URL/admin/api/v1/aliases/$QA_ANTHROPIC_ALIAS"
713751
```
714752

715753
## 11. Audit failure coverage
716754

717-
### S61 Unsupported translated model is still written to audit log
755+
### S61 Unsupported translated model is still visible in audit log search
718756

719-
Checks that a rejected translated request is still visible in audit logs with the requested model and error type.
757+
Checks that a rejected translated request is still visible in audit-log search by request ID with the requested model and error type.
720758

721759
```bash
722-
REQUEST_ID="qa-invalid-model-$(date +%s)"
760+
REQUEST_ID="qa-invalid-model-$(date +%s)-$$"
723761
curl -sS -i "$BASE_URL/v1/chat/completions" \
724762
-H 'Content-Type: application/json' \
725763
-H "X-Request-ID: $REQUEST_ID" \
726764
-d '{"model":"does-not-exist-model","messages":[{"role":"user","content":"Reply with exactly QA_INVALID_MODEL"}],"max_tokens":20}' && echo
727765
sleep 6
728-
curl -sS "$BASE_URL/admin/api/v1/audit/log?request_id=$REQUEST_ID&limit=5" \
729-
| jq '{total,entries:(.entries|map({request_id,path,model,resolved_model,provider,status_code,error_type}))}'
766+
curl -sS "$BASE_URL/admin/api/v1/audit/log?search=$REQUEST_ID&limit=5" \
767+
| jq --arg request_id "$REQUEST_ID" '{total:(.entries|map(select(.request_id==$request_id))|length),entries:(.entries|map(select(.request_id==$request_id))|map({request_id,path,model,resolved_model,provider,status_code,error_type}))}'
730768
```
731769

732-
### S62 Unsupported passthrough provider is still written to audit log
770+
### S62 Unsupported passthrough provider is still visible in audit log search
733771

734-
Checks that a rejected passthrough request is still visible in audit logs with the provider parsed from the path.
772+
Checks that a rejected passthrough request is still visible in audit-log search by request ID with the provider parsed from the path.
735773

736774
```bash
737-
REQUEST_ID="qa-invalid-provider-$(date +%s)"
775+
REQUEST_ID="qa-invalid-provider-$(date +%s)-$$"
738776
curl -sS -i "$BASE_URL/p/not-a-real-provider/responses" \
739777
-H 'Content-Type: application/json' \
740778
-H "X-Request-ID: $REQUEST_ID" \
741779
-d '{"model":"gpt-4.1-nano","input":"Reply with exactly QA_INVALID_PROVIDER"}' && echo
742780
sleep 6
743-
curl -sS "$BASE_URL/admin/api/v1/audit/log?request_id=$REQUEST_ID&limit=5" \
744-
| jq '{total,entries:(.entries|map({request_id,path,model,provider,status_code,error_type}))}'
781+
curl -sS "$BASE_URL/admin/api/v1/audit/log?search=$REQUEST_ID&limit=5" \
782+
| jq --arg request_id "$REQUEST_ID" '{total:(.entries|map(select(.request_id==$request_id))|length),entries:(.entries|map(select(.request_id==$request_id))|map({request_id,path,model,provider,status_code,error_type}))}'
745783
```
746784

747785
## 12. Authenticated runtime features
@@ -758,7 +796,7 @@ curl -sS "$AUTH_BASE_URL/admin/api/v1/dashboard/config" \
758796

759797
### S64 Create managed API key
760798

761-
Creates one managed API key scoped to a release-specific user path and stores the one-time secret under `/tmp`.
799+
Creates one managed API key scoped to a release-specific user path and stores the one-time secret under `QA_RUN_DIR`.
762800

763801
```bash
764802
AUTH_KEY_JSON=$(curl -sS -X POST "$AUTH_BASE_URL/admin/api/v1/auth-keys" \
@@ -850,12 +888,13 @@ curl -sS -D - "$AUTH_BASE_URL/v1/chat/completions" \
850888

851889
### S70 Audit evidence for managed-key scoped workflow
852890

853-
Confirms that auth method, managed auth key ID, normalized user path, workflow ID, and no cache hit are all recorded together.
891+
Confirms through audit-log search that auth method, managed auth key ID, normalized user path, workflow ID, and no cache hit are all recorded together.
854892

855893
```bash
856-
curl -sS "$AUTH_BASE_URL/admin/api/v1/audit/log?request_id=$QA_AUTH_REQ2&limit=5" \
894+
sleep 6
895+
curl -sS "$AUTH_BASE_URL/admin/api/v1/audit/log?search=$QA_AUTH_REQ2&limit=5" \
857896
-H "$ADMIN_AUTH_HEADER" \
858-
| jq '{total,entries:(.entries|map({request_id,status_code,auth_method,auth_key_id,user_path,execution_plan_version_id,cache_type,answer:.data.response_body.choices[0].message.content}))}'
897+
| jq --arg request_id "$QA_AUTH_REQ2" '{total:(.entries|map(select(.request_id==$request_id))|length),entries:(.entries|map(select(.request_id==$request_id))|map({request_id,status_code,auth_method,auth_key_id,user_path,execution_plan_version_id,cache_type,answer:.data.response_body.choices[0].message.content}))}'
859898
```
860899

861900
### S71 Global cache warm request with explicit user path

0 commit comments

Comments
 (0)