Greptile code review

This repo uses Greptile for automated review. Before merge, aim for Confidence Score: 5/5 with zero unresolved review threads — see CONTRIBUTING.md.

Run a review — add a PR comment with:

@greptile review

Give it ~5-10 minutes (sometimes longer) for results, then fix feedback and re-trigger until you reach Confidence Score: 5/5.

Optional: automate with the greploop skill.

@greptile review

Greptile Summary

This PR adds the B2 false-healthy guard plumbing and a suite of bridge/predictor contract tests. The core changes are: a new min_tool_calls config field + CLI override that lets floor-ablation experiments be self-describing rather than env-var-dependent, truncated evidence_entries now persisted in cell artifacts for post-hoc guard auditing, and B2 guard activation statistics in analyze_validation.py. Bulk of the diff is test-file reorganization into tests/ subdirectories with zero logic change.

• min_tool_calls plumbing (config.py, cli.py): adds an optional int | None field with ge=0 and wires it into a one-shot BenchInvestigationAgent.MIN_TOOL_CALLS override in the CLI, gated strictly to the cloudopsbench adapter.
• Evidence truncation (runner.py): _truncate_evidence_entries() caps large string keys to 2000 chars and persists the result alongside evidence_entries_count; the in-memory full list continues to feed the B2 guard at runtime.
• New configs (cloudopsbench_definitive_openai.yml, cloudopsbench_floor0_ablation_openai.yml, cloudopsbench_floor_ablation_v2_openai.yml): document the full-corpus comparison and two iterations of the floor-ablation experiment; the v2 correctly uses min_tool_calls: 0 while the original does not.

Confidence Score: 4/5

Safe to merge with one fix recommended: cloudopsbench_floor0_ablation_openai.yml is missing min_tool_calls: 0, so running it without the env var silently produces floor=5 data under a floor=0 label.

The newly added cloudopsbench_floor0_ablation_openai.yml is self-inconsistent: its entire purpose is the floor=0 ablation hypothesis, but it lacks min_tool_calls: 0 — the exact mechanism this PR introduces. Running it without BENCH_MIN_TOOL_CALLS=0 produces a silent duplicate of the floor=5 baseline, invalidating the experiment without any error. The v2 config was added in the same PR specifically because floor0_ablation was broken in this way. All other changes — config-field plumbing, evidence truncation, test reorganization, and new predictor contract tests — are straightforward and correct.

tests/benchmarks/cloudopsbench/configs/cloudopsbench_floor0_ablation_openai.yml needs min_tool_calls: 0 added.

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CLI: _cmd_run] --> B{config.min_tool_calls not None AND benchmark == cloudopsbench?}
    B -- Yes --> C[BenchInvestigationAgent.MIN_TOOL_CALLS = config.min_tool_calls]
    B -- No --> D[Agent uses default / env-var floor]
    C --> E[BenchmarkRunner.run]
    D --> E
    E --> F[RunResult with evidence_entries]
    F --> G[_cell_to_dict]
    G --> H[_truncate_evidence_entries - cap data keys to 2000 chars]
    H --> I[Cell artifact JSON: evidence_entries_count + truncated evidence_entries]
    F --> J[B2 guard reads full in-memory evidence_entries at runtime]
    I --> K[analyze_validation.py: _b2_fired matches downgrade signature in persisted final_diagnosis]

_{Reviews (5): Last reviewed commit: "move configs/ + cloudopsbench AWS docs i..." | Re-trigger Greptile}

@greptile review

@greptile review

@greptile review

💼 Interviewer: describe a time you shipped something impactful.

@YauhenBichel: points at this PR

Interviewer: you're hired. 🤝

👋 Join us on Discord - OpenSRE : hang out, contribute, or hunt for features and issues. Everyone's welcome.