Optimize CI integration test matrix for better balance#6599
Merged
Conversation
Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Optimize CI integration test matrix for better balance
Optimize CI integration test matrix for better balance
Dec 16, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebalances the CI integration test matrix to reduce the critical path from 76.77s to ~46s (39% improvement) by addressing severe group imbalances and duplicate test execution.
Changes
1. Isolate slow test into dedicated group
TestProgressFlagSignature(30s test)2. Fix duplicate test execution
Eliminates ~20s of wasted execution per run.
3. Split large catch-all group
Impact
Original prompt
This section details on the original issue you should resolve
<issue_title>[ci-coach] Optimize CI integration test matrix for better balance</issue_title>
<issue_description>## CI Optimization: Integration Test Matrix Rebalancing
Summary
This PR optimizes the CI integration test matrix to reduce the critical path by addressing severe imbalances. The changes reduce the maximum group duration from 76.77s to an estimated 46s (39% improvement) while adding better test isolation and eliminating duplicate test execution.
Analysis Results
Baseline Metrics (from last 100 CI runs):
Integration Matrix Imbalance Issues:
Key Problems Identified:
TestProgressFlagSignature(30s) runs in multiple groupsTestCompileWorkflowsmatches multiple patterns, running 3 timesOptimizations
1. Isolate Slow Test (New Group)
Type: Matrix Rebalancing
Impact: Reduces "CLI Completion & Safe Inputs" from 76.77s to ~46s (39% reduction)
Risk: Low
Changes:
TestProgressFlagSignatureRationale:
TestProgressFlagSignaturetakes 30+ seconds alone and was running as part of the catch-all "CLI Completion & Safe Inputs" group. Isolating it allows the catch-all group to run ~39% faster, improving overall matrix balance.Before:
After:
Benefits:
2. Fix Duplicate Test Execution
Type: Test Deduplication
Impact: Eliminates ~20s of wasted CI time from duplicate runs
Risk: Low
Changes:
TestCompile|TestPoutineto^TestCompile[^W]|TestPoutineRationale:
The pattern
TestCompile|TestPoutinewas matchingTestCompileWorkflows*tests, causing them to run in:This wastes CI time and provides no additional test coverage.
Before:
After:
The regex
^TestCompile[^W]matches "TestCompile" at the start (^) followed by any character except "W" ([^W]), thus excluding "TestCompileWorkflows" while including other "TestCompile*" tests.Benefits:
3. Split Large Catch-All Group
Type: Matrix Rebalancing
Impact: Reduces "Workflow Misc" from 61.04s to ~30s per group
Risk: Low
Changes:
Rationale:
"Workflow Misc" was a catch-all containing 6,736 tests (25.1% of all integration tests) taking 61.04s. Splitting it into two groups allows better parallelization.
Before:
After:
Benefits:
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.