-
Notifications
You must be signed in to change notification settings - Fork 9
Support nested parallel tasks (parallel task referencing another parallel task) #145
Description
Problem
Currently, rr does not allow a parallel task to reference another parallel task. This limitation requires users to manually flatten their task hierarchies, which leads to repetition and harder maintenance.
Current Behavior
When you define:
tasks:
test-opendata:
parallel:
- test-opendata-1
- test-opendata-2
- test-opendata-3
test:
parallel:
- test-opendata # ERROR: can't reference parallel task
- test-backend
- test-frontendYou get: parallel task 'test' can't reference another parallel task 'test-opendata'
Workaround
Users must manually flatten the hierarchy:
tasks:
test:
parallel:
- test-opendata-1
- test-opendata-2
- test-opendata-3
- test-backend-1
- test-backend-2
- test-backend-3
- test-frontendThis is verbose and error-prone. If you add a 4th split to test-opendata, you must remember to update every parent task that references it.
Proposed Solution
Allow parallel tasks to reference other parallel tasks. When rr encounters a nested parallel reference, it should flatten the task tree before execution.
Expected Behavior
tasks:
test-opendata:
parallel:
- test-opendata-1
- test-opendata-2
- test-opendata-3
test-backend:
parallel:
- test-backend-1
- test-backend-2
- test-backend-3
test:
parallel:
- test-opendata # Expands to: test-opendata-1, test-opendata-2, test-opendata-3
- test-backend # Expands to: test-backend-1, test-backend-2, test-backend-3
- test-frontend # Simple task, no expansionWhen running rr test, rr would:
- Detect that
test-opendataandtest-backendare parallel tasks - Expand them to their constituent subtasks
- Execute all 7 tasks in parallel:
test-opendata-1,test-opendata-2,test-opendata-3,test-backend-1,test-backend-2,test-backend-3,test-frontend
Implementation Notes
- Cycle detection: Prevent infinite recursion if tasks reference each other
- Depth limit: Optionally limit nesting depth (2-3 levels should be plenty)
- Task listing:
rr taskscould show expanded view for parallel tasks - Logging: When expanding, log which tasks were flattened for clarity
Alternative Considered
YAML anchors: Users could use YAML anchors to avoid repetition:
tasks:
test:
parallel:
- *opendata-subtasks
- *backend-subtasks
- test-frontendHowever, this is less intuitive and doesn't work well with rr's task model where each subtask is a named entity that can be run independently.
Use Case
This is particularly useful when test suites are split across multiple hosts for parallelization. For example, splitting pytest runs using --test-group-count 3 --test-group N:
test-opendata-1,test-opendata-2,test-opendata-3(33% each)test-backend-1,test-backend-2,test-backend-3(33% each)
Users want to run:
rr test-opendata→ runs the 3 opendata splits in parallelrr test-backend→ runs the 3 backend splits in parallelrr test→ runs ALL 7 tasks (both suites + frontend) in parallel
Without nested parallel support, the test task becomes a maintenance burden.