Skip to content

Support nested parallel tasks (parallel task referencing another parallel task) #145

@rileyhilliard

Description

@rileyhilliard

Problem

Currently, rr does not allow a parallel task to reference another parallel task. This limitation requires users to manually flatten their task hierarchies, which leads to repetition and harder maintenance.

Current Behavior

When you define:

tasks:
  test-opendata:
    parallel:
      - test-opendata-1
      - test-opendata-2
      - test-opendata-3

  test:
    parallel:
      - test-opendata    # ERROR: can't reference parallel task
      - test-backend
      - test-frontend

You get: parallel task 'test' can't reference another parallel task 'test-opendata'

Workaround

Users must manually flatten the hierarchy:

tasks:
  test:
    parallel:
      - test-opendata-1
      - test-opendata-2
      - test-opendata-3
      - test-backend-1
      - test-backend-2
      - test-backend-3
      - test-frontend

This is verbose and error-prone. If you add a 4th split to test-opendata, you must remember to update every parent task that references it.

Proposed Solution

Allow parallel tasks to reference other parallel tasks. When rr encounters a nested parallel reference, it should flatten the task tree before execution.

Expected Behavior

tasks:
  test-opendata:
    parallel:
      - test-opendata-1
      - test-opendata-2
      - test-opendata-3

  test-backend:
    parallel:
      - test-backend-1
      - test-backend-2
      - test-backend-3

  test:
    parallel:
      - test-opendata  # Expands to: test-opendata-1, test-opendata-2, test-opendata-3
      - test-backend   # Expands to: test-backend-1, test-backend-2, test-backend-3
      - test-frontend  # Simple task, no expansion

When running rr test, rr would:

  1. Detect that test-opendata and test-backend are parallel tasks
  2. Expand them to their constituent subtasks
  3. Execute all 7 tasks in parallel: test-opendata-1, test-opendata-2, test-opendata-3, test-backend-1, test-backend-2, test-backend-3, test-frontend

Implementation Notes

  1. Cycle detection: Prevent infinite recursion if tasks reference each other
  2. Depth limit: Optionally limit nesting depth (2-3 levels should be plenty)
  3. Task listing: rr tasks could show expanded view for parallel tasks
  4. Logging: When expanding, log which tasks were flattened for clarity

Alternative Considered

YAML anchors: Users could use YAML anchors to avoid repetition:

tasks:
  test:
    parallel:
      - *opendata-subtasks
      - *backend-subtasks
      - test-frontend

However, this is less intuitive and doesn't work well with rr's task model where each subtask is a named entity that can be run independently.

Use Case

This is particularly useful when test suites are split across multiple hosts for parallelization. For example, splitting pytest runs using --test-group-count 3 --test-group N:

  • test-opendata-1, test-opendata-2, test-opendata-3 (33% each)
  • test-backend-1, test-backend-2, test-backend-3 (33% each)

Users want to run:

  • rr test-opendata → runs the 3 opendata splits in parallel
  • rr test-backend → runs the 3 backend splits in parallel
  • rr test → runs ALL 7 tasks (both suites + frontend) in parallel

Without nested parallel support, the test task becomes a maintenance burden.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions