[1.10.0 Release] `stress_test_many_tasks` regression

Context: https://github.com/ray-project/ray/pull/21512#discussion_r788310272

Result from 1.10.0:
```
"stage_3_creation_time": 0.06580400466918945,
"stage_3_time": 2932.907886981964,
```
 

This is a significant regression from 1.9:
```
"stage_3_creation_time": 0.05767488479614258, 
"stage_3_time": 1949.1530861854553
```