🐛 Describe the bug

Have a look at https://hud.pytorch.org/ci/pytorch/pytorch/master?name_filter=asan:
Typical run time for shard1 is 3h30m for shard2 is 1h30m and for shard3 is 1h20m

• Can it be because shard1 runs c++ tests?
• Or because sharding calculations are made based on runtime of optimized builds (i.e. windows and asan shards would always be scewed towards linux-cuda tests runtimes?)
• Is there a bug in sharing algorithm?
  (Perhaps it ignores skipped/slow test metrics when calculating testfile runtimes?)

Tests (at the file granularity) are selected for each shards based on previous nightly runtime using following function

Versions

N/A it relates to CI

cc @seemethere @malfet @pytorch/pytorch-dev-infra