Unit Test Results

       15 files  ±    0         15 suites  ±0   6h 16m 48s ⏱️ + 8m 48s
  2 853 tests ±    0    2 772 ✔️ +    2       81 💤  -   1  0 ❌  - 1 
21 586 runs  +449  20 554 ✔️ +365  1 032 💤 +85  0 ❌  - 1 

Results for commit 46230a8. ± Comparison against base commit 3468601.

♻️ This comment has been updated with latest results.

I ran this 1000 times locally and saw no issues.

Planning to merge EOD

I ran this 1000 times locally and saw no issues.

Did you run main 1000 times locally and see issues? I was never able to reproduce the actual issue we're seeing in CI (subprocess.TimeoutExpired error) locally.

We've already been working on this test and others #6395. See #6483.

In one failed output we can see that the scheduler is up but the Client times out

In this case, you also see

2022-06-02 06:11:40,414 - distributed.scheduler - INFO - Receive client connection: Client-dccea0ee-e23a-11ec-bafa-005056bd5fdc
...
2022-06-02 06:11:48,470 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 8.07s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.

So

1. The client did connect, therefore trying to wait for the scheduler to come up by parsing logs is unnecessary
2. As you can see on this line, the timeout occurred when the client was waiting for the scheduler to respond, after it had already connected
3. The unresponsiveness in the scheduler is likely the problem

If you look at the test report, every failure of test_dashboard_port_zero is a subprocess.TimeoutExpired, not an asyncio.exceptions.TimeoutError like the case this test is trying to fix (and as I explained, I don't believe the changes here would fix that problem).

As discussed in #6483, piping stdout and parsing logs is better to avoid when it's not strictly necessary for the test.