Fix flaky test_close_gracefully and test_lifetime by crusaderky · Pull Request #5677 · dask/distributed

crusaderky · 2022-01-21T12:13:41Z

crusaderky · 2022-01-21T12:18:59Z

distributed/tests/test_worker.py

-        futures = c.map(slowinc, range(200), delay=0.1, worker=[b.address])
-        await asyncio.sleep(1.5)
-        assert b.status not in (Status.running, Status.paused)
+        futures = c.map(slowinc, range(200), delay=0.1, workers=[b.address])


fix spelling error worker->workers! All tasks were failing before!

something like await next(as_completed(futures)) would protect us there and may remove the necessity for sleep below. I'm fine without it

crusaderky · 2022-01-21T14:32:02Z

distributed/tests/test_worker.py

+    async with Worker(s.address) as a, Worker(s.address, lifetime="2 seconds") as b:
+        futures = c.map(slowinc, range(200), delay=0.1, workers=[b.address])
+        await asyncio.sleep(1)
+        assert not a.data


After fixing the misspelling above, the test was occasionally failing this assertion with lifetime="1 seconds" and sleep(0.5)

crusaderky · 2022-01-21T16:51:24Z

All test failures are unrelated; ready for review and merge

fjetter · 2022-01-24T12:36:32Z

distributed/tests/test_worker.py

-    proc = {ts for ts in b.tasks.values() if ts.state == "executing"}
-    assert proc
+
+    assert any(ts for ts in b.tasks.values() if ts.state == "executing")


I think there is a tiny time window possible where no task is in executing state until the next ensure_computing is called

fjetter · 2022-01-24T12:38:32Z

distributed/tests/test_worker.py

+            a.story(key),
+            [
+                (key, "put-in-memory"),
+                (key, "receive-from-scatter"),


why is this a scatter event?

legacy (pre-AMM) Scheduler.gather_on_worker, which underlies both Scheduler.replicate and Scheduler.retire_workers

fjetter · 2022-01-24T12:41:31Z

distributed/tests/test_worker.py

-        futures = c.map(slowinc, range(200), delay=0.1, worker=[b.address])
-        await asyncio.sleep(1.5)
-        assert b.status not in (Status.running, Status.paused)
+        futures = c.map(slowinc, range(200), delay=0.1, workers=[b.address])


something like await next(as_completed(futures)) would protect us there and may remove the necessity for sleep below. I'm fine without it

crusaderky · 2022-01-24T19:07:16Z

All review comments have been incorporated.
All test failures are unrelated.

commit 14bddba1482de035ca416368f918d94b115d3660 Merge: df079fc 9a66b71 Author: crusaderky <crusaderky@gmail.com> Date: Tue Jan 25 17:16:58 2022 +0000 Merge branch 'main' into AMM/RetireWorker commit df079fc Merge: 9676934 5835ce3 Author: crusaderky <crusaderky@gmail.com> Date: Mon Jan 24 15:08:51 2022 +0000 Merge branch 'AMM/staging' into AMM/RetireWorker commit 5835ce3 Merge: ee68b02 b0b8e95 Author: crusaderky <crusaderky@gmail.com> Date: Mon Jan 24 15:07:27 2022 +0000 Merge branch 'AMM/test_close_gracefully' into AMM/staging commit ee68b02 Merge: 7d4e6ee ccad288 Author: crusaderky <crusaderky@gmail.com> Date: Mon Jan 24 15:06:50 2022 +0000 Merge branch 'main' into AMM/staging commit b0b8e95 Author: crusaderky <crusaderky@gmail.com> Date: Mon Jan 24 14:32:29 2022 +0000 Code review commit 9676934 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 14:50:53 2022 +0000 AMM to manage retire_workers() commit 7d4e6ee Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 14:39:41 2022 +0000 Fix flaky test_close_gracefully and test_lifetime (dask#5677) commit 7faab51 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 14:39:41 2022 +0000 harden test commit aef3b71 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 14:29:10 2022 +0000 Increase resilience on slow CI commit af84e40 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:39:18 2022 +0000 Dump cluster state on all test failures (dask#5674) commit 5054c19 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:38:38 2022 +0000 Paused workers shouldn't steal tasks (dask#5665) commit eadb35f Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:37:48 2022 +0000 transition flight to missing if no who_has (dask#5653) commit 581aee8 Merge: 940bb45 60c0d60 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:36:09 2022 +0000 Merge branch 'main' into AMM/test_close_gracefully commit 940bb45 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:20:10 2022 +0000 tweak comment commit 731d132 Author: crusaderky <crusaderky@gmail.com> Date: Fri Jan 21 12:12:03 2022 +0000 Fix flaky test_close_gracefully and test_lifetime

Fix flaky test_close_gracefully and test_lifetime

731d132

crusaderky self-assigned this Jan 21, 2022

crusaderky marked this pull request as draft January 21, 2022 12:13

crusaderky commented Jan 21, 2022

View reviewed changes

tweak comment

940bb45

crusaderky added the flaky test Intermittent failures on CI. label Jan 21, 2022

This was referenced Jan 21, 2022

Paused workers shouldn't steal tasks #5665

Merged

AMM: Graceful Worker Retirement #5381

Merged

Merge branch 'main' into AMM/test_close_gracefully

581aee8

crusaderky added a commit to crusaderky/distributed that referenced this pull request Jan 21, 2022

Fix flaky test_close_gracefully and test_lifetime (dask#5677)

d75e3ad

crusaderky marked this pull request as ready for review January 21, 2022 14:20

Increase resilience on slow CI

aef3b71

crusaderky commented Jan 21, 2022

View reviewed changes

harden test

7faab51

crusaderky added a commit to crusaderky/distributed that referenced this pull request Jan 21, 2022

Fix flaky test_close_gracefully and test_lifetime (dask#5677)

7d4e6ee

fjetter reviewed Jan 24, 2022

View reviewed changes

Code review

b0b8e95

fjetter approved these changes Jan 25, 2022

View reviewed changes

fjetter merged commit 0f6a4df into dask:main Jan 25, 2022

crusaderky deleted the AMM/test_close_gracefully branch January 25, 2022 13:44

gjoseph92 pushed a commit to gjoseph92/distributed that referenced this pull request Feb 1, 2022

Fix flaky test_close_gracefully and test_lifetime (dask#5677)

3c0b483

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix flaky test_close_gracefully and test_lifetime#5677

Fix flaky test_close_gracefully and test_lifetime#5677
fjetter merged 6 commits intodask:mainfrom
crusaderky:AMM/test_close_gracefully

crusaderky commented Jan 21, 2022

Uh oh!

crusaderky Jan 21, 2022 •

edited

Loading

Uh oh!

fjetter Jan 24, 2022

Uh oh!

crusaderky Jan 24, 2022

Uh oh!

crusaderky Jan 21, 2022

Uh oh!

crusaderky commented Jan 21, 2022

Uh oh!

fjetter Jan 24, 2022

Uh oh!

crusaderky Jan 24, 2022

Uh oh!

fjetter Jan 24, 2022

Uh oh!

crusaderky Jan 24, 2022

Uh oh!

fjetter Jan 24, 2022

Uh oh!

crusaderky commented Jan 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

crusaderky commented Jan 21, 2022

Uh oh!

crusaderky Jan 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crusaderky commented Jan 21, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crusaderky commented Jan 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

crusaderky Jan 21, 2022 •

edited

Loading