Skip to content

Change memory bars color on spilling/paused status#6959

Merged
crusaderky merged 1 commit intodask:mainfrom
crusaderky:pause_color
Aug 26, 2022
Merged

Change memory bars color on spilling/paused status#6959
crusaderky merged 1 commit intodask:mainfrom
crusaderky:pause_color

Conversation

@crusaderky
Copy link
Copy Markdown
Collaborator

@crusaderky crusaderky commented Aug 25, 2022

Turn the GUI memory bar for a worker to orange when it passes the target threshold, and to red when it pauses or enters its graceful retirement stage.
The cluster bar changes accordingly to orange if there's at least one worker spilling and to red if there's at least one worker paused or retiring.

In main: individual worker bars change to orange when above 50% and to red when above 100%.
The cluster bar changes to orange when the total cluster memory is above 50% and to red when above 100%.

Demo

import dask
import distributed
import dask.array as da

dask.config.set(
    {
        "distributed.worker.memory.max_spill": "250 MiB",
        "distributed.worker.memory.terminate": False,
    }
)
c = distributed.Client(
    threads_per_worker=4, n_workers=4, memory_limit="4 GiB"
)

rng = da.random.RandomState(0)
a = rng.random((16_000, 16_000), chunks=(4000, 4000))
b = (a @ a.T).sum().round(3)
b.compute()

Peek 2022-08-25 18-12

Copy link
Copy Markdown
Collaborator

@gjoseph92 gjoseph92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love this change and will be so happy to see it.

@github-actions
Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 46m 5s ⏱️ + 5m 1s
  3 045 tests +1    2 960 ✔️ ±0    84 💤 +1  1 ±0 
22 521 runs  +7  21 545 ✔️ +5  975 💤 +2  1 ±0 

For more details on these failures, see this check.

Results for commit 55d9f93. ± Comparison against base commit 31cd316.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants