There as a report about disk being awfully slow a while ago, see dask/distributed#5554 which came up in the context of dask/distributed#5502 (which is basically what this issue is about)

my guess is that this is not just true for the groupby but everything. we might want to consider dropping disk entirely or change the default for everything to tasks. this obviously requires more benchmarking

The reason why I dislike picking tasks for the one operation but disk for the other is because performance characteristics for both algorithms are more or less exclusively depending on the data distribution and number of partitions (apart from hardware performance) and not on whether we're performing a set-index, shuffle, groupby, etc.
My guess is your measurements are using a similar asymmetric data distribution as described in dask/distributed#5554. Is your benchmark code/data still available somewhere?

I'll revert the behavioral change for now. I don't want this PR to be blocked by this

The reason why I dislike picking tasks for the one operation but disk for the other is because performance characteristics for both algorithms are more or less exclusively depending on the data distribution and number of partitions..

I agree that it feels messy to use a different shuffle default for different algorithms. With that said, I do think it is safe to consider the groupby case to be somewhat "special". The difference is that the true default behavior for groupby is to not include any shuffle at all. It just so happens that it is possible to boost the performance of groupby(...).agg(.., split_out>1) by leveraging a task-based shuffle (but not a disk-based shuffle).

Is your benchmark code/data still available somewhere?

I don't think you need a particularly unusual data distribution to show that the shuffle based groupby is only better than the conventional ACA groupby when shuffle is not set to "disk". For example, I'm sure the performance of shuffle="disk" would be pretty low across the board for Ian's results shared in #9302 (comment). Another example (a high-cardinality groupby where a shuffle-based reduction should outperform shuffle=False):

import time
from dask.datasets import timeseries

def run(shuffle="tasks"):
    ddf = timeseries(
        id_lam=int(1e18),
        start='2000-01-01',
        end='2001-06-01',
        freq='1s',
        seed=15,
    )
    agg = ddf.groupby(
        "id",
        sort=False,
    ).agg(
        {"x": "sum", "y": "mean"},
        split_out=4, 
        shuffle=shuffle,
    )

    t0 = time.time()
    result = agg.compute()
    total_time = time.time() - t0
    print(f"shuffle: {shuffle}\ntime: {total_time} sec\nnum-rows: {len(result)}\n")

run(shuffle="tasks")  # time: 10.846901416778564 sec
run(shuffle=False)  # time: 18.84290647506714sec
run(shuffle="disk")  # time: 26.46630048751831 sec

Here we see why we probably shouldn't use shuffle="disk" over shuffle=False as a default.

don't think you need a particularly unusual data distribution to show that the shuffle based groupby is only better than the conventional ACA groupby when shuffle is not set to "disk".

This example lets me wonder whether we should drop support for disk entirely. Are there any known cases where it does perform better?

Anyhow, let's not blow up the scope of this PR, please.

I restored the behavior as before but removed the need for the config to be set. This PR is now blocking dask/distributed#7482