It looks like we need to inline this elsewhere. We're getting

E   ImportError: cannot import name 'BIG_BYTES_SHARD_SIZE' from 'distributed.protocol.utils' (/home/runner/work/distributed/distributed/distributed/protocol/utils.py)

in CI

Can we add a small assert here to ensure/confirm that frame is 8 MiB?

In [1]: frame = b"1234abcd" * (2 ** 20) # 8 MiB

In [2]: from dask.utils import parse_bytes

In [3]: from dask.sizeof import sizeof

In [4]: assert sizeof(frame) == parse_bytes("8 MiB")

Sure, happy to do that. I sort of figured that the test itself functioned as an assert (3*3 ~ 8 and 2*5 ~ 8), but an assertion is more explicit

This happens several times on every message sent. There may be thousands of messages per second. It may be worth considering performance here

In [1]: import dask, distributed

In [2]: %timeit dask.utils.parse_bytes(dask.config.get("distributed.comm.shard"))
2.2 µs ± 8.69 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [3]: %timeit dask.config.get("distributed.comm.shard") # maybe we choose to add an lru cache on parse_bytes
755 ns ± 4.08 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

This makes me slightly nervous

To solve our immediate pain my preference would be something like #5052

Hmm, that's a good point. Perhaps we can read the config somewhat higher in the stack and pass it down to the protocol machinery

Passing down the shard size is also done in the linked PR.

Doing this per-message would be good, and probably reduce the pain by 4x or so. It's still at a level that we might want to avoid though.

This just isn't that common of a use case. I don't think that people are likely to be dynamically changing shard size. I think that we want to set a conservative default for websockets, and then use environment variables / config options stored in yaml for other cases.

That's fair. I agree it's not a common use-case, and it's not worth incurring the performance hit.

The semi-configurability of the shard size did cause me some pain today when I was testing some things, and for a while was running in circles trying to figure out why the config changes wasn't taking effect.