This is a WIP PR. The logic to support entry-points is, as you can see, very simple.

At the moment, entrypoints are loaded as soon as dask.sizeof is imported. It would be possible to modify the Dispatch object such that we only load entry-points when there is no dispatch for a given type. However, if entry-points are needed by an application, this just moves the cost of reading entry_points to the point at which sizeof is dispatched upon. This would eliminate the cost for applications which do not need entry-point based implementations.

I decided not to use the "name" field for anything practical; it would be possible to use the name as the import path to a Python class object, such that Dask would be responsible for calling sizeof.register. However precludes the ability to use register_lazy, or to register multiple plugins.

I haven't added any tests yet, and I didn't see anywhere in the docs to put any new documentation on this. Any suggestions here would be helpful!

@mrocklin @jsignell @jrbourbeau I'd appreciate any of your thoughts on this 👋

@agoose77 we are in agreement that this is a nice improvement, so you should feel free to finish it up whenever you get a chance.

You can put the documentation in the docstring or you could write a new section on https://docs.dask.org/en/latest/develop.html#

@jsignell how does the documentation look? Is that sufficiently detailed?

Thanks for the updates @agoose77! For reference, we run a few code style checks which are outlined at https://docs.dask.org/en/latest/develop.html#code-formatting. In particular, I recommend using the (optional) pre-commit hooks, as they will ensure all style checks are automatically run when you make a git commit locally

Thanks for the updates @agoose77! For reference, we run a few code style checks which are outlined at https://docs.dask.org/en/latest/develop.html#code-formatting. In particular, I recommend using the (optional) pre-commit hooks, as they will ensure all style checks are automatically run when you make a git commit locally

Huh, I didn't notice the .pre-commit-config.yaml, my bad. I'll run the linters.

Thanks @agoose77! Can we add a test to ensure that sizeof entrypoints are properly registered?

What do you mean by this? I already test for the group key, would you like a warning?

Sorry, I mean can we add a test to our test suite for this? Right now we don't have any tests which check that this functionality works as we expect it to

@jrbourbeau I'm not against it in principle, but I'm not even sure how we would test this - really we would just be testing importlib_metadata - this code just invokes the given entry-point function with a single arg, and so it's really only external user code that is testable.

I agree there's no need to test importlib_metadata functionality. Though we should test that calling _register_entry_point_plugins results in sizeof entrypoints actually being registered with the dask.sizeof.sizeof dispatch object.

One possible test would be to do something like:

1. Create some custom sizeof entrypoint implementation
2. Call _register_entry_point_plugins
3. assert that the dask.sizeof.sizeof dispatch object has a corresponding implementation for the custom entrypoint

We have some support for entrypoints in dask/distributed. Perhaps test_register_backend_entrypoint over there could help inform the test we add here.

Can one of the admins verify this patch?

@jrbourbeau finally got around to this. I've made the changes you mentioned earlier!

ok to test

@graingert thanks, I've made those changes. I've reabsed on main too.

I am not sure where is best to add the link for the small docs that I wrote. Have you any suggestions?

For docs, I could imagine writing a new how-to section, those can be as short as you like, but I can't think of another place where it would fit.

@jsignell I think this is done, or at least ready for feedback!

@jsignell Just pinging again to see whether I need to do anything further here? Let me know if there's no scope for looking at this right now, I can put it on the backburner.

Please, no need to apologise, this is a big project and a small PR! I'll keep my eye on the issue :)

Seems like something in that last commit broke linting :)

Ah, my mistake - I assumed that the linters were also run by precommit, but it looks like this particular linter was added recently.

This seems like a great first step. We can always iterate and refactor if we like. But for now merging! Thanks for sticking with this @agoose77! And thank you @graingert and @jakirkham for reviewing.

Thanks all for helping get this merged!