Skip to content

Add support for operator plugins#532

Merged
jacobtomlinson merged 10 commits intodask:mainfrom
jacobtomlinson:operator-plugins
Aug 22, 2022
Merged

Add support for operator plugins#532
jacobtomlinson merged 10 commits intodask:mainfrom
jacobtomlinson:operator-plugins

Conversation

@jacobtomlinson
Copy link
Member

@jacobtomlinson jacobtomlinson commented Jul 4, 2022 •
edited
Loading

This PR is the result of a chat with @bstadlbauer about extending the functionality of the operator. The specific context is what if the user wants the operator to create other resources like Istio VirtualSerivce, Gateway and Certificate resources. Extra resources like this may end up being a common requirement, but given the endless possibilities of k8s cluster setups it's hard to make this configurable.

Full Slack conversation

Bernhard Stadlbauer
09:21
Hi Jacob!
I would have one really small question on the operator. We're using istio to manage our ingress and I was wondering if you could think of a way of creating a VirtualSerivce, Gateway and Certificate to create easy access to the scheduler dashboard for cluster?
Happy to also contribute there 🙂

Jacob Tomlinson
13:39
I had imagined folks would just create these resources alongside the DaskCluster resources. We should probably document that though. Do you mean also doing this via the Python API?

Bernhard Stadlbauer
07:40
There is probably no way around folks creating these resources themselves (I'd assume; because there are multiple ways to do ingress; etc.). I was wondering whether you could think of a nice way to add this? E.g. in case of a single-cluster helm deployment, I would probably create my own chart with dask/dask as subchart. I cannot think of a nice way of doing something similar with the operator though, as these resources would ideally always be created alongside the CRD, no matter where that comes from

Jacob Tomlinson
11:04
Documentation is one of the only ways to handle this to be honest. For folks using manifests with directly or via kustomize then you can create all your manifests by hand. If you're using a helm chart you can do the same thing (we have a goal to create a single cluster chart like dask/dask that uses the new CRDS). Then if you're using the Python API then you could use pykube or directly use the kubernetes Python library to create additional resources.
We do add a certain amount of additional extras, like the helm chart has some optional metrics resources. But given the almost infinite configurability I think the only thing we can do is document the Service resources that you need to care about and leave folks to do what makes sense for them.

Bernhard Stadlbauer
16:43
Thank you for the detailed answer!
We might end up with our own small (kopf based) operator then, which listens to the DaskCluster CR and manages our ingress resources accordingly. Because when the Flyte plugin creates the CR, there is also no way to add those (if we assume a general Flyte plugin) (edited)

Jacob Tomlinson
16:45
I bet it wouldn't take much effort for us to add plugin support to the Dask operator. We could do it with entrypoints so you just need to install your plugin in the docker image and it would get loaded.

Bernhard Stadlbauer
16:51
Uh that might be nice!

One option would be to allow users to create an extension to the operator that creates these resources automatically.

This PR loads in additional modules via a dask_operator_plugin entrypoint. Any code in that module that uses kopf decorators to register event handlers also gets loaded.

# setup.py

setup(
...
    entry_points="""
        [dask_operator_plugin]
        foo=dask_kubernetes.operator.plugins.foo
      """,
)
# foo.py

import kopf


@kopf.on.startup()
async def foo_startup(logger, **kwargs):
    logger.info("Plugin 'foo' running.")

The core Dask operator with handle authentication and all that so if you want additional actions to happen when a daskcluster resource is created for example then you can write a plugin this was and watch for the right event.

import kopf

@kopf.on.create("daskcluster")
async def daskcluster_create(spec, name, namespace, logger, **kwargs):
    # Handle event

Or more specifically you could watch for the scheduler service getting created.

import kopf

@kopf.on.create("service", labels={"dask.org/component": "scheduler"})
async def handle_scheduler_service_create(meta, new, namespace, logger, **kwargs):
    # Create additional resources here

So to create a plugin you just create a Python package with an entrypoint and to install it you just install that package in the operator container image.

FROM ghcr.io/dask/dask-kubernetes-operator:latest

RUN pip install myawesomeplugin

@jacobtomlinson jacobtomlinson mentioned this pull request Aug 17, 2022
Merged
@jacobtomlinson jacobtomlinson marked this pull request as ready for review August 22, 2022 13:13
@jacobtomlinson jacobtomlinson changed the title [Experiment] Add operator plugins Add support for operator plugins Aug 22, 2022
@jacobtomlinson jacobtomlinson merged commit 4a69e3a into dask:main Aug 22, 2022
@jacobtomlinson jacobtomlinson deleted the operator-plugins branch August 22, 2022 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jacobtomlinson