Skip to content

[Serve] Reimplement BackendConfig as pydantic model#10389

Merged
edoakes merged 43 commits intoray-project:masterfrom
architkulkarni:pydantic-backend
Sep 4, 2020
Merged

[Serve] Reimplement BackendConfig as pydantic model#10389
edoakes merged 43 commits intoray-project:masterfrom
architkulkarni:pydantic-backend

Conversation

@architkulkarni
Copy link
Copy Markdown
Contributor

Why are these changes needed?

Previously the user of the API would pass in an untyped dict to configure the backend. This PR exposes a typed BackendConfig class to the user. We use pydantic for type hints and validation.

This is a work in progress. What remains to be done is to fully update the Serve documentation for this change.

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • (WIP) I've included any doc changes needed for https://docs.ray.io/en/latest/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested (please justify below)

Copy link
Copy Markdown
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really nice, I'm excited! A few smallish things to clean up.

max_batch_size: Optional[PositiveInt] = None
batch_wait_timeout: float = 0
max_concurrent_queries: Optional[int] = None
autoscaling_config: Optional[Dict[str, Any]] = None
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid exposing this to users for now somehow? Else we should mark it as experimental somehow.

Also, is it possible to add a description for each of these fields that will be auto-documented?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're just talking about the autoscaling_config? Would putting an underscore before the name be enough?

For the auto-documentation, yes, I'll add it. And the autodoc will also hide variables whose name starts with an underscore.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

underscore sounds good.

Copy link
Copy Markdown
Contributor Author

@architkulkarni architkulkarni Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out that with our current version of Sphinx (1.8.5), there's no way to autodocument type hints, nor is there a way to autodocument default values for instance attributes. So I'll just use a docstring for the class BackendConfig.

@edoakes edoakes added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 28, 2020
@architkulkarni
Copy link
Copy Markdown
Contributor Author

@simon-mo All serve/tests/ tests pass on my laptop, but we seem to be running into the same cloudpickle Union type issue on the CI. If we don't upgrade cloudpickle, we would somehow have to work around typing the config parameter as Union[BackendConfig, dict] in the API. I don't know how to do this without breaking backwards compatibility--thoughts?

architkulkarni and others added 2 commits August 28, 2020 10:36
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
max_batch_size: Optional[PositiveInt] = None
batch_wait_timeout: float = 0
max_concurrent_queries: Optional[int] = None
autoscaling_config: Optional[Dict[str, Any]] = None
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

underscore sounds good.

# This is not a pydantic validator, so that we may skip this method when
# creating partially filled BackendConfig objects to pass as updates--for
# example, BackendConfig(max_batch_size=5).
def _validate_complete(self):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • let's call it _validate_max_batch_size or something similar?

Copy link
Copy Markdown
Contributor Author

@architkulkarni architkulkarni Aug 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true that currently max_batch_size is the only thing being validated here, but potentially we could put any kind of validation here that we only want to run on completely filled BackendConfigs. If we call it _validate_max_batch_size(), I worry it might look kind of arbitrary when we call it elsewhere in serve--like, why are they only validating this specific parameter?

Would putting this logic in _validate_max_batch_size() and then making _validate_complete() call _validate_max_batch_size() be reasonable?

@simon-mo
Copy link
Copy Markdown
Contributor

I think one alternative is to use string literal as types. Maybe we can try that:

- config: Optional[Dict]
+ config: "Optional[Dict]"

@architkulkarni architkulkarni added tests-ok The tagger certifies test failures are unrelated and assumes personal liability. and removed @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. labels Sep 1, 2020
@architkulkarni architkulkarni added @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. and removed tests-ok The tagger certifies test failures are unrelated and assumes personal liability. labels Sep 1, 2020
@architkulkarni architkulkarni removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 2, 2020
@architkulkarni architkulkarni added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Sep 2, 2020
@simon-mo simon-mo changed the title [WIP] [Serve] Reimplement BackendConfig as pydantic model [Serve] Reimplement BackendConfig as pydantic model Sep 3, 2020
@simon-mo
Copy link
Copy Markdown
Contributor

simon-mo commented Sep 3, 2020

Lint is failing, can you add pydantic to requirements-doc?
https://travis-ci.com/github/ray-project/ray/jobs/380452414

@edoakes edoakes merged commit 0d93e92 into ray-project:master Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests-ok The tagger certifies test failures are unrelated and assumes personal liability.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants