Skip to content

[tune] docker syncer#11035

Merged
richardliaw merged 9 commits intoray-project:masterfrom
krfricke:ray-docker-syncer-rev
Oct 1, 2020
Merged

[tune] docker syncer#11035
richardliaw merged 9 commits intoray-project:masterfrom
krfricke:ray-docker-syncer-rev

Conversation

@krfricke
Copy link
Copy Markdown
Contributor

@krfricke krfricke commented Sep 25, 2020

Why are these changes needed?

Add support for syncing between docker containers in Ray Tune.

Related issue number

Replaces and closes #10800
Also, closes #4403

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

krfricke and others added 3 commits September 27, 2020 19:43
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
@krfricke
Copy link
Copy Markdown
Contributor Author

I'll add unittesting for the integration tomorrow (similar to the Kubernetes test).

@krfricke krfricke self-assigned this Sep 28, 2020
@krfricke krfricke added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Sep 28, 2020
Comment on lines +16 to +22
from ray.autoscaler._private.util import validate_config
from ray.autoscaler.node_provider import _get_node_provider

with open(config_path) as f:
new_config = yaml.safe_load(f.read())
validate_config(new_config)
self._config = new_config
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI @ericl for Tune to sync between docker containers, we leverage some of the autoscaler's private interfaces.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please advise if there's a better alternative

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you instead add validate_config to the public API (sdk.py)?

ericl
ericl previously requested changes Sep 28, 2020
Copy link
Copy Markdown
Contributor

@ericl ericl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@richardliaw @krfricke can we have a design document on this? It looks like this is a pretty complex piece of machinery here and it's concerning it is peeking into the internals of the autoscaler. This makes it likely to be broken by future changes.

@richardliaw
Copy link
Copy Markdown
Contributor

Let's use #11122 as the API for rsync.

Comment on lines +62 to +63
from ray.tune.integration import docker
docker.rsync = self.mock_command
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, - generally you'll want to use mock.patch instead

@richardliaw richardliaw dismissed ericl’s stale review October 1, 2020 16:38

addressed comments, fully public api

@richardliaw richardliaw merged commit bdf647c into ray-project:master Oct 1, 2020
@krfricke krfricke deleted the ray-docker-syncer-rev branch October 1, 2020 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests-ok The tagger certifies test failures are unrelated and assumes personal liability.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tune/autoscaler] _LogSyncer cannot rsync with Docker

3 participants