Skip to content

[Java] Simplify Ray.init() by invoking ray start internally#10762

Merged
raulchen merged 52 commits intoray-project:masterfrom
antgroup:remove_run_manager
Dec 4, 2020
Merged

[Java] Simplify Ray.init() by invoking ray start internally#10762
raulchen merged 52 commits intoray-project:masterfrom
antgroup:remove_run_manager

Conversation

@kfstorm
Copy link
Copy Markdown
Member

@kfstorm kfstorm commented Sep 13, 2020

Why are these changes needed?

There are lots of duplicated logic copied from Python to Java when initializing Ray runtime. e.g.,

  • to spawn processes like Raylet, redis-server, GCS, etc.
  • to get the required arguments for a driver to connect to an existing cluster, like socket paths and node manager port.
  • to generate session dir based on current time.

This PR updates the implementation of Ray.init() to invoke ray start and ray stop internally and try the best to leverage existing code in Python.

pros

  • ~1000 lines of code is removed.
  • Some Java config items are removed.
  • The size of ray-runtime jar is reduced due to the removal of some native libs.
  • No more inconsistency when starting Ray on a single machine. e.g. memory resource tag is now available when the Ray cluster is booted by a Java driver.

cons

  • pip install ray is required before when starting Ray on a single machine by a Java driver or Java unit test.

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/latest/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@chaokunyang
Copy link
Copy Markdown
Member

This is great. The duplicated code is easy to diverge and hard to maintain. My concern is that the user can't use ray java out of box. They must setup a python enviroment. And some users may not have python programing experience which increases the learning curve for those users. Is that acceptable?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants