Skip to content

[raysgd] Custom training operator#7211

Merged
richardliaw merged 67 commits intoray-project:masterfrom
richardliaw:raysgd-operator
Mar 2, 2020
Merged

[raysgd] Custom training operator#7211
richardliaw merged 67 commits intoray-project:masterfrom
richardliaw:raysgd-operator

Conversation

@richardliaw
Copy link
Copy Markdown
Contributor

@richardliaw richardliaw commented Feb 18, 2020 •
edited
Loading

Why are these changes needed?

Provide an abstraction for stateful custom training loops. See requirement doc here:

https://docs.google.com/document/d/1Cjp8Xtu9dG_vOO9yyrEIGL2MypgSsJ5wApzBa_BkUOQ/edit#

Need to:

  • write documentation
  • write tests

This PR leaves a couple usability stories to do:

  1. train_epoch(num_steps) should let you intercept training during an epoch for custom methods such as checkpoints. Do we continue training from the point we left off?
  2. If my 100 batches returns some loss, how do I aggregate this?
  3. In the current interface, custom learning rate schedulers are unwieldy. Is there a clear way of handling them?

Related issue number

Checks

richardliaw and others added 30 commits February 5, 2020 09:24
@richardliaw
Init fp16
df57735
@richardliaw
fp16 and schedulers
2342e76
@richardliaw
scheduler linking and fp16
49abded
@richardliaw
to fp16
cc4465f
@richardliaw
loss scaling and documentation
00f9926
@richardliaw
more documentation
bee750a
@richardliaw
add tests, refactor config
80e43a8
@richardliaw
moredocs
9a89b89
@richardliaw
more docs
1d20075
@richardliaw
fix logo, add test mode, add fp16 flag
3e98f6d
@richardliaw
fix tests
168b813
@richardliaw
fix scheduler
9a77e49
@richardliaw
fix apex
fa1eaaa
@richardliaw
improve safety
bea67f6
@richardliaw
fix tests
420a2f2
@richardliaw
fix tests
f53e9e6
@richardliaw
remove pin memory default
fce88a5
@richardliaw
rm
39f5328
@richardliaw
fix
f8caf4b
@richardliaw
Update doc/examples/doc_code/raysgd_torch_signatures.py
9d76e21
@richardliaw
fix
b7ca982
@richardliaw
Merge branch 'fp16_pytorch' of https://github.com/richardliaw/ray int…
c278e9b 
…o fp16_pytorch
@richardliaw
migrate changes from other PR
dff9521
@richardliaw
ok thanks
20428a7
@richardliaw
pass
62ccf6d
@richardliaw
signatures
0c388fa
@richardliaw
lint'
beeae3c
@richardliaw
Update python/ray/experimental/sgd/pytorch/utils.py
748a553
@richardliaw @edoakes
Apply suggestions from code review
acacee0 
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
@richardliaw
should address most comments
daf5794
@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22451/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 27, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22469/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 27, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22471/
Test FAILed.

maximsmol
maximsmol requested changes Feb 27, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@maximsmol maximsmol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggesting some documentation clarification.

edoakes
edoakes reviewed Feb 28, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Sorry for the delay on reviewing.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 29, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22572/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 29, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22576/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 29, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22573/
Test FAILed.

edoakes
edoakes approved these changes Mar 1, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Mar 2, 2020

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22594/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Mar 2, 2020

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22602/
Test PASSed.

@richardliaw richardliaw merged commit 48cdca8 into ray-project:master Mar 2, 2020
@richardliaw richardliaw deleted the raysgd-operator branch March 2, 2020 05:22
ffbin pushed a commit to antgroup/ant-ray that referenced this pull request Mar 20, 2020
@richardliaw
[raysgd] Custom training operator (ray-project#7211)
a7dc444
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maximsmol maximsmol maximsmol requested changes

@edoakes edoakes edoakes approved these changes

Assignees

@edoakes edoakes

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@richardliaw @AmplabJenkins @maximsmol @edoakes