Skip to content

Add distributed multi-node cpu only support (MULTI_CPU)#63

Merged
sgugger merged 1 commit intohuggingface:mainfrom
ddkalamk:multi_cpu
May 4, 2021
Merged

Add distributed multi-node cpu only support (MULTI_CPU)#63
sgugger merged 1 commit intohuggingface:mainfrom
ddkalamk:multi_cpu

Conversation

@ddkalamk
Copy link
Copy Markdown
Contributor

@ddkalamk ddkalamk commented May 3, 2021

Possible fix for #32

sgugger
sgugger reviewed May 3, 2021
View reviewed changes
Copy link
Copy Markdown
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for digging into this! It looks great and with a little bit of polishing (mostly naming), it should be ready to be merged soon!

You should add a line in the main README in the list at the end with - multinode CPU! Also it looks like your suggested integration supports different types of launchers (from the number of env variables looked at). Could you document this somewhere?

Comment thread src/accelerate/accelerator.py Outdated
Comment thread src/accelerate/commands/config/cluster.py
Comment thread src/accelerate/state.py Outdated
Comment thread src/accelerate/commands/config/config_utils.py
Comment thread src/accelerate/state.py Outdated
Comment thread src/accelerate/state.py Outdated
Comment on lines +174 to +179
print("Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default")
print("If this run hangs, try exporting rank 0's hostname as MASTER_ADDR")
Copy link
Copy Markdown
Collaborator

@sgugger sgugger May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thin we should raise a ValueError here telling the user to pass a MASTER_ADDR instead of relying on a default.

Copy link
Copy Markdown
Contributor Author

@ddkalamk ddkalamk May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some MPI-like custom backends can initialize without MASTER_ADDR so I just put a warning. I can change to ValueError if you think that's a better choice.

Copy link
Copy Markdown
Collaborator

@sgugger sgugger May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are in a test where backend != "mpi", so I would adapt the error message (to say the backend needs a MASTER_ADDR) and raise an error. For MPI, it will still work without the MASTER_ADDR.

Copy link
Copy Markdown
Contributor Author

@ddkalamk ddkalamk May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the change.

Comment thread src/accelerate/state.py Outdated
Comment thread src/accelerate/state.py Outdated
@ddkalamk ddkalamk force-pushed the multi_cpu branch 2 times, most recently from bf91de5 to 276bbb8 Compare May 3, 2021 17:00
@ddkalamk
Copy link
Copy Markdown
Contributor Author

ddkalamk commented May 3, 2021

I have made the suggested changes. Please take a look.
Thanks.

sgugger
sgugger approved these changes May 3, 2021
View reviewed changes
Copy link
Copy Markdown
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the adjustments, two more little things and we should be good to merge!

Comment thread README.md
On your cluster just run:

```bash
mpirun -np 2 python examples/nlp_example.py
Copy link
Copy Markdown
Collaborator

@sgugger sgugger May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this require an install of some library? Let's add how to!

Copy link
Copy Markdown
Contributor Author

@ddkalamk ddkalamk May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple options e.g. Intel MPI, OpenMPI or MVAPICH. I assume, typical HPC clusters would have some version available or users can install them in userspace easily. example instructions for Open-MPI are here.

Copy link
Copy Markdown
Contributor Author

@ddkalamk ddkalamk May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, added a line on how to get OpenMPI...

Comment thread src/accelerate/state.py Outdated
@sgugger
Copy link
Copy Markdown
Collaborator

sgugger commented May 3, 2021

Last step, could you run make style on your branch? This should take care of the failing test.
Let me know if run into any trouble and I can push it on the PR instead.

@ddkalamk
Copy link
Copy Markdown
Contributor Author

ddkalamk commented May 3, 2021

Sorry, my system doesn't have black. Make gave error.

black tests src examples
make: black: Command not found
make: *** [style] Error 127

Can you please go ahead and push it to PR?
Thanks.

@ddkalamk
Copy link
Copy Markdown
Contributor Author

ddkalamk commented May 3, 2021

Just installed black and fixed formatting... It is still giving an error about torch_ccl import but I don't know how to fix it...

sgugger
sgugger reviewed May 3, 2021
View reviewed changes
Comment thread src/accelerate/state.py Outdated
@ddkalamk
Copy link
Copy Markdown
Contributor Author

ddkalamk commented May 4, 2021

Fixed issue with flake8 and squashed to single commit... Should be good to merge now.

@sgugger sgugger merged commit df260fa into huggingface:main May 4, 2021
@sgugger
Copy link
Copy Markdown
Collaborator

sgugger commented May 4, 2021

Thanks again for all your work on this!

@ddkalamk ddkalamk deleted the multi_cpu branch May 4, 2021 14:38
@sgugger sgugger mentioned this pull request May 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgugger sgugger sgugger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ddkalamk @sgugger