add cuda memory and distributed metadata by guotuofeng · Pull Request #57252 · pytorch/pytorch

guotuofeng · 2021-04-29T06:17:15Z

Implementation for pytorch/kineto#155

facebook-github-bot · 2021-04-29T06:17:21Z

💊 CI failures summary and remediations

As of commit 853cb0f (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

…rofiler

codecov · 2021-04-29T15:10:35Z

Codecov Report

Merging #57252 (853cb0f) into master (3a66a1c) will decrease coverage by 0.00%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##           master   #57252      +/-   ##
==========================================
- Coverage   76.82%   76.82%   -0.01%     
==========================================
  Files        1984     1984              
  Lines      197163   197190      +27     
==========================================
+ Hits       151465   151484      +19     
- Misses      45698    45706       +8

facebook-github-bot · 2021-05-07T19:28:50Z

@ilia-cher has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/csrc/autograd/profiler_kineto.cpp

gdankel · 2021-05-07T21:05:44Z

@guotuofeng can you add an example json or screenshot of the metadata section (when pressing M button in chrome trace)?

guotuofeng · 2021-05-08T03:47:09Z

@guotuofeng can you add an example json or screenshot of the metadata section (when pressing M button in chrome trace)?

The metadata is like:

"metadata": {
"devices": "[{"id": 0, "name": "Tesla V100-DGXS-32GB", "multi_processor_count": 80, "total_memory": 34084028416}, {"id": 1, "name": "Tesla V100-DGXS-32GB", "multi_processor_count": 80, "total_memory": 34087305216}, {"id": 2, "name": "Tesla V100-DGXS-32GB", "multi_processor_count": 80, "total_memory": 34087305216}, {"id": 3, "name": "Tesla V100-DGXS-32GB", "multi_processor_count": 80, "total_memory": 34087305216}]",
"distributed": "{"backend": "nccl", "rank": 0, "world_size": 2}"
},

facebook-github-bot · 2021-05-08T06:41:37Z

@ilia-cher has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-05-10T20:30:27Z

@ilia-cher merged this pull request in 98fcdb8.

facebook-github-bot · 2021-05-10T22:30:47Z

This pull request has been reverted by 0361671.

ilia-cher · 2021-05-11T00:31:33Z

(will resend)

Summary: Resending #57252 Test Plan: CI [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI ghstack-source-id: 6f18720 Pull Request resolved: #58010

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending pytorch#57252 Test Plan: CI ghstack-source-id: 2e63611 Pull Request resolved: pytorch#58010

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

…tributed metadata" Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Resending #57252 Test Plan: CI Differential Revision: [D28345161](https://our.internmc.facebook.com/intern/diff/D28345161) [ghstack-poisoned]

Summary: Pull Request resolved: #58010 Resending #57252 Test Plan: CI Reviewed By: gdankel Differential Revision: D28345161 Pulled By: ilia-cher fbshipit-source-id: 18be07b275403205f5b5487ae3589bd39a8eac96

Summary: Implementation for pytorch/kineto#155 Pull Request resolved: pytorch#57252 Reviewed By: gdankel Differential Revision: D28294662 Pulled By: ilia-cher fbshipit-source-id: 3c71ffa333e341ff8113e891681a4905f54802dc

…58010) Summary: Pull Request resolved: pytorch#58010 Resending pytorch#57252 Test Plan: CI Reviewed By: gdankel Differential Revision: D28345161 Pulled By: ilia-cher fbshipit-source-id: 18be07b275403205f5b5487ae3589bd39a8eac96

add cuda memory and distributed metadata

d4725e2

guotuofeng requested review from albanD and soulitzer as code owners April 29, 2021 06:17

facebook-github-bot added the cla signed label Apr 29, 2021

pytorchbot added the open source label Apr 29, 2021

guotuofeng added 2 commits April 29, 2021 18:11

add device index

6084149

correct the exception type when catching in pybind layer for kineto p…

7b51ddf

…rofiler

gchanan requested a review from gdankel April 29, 2021 16:44

gchanan added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Apr 29, 2021

ilia-cher reviewed May 7, 2021

View reviewed changes

torch/csrc/autograd/profiler_kineto.cpp Outdated Show resolved Hide resolved

guotuofeng added 2 commits May 8, 2021 11:49

exception error message fix

ecc16a1

Merge remote-tracking branch 'upstream/master' into myguo/dist

853cb0f

ilia-cher self-requested a review May 10, 2021 19:16

facebook-github-bot closed this in 98fcdb8 May 10, 2021

facebook-github-bot added the Merged label May 10, 2021

facebook-github-bot added the Reverted label May 10, 2021

guotuofeng mentioned this pull request May 11, 2021

dump cudaOccDeviceProp pytorch/kineto#209

Closed

ilia-cher mentioned this pull request May 11, 2021

[profiler][resend] Add cuda memory and distributed metadata #58010

Closed

ilia-cher pushed a commit that referenced this pull request May 11, 2021

[profiler][resend] Add cuda memory and distributed metadata

efada45

Summary: Resending #57252 Test Plan: CI [ghstack-poisoned]

ilia-cher pushed a commit that referenced this pull request May 11, 2021

[profiler][resend] Add cuda memory and distributed metadata

fea5a4c

Summary: Resending #57252 Test Plan: CI ghstack-source-id: 6f18720 Pull Request resolved: #58010

dgl-intel pushed a commit to dgl-intel/pytorch that referenced this pull request May 11, 2021

[profiler][resend] Add cuda memory and distributed metadata

52dfd9f

Summary: Resending pytorch#57252 Test Plan: CI ghstack-source-id: 2e63611 Pull Request resolved: pytorch#58010

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add cuda memory and distributed metadata#57252

add cuda memory and distributed metadata#57252
guotuofeng wants to merge 5 commits intopytorch:masterfrom
guotuofeng:myguo/dist

guotuofeng commented Apr 29, 2021

Uh oh!

facebook-github-bot commented Apr 29, 2021 •

edited

Loading

Uh oh!

codecov bot commented Apr 29, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented May 7, 2021

Uh oh!

Uh oh!

gdankel commented May 7, 2021

Uh oh!

guotuofeng commented May 8, 2021

Uh oh!

facebook-github-bot commented May 8, 2021

Uh oh!

facebook-github-bot commented May 10, 2021

Uh oh!

facebook-github-bot commented May 10, 2021

Uh oh!

ilia-cher commented May 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

guotuofeng commented Apr 29, 2021

Uh oh!

facebook-github-bot commented Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

codecov bot commented Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

facebook-github-bot commented May 7, 2021

Uh oh!

Uh oh!

gdankel commented May 7, 2021

Uh oh!

guotuofeng commented May 8, 2021

Uh oh!

facebook-github-bot commented May 8, 2021

Uh oh!

facebook-github-bot commented May 10, 2021

Uh oh!

facebook-github-bot commented May 10, 2021

Uh oh!

ilia-cher commented May 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

facebook-github-bot commented Apr 29, 2021 •

edited

Loading

codecov bot commented Apr 29, 2021 •

edited

Loading