Skip to content

Upload MPS benchmark results#141087

Closed
huydhn wants to merge 6 commits intomainfrom
upload-mps-benchmark-results
Closed

Upload MPS benchmark results#141087
huydhn wants to merge 6 commits intomainfrom
upload-mps-benchmark-results

Conversation

@huydhn
Copy link
Copy Markdown
Contributor

@huydhn huydhn commented Nov 20, 2024

This uploads the MPS benchmark results to benchmark database. The data can then be queried, for example:

select benchmark, model, metric from oss_ci_benchmark_v3 where head_sha = '99a133116fee15aa1467165f2b209b37da53f189' and metric.name in ['eager_peak_mem', 'dynamo_peak_mem', 'speedup'] and model.name = 'BERT_pytorch'

I'm documenting the JSON format at https://github.com/pytorch/pytorch/wiki/How-to-integrate-with-PyTorch-OSS-benchmark-database

Testing

Locally,

PYTHONPATH=/Users/huydo/Storage/mine/benchmark python benchmarks/dynamo/torchbench.py --performance --only resnet152 --backend eager --training --devices mps --output test/test-reports/torchbench_training.csv

Workflow dispatch https://github.com/pytorch/pytorch/actions/runs/11927990520

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Nov 20, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141087

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit 16a4c61 with merge base 4f2543c (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

huydhn added a commit to pytorch/test-infra that referenced this pull request Nov 20, 2024
This is the format used by ClickHouse, so it's useful to support this
too in addition to a valid JSON list of benchmark records.

### Testing

* Adding a test case
https://github.com/pytorch/test-infra/actions/runs/11927946564/job/33244134505#step:5:124
* I'm also testing this in
pytorch/pytorch#141087
@huydhn huydhn marked this pull request as ready for review November 20, 2024 18:10
@huydhn huydhn requested a review from a team as a code owner November 20, 2024 18:10
@malfet
Copy link
Copy Markdown
Contributor

malfet commented Nov 20, 2024

@pytorchbot merge -f "Lint is green and then job is periodic anyway"

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorchmergebot pushed a commit that referenced this pull request Nov 20, 2024
facebook-github-bot pushed a commit to pytorch/benchmark that referenced this pull request Nov 20, 2024
Summary:
This uploads the MPS benchmark results to benchmark database.  The data can then be queried, for example:

```
select benchmark, model, metric from oss_ci_benchmark_v3 where head_sha = '99a133116fee15aa1467165f2b209b37da53f189' and metric.name in ['eager_peak_mem', 'dynamo_peak_mem', 'speedup'] and model.name = 'BERT_pytorch'
```

I'm documenting the JSON format at https://github.com/pytorch/pytorch/wiki/How-to-integrate-with-PyTorch-OSS-benchmark-database

### Testing

Locally,

```
PYTHONPATH=/Users/huydo/Storage/mine/benchmark python benchmarks/dynamo/torchbench.py --performance --only resnet152 --backend eager --training --devices mps --output test/test-reports/torchbench_training.csv
```

Workflow dispatch https://github.com/pytorch/pytorch/actions/runs/11927990520

X-link: pytorch/pytorch#141087
Approved by: https://github.com/malfet

Reviewed By: izaitsevfb

Differential Revision: D66262910

Pulled By: huydhn

fbshipit-source-id: 318fbb0ded0e85742aa441e023f5abf906398997
facebook-github-bot pushed a commit to pytorch/benchmark that referenced this pull request Nov 20, 2024
Summary:
A small fix I just realize after pytorch/pytorch#141087.

X-link: pytorch/pytorch#141145
Approved by: https://github.com/malfet

Reviewed By: izaitsevfb

Differential Revision: D66262943

Pulled By: huydhn

fbshipit-source-id: 4ccec00d23bac753fbba3ad1a39de1569efd11f8
pytorchmergebot pushed a commit that referenced this pull request Dec 2, 2024
)

Similar to #141087, this uploads the micro benchmark results to benchmark database with its new schema v3. The data can then be queried.

~I'm testing with `inductor-micro-benchmark-x86` which should be sufficient because `inductor-micro-benchmark` is broken atm.  The CSV output stays for now until the dashboard is migrated to schema v3.~ #141747 has been resolved, so inductor-micro-benchmark should work now

Pull Request resolved: #141745
Approved by: https://github.com/yanboliang
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
This uploads the MPS benchmark results to benchmark database.  The data can then be queried, for example:

```
select benchmark, model, metric from oss_ci_benchmark_v3 where head_sha = '99a133116fee15aa1467165f2b209b37da53f189' and metric.name in ['eager_peak_mem', 'dynamo_peak_mem', 'speedup'] and model.name = 'BERT_pytorch'
```

I'm documenting the JSON format at https://github.com/pytorch/pytorch/wiki/How-to-integrate-with-PyTorch-OSS-benchmark-database

### Testing

Locally,

```
PYTHONPATH=/Users/huydo/Storage/mine/benchmark python benchmarks/dynamo/torchbench.py --performance --only resnet152 --backend eager --training --devices mps --output test/test-reports/torchbench_training.csv
```

Workflow dispatch https://github.com/pytorch/pytorch/actions/runs/11927990520

Pull Request resolved: pytorch#141087
Approved by: https://github.com/malfet
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
…rch#141745)

Similar to pytorch#141087, this uploads the micro benchmark results to benchmark database with its new schema v3. The data can then be queried.

~I'm testing with `inductor-micro-benchmark-x86` which should be sufficient because `inductor-micro-benchmark` is broken atm.  The CSV output stays for now until the dashboard is migrated to schema v3.~ pytorch#141747 has been resolved, so inductor-micro-benchmark should work now

Pull Request resolved: pytorch#141745
Approved by: https://github.com/yanboliang
@github-actions github-actions bot deleted the upload-mps-benchmark-results branch December 21, 2024 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants