Skip to content

Checkout torchbench with only needed models#94578

Closed
weiwangmeta wants to merge 5 commits intomasterfrom
weiwangmeta/slim_checkout_torchbench
Closed

Checkout torchbench with only needed models#94578
weiwangmeta wants to merge 5 commits intomasterfrom
weiwangmeta/slim_checkout_torchbench

Conversation

@weiwangmeta
Copy link
Copy Markdown
Contributor

@weiwangmeta weiwangmeta commented Feb 10, 2023 •
edited
Loading

Addresses (#93395 (comment)) The perf smoke test is supposed to be around one minute. But the torchbench checkout process is taking more than 15 minutes. This PR explores a way to just checkout torchbench with only needed models that are later used to do perf smoke test and memory compression ratio check.

Torchbench installation has "python install.py models model1 model 2 model3" support to just install model1 model2 and model3, not providing "models model1 model2 model3" would install all models by default.

Before this PR, inductor job takes about 27 minutes (21 minutes spent in testing phase) https://github.com/pytorch/pytorch/actions/runs/4149154553/jobs/7178024253
After this PR, inductor job takes about 19 minutes (12 minutes spent in testing phase), pytorch checkout and docker image pull takes about 5 - 6 minutes total. https://github.com/pytorch/pytorch/actions/runs/4149155814/jobs/7178735494

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Feb 10, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94578

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3bb2749:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@weiwangmeta weiwangmeta added ciflow/inductor topic: not user facing topic category ciflow/trunk Trigger trunk jobs on your pull request labels Feb 10, 2023
@weiwangmeta weiwangmeta force-pushed the weiwangmeta/slim_checkout_torchbench branch from d8f39ed to d116d4f Compare February 10, 2023 18:43
@weiwangmeta weiwangmeta changed the title [Draft] Checkout torchbench with only one model Checkout torchbench with only one model Feb 11, 2023
@weiwangmeta weiwangmeta changed the title Checkout torchbench with only one model Checkout torchbench with only needed models Feb 11, 2023
@weiwangmeta
Copy link
Copy Markdown
Contributor Author

weiwangmeta commented Feb 11, 2023

@pytorchbot rebase

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 11, 2023

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 11, 2023

Successfully rebased weiwangmeta/slim_checkout_torchbench onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weiwangmeta/slim_checkout_torchbench && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the weiwangmeta/slim_checkout_torchbench branch from c651c4f to 058cc9c Compare February 11, 2023 07:09
@weiwangmeta
Copy link
Copy Markdown
Contributor Author

weiwangmeta commented Feb 12, 2023

@pytorchbot rebase

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 12, 2023

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 12, 2023

Successfully rebased weiwangmeta/slim_checkout_torchbench onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weiwangmeta/slim_checkout_torchbench && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the weiwangmeta/slim_checkout_torchbench branch from 058cc9c to ab2d70d Compare February 12, 2023 00:38
@weiwangmeta
Copy link
Copy Markdown
Contributor Author

weiwangmeta commented Feb 12, 2023

@pytorchbot rebase

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 12, 2023

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 12, 2023

Successfully rebased weiwangmeta/slim_checkout_torchbench onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weiwangmeta/slim_checkout_torchbench && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the weiwangmeta/slim_checkout_torchbench branch from ab2d70d to 3bb2749 Compare February 12, 2023 16:28
@weiwangmeta
Copy link
Copy Markdown
Contributor Author

weiwangmeta commented Feb 13, 2023

@pytorchbot merge

1 similar comment
@weiwangmeta
Copy link
Copy Markdown
Contributor Author

weiwangmeta commented Feb 13, 2023

@pytorchbot merge

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Feb 13, 2023

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@weiwangmeta weiwangmeta deleted the weiwangmeta/slim_checkout_torchbench branch February 13, 2023 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malfet malfet malfet approved these changes

@desertfire desertfire desertfire approved these changes

+1 more reviewer

@orionr orionr orionr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@weiwangmeta @pytorchmergebot @orionr @malfet @desertfire