Skip to content

[Regression] Local performance regression#321

Merged
yaoyaoding merged 15 commits intohidet-org:mainfrom
hjjq:regression
Jul 20, 2023
Merged

[Regression] Local performance regression#321
yaoyaoding merged 15 commits intohidet-org:mainfrom
hjjq:regression

Conversation

@hjjq
Copy link
Copy Markdown
Collaborator

@hjjq hjjq commented Jul 19, 2023

python ./scripts/regression/run.py -h at the repo root for usage.
Currently only some operator shapes and models, with 3090 latency data, are added. More operator/shapes/models/GPUs to be added.
Tolerance to report speedup/slowdown is set at +-8%. May adjust in the future accordingly.

yaoyaoding
yaoyaoding reviewed Jul 19, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@yaoyaoding yaoyaoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @hjjq, thanks!

Just one suggestion. Can we store the shapes to profile in a seperate json file instead of using the regression_data.json, which also records the profile data for previous runs.

  • configs.json: store all the shape information that we want to benchmark. (This information can also be written in the python source code directly).
  • regression_data.json: (optional) store the benchmark data for previous run, if have. We can generate one for the first run.

@hjjq
Copy link
Copy Markdown
Collaborator Author

hjjq commented Jul 19, 2023

Hi @yaoyaoding , I generated this current version of regression_data.json with a local run on my 3090.
What I did was similar to what triton did, where the latency numbers are stored in the repository together with the shapes.
I can create configs.json as you described, but regression_data.json will still be needed in order to compare against a reference run, and the reference latencies need to be updated in the repo every time we achieve a speedup (It will always contain the all-time-high peformance that Hidet can achieve on a particular GPU.)

@yaoyaoding
Copy link
Copy Markdown
Member

yaoyaoding commented Jul 19, 2023

I see, that makes sense.

@yaoyaoding yaoyaoding merged commit 73b7fdb into hidet-org:main Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaoyaoding yaoyaoding yaoyaoding approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hjjq @yaoyaoding