Skip to content

[benchmarks] Fix verifier error handling and OOM.#7777

Merged
ysiraichi merged 4 commits intomasterfrom
ysiraichi/skip-experiment-if-verify
Aug 5, 2024
Merged

[benchmarks] Fix verifier error handling and OOM.#7777
ysiraichi merged 4 commits intomasterfrom
ysiraichi/skip-experiment-if-verify

Conversation

@ysiraichi
Copy link
Copy Markdown
Collaborator

In summary, this PR:

  • Skips the actual experiment when running the verifier, i.e. goes directly into the verifier code, instead of running the experiment first (the experiment is still run in the end of the verifier, though)
  • Handles errors when running the model on fp64, turning cosine similarity verification ON on any exception
  • Renames a few confusing VerificationCode enum names

cc @miladm @JackCaoG @zpcore

@ysiraichi ysiraichi requested review from miladm and zpcore July 30, 2024 20:36
Comment thread benchmarks/verifier.py Outdated
Comment thread benchmarks/experiment_runner.py
Comment thread benchmarks/experiment_runner.py
Copy link
Copy Markdown
Member

@zpcore zpcore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants