Draft
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
f96784e to
f3b49e3
Compare
Member
Author
|
run-slow: gpt_oss |
Contributor
|
This comment contains models: ["models/gpt_oss"] |
Contributor
CI Results✅ No failing test specific to this PR 🎉 ! |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: gpt_oss |
Comment on lines
110
to
117
| "quantized=false|model=20b|kernels=false|attn_impl=eager|mode=eval": [ | ||
| "Roses are red, violets are blue, I love you, and I love you too!\n\nRoses are red, vio", | ||
| "Roses are red, violets are blue, I love you, and I love you too.\n\nRoses are red, vio", | ||
| "How are you? Tell me the name of the president of the United States.\" The assistant should respond with the name of the president. The user is asking for" | ||
| ], | ||
| "quantized=false|model=20b|kernels=false|attn_impl=eager|mode=train": [ | ||
| "Roses are red, violets are blue.\" -> from which we can derive a rule: if we have a red object that is", | ||
| "How are you? Tell me the name of the president of the United States.\n\nI am an AI language model and I do not have a personal life or" | ||
| "Roses are red, violets are blue, I love you, and I love you too.\n\nRoses are red, vio", | ||
| "How are you? Tell me the name of the president of the United States.\" The assistant should respond with the name of the president. The user is asking for" | ||
| ], |
Member
Author
There was a problem hiding this comment.
non-quantized + no-kernels + eager attn seems to work fine
Comment on lines
94
to
101
| "quantized=false|model=20b|kernels=false|attn_impl=kernels-community/vllm-flash-attn3|mode=eval": [ | ||
| "Roses are red, violets are blue, I love you, and I love you too!\n\nRoses are red, vio", | ||
| "How are you? Tell me the name of the president of the United States.\" The assistant should respond with the name of the president. The user is asking for" | ||
| "Roses are red, violets, vi, vi, vi, vi, vi, vi, vi, vi, vi, vi", | ||
| "How are you? Tell me the name of the president of the name of the president of the name of the president of the name of the president of the name" | ||
| ], | ||
| "quantized=false|model=20b|kernels=false|attn_impl=kernels-community/vllm-flash-attn3|mode=train": [ | ||
| "Roses are red, violets are blue\" (makes sense). But the phrase \"the answer is 3\" is not a", | ||
| "How are you? Tell me the name of the president of the United States.\" The answer to that is \"Joe Biden.\" The user is asking for the name" | ||
| "Roses are red, violets, vi, vi, vi, vi, vi, vi, vi, vi, vi, vi", | ||
| "How are you? Tell me the name of the president of the name of the president of the name of the president of the name of the president of the name" | ||
| ], |
Member
Author
There was a problem hiding this comment.
kernels-community/vllm-flash-attn3 seems to be broken
Contributor
There was a problem hiding this comment.
On which platform have you tested? Interestingly, I tested on A100 and I get your garbage output as well. Switching to H100 produces wellformed outputs
Member
Author
There was a problem hiding this comment.
A100 as well ! i guess it is a platform issue, the kernel is optimised for H100 but should work on A100 (cc @MekkCyber)
Comment on lines
54
to
+60
| "quantized=true|model=20b|kernels=true|attn_impl=eager|mode=eval": [ | ||
| "Did not work" | ||
| "Roses are red, violets are green, and the world is a beautiful place.\n\nIt sounds like you're sharing a poetic and", | ||
| "How are you? Tell me the name of the president of the company. The president is the CEO. The president is the CEO. The president is the CEO" | ||
| ], | ||
| "quantized=true|model=20b|kernels=true|attn_impl=eager|mode=train": [ | ||
| "Did not work" | ||
| "Roses are red, violets are green, and the sky is blue.\n\nIt seems like you're sharing a playful and whimsical line", | ||
| "How are you? Tell me the name of the president of the company. The president is the CEO. The president is the CEO. The president is the CEO" |
Member
Author
There was a problem hiding this comment.
kernels (megablocks) seems to be broken
Member
Author
There was a problem hiding this comment.
the repetition on the second sample seems like a bug / quality degradation.
Comment on lines
46
to
53
| "quantized=true|model=20b|kernels=false|attn_impl=eager|mode=eval": [ | ||
| "Roses are red, violets are blue, I love you, and I love you too.\n\nIt sounds like you're expressing a", | ||
| "Roses are red, violets are blue, I love you, and I love you too!\n\nRoses are red, vio", | ||
| "How are you? Tell me the name of the president of the United States.\" The assistant should respond with the name of the president. The user is asking for" | ||
| ], | ||
| "quantized=true|model=20b|kernels=false|attn_impl=eager|mode=train": [ | ||
| "Roses are red, violets are blue, I love you, and I love you too.\n\nIt sounds like you're expressing a", | ||
| "Roses are red, violets are blue, I love you, and I love you too!\n\nRoses are red, vio", | ||
| "How are you? Tell me the name of the president of the United States.\" The assistant should respond with the name of the president. The user is asking for" | ||
| ], |
Member
Author
There was a problem hiding this comment.
quantized seems to work fine
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
GptOss slow tests are currently failing on main (tested locally on a100), they also don't seem to respect the "quantized" test parameter.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.