Fix: Mamba2 generation mismatch between input_ids and inputs_embeds#32694
Fix: Mamba2 generation mismatch between input_ids and inputs_embeds#32694ArthurZucker merged 2 commits intohuggingface:mainfrom
Conversation
|
Lmk if there's a better way to do this but I think this should be ok. At least the cause of the issue is apparent now. |
|
That's a nice fix, thanks! I wonder if we can use |
ArthurZucker
left a comment
There was a problem hiding this comment.
Can you add a small test. They exist for other model in the Mixin, https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L2777 should be tested for mamba models
| if input_ids.shape[1] == 0: | ||
| past_len = inputs_embeds.shape[1] | ||
| if inputs_embeds is not None: | ||
| past_len = inputs_embeds.shape[1] + input_ids.shape[1] |
There was a problem hiding this comment.
this is no longer the past length, but the current length 😉 Otherwise as this is used to create the cache positions, this is valid 😉
|
@ArthurZucker Isn't it also tested for all mamba-related models already? I've run the mamba2 tests locally for example and the test you linked was included in them (can also be seen in the CI of this PR over here). Maybe I've misunderstood something 😅 Could you clarify the test? |
|
You are right I forgot that it was failing ever since we added it! |
|
Thanks @vasqu for maintaining mamba2 😉 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Fixes the failing generation test in mamba2 comparing the generation between passing input ids or passing the embeds directly. The cause of the mismatch was a false calculation of
past_lenwhich can be solved by changing the if check slightly.Fixes #32555
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@molbap @ArthurZucker