Conversation
|
Feel free to ping me for a review whenever this is ready 🤗 |
@ArthurZucker , I only identified 3 applicable model classes and made changes. Please check my logic in classes selection in my big first message above. |
ArthurZucker
left a comment
There was a problem hiding this comment.
#28132 might be relevant to you! I'll make sure 4d can still be used but will make things a lot easier I think.
LGTM otherwise!
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Has something like this or equivalent been merged? I don't see much documentation for 4D masks but would find useful! |
|
Yep sorry, it's supported for some models. doc is a bit scarce, feel free to open a pr to add this! |
following #27539 this PR adds updates to transformers documentation to reflect possibility of utilizing 4D masks.
Plan:
position_idsargument) and updating their docstrings. Classes that need updates:Update 20.12.2023:
to find out which models require docstring changes, I scanned all model classes in transformers insing inspect.
position_idsargument in.forward()- can't use 4D mask effectively_prepare_4d_attention_maskmethod - need different code change to use 4D maskwhat is left is LlamaModel, FalconModel and XGLMModel.
cc @ArthurZucker