Skip to content

Gemma3 is Torch Exportable#37728

Merged
Cyrilvallez merged 2 commits intohuggingface:mainfrom
guangy10:gemma3_executorch
Apr 28, 2025
Merged

Gemma3 is Torch Exportable#37728
Cyrilvallez merged 2 commits intohuggingface:mainfrom
guangy10:gemma3_executorch

Conversation

@guangy10
Copy link
Contributor

@guangy10 guangy10 commented Apr 23, 2025 •
edited
Loading

What does this PR do?

Initial effort to add torch.expport support for the Gemma3 model!
Gemma3 provides a 1b variant that is suitable for ExecuTorch to bring it for on-device use-case. This PR is focusing on creating the export recipe and validate the exported model can produce same output as eager.

Expand support to other models that utilize HybridCache as well including gemma2 and cohere2.

End2end Test Validation with Exported Graph

RUN_SLOW=1 pytest tests/models/gemma3/test_modeling_gemma3.py -s -v -k test_export_text_only_with_hybrid_cache

Export generated texts: 'What is the capital of France?

The capital of France is Paris.

Final Answer: The final answer is $\boxed{Paris'


Eager generated texts: 'What is the capital of France?

The capital of France is Paris.

Final Answer: The final answer is $\boxed{Paris'

PASSED
======================================================= 1 passed, 319 deselected, 108 warnings in 26.95s =======================================================

Ene2end Validation in Optimum-ExecuTorch

Screenshot 2025-04-24 at 2 01 49 PM

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @gante @qubvel

@github-actions
Copy link
Contributor

github-actions bot commented Apr 23, 2025

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@github-actions github-actions bot marked this pull request as draft April 23, 2025 23:50
@guangy10 guangy10 mentioned this pull request Apr 23, 2025
Open
33 tasks
@guangy10 guangy10 force-pushed the gemma3_executorch branch from 41ccd3d to e120479 Compare April 23, 2025 23:53
guangy10
guangy10 commented Apr 23, 2025
View reviewed changes
guangy10
guangy10 commented Apr 23, 2025
View reviewed changes
@guangy10 guangy10 force-pushed the gemma3_executorch branch from e120479 to c3a03f9 Compare April 24, 2025 00:01
@guangy10
Copy link
Contributor Author

guangy10 commented Apr 24, 2025

CC: @tugsbayasgalan for review as well

@guangy10 guangy10 marked this pull request as ready for review April 24, 2025 00:09
@guangy10 guangy10 force-pushed the gemma3_executorch branch from c3a03f9 to c126d47 Compare April 24, 2025 18:04
@guangy10
Copy link
Contributor Author

guangy10 commented Apr 24, 2025 •
edited
Loading

removed unused buffer

@guangy10 guangy10 force-pushed the gemma3_executorch branch from c126d47 to 98df630 Compare April 24, 2025 20:53
@guangy10
Copy link
Contributor Author

guangy10 commented Apr 24, 2025

Fixed in the ExportableModule to make gemma3 lowerable to ExecuTorch

@guangy10 guangy10 force-pushed the gemma3_executorch branch 6 times, most recently from 13dbffd to 6ec580f Compare April 25, 2025 01:09
@guangy10 guangy10 mentioned this pull request Apr 25, 2025
Merged
2 tasks
@guangy10
Copy link
Contributor Author

guangy10 commented Apr 25, 2025

I've ran make fixup but it doesn't fixed the linter.

@guangy10 guangy10 force-pushed the gemma3_executorch branch from 6ec580f to 6f389db Compare April 25, 2025 01:18
guangy10
guangy10 commented Apr 25, 2025
View reviewed changes
@guangy10 guangy10 force-pushed the gemma3_executorch branch from 6f389db to bff3c2f Compare April 25, 2025 01:20
@Cyrilvallez
Copy link
Member

Cyrilvallez commented Apr 25, 2025

Hey @guangy10! Thanks for the PR! The changes LGTM, however the gemma3 change should be reflected in modular_gemma3.py, which is the source of modeling_gemma3.py! 🤗
Any way you could downstream the small change to other models using Hybridcache as well? They should be the models that were changed in #37447 👌

@Cyrilvallez
Copy link
Member

Cyrilvallez commented Apr 25, 2025

It's the source of the issue in check_repo_consistency 😉

@Cyrilvallez
Copy link
Member

Cyrilvallez commented Apr 25, 2025

cc @gante as well for viz!

@guangy10
Copy link
Contributor Author

guangy10 commented Apr 25, 2025

Hey @guangy10! Thanks for the PR! The changes LGTM, however the gemma3 change should be reflected in modular_gemma3.py, which is the source of modeling_gemma3.py! 🤗 Any way you could downstream the small change to other models using Hybridcache as well? They should be the models that were changed in #37447 👌

Done. Could you take another look?

@guangy10 guangy10 force-pushed the gemma3_executorch branch from 0729eff to ae84d3f Compare April 25, 2025 19:34
@guangy10 guangy10 mentioned this pull request Apr 25, 2025
Closed
Cyrilvallez
Cyrilvallez approved these changes Apr 28, 2025
View reviewed changes
Copy link
Member

@Cyrilvallez Cyrilvallez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks a lot! Great work, super clean 🤗
Merging

@Cyrilvallez Cyrilvallez merged commit 816b370 into huggingface:main Apr 28, 2025
13 checks passed
sushmanthreddy pushed a commit to sushmanthreddy/transformers that referenced this pull request Apr 28, 2025
@guangy10 @sushmanthreddy
Gemma3 is Torch Exportable (huggingface#37728)
1b773ba 
* Gemma3 is Torch Exportable

* Expand the support to other mdoels using HybridCache

---------

Co-authored-by: Guang Yang <guangyang@fb.com>
This was referenced Apr 28, 2025
Merged
Merged
zucchini-nlp pushed a commit to zucchini-nlp/transformers that referenced this pull request May 14, 2025
@guangy10 @zucchini-nlp
Gemma3 is Torch Exportable (huggingface#37728)
21067f2 
* Gemma3 is Torch Exportable

* Expand the support to other mdoels using HybridCache

---------

Co-authored-by: Guang Yang <guangyang@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Cyrilvallez Cyrilvallez Cyrilvallez approved these changes

+1 more reviewer

@tugsbayasgalan tugsbayasgalan tugsbayasgalan left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@guangy10 @Cyrilvallez @tugsbayasgalan