Add script to convert T5X T5 (v1.0 and v1.1) checkpoints to PyTorch by bastings · Pull Request #20801 · huggingface/transformers

bastings · 2022-12-16T13:31:08Z

What does this PR do?

Adds a script that can convert Google T5X (Flax) T5 and T5-v1.1 checkpoints into PyTorch checkpoints.
This allows users to convert non-standard checkpoints that have been trained with T5X and use them with the Transformers library in PyTorch.

Usage:

In case you don't have gsutil, install according to https://cloud.google.com/storage/docs/gsutil_install
Native T5X checkpoints are at https://github.com/google-research/t5x/blob/main/docs/models.md#t5-11-checkpoints. Example:
gsutil -m cp -r gs://t5-data/pretrained_models/t5x/t5_1_1_small $HOME/
Create a corresponding config.json for the downloaded checkpoint. Often one already exists, e.g. here we can use https://huggingface.co/google/t5-v1_1-small/blob/main/config.json
Finally python3 convert_t5x_checkpoint_to_pytorch.py --t5x_checkpoint_path=$HOME/t5_1_1_small --config_file=config.json --pytorch_dump_path=$HOME/t5_1_1_small_pt

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. Discussed with @thomwolf .
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests? The code is tested but not part of this PR, since the test requires manually downloading the T5X checkpoints from a cloud bucket.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten
@sanchit-gandhi
@ArthurZucker
@younesbelkada

HuggingFaceDocBuilderDev · 2022-12-16T13:55:48Z

The documentation is not available anymore as the PR was closed or merged.

bastings · 2022-12-18T09:07:22Z

I could use some clarification on the following: I'm missing a configuration option for T5 for the 1.0/original T5 checkpoints to have an lm_head that shares parameters with the token embeddings.

Currently there is T5Model (which returns hidden states) and T5ForConditionalGeneration (which returns logits, used for T5 v1.1 models among others). The latter assumes there is an lm_head layer, but for the 1.0 checkpoints there is no such thing, it reuses the embedding matrix to map to the vocab space.

patrickvonplaten

Thanks a lot for adding this @bastings

cc @ArthurZucker

ArthurZucker · 2022-12-19T12:31:30Z

Hey @bastings, when there is no lm_head you have to set the tie_word_embeddings to True

ArthurZucker

It's very clean, thanks a lot for the addition.

sanchit-gandhi

Very cool PR @bastings! Thanks for the addition! Do you have a set of example args I could use just to try the script out once for myself? Thanks! 🙌

bastings · 2022-12-21T12:49:34Z

I added the instructions to the top docstring. Maybe it's ready? :-)

ArthurZucker · 2022-12-21T15:33:35Z

A last nit and we can merge! Thanks a lot for bearing with me 😄

bastings · 2022-12-21T15:42:50Z

Thanks! Committed your suggestion :)

sanchit-gandhi · 2022-12-21T16:35:52Z

LGTM! Thanks!

sanchit-gandhi · 2022-12-21T16:37:52Z

Once the quality tests are green (requires make fixup) we can merge!

bastings · 2022-12-21T17:04:03Z

Oh looks like the suggestion made it fail ;)

ArthurZucker · 2022-12-21T17:17:13Z

Ah, sorry then ahha, I guess the make stylewill correct that 😅

bastings · 2022-12-22T19:37:15Z

Ah, sorry then ahha, I guess the make stylewill correct that 😅

Fixed! :)

…uggingface#20801) * Add script to convert T5X T5 (v1.0 and v1.1) checkpoints to PyTorch * Remove unnecessary check and update docstring * Format docstring * Fix whitespace in docstring

Add script to convert T5X T5 (v1.0 and v1.1) checkpoints to PyTorch

f342f24

bastings force-pushed the t5x_to_pytorch branch from ff2c4af to f342f24 Compare December 16, 2022 19:27

patrickvonplaten approved these changes Dec 19, 2022

View reviewed changes

patrickvonplaten requested a review from ArthurZucker December 19, 2022 11:44

ArthurZucker approved these changes Dec 19, 2022

View reviewed changes

Comment thread src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py Outdated

sanchit-gandhi approved these changes Dec 19, 2022

View reviewed changes

Comment thread src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py Outdated

Comment thread src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py

bastings force-pushed the t5x_to_pytorch branch from a3c1dd9 to 0d9d0af Compare December 21, 2022 10:14

Remove unnecessary check and update docstring

7fa9b8d

bastings force-pushed the t5x_to_pytorch branch 2 times, most recently from ea37c40 to c529472 Compare December 21, 2022 12:28

Format docstring

6043de3

bastings force-pushed the t5x_to_pytorch branch from c529472 to 6043de3 Compare December 21, 2022 12:37

ArthurZucker reviewed Dec 21, 2022

View reviewed changes

Comment thread src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py Outdated

sanchit-gandhi reviewed Dec 21, 2022

View reviewed changes

Comment thread src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py Outdated

sanchit-gandhi Dec 21, 2022

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

Fix whitespace in docstring

7032ed3

bastings force-pushed the t5x_to_pytorch branch from 0f279bb to 7032ed3 Compare December 22, 2022 13:16

ArthurZucker merged commit efed8a2 into huggingface:main Dec 23, 2022

soyoung97 mentioned this pull request Oct 4, 2023

Fix encoder->decoder typo bug in convert_t5x_checkpoint_to_pytorch.py #26587

Merged

5 tasks

Conversation

bastings commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bastings commented Dec 18, 2022

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker commented Dec 19, 2022

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sanchit-gandhi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bastings commented Dec 21, 2022

Uh oh!

Uh oh!

ArthurZucker commented Dec 21, 2022

Uh oh!

bastings commented Dec 21, 2022

Uh oh!

sanchit-gandhi Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi commented Dec 21, 2022

Uh oh!

bastings commented Dec 21, 2022

Uh oh!

ArthurZucker commented Dec 21, 2022

Uh oh!

bastings commented Dec 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bastings commented Dec 16, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 16, 2022 •

edited

Loading