Skip to content

Add a transformers example#322

Merged
nzw0301 merged 19 commits intooptuna:mainfrom
ParagEkbote:transformers-example-for-optuna
May 16, 2025
Merged

Add a transformers example#322
nzw0301 merged 19 commits intooptuna:mainfrom
ParagEkbote:transformers-example-for-optuna

Conversation

@ParagEkbote
Copy link
Copy Markdown
Contributor

@ParagEkbote ParagEkbote commented Apr 26, 2025 •
edited
Loading

Description of the changes

Refs the following issue.

This PR adds an example showing the integration of transformers with optuna. We use the imdb dataset and prajjwal1/bert-tiny as the model. The Best trial is:
BestRun(run_id='1', objective=0.622, hyperparameters={'learning_rate': 3.341916221533824e-05, 'per_device_train_batch_size': 16}, run_summary=None)

Could you please review?

cc: @c-bata

@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented Apr 26, 2025 •
edited
Loading

It is important to note that since we are using a big model (~67M params) and CPU based systems, the CI checks will take a long time. The example can be executed on a T4 GPU with a duration of (~10 mins).

@nabenabe0928
Copy link
Copy Markdown
Contributor

nabenabe0928 commented Apr 28, 2025

@kAIto47802 Could you review this PR?

@nabenabe0928
Copy link
Copy Markdown
Contributor

nabenabe0928 commented Apr 28, 2025

@ParagEkbote

Could you reduce the computational time by using a much smaller model?

#322 (comment)

@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented Apr 28, 2025

I have updated the example to run on a much smaller model. Now the CI is being completed in a reasonable amount of time :)

cc: @nabenabe0928

@github-actions
Copy link
Copy Markdown

github-actions bot commented May 5, 2025

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label May 5, 2025
@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented May 6, 2025

Not stale.

@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented May 6, 2025

Could you please review the changes?

cc: @kAIto47802

@github-actions github-actions bot removed the stale Exempt from stale bot labeling. label May 6, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented May 13, 2025

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label May 13, 2025
@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented May 14, 2025

Not stale.

@ParagEkbote
Copy link
Copy Markdown
Contributor Author

ParagEkbote commented May 14, 2025

Could you please review?

cc: @kAIto47802

nzw0301
nzw0301 reviewed May 14, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@nzw0301 nzw0301 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using transformers nowadays, so let me leave a few suggestions.

nzw0301
nzw0301 reviewed May 14, 2025
View reviewed changes
ParagEkbote and others added 3 commits May 14, 2025 23:09
@ParagEkbote @nzw0301
Apply suggestions as per code review
cb3e0ac 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote @nzw0301
Update transformers/transformers_integration.py
34b32dd 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote
update
bc51973
@github-actions github-actions bot removed the stale Exempt from stale bot labeling. label May 14, 2025
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
@c-bata c-bata assigned nzw0301 and unassigned kAIto47802 May 15, 2025
@c-bata
Copy link
Copy Markdown
Member

c-bata commented May 15, 2025

Let me change the reviewer since @nzw0301 started the review for this PR.

@ParagEkbote ParagEkbote requested a review from nzw0301 May 15, 2025 06:46
nzw0301
nzw0301 requested changes May 15, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@nzw0301 nzw0301 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for quickly updating the PR. I would appreciate it if you followed my suggestions carefully, so as not to repeat a similar suggestion or the same suggestion.

ParagEkbote and others added 3 commits May 15, 2025 17:29
@ParagEkbote @nzw0301
Apply suggestions from code review-3
15ba919 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote @nzw0301
Update transformers/transformers_simple.py
3d83a90 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote
update.
b16bc25
@ParagEkbote ParagEkbote requested a review from nzw0301 May 15, 2025 12:39
nzw0301
nzw0301 requested changes May 15, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@nzw0301 nzw0301 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. let me leave a few more comments.

@nzw0301
Copy link
Copy Markdown
Member

nzw0301 commented May 15, 2025 •
edited
Loading

Please kindly stop commenting just Updated as discussed.. I've got too many notifications with uninformative content.

Best regards,

ParagEkbote and others added 2 commits May 15, 2025 19:39
@ParagEkbote @nzw0301
Apply suggestions from code review
dcd9202 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote
update the example.
f1bcc88
@ParagEkbote ParagEkbote requested a review from nzw0301 May 15, 2025 14:55
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
nzw0301
nzw0301 reviewed May 15, 2025
View reviewed changes
ParagEkbote and others added 2 commits May 15, 2025 21:00
@ParagEkbote @nzw0301
Apply suggestions from code review
b39aa94 
Co-authored-by: Kento Nozawa <k_nzw@klis.tsukuba.ac.jp>
@ParagEkbote
make style and update the example.
858a0db
@ParagEkbote ParagEkbote requested a review from nzw0301 May 15, 2025 16:12
@nzw0301 nzw0301 changed the title Add transformers Example for Optuna Add a transformers example May 16, 2025
nzw0301
nzw0301 approved these changes May 16, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@nzw0301 nzw0301 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks

@nzw0301 nzw0301 merged commit d55f622 into optuna:main May 16, 2025
5 checks passed
@nzw0301 nzw0301 added this to the v4.4.0 milestone May 16, 2025
@nzw0301 nzw0301 mentioned this pull request May 16, 2025
Merged
@ParagEkbote ParagEkbote deleted the transformers-example-for-optuna branch May 16, 2025 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nzw0301 nzw0301 nzw0301 approved these changes

Assignees

@nzw0301 nzw0301

Labels

None yet

Projects

None yet

Milestone

v4.4.0

Development

Successfully merging this pull request may close these issues.

5 participants

@ParagEkbote @nabenabe0928 @c-bata @nzw0301 @kAIto47802