Memory Efficient GRPO by danielhanchen · Pull Request #1773 · unslothai/unsloth

danielhanchen · 2025-02-20T12:22:35Z

No description provided.

Fix typo in comment: know -> now. This was printed when running the Llama3.1_(8B)-GRPO.ipynb example notebook, so I'd expect others to run into it as well.

* fix an import error * Delete .gitignore * Update loader.py * Update save.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Add latest xformers * Add a couple of lines to docs

…ightly

* Update __init__.py * Update loader.py * Update rl.py * Update rl.py * Update _utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Better TRL handling * Update rl.py * Update tokenizer_utils.py * Auto patching * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update rl.py * Update tokenizer_utils.py * Update rl.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update tokenizer_utils.py * Update rl.py * Update rl.py * Update rl.py * max seq length * Update rl.py * Update rl.py * Patching * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * NEFTune * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Extra replacements * Update rl_replacements.py * Update rl.py * extra RL replacements * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update _utils.py * Update loader_utils.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * autocast * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update pyproject.toml * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update _utils.py * Update llama.py * Update _utils.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * GRPO optimized * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Selective Log softmax * Fix GRPO bsz * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Fix TRL * Metrics GRPO * Update rl_replacements.py * Update rl_replacements.py * No compile * Update rl.py * Remove docs * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * llama-quantize on WINDOWS WSL error fix - edit save.py (gguf saving breaks) (unslothai#1649) * edit save.py to fix gguf saving breaks. * add check for .exe or not exe file extension for linux and windows * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * unsloth_num_chunks * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py (unslothai#1754) Fix typo in comment: know -> now. This was printed when running the Llama3.1_(8B)-GRPO.ipynb example notebook, so I'd expect others to run into it as well. * Optional logits * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * fix an import error (unslothai#1767) * fix an import error * Delete .gitignore * Update loader.py * Update save.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * SamplingParams * Convert mask to float (unslothai#1762) * [Windows Support] Add latest `xformers` wheels to pyproject.toml (unslothai#1753) * Add latest xformers * Add a couple of lines to docs * vLLMSamplingParams * Update __init__.py * default num_chunks == -1 * Versioning --------- Co-authored-by: Gennadii Manzhos <105049664+everythingisc00l@users.noreply.github.com> Co-authored-by: Seth Weidman <seth@sethweidman.com> Co-authored-by: Nino Risteski <95188570+NinoRisteski@users.noreply.github.com> Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Ben <6579034+versipellis@users.noreply.github.com>

danielhanchen added 30 commits February 6, 2025 02:32

Update __init__.py

cfb1a00

Update loader.py

1f5a418

Merge branch 'main' into nightly

7a8d910

Update rl.py

34d92aa

Update rl.py

8b7c3af

Update _utils.py

066ec25

Merge branch 'main' into nightly

cb90aae

Merge branch 'main' into nightly

af8a21b

Update tokenizer_utils.py

052b93f

Update tokenizer_utils.py

fdac025

Better TRL handling

ade058e

Update rl.py

15073c0

Update tokenizer_utils.py

0c54b1e

Auto patching

a820ac6

Update tokenizer_utils.py

15c5220

Update tokenizer_utils.py

92a9f0b

Update tokenizer_utils.py

61b1853

Update rl.py

ea8739d

Update tokenizer_utils.py

61699bf

Update rl.py

acbf23f

Update tokenizer_utils.py

b1b9af3

Update tokenizer_utils.py

fee37b0

Update tokenizer_utils.py

ff27094

Update tokenizer_utils.py

6ab51be

Update tokenizer_utils.py

b45f633

Update tokenizer_utils.py

fd9e677

Update tokenizer_utils.py

b9b3166

Update tokenizer_utils.py

7fdab17

Update rl.py

2595971

Update rl.py

f470f55

danielhanchen and others added 27 commits February 18, 2025 00:17

Update rl_replacements.py

6732822

Update rl_replacements.py

5efe9f3

Update rl_replacements.py (#1754)

15442d1

Fix typo in comment: know -> now. This was printed when running the Llama3.1_(8B)-GRPO.ipynb example notebook, so I'd expect others to run into it as well.

Optional logits

91ab43d

Update rl.py

a6a5f60

Update rl.py

83ce085

Update rl.py

8ece11f

Update rl.py

bc6bfae

Update rl.py

95fb6a4

Update rl.py

ba01cf5

Update rl.py

eb48b98

Update rl.py

3c750a1

Update rl_replacements.py

515cf5a

Update rl.py

2cf4349

Update rl.py

ae8bf68

Update rl.py

e07f4bc

Merge branch 'main' into nightly

f11e5ab

Update rl.py

3fccf5d

fix an import error (#1767)

798ad95

* fix an import error * Delete .gitignore * Update loader.py * Update save.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>

SamplingParams

2957d89

Convert mask to float (#1762)

19d57bc

[Windows Support] Add latest xformers wheels to pyproject.toml (#1753)

07aea40

* Add latest xformers * Add a couple of lines to docs

Merge branch 'nightly' of https://github.com/unslothai/unsloth into n…

77109a4

…ightly

vLLMSamplingParams

f3d9efb

Update __init__.py

6d5caca

default num_chunks == -1

3a5610e

Versioning

0362bd2

danielhanchen merged commit f29da34 into main Feb 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Memory Efficient GRPO#1773

Memory Efficient GRPO#1773
danielhanchen merged 459 commits into
mainfrom
nightly

danielhanchen commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

danielhanchen commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants