Skip to content

[1/2] Enable keyword arguments for liger functional#399

Closed
hongpeng-guo wants to merge 6 commits into
linkedin:mainfrom
hongpeng-guo:hpguo/functional_revamp
Closed

[1/2] Enable keyword arguments for liger functional#399
hongpeng-guo wants to merge 6 commits into
linkedin:mainfrom
hongpeng-guo:hpguo/functional_revamp

Conversation

@hongpeng-guo

@hongpeng-guo hongpeng-guo commented Nov 20, 2024

Copy link
Copy Markdown
Contributor

Summary

This PR is the first step to enable the keyword arguments of liger functional #368.

  1. Warp the Liger Operator Functions (torch.autograd.Function) with an extra layer that can take key word arguments.
  2. For each of the liger functions, updating its unit test function test_{operator_name}.py::test_correctness_functional to reflect that keyword args can be accepted.
  3. Next step is do apply the new op functionals in transformers/monkey_patch.py

Testing Done

  • Hardware Type: A10G
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
@hongpeng-guo

Copy link
Copy Markdown
Contributor Author

@ByronHsu I would like to split the functional revamp into two PRs. This is the first one that simply defines the kwarg functions, as well as the unit test. The next PR will try to apply these functions to monkey patch the transformers.

@ByronHsu

Copy link
Copy Markdown
Contributor

It looks good!

  1. Can you create a branch in the main repo and open a new PR, so you can run CI?
  2. We don't have to change monkey patch file because only the new cross entropy relies on functional CE and i have already fixed it.

@hongpeng-guo

Copy link
Copy Markdown
Contributor Author

It looks good!

  1. Can you create a branch in the main repo and open a new PR, so you can run CI?
  2. We don't have to change monkey patch file because only the new cross entropy relies on functional CE and i have already fixed it.

Sure! That sounds good!

@hongpeng-guo

Copy link
Copy Markdown
Contributor Author

Close this PR, pivot for #400 that is created in the main repo to run the tests.

@hongpeng-guo hongpeng-guo deleted the hpguo/functional_revamp branch November 20, 2024 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants