fix splt shape < 64 issue & add num_kv_heads to mp_params #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

delock merged 1 commit into delock:gma/run-opt-branch from Yejing-Lai:lyj/fix_falcon-40b_issue

Nov 21, 2023

Yejing-Lai commented Nov 21, 2023

When split shape(total_size) < 64, tp sharding will fail. We have added support for split_shape < 64.
falcon-40b will fail on SNC3. This reason is falcon-40b kv_head_names is "num_kv_heads". Need to add this name in mp_param list and kv_head_names list.


          fix splt shape < 64 issue & add num_kv_heads to mp_params

c2ae7f0

delock merged commit 14f5058 into delock:gma/run-opt-branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet