FP6 quantization end-to-end. #5234
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Initialize the fp6-quant-kernel integration. * Add necessary parameters of kernel interfaces and the linear layer selection logic. * upload kernel code * The simple script for debugging. * fix typo * update * fix split k * Fix some errors and add test case. * Workspace for Inference Kernels (#1) * Add transform_param functions and update format. * kernel debug * fix include * Update core_ops.cpp * Add split k support * fix * Fix kernel error * update * update * Fix rebase errors. * Add missed include. * Fix the bug that the attribute uses the weight information for mem alloc. * Avoid GPU preallocation during weight loading. * Add support of larger shapes for gated activation kernel. * update * model update * fix all weight preprocessing * Add split-k heuristic. * Avoid reading scale attribute on non-quantized tensors. * Change the scales from attributes to new tensors. Provide the end-to-end script given HuggingFace model id. * Hard-coded commented out the scales in the kernel to workaround the bug. * Support the user config for quantization. Fix kernel bug. * Per operator test functions. * Multiply scales by 1e12 according to the kernel design. * Revert "Workspace for Inference Kernels (#1)". This reverts commit 1528732. * Remove the format-only changes. * Put the quantization into the transform_param function. --------- Co-authored-by: Shiyang Chen <csycfl@gmail.com> Co-authored-by: Haojun Xia <xhjustc@mail.ustc.edu.cn>
xiaoxiawu-microsoft
approved these changes
Mar 6, 2024
mrwyattii
approved these changes
Mar 6, 2024
arashb
approved these changes
Mar 6, 2024
Collaborator
Author
|
A6000 CI that points to the required branch for MII is here |
ShellyNR
pushed a commit
to ShellyNR/DeepSpeed
that referenced
this pull request
Mar 11, 2024
The user interface: deepspeedai/DeepSpeed-MII#433 nv-a6000 ci running against the MII branch linked above is [here](https://github.com/microsoft/DeepSpeed/actions/runs/8192124606) Co-authored-by: Zhen Zheng [zhengzhen@microsoft.com](mailto:zhengzhen@microsoft.com) Co-authored-by: Shiyang Chen [csycfl@gmail.com](mailto:csycfl@gmail.com) Co-authored-by: Arash Bakhtiari [abakhtiari@microsoft.com](mailto:abakhtiari@microsoft.com) Co-authored-by: Haojun Xia [xhjustc@mail.ustc.edu.cn](mailto:xhjustc@mail.ustc.edu.cn) --------- Co-authored-by: ZHENG, Zhen <zhengzhen.z@qq.com> Co-authored-by: Shiyang Chen <csycfl@gmail.com> Co-authored-by: Haojun Xia <xhjustc@mail.ustc.edu.cn> Co-authored-by: Arash Bakhtiari <arash@bakhtiari.org> Co-authored-by: Michael Wyatt <mrwyattii@gmail.com> Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
loadams
added a commit
that referenced
this pull request
Apr 4, 2024
rraminen
pushed a commit
to ROCm/DeepSpeed
that referenced
this pull request
May 9, 2024
The user interface: deepspeedai/DeepSpeed-MII#433 nv-a6000 ci running against the MII branch linked above is [here](https://github.com/microsoft/DeepSpeed/actions/runs/8192124606) Co-authored-by: Zhen Zheng [zhengzhen@microsoft.com](mailto:zhengzhen@microsoft.com) Co-authored-by: Shiyang Chen [csycfl@gmail.com](mailto:csycfl@gmail.com) Co-authored-by: Arash Bakhtiari [abakhtiari@microsoft.com](mailto:abakhtiari@microsoft.com) Co-authored-by: Haojun Xia [xhjustc@mail.ustc.edu.cn](mailto:xhjustc@mail.ustc.edu.cn) --------- Co-authored-by: ZHENG, Zhen <zhengzhen.z@qq.com> Co-authored-by: Shiyang Chen <csycfl@gmail.com> Co-authored-by: Haojun Xia <xhjustc@mail.ustc.edu.cn> Co-authored-by: Arash Bakhtiari <arash@bakhtiari.org> Co-authored-by: Michael Wyatt <mrwyattii@gmail.com> Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The user interface: deepspeedai/DeepSpeed-MII#433
nv-a6000 ci running against the MII branch linked above is here
Co-authored-by: Zhen Zheng zhengzhen@microsoft.com
Co-authored-by: Shiyang Chen csycfl@gmail.com
Co-authored-by: Arash Bakhtiari abakhtiari@microsoft.com
Co-authored-by: Haojun Xia xhjustc@mail.ustc.edu.cn