Skip to content

Integrate high-performance x64 gemm library to MLAS #17669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yufenglee merged 115 commits into from
Dec 19, 2023
Merged

Integrate high-performance x64 gemm library to MLAS #17669

yufenglee merged 115 commits into from
Dec 19, 2023

Conversation

@luoyu-intel
Copy link
Contributor

@luoyu-intel luoyu-intel commented Sep 22, 2023 •
edited
Loading

Description

Improve MLAS to support high-performance x64 INT4 kernels

Motivation and Context

  1. improve LLM inference performance on Intel CPUs.
  2. support more 4bit quantization types: nf4, fp4
  3. support dynamic block size: block size aligned with kernel's tiling size(e.g. 4 for VNNI kernel), per channel on N dimension
  4. support most Intel ISAs: avx2, avx_vnni, avx512f, avx512_vnni, amx_bf16, amx_int8, avx512_fp16
  5. support MatMulNBits' data format

Tasks

  • support block_size: 32, 128, -1(per channel)
  • get weight pack size without memory allocation
  • use ort's thread pool for parallelism
  • support ISAs: avx2, avx512f, avx_vnni, avx512_vnni, amx_int8

Benchmark

Ubuntu 20.22 + Intel(R) Xeon(R) Platinum 8480+ 56 cores

Benchmark Time CPU Iterations
Q4GEMM_Jblas/Q4G32SymInt8/M:1/N:4096/K:4096/Threads:56/real_time 47613 47401 12970
Q4GEMM_Jblas/Q4G32SymInt8/M:1024/N:4096/K:4096/Threads:56/real_time 6347792 6317562 109
Q4GEMM_Jblas/Q4G32SymInt8/M:2048/N:4096/K:4096/Threads:56/real_time 11814014 11757847 59
Q4GEMM_Jblas/Q4G128SymInt8/M:1/N:4096/K:4096/Threads:56/real_time 50222 50031 13759
Q4GEMM_Jblas/Q4G128SymInt8/M:1024/N:4096/K:4096/Threads:56/real_time 2038222 2028743 341
Q4GEMM_Jblas/Q4G128SymInt8/M:2048/N:4096/K:4096/Threads:56/real_time 3792832 3774485 191
Q4GEMM_Jblas/Q4GPerNSymInt8/M:1/N:4096/K:4096/Threads:56/real_time 58717 58501 11467
Q4GEMM_Jblas/Q4GPerNSymInt8/M:1024/N:4096/K:4096/Threads:56/real_time 1360846 1354598 543
Q4GEMM_Jblas/Q4GPerNSymInt8/M:2048/N:4096/K:4096/Threads:56/real_time 2564232 2551365 266
Q4GEMM_Jblas/Q4G32SymFp32/M:1/N:4096/K:4096/Threads:56/real_time 57929 57694 12047
Q4GEMM_Jblas/Q4G32SymFp32/M:1024/N:4096/K:4096/Threads:56/real_time 5495330 5465810 126
Q4GEMM_Jblas/Q4G32SymFp32/M:2048/N:4096/K:4096/Threads:56/real_time 10676240 10617817 66
Q4GEMM_Jblas/Q4G128SymFp32/M:1/N:4096/K:4096/Threads:56/real_time 68305 68047 10026
Q4GEMM_Jblas/Q4G128SymFp32/M:1024/N:4096/K:4096/Threads:56/real_time 5504862 5476215 126
Q4GEMM_Jblas/Q4G128SymFp32/M:2048/N:4096/K:4096/Threads:56/real_time 11758623 11697337 66
Q4GEMM_Jblas/Q4GPerNSymFp32/M:1/N:4096/K:4096/Threads:56/real_time 67713 67451 10298
Q4GEMM_Jblas/Q4GPerNSymFp32/M:1024/N:4096/K:4096/Threads:56/real_time 5508325 5480237 126
Q4GEMM_Jblas/Q4GPerNSymFp32/M:2048/N:4096/K:4096/Threads:56/real_time 10738528 10681656 64
Q4GEMM_Jblas/Q4G32AsymFp32/M:1/N:4096/K:4096/Threads:56/real_time 60708 60486 11321
Q4GEMM_Jblas/Q4G32AsymFp32/M:1024/N:4096/K:4096/Threads:56/real_time 5523784 5495736 126
Q4GEMM_Jblas/Q4G32AsymFp32/M:2048/N:4096/K:4096/Threads:56/real_time 10829633 10772161 67

Reference:

Benchmark Time CPU Iterations
Q4GEMM/Q4Sym/M:1/N:4096/K:4096/Threads:56/real_time 53088 52911 13364
Q4GEMM/Q4Sym/M:1024/N:4096/K:4096/Threads:56/real_time 6268981 6230335 110
Q4GEMM/Q4Sym/M:2048/N:4096/K:4096/Threads:56/real_time 11701237 11632339 59

Win11+12900K 8 cores:

Benchmark Time CPU Iterations
Q4GEMM_Jblas/Q4G32SymInt8/M:1/N:4096/K:4096/Threads:8/real_time 215976 211295 2884
Q4GEMM_Jblas/Q4G32SymInt8/M:1024/N:4096/K:4096/Threads:8/real_time 60960590 60937500 10
Q4GEMM_Jblas/Q4G32SymInt8/M:2048/N:4096/K:4096/Threads:8/real_time 1.18E+08 1.19E+08 5
Q4GEMM_Jblas/Q4G32SymInt8/M:1/N:11008/K:4096/Threads:8/real_time 470377 453059 1414
Q4GEMM_Jblas/Q4G32SymInt8/M:1024/N:11008/K:4096/Threads:8/real_time 1.54E+08 1.53E+08 5
Q4GEMM_Jblas/Q4G32SymInt8/M:2048/N:11008/K:4096/Threads:8/real_time 3.18E+08 3.13E+08 2
Q4GEMM_Jblas/Q4G32SymInt8/M:1/N:4096/K:11008/Threads:8/real_time 569072 559398 1229
Q4GEMM_Jblas/Q4G32SymInt8/M:1024/N:4096/K:11008/Threads:8/real_time 1.54E+08 1.52E+08 4
Q4GEMM_Jblas/Q4G32SymInt8/M:2048/N:4096/K:11008/Threads:8/real_time 3.22E+08 3.28E+08 2
Q4GEMM_Jblas/Q4G32SymInt8/M:1/N:11008/K:11008/Threads:8/real_time 1486055 1473325 403
Q4GEMM_Jblas/Q4G32SymInt8/M:1024/N:11008/K:11008/Threads:8/real_time 4.14E+08 4.14E+08 2
Q4GEMM_Jblas/Q4G32SymInt8/M:2048/N:11008/K:11008/Threads:8/real_time 8.88E+08 8.59E+08 1

snnn
snnn reviewed Sep 25, 2023
View reviewed changes
snnn
snnn reviewed Sep 25, 2023
View reviewed changes
@mengniwang95 mengniwang95 mentioned this pull request Nov 10, 2023
Merged
@luoyu-intel luoyu-intel marked this pull request as ready for review November 15, 2023 15:03
@luoyu-intel luoyu-intel requested a review from a team as a code owner November 15, 2023 15:03
luoyu-intel and others added 24 commits November 16, 2023 11:18
@luoyu-intel
add jblas; add q4 perchannel of jblas to kernel, benchmark and test; …
41a8f2a 
…clang-format all
@luoyu-intel
workaround the compilation on gcc9
82720d0
@luoyu-intel
sync compiler flags
c71490a
@luoyu-intel
get memory size without memory allocation
0ee5377
@luoyu-intel
add amx_int8 kernel for computation
44846fe
@luoyu-intel
fix path error
2f9b26a
@luoyu-intel
no need for AMX syscall
8a30f65
@mengniwang95 @luoyu-intel
Add MatMulNBitsCPU op
483818c 
Signed-off-by: Mengni Wang <mengni.wang@intel.com>
@luoyu-intel
pass compile
e48430a
@luoyu-intel
update jblas. add comp_dtype=fp32
a15442f
@luoyu-intel
add more test cases
438fb72
@luoyu-intel
add ut for mamtmul_nbits_cpu
b122560
@luoyu-intel
fix compile errors
c758d53
@luoyu-intel
clang-format. update UT
20f2bb2
@luoyu-intel
update UT reference value.
4b81a1d
@luoyu-intel
update jblas for MatMulNBits prepack
cb12f02
@luoyu-intel
add jblas execute path in matmul_nbits.cc
a31a24d
@luoyu-intel
add UT for matmul_nbits
0e5c032
@luoyu-intel
pass pre-pack UT of matmul_nbits
9aea379
@luoyu-intel
pass UT for comp_int8
37a800c
@luoyu-intel
revert matmul_nbits_cpu
31df05b
@luoyu-intel
revert code
35db583
@luoyu-intel
revert code
438a9e0
@luoyu-intel
fix benchmark error
c77403d
@yufenglee
Copy link
Member

yufenglee commented Dec 14, 2023

/azp run Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

@yufenglee
Copy link
Member

yufenglee commented Dec 14, 2023

/azp run Windows x64 QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented Dec 14, 2023

Azure Pipelines successfully started running 1 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented Dec 14, 2023

Azure Pipelines successfully started running 8 pipeline(s).

@yufenglee
Copy link
Member

yufenglee commented Dec 14, 2023

/azp run Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented Dec 14, 2023

Azure Pipelines successfully started running 7 pipeline(s).

@yihonglyu
Copy link
Contributor

yihonglyu commented Dec 15, 2023

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Linux QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented Dec 15, 2023

Azure Pipelines successfully started running 9 pipeline(s).

@yufenglee yufenglee merged commit 5f00bc9 into microsoft:main Dec 19, 2023
@yufenglee
Copy link
Member

yufenglee commented Dec 19, 2023

Thanks Louyu!

@edgchen1 edgchen1 mentioned this pull request Jan 5, 2024
Merged
edgchen1 added a commit that referenced this pull request Jan 5, 2024
@edgchen1
Add MatMulNBits accuracy_level parameter to quantization utilities. (#…
4190c29 
…19015)

Allow MatMulNBits `accuracy_level` attribute (added in #17669) to be set to a particular value when the model is quantized.
@luoyu-intel luoyu-intel mentioned this pull request Feb 2, 2024
Merged
snnn pushed a commit that referenced this pull request Feb 7, 2024
@luoyu-intel
Revert NeuralSpeed code for x64 MatMulNBits (#19382)
0d10c7f 
### Description
<!-- Describe your changes. -->
Revert PR#19016 #19016
Revert PR#17669 #17669
skottmckay pushed a commit that referenced this pull request Feb 15, 2024
@luoyu-intel @skottmckay
Revert NeuralSpeed code for x64 MatMulNBits (#19382)
c10d6a4 
### Description
<!-- Describe your changes. -->
Revert PR#19016 #19016
Revert PR#17669 #17669
jslap-ubi pushed a commit to cgaudreau-ubisoft/onnxruntime that referenced this pull request Apr 5, 2024
@edgchen1 @jslap-ubi
Add MatMulNBits accuracy_level parameter to quantization utilities. (m…
8fff1e1 
…icrosoft#19015)

Allow MatMulNBits `accuracy_level` attribute (added in microsoft#17669) to be set to a particular value when the model is quantized.
rohan11235813 pushed a commit to quadric-io/onnxruntime that referenced this pull request Aug 19, 2025
@luoyu-intel @rohan11235813
Revert NeuralSpeed code for x64 MatMulNBits (#19382)
ba5542c 
### Description
<!-- Describe your changes. -->
Revert PR#19016 microsoft/onnxruntime#19016
Revert PR#17669 microsoft/onnxruntime#17669
rohan11235813 pushed a commit to quadric-io/onnxruntime that referenced this pull request Sep 15, 2025
@luoyu-intel @rohan11235813
Revert NeuralSpeed code for x64 MatMulNBits (#19382)
195126e 
### Description
<!-- Describe your changes. -->
Revert PR#19016 microsoft/onnxruntime#19016
Revert PR#17669 microsoft/onnxruntime#17669
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edgchen1 edgchen1 edgchen1 left review comments

@yufenglee yufenglee yufenglee approved these changes

+3 more reviewers

@guotuofeng guotuofeng guotuofeng left review comments

@snnn snnn snnn left review comments

@yihonglyu yihonglyu yihonglyu left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@luoyu-intel @yufenglee @yihonglyu @guotuofeng @snnn @edgchen1 @mengniwang95