Efficient Inference Kernel for SpQR by elvircrn · Pull Request #34976 · huggingface/transformers

elvircrn · 2024-11-27T15:40:28Z

What does this PR do?

Adds support for efficient single-batch inference for SpQR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @MekkCyber

MekkCyber · 2024-11-29T11:05:57Z

Hey @elvircrn, Thanks for adding this quantization method ! A very smooth integration 🔥! Just left a few comments

src/transformers/integrations/spqr.py

MekkCyber · 2024-11-29T10:19:29Z

src/transformers/integrations/spqr.py

No need to add the checks here, validate_environment in the quantizer will take care of that

Resolved by removing the checks.

MekkCyber · 2024-11-29T10:27:46Z

src/transformers/quantizers/quantizer_spqr.py

Not used, to be removed

Resolved by removing

MekkCyber · 2024-11-29T10:29:44Z

src/transformers/quantizers/quantizer_spqr.py

let's raise a warning here if you need to override the torch_dtype like in other quantizers

Since SpQR really only supports float16, I raise an error if it's None or anything other than float16. Is this the desired behavior?

src/transformers/utils/quantization_config.py

MekkCyber · 2024-11-29T10:47:33Z

tests/quantization/spqr_integration/test_spqr.py

Sorry I can't find this model on the hub 🤔

Ah, didn't expect a review so soon. Thank you!

I've uploaded the model and updated this part of the test.

No worries 😊 take your time and ping me when you finish the integration

MekkCyber · 2024-11-29T10:52:23Z

tests/quantization/spqr_integration/test_spqr.py

No need to add the requirement here, it's already verified with the decorator require_spqr

Resolved by removing.

MekkCyber · 2024-11-29T10:57:12Z

docs/source/en/quantization/spqr.md

We can also add a small section about how to load a model quantized using SpQR using transformers

Resolved by updating the README.

MekkCyber · 2024-11-29T10:59:37Z

src/transformers/quantizers/quantizer_spqr.py

if the quantization method requires a gpu we need to check if it's availabe here

Resolved by adding an is_cuda check.

MekkCyber · 2024-11-29T11:03:30Z

tests/quantization/spqr_integration/test_spqr.py

Is it possible to add tests for multi_gpu setups ?

Resolving by bringing this back form AQLM. Will the CI be able to pick this up? I don't have the means of testing this.

Yes it does 😄

+1 on that, you can add the test and we will check to make sure it to pass the right expected output

elvircrn · 2024-12-02T15:17:40Z

@MekkCyber I've removed the Draft tag from the PR.

It seems that I am unable to pass the test_raise_if_non_quantized. Could you give me some pointers on this?

MekkCyber · 2024-12-06T12:47:27Z

Could you rebase @elvircrn to update the branch :)

elvircrn · 2024-12-06T13:10:27Z

@MekkCyber Rebase done.

MekkCyber · 2024-12-09T08:13:48Z

Hey @elvircrn, thanks for iterating ! It looks great I just left some minor comments

MekkCyber · 2024-12-09T07:37:29Z

src/transformers/integrations/spqr.py

We need to add the checks here if the accelerate and spqr_quant packages are available before doing the import, what we don't need to do is raising errors, because the packages are assumed to be installed when executing the function

Not sure if I understood this, please double-check my newest change.

MekkCyber · 2024-12-09T07:40:19Z

src/transformers/quantizers/quantizer_spqr.py

We are not assuming here because we already validated the environment 😉

MekkCyber · 2024-12-09T07:42:24Z

src/transformers/quantizers/quantizer_spqr.py

Suggested change

elif torch_dtype != torch.float16:

raise ValueError(

"You cannot any type other than torch.float16 for SpQR. Please either leave it None or set it to"

"torch.float16 explicitly."

)

return torch_dtype

elif torch_dtype != torch.float16:

raise ValueError(

"You cannot use any type other than torch.float16 for SpQR. Please either leave it None or set it to"

"torch.float16 explicitly."

)

return torch_dtype

MekkCyber · 2024-12-09T07:55:44Z

src/transformers/integrations/spqr.py

Just FMI, are the beta and shapes parameters used solely for loading, or do they also play a role in quantization? In other words, if a model is quantized using specific beta or shapes values, can it only be loaded with those same parameters?

SpQR quantized weights are a product of tile-based (beta corresponds to the dimension of this tile where beta1, beta2 represent tile width and height respectively) bi-level quantization. You always specify beta before the quantization starts.

Each SpQR weight also comes with a sparse tensor representing the outlier weights. As this tensor is unstructured, one had to keep track of the size of this matrix during loading.

The following is a visualization of the compression format from the original publication (https://arxiv.org/pdf/2306.03078):

Thanks for the detailed explanation

Let's add it to the doc!

MekkCyber · 2024-12-09T07:59:14Z

src/transformers/integrations/spqr.py

in the shapes attribute do we need to specify the shape for all the tensors ?

The colval size needs to be specified ahead of time for all weights. The rest can be computed from m, n and bits. Can we keep it the way it is now?

I can do it both ways easily, not sure what is preferred here.

I'm just worried about the quantization config, I'm not sure if we can verify that shapes contains all the necessary elements in the post_init. Maybe before trying to access the key in replace_with_spqr_linear we verify if it's there or we raise an error to inform the user there is something wrong with the config.json they are using, wdyt @SunMarc

In my estimate, these configurations are directly tied to the quantized weights. This would be a fatal error in quantization serialization if the shapes don't cover the full set of quantized weights. Perhaps this is something that should be done on the side of SpQR (https://pypi.org/project/spqr-quant/)?

If we were to do it on huggingface/transformers-side:

a) during replace_with_linear, we raise a fatal error (please let me know which one would be appropriate here) if the quantized weights don't have the corresponding key in the shapes config.
b) we do a full check before conducting the replacement in order to conserve memory/device cycles.

Let me know what best suits transformers here.

I didn't see the tag to @SunMarc - I thought the wdyt referred to me.

MekkCyber · 2024-12-09T08:00:17Z

tests/quantization/spqr_integration/test_spqr.py

Yes it does 😄

tests/quantization/spqr_integration/test_spqr.py

MekkCyber · 2024-12-09T08:10:59Z

tests/quantization/spqr_integration/test_spqr.py

If possible it's better to use small models for the CI like Qwen 0.5, Bloom 560M, SmolLM 135M...

Will have to get back to you on this (might require a significant time investment).

Is this a blocker for the merge?

Not really a blocker for now

elvircrn · 2024-12-09T10:04:52Z

@MekkCyber The latest batch of comments are addressed.

MekkCyber · 2024-12-09T12:58:47Z

Thanks for the quick iteration @elvircrn 🔥 ! can you re-rebase please 😅

MekkCyber · 2024-12-09T12:19:44Z

src/transformers/integrations/spqr.py

I'm just worried about the quantization config, I'm not sure if we can verify that shapes contains all the necessary elements in the post_init. Maybe before trying to access the key in replace_with_spqr_linear we verify if it's there or we raise an error to inform the user there is something wrong with the config.json they are using, wdyt @SunMarc

MekkCyber · 2024-12-09T12:42:02Z

src/transformers/integrations/spqr.py

Thanks for the detailed explanation

MekkCyber · 2024-12-09T12:43:28Z

src/transformers/integrations/spqr.py

Yes that's exactly what I meant, thank you for fixing it !

MekkCyber · 2024-12-09T12:46:26Z

tests/quantization/spqr_integration/test_spqr.py

Not really a blocker for now

elvircrn · 2024-12-09T13:01:21Z

@MekkCyber rebased.

elvircrn · 2024-12-09T13:10:13Z

@MekkCyber Resolved the latest batch of comments.

elvircrn · 2024-12-09T14:30:59Z

@SunMarc @MekkCyber
I've pushed a check for keys() in shapes in replace_with_spqr_linear. Let me know if something in this ballpark works.

SunMarc · 2025-01-15T13:36:30Z

pinging @ArthurZucker to have a quick look !

SunMarc · 2025-01-15T13:36:41Z

Could you fix the conflits @elvircrn ?

elvircrn · 2025-01-15T13:38:59Z

@SunMarc resolved quickly, as it was just an md conflict in the overview table.

* gptqmodel Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix format Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update readme Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * gptqmodel need use checkpoint_format (huggingface#1) * gptqmodel need use checkpoint_format * fix quantize * Update quantization_config.py * Update quantization_config.py * Update quantization_config.py --------- Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai> Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> * Revert quantizer_gptq.py (huggingface#2) * revert quantizer_gptq.py change * pass **kwargs * limit gptqmodel and optimum version Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix format Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix warning Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix version check Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * revert unrelated changes Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * enable gptqmodel tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix requires gptq Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * Fix Transformer compat (huggingface#3) * revert quantizer_gptq.py change * pass **kwargs * add meta info * cleanup * cleanup * Update quantization_config.py * hf_select_quant_linear pass checkpoint_format and meta * fix GPTQTestCUDA * Update test_gptq.py * gptqmodel.hf_select_quant_linear() now does not select ExllamaV2 * cleanup * add backend * cleanup * cleanup * no need check exllama version * Update quantization_config.py * lower checkpoint_format and backend * check none * cleanup * Update quantization_config.py * fix self.use_exllama == False * spell * fix unittest * fix unittest --------- Co-authored-by: LRL <lrl@lbx.dev> Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> * fix format Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix format again Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update gptqmodel version (huggingface#6) * update gptqmodel version * update gptqmodel version * fix unit test (huggingface#5) * update gptqmodel version * update gptqmodel version * "not self.use_exllama" is not equivalent to "self.use_exllama==False" * fix unittest * update gptqmodel version * backend is loading_attibutes (huggingface#7) * fix format and tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix memory check Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix device mismatch Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix result check Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * Update src/transformers/quantizers/quantizer_gptq.py Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com> * Update src/transformers/quantizers/quantizer_gptq.py Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com> * Update src/transformers/quantizers/quantizer_gptq.py Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com> * update tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * review: update docs (huggingface#10) * review: update docs (huggingface#12) * review: update docs * fix typo * update tests for gptqmodel Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update document (huggingface#9) * update overview.md * cleanup * Update overview.md * Update overview.md * Update overview.md * update gptq.md * Update gptq.md * Update gptq.md * Update gptq.md * Update gptq.md * Update gptq.md * Update gptq.md --------- Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> * typo * doc note for asymmetric quant * typo with apple silicon(e) * typo for marlin * column name revert: review * doc rocm support * Update docs/source/en/quantization/gptq.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/quantization/gptq.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/quantization/gptq.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/quantization/gptq.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/quantization/overview.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/quantization/overview.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com> Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai> Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> Co-authored-by: ZX-ModelCloud <165115237+ZX-ModelCloud@users.noreply.github.com> Co-authored-by: LRL <lrl@lbx.dev> Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com> Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fixing nemotron processor * make style

elvircrn · 2025-02-07T13:59:18Z

@SunMarc @ArthurZucker Conflicts resolved.

ArthurZucker

LGTM!

docs/source/en/quantization/spqr.md

ArthurZucker · 2025-01-16T16:39:21Z

docs/source/en/quantization/spqr.md

+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf")
+```


we can't quantize a model on the fly with quantization_config=... using spqr as the quantize type?

We can't, this is not supported as of yet. This PR only adds Inference support.

ArthurZucker · 2025-02-13T09:17:21Z

src/transformers/integrations/spqr.py

Let's add it to the doc!

ArthurZucker · 2025-02-13T09:18:46Z

Can you just fix this:

RuntimeError: The following files are not present in the table of contents:
- quantization/spqr
Add them to ../transformers/docs/source/en/_toctree.yml.

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

elvircrn · 2025-02-13T09:32:08Z

@ArthurZucker Is the PR in order now?

elvircrn · 2025-02-13T09:47:27Z

@ArthurZucker Erreneous rebase issue resolved.

MekkCyber · 2025-02-13T12:08:24Z

Hi @elvircrn, can you just resolve the conflicts and once the CI is green we can merge ! Again sorry for the delay 🙏 and thanks for your patience

elvircrn · 2025-02-13T12:35:50Z

@MekkCyber Done.

elvircrn · 2025-02-13T12:58:29Z

@MekkCyber The failing tests, based on the logs, seem to be unrelated to my own source code.

MekkCyber · 2025-02-13T14:31:09Z

Yes they seem unrelated @elvircrn, i tried updating the branch I hope it works

elvircrn · 2025-02-13T14:37:00Z

@MekkCyber I don't think that did it - unrelated tests are still failing.

elvircrn · 2025-02-13T14:49:50Z

@MekkCyber Seems that everything is green now. :)

elvircrn · 2025-02-13T15:24:19Z

Thanks everyone!

HuggingFaceDocBuilderDev · 2025-02-13T15:25:34Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

elvircrn marked this pull request as draft November 27, 2024 15:40

elvircrn changed the title ~~Spqr quantizer~~ Efficient Inference Kernel for SpQR Nov 27, 2024

SunMarc requested a review from MekkCyber November 28, 2024 14:45

MekkCyber reviewed Nov 29, 2024

View reviewed changes

This was referenced Nov 30, 2024

Any update on inference code? Vahe1994/SpQR#44

Closed

has the inference code released? Vahe1994/SpQR#42

Closed

How to test inference speed? Vahe1994/SpQR#12

Closed

elvircrn force-pushed the spqr-quantizer branch from fda6b66 to ee10336 Compare December 2, 2024 15:08

elvircrn marked this pull request as ready for review December 2, 2024 15:09

elvircrn force-pushed the spqr-quantizer branch 4 times, most recently from 56c016d to e1a91fa Compare December 5, 2024 17:16

elvircrn force-pushed the spqr-quantizer branch from e1a91fa to 4ea16a9 Compare December 6, 2024 13:10

elvircrn force-pushed the spqr-quantizer branch from 4ea16a9 to 51c9f8d Compare December 6, 2024 13:52

MekkCyber reviewed Dec 9, 2024

View reviewed changes

elvircrn force-pushed the spqr-quantizer branch from 436697e to 16c0278 Compare December 9, 2024 10:04

MekkCyber reviewed Dec 9, 2024

View reviewed changes

elvircrn force-pushed the spqr-quantizer branch from 16c0278 to efd9f20 Compare December 9, 2024 13:01

MekkCyber requested a review from SunMarc December 9, 2024 13:04

jiqing-feng and others added 2 commits February 7, 2025 14:53

Fix : Nemotron Processor in GGUF conversion (huggingface#35708)

c4273e2

* fixing nemotron processor * make style

elvircrn force-pushed the spqr-quantizer branch from 64d2039 to c4273e2 Compare February 7, 2025 13:57

Merge branch 'main' into spqr-quantizer

5aac5e3

ArthurZucker approved these changes Feb 13, 2025

View reviewed changes

elvircrn and others added 2 commits February 13, 2025 10:26

Update docs/source/en/quantization/spqr.md

95d2e74

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

Add missing TOC to doc

3fdc0c3

Merge branch 'huggingface:main' into spqr-quantizer

14f21c1

elvircrn force-pushed the spqr-quantizer branch 2 times, most recently from 0340152 to 14f21c1 Compare February 13, 2025 09:42

Merge branch 'main' into spqr-quantizer

cdefeaf

Merge branch 'main' into spqr-quantizer

8da4a66

Merge branch 'main' into spqr-quantizer

afff70e

SunMarc approved these changes Feb 13, 2025

View reviewed changes

MekkCyber merged commit 845b0a2 into huggingface:main Feb 13, 2025
25 checks passed

-        elif torch_dtype != torch.float16:
-            raise ValueError(
-                "You cannot any type other than torch.float16 for SpQR. Please either leave it None or set it to"
-                "torch.float16 explicitly."
-            )
-        return torch_dtype
+        elif torch_dtype != torch.float16:
+            raise ValueError(
+                "You cannot use any type other than torch.float16 for SpQR. Please either leave it None or set it to"
+                "torch.float16 explicitly."
+            )
+        return torch_dtype

Conversation

elvircrn commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

MekkCyber commented Nov 29, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elvircrn commented Dec 2, 2024

Uh oh!

MekkCyber commented Dec 6, 2024

Uh oh!

elvircrn commented Dec 6, 2024

Uh oh!

MekkCyber commented Dec 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

elvircrn commented Nov 27, 2024 •

edited

Loading

elvircrn commented Dec 9, 2024 •

edited

Loading

MekkCyber commented Dec 9, 2024 •

edited

Loading