We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

Let me know how that goes for you

Thanks!

Uh oh!

OOM error when i tried to save model in q_8 gguf #1721

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions