Skip to content

Error while saving to GGUF model #626

@MaheshAwasare

Description

@MaheshAwasare

Hi @danielhanchen ,
Tried to save GGUF model but got error for following codeblock.

Save to 8bit Q8_0

if True: model.save_pretrained_gguf("model", tokenizer,)

Following error is thrown

/usr/local/lib/python3.10/dist-packages/unsloth/save.py in save_to_gguf(model_type, model_dtype, is_sentencepiece, model_directory, quantization_method, first_conversion, _run_installer)
935 elif first_conversion == "q8_0" : pass
936 else:
--> 937 raise RuntimeError(
938 f"Unsloth: first_conversion can only be one of ['f16', 'bf16', 'f32', 'q8_0'] and not {first_conversion}."
939 )

RuntimeError: Unsloth: first_conversion can only be one of ['f16', 'bf16', 'f32', 'q8_0'] and not f16.

URL - https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing#scrollTo=FqfebeAdT073

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions