Hi @danielhanchen ,
Tried to save GGUF model but got error for following codeblock.
Save to 8bit Q8_0
if True: model.save_pretrained_gguf("model", tokenizer,)
Following error is thrown
/usr/local/lib/python3.10/dist-packages/unsloth/save.py in save_to_gguf(model_type, model_dtype, is_sentencepiece, model_directory, quantization_method, first_conversion, _run_installer)
935 elif first_conversion == "q8_0" : pass
936 else:
--> 937 raise RuntimeError(
938 f"Unsloth: first_conversion can only be one of ['f16', 'bf16', 'f32', 'q8_0'] and not {first_conversion}."
939 )
RuntimeError: Unsloth: first_conversion can only be one of ['f16', 'bf16', 'f32', 'q8_0'] and not f16.
URL - https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing#scrollTo=FqfebeAdT073
Hi @danielhanchen ,
Tried to save GGUF model but got error for following codeblock.
Save to 8bit Q8_0
if True: model.save_pretrained_gguf("model", tokenizer,)
Following error is thrown
/usr/local/lib/python3.10/dist-packages/unsloth/save.py in save_to_gguf(model_type, model_dtype, is_sentencepiece, model_directory, quantization_method, first_conversion, _run_installer)
935 elif first_conversion == "q8_0" : pass
936 else:
--> 937 raise RuntimeError(
938 f"Unsloth:
first_conversioncan only be one of ['f16', 'bf16', 'f32', 'q8_0'] and not{first_conversion}."939 )
RuntimeError: Unsloth:
first_conversioncan only be one of ['f16', 'bf16', 'f32', 'q8_0'] and notf16.URL - https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing#scrollTo=FqfebeAdT073