Skip to content

ValueError: Unknown quantization method: bitsandbytes. Must be one of ['awq', 'gptq', 'squeezellm', 'marlin']. #482

@manliu1225

Description

@manliu1225

When I want to inference the finetuned model with vLLM, I got this error.
I have saved unsloth finetuned model to HF model already.
vLLM==0.4.0+cu118
unsloth==2024.5
transformers==4.40.2

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions