Amazing job with the swift port @johnmai-dev! 🔥🚀

The quantization here is done normally like any other model. So you can consider it done ✅

Amazing job with the swift port @johnmai-dev! 🔥🚀

The quantization here is done normally like any other model. So you can consider it done ✅

Thank you! ❤️
Yes, you are right. I still need to adjust some details and expect to finish later today.

Hello，@Blaizzy

What is the difference between quantization and quantization_config in config.json? But there is only quantization_config in microsoft/bitnet-b1.58-2B-4T.

I see that apply_hf_quantization only uses quantization_config
https://github.com/ml-explore/mlx-lm/blob/4fab6fcbc9dd63dea229692f91028d33f7532fd6/mlx_lm/quant/utils.py#L56-L72

    "quantization": {
        "group_size": 64,
        "bits": 4,
        "quant_method": "bitnet",
        "linear_class": "autobitlinear",
        "quantization_mode": "offline"
    },
    "quantization_config": {
        "group_size": 64,
        "bits": 4,
        "quant_method": "bitnet",
        "linear_class": "autobitlinear",
        "quantization_mode": "offline"
    },

Looking forward to your reply, thanks ❤️

Hello，@Blaizzy

What is the difference between quantization and quantization_config in config.json? But there is only quantization_config in microsoft/bitnet-b1.58-2B-4T.

I see that apply_hf_quantization only uses quantization_config https://github.com/ml-explore/mlx-lm/blob/4fab6fcbc9dd63dea229692f91028d33f7532fd6/mlx_lm/quant/utils.py#L56-L72

    "quantization": {
        "group_size": 64,
        "bits": 4,
        "quant_method": "bitnet",
        "linear_class": "autobitlinear",
        "quantization_mode": "offline"
    },
    "quantization_config": {
        "group_size": 64,
        "bits": 4,
        "quant_method": "bitnet",
        "linear_class": "autobitlinear",
        "quantization_mode": "offline"
    },

Looking forward to your reply, thanks ❤️

Currently, Quantization does not support decoding quant_method, linear_class, or quantization_mode.

I am considering whether to adjust Quantization or add a new QuantizationConfig struct.

Failed: configurationDecodingError("config.json", "mlx-community/bitnet-b1.58-2B-4T-4bit", Swift.DecodingError.typeMismatch(Swift.Dictionary<Swift.String, Any>, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "quantization", intValue: nil), _DictionaryCodingKey(stringValue: "quant_method", intValue: nil)], debugDescription: "Expected to decode Dictionary<String, Any> but found a string instead.", underlyingError: nil)))

#295

Cut a tag on mlx-swift for the relu squared: 0.25.5

What is the difference between quantization and quantization_config in config.json?

I can say a little about that. MLX originally added the quantization field to the config.json. But Hugging Face uses a field called quantization_config to understand metadata about the model (e.g. if it's a quant of another model to display in the UI). So we now add both in order to maintain back-compatibility and set the right field for Hugging Face. So for any MLX model they should be the same.

For non MLX models they will probably just use the quantization_config. And that may or may not be compatible with MLX depending on the quant format.

What is the difference between quantization and quantization_config in config.json?

I can say a little about that. MLX originally added the quantization field to the config.json. But Hugging Face uses a field called quantization_config to understand metadata about the model (e.g. if it's a quant of another model to display in the UI). So we now add both in order to maintain back-compatibility and set the right field for Hugging Face. So for any MLX model they should be the same.

For non MLX models they will probably just use the quantization_config. And that may or may not be compatible with MLX depending on the quant format.

Thank you for your answer! ♥️

Thank you very much!!! @angeloskath
Speed increased 2x!!! 🚀🚀🚀

I think it's ready to merge.
cc @davidkoski @awni