How can this run MLX #2104

TomLucidor · 2025-11-04T04:13:24Z

TomLucidor
Nov 4, 2025

Referring to the following models

Got the following

 ~ % ramalama --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
Downloading Kimi-Linear-48B-A3B-Instruct-4bit
100% |████████████████████████████|   98.71 KB/  98.71 KB   3.78 MB/s        0s
Traceback (most recent call last):
  File "/Users/user/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/user.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 165, in load_model
    config = load_config(model_path)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 127, in load_config
    with open(model_path / "config.json", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: '/Users/user/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f/config.json'

Answered by kush-gupt

Nov 5, 2025

@TomLucidor So Kimi Linear just isn't in MLX yet unless you build off that PR because of how new it is. Once it's in MLX, you should be able to run uv tool upgrade mlx-lm and then the ramalama commands.

Here is Qwen3 Coder working just fine though:

❯ ramalama --runtime=mlx run hf://mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
🍏 > Create an infinite python loop
Here are several ways to create infinite loops in Python:

## Method 1: Using `while True` (Most Common)
```python
while True:
    print("This loop will run forever")
    # Add your code here
    # Use break condition when needed

Method 2: Using `while 1` (Same as True)

View full answer

rhatdan · 2025-11-04T12:51:37Z

rhatdan
Nov 4, 2025
Maintainer

@engelmi PTAL, looks like we are handing MLX the wrong file?

0 replies

engelmi · 2025-11-04T15:16:18Z

engelmi
Nov 4, 2025
Maintainer

@TomLucidor Can you provide the debug output?
ramalama --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit

It seems we pass each file individually to the mlx CLI and the mlx library expects a directory containing all files to be passed - and has expectations on the naming. I can't test it, unfortunately. Maybe @kush-gupt can help? I'll still have a look at it.

4 replies

kush-gupt Nov 4, 2025

The URL pull fails super quick:

❯ ramalama --debug --runtime=mlx serve https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:35:44 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:35:44 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:44 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:44 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:35:44 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:44 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:44 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:35:44 - DEBUG - env: None
DEBUG:ramalama:env: None
Downloading Kimi-Linear-48B-A3B-Instruct-4bit
2025-11-04 14:35:44 - DEBUG - Running urlopen https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit with headers: {'Range': 'bytes=0-'}
DEBUG:ramalama:Running urlopen https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit with headers: {'Range': 'bytes=0-'}
100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████|   98.72 KB/  98.72 KB  11.91 MB/s        0s
2025-11-04 14:35:45 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:35:45 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:35:45 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:35:45 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:35:45 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:35:45 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f --temp 0.8 --host 0.0.0.0 --port 8080
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f --temp 0.8 --host 0.0.0.0 --port 8080
Traceback (most recent call last):
  File "/Users/kugupta/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 165, in load_model
    config = load_config(model_path)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 127, in load_config
    with open(model_path / "config.json", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: '/Users/kugupta/.local/share/ramalama/store/https/hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/blobs/sha256-2a8496563bad6a7f2a3d9537b514826a4fd474e41ea88c2e0a5f51c58420964f/config.json'

engelmi Nov 5, 2025
Maintainer

Thanks! @kush-gupt
When pulled via URL, then the file type is not set to be a safetensor. In case of safetensors, the whole snapshot directory is passed to mlx. In contrast, for normal files the specific blob is passed. I'll have a look at this, fix should be easy to enable this for URL pulls as well.

engelmi Nov 5, 2025
Maintainer

Rechecking again, the issue is that only one file is being pulled by RamaLama: the .html of the hf website.
Which makes sense. We could do a bit of preprocessing for the model input (similar to the shortnames) and detect the host (huggingface.co) and change the model input accordingly - so it wouldn't use the http pull, but the hf one (which would fall back to the hf cli).

engelmi Nov 7, 2025
Maintainer

@TomLucidor @kush-gupt Opened a PR to map certain https:// usages internally to huggingface/ollama and pull the whole model: #2117

kush-gupt · 2025-11-04T19:22:18Z

kush-gupt
Nov 4, 2025

It looks like the URL pull https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit versus the hf pull ramalama --debug --runtime=mlx run hf://mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit causes the issue here. Have not tested URL pull for MLX before so that's not unexpected, but pulling down the model successfully on the latest released Ramalama.

The next crash I ran into was the following, seems to be an issue at the MLX layer:

❯ ramalama --debug --runtime=mlx run hf://mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:20:10 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - MLX server not ready, waiting... (attempt 1/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 1/10)
2025-11-04 14:20:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:20:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:20:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:20:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:20:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:20:10 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/snapshots/sha256-e0f9962de48aca5482b3f9308b9ef4719df02f51 --temp 0.8 --port 8080
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit/snapshots/sha256-e0f9962de48aca5482b3f9308b9ef4719df02f51 --temp 0.8 --port 8080
2025-11-04 14:20:11,477 - ERROR - Model type kimi_linear not supported.
Traceback (most recent call last):
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 68, in _get_classes
    arch = importlib.import_module(f"mlx_lm.models.{model_type}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mlx_lm.models.kimi_linear'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kugupta/.local/bin/mlx_lm.server", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 1083, in main
    run(args.host, args.port, ModelProvider(args))
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 168, in __init__
    self.load(self.cli_args.model, draft_model_path="default_model")
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py", line 196, in load
    model, tokenizer = load(
                       ^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 266, in load
    model, config = load_model(model_path, lazy, model_config=model_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 178, in load_model
    model_class, model_args_class = get_model_classes(config=config)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/utils.py", line 72, in _get_classes
    raise ValueError(msg)
ValueError: Model type kimi_linear not supported.

From uv tool list:

mlx-lm v0.28.3
- mlx_lm.awq
- mlx_lm.benchmark
- mlx_lm.cache_prompt
- mlx_lm.chat
- mlx_lm.convert
- mlx_lm.dwq
- mlx_lm.dynamic_quant
- mlx_lm.evaluate
- mlx_lm.fuse
- mlx_lm.generate
- mlx_lm.gptq
- mlx_lm.lora
- mlx_lm.manage
- mlx_lm.perplexity
- mlx_lm.server
- mlx_lm.upload
ramalama v0.14.0
- ramalama

5 replies

kush-gupt Nov 4, 2025

Seems to be tracking at this PR here: ml-explore/mlx-lm#577

kush-gupt Nov 4, 2025

Seems OSS works perfectly fine since it has been implemented within MLX for a while, example:

❯ ramalama --debug --runtime=mlx run hf://mlx-community/gpt-oss-20b-MXFP4-Q4
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
2025-11-04 14:29:10 - DEBUG - Checking if 8080 is available
DEBUG:ramalama:Checking if 8080 is available
2025-11-04 14:29:10 - DEBUG - Checking if 8082 is available
DEBUG:ramalama:Checking if 8082 is available
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
DEBUG:ramalama:run_cmd: podman inspect quay.io/ramalama/ramalama:0.14
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: True
DEBUG:ramalama:Ignore all: True
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - MLX server not ready, waiting... (attempt 1/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 1/10)
2025-11-04 14:29:10 - DEBUG - run_cmd: npu-smi info
DEBUG:ramalama:run_cmd: npu-smi info
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - run_cmd: mthreads-gmi
DEBUG:ramalama:run_cmd: mthreads-gmi
2025-11-04 14:29:10 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-04 14:29:10 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-04 14:29:10 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-04 14:29:10 - DEBUG - env: None
DEBUG:ramalama:env: None
2025-11-04 14:29:10 - DEBUG - exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/gpt-oss-20b-MXFP4-Q4/snapshots/sha256-a35c077138d6f155efd51134df24d477c0b1fc19 --temp 0.8 --port 8082
DEBUG:ramalama:exec_cmd: mlx_lm.server --model /Users/kugupta/.local/share/ramalama/store/huggingface/mlx-community/gpt-oss-20b-MXFP4-Q4/snapshots/sha256-a35c077138d6f155efd51134df24d477c0b1fc19 --temp 0.8 --port 8082
2025-11-04 14:29:13 - DEBUG - MLX server not ready, waiting... (attempt 2/10)
DEBUG:ramalama:MLX server not ready, waiting... (attempt 2/10)
/Users/kugupta/.local/share/uv/tools/mlx-lm/lib/python3.12/site-packages/mlx_lm/server.py:973: UserWarning: mlx_lm.server is not recommended for production as it only implements basic security checks.
  warnings.warn(
2025-11-04 14:29:13,909 - INFO - Starting httpd at 127.0.0.1 on port 8082...
🍏 > Hello!
2025-11-04 14:29:22 - DEBUG - Request: URL=http://127.0.0.1:8082/v1/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.8}', Headers={'Content-Type': 'application/json'}
DEBUG:ramalama:Request: URL=http://127.0.0.1:8082/v1/chat/completions, Data=b'{"stream": true, "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.8}', Headers={'Content-Type': 'application/json'}
127.0.0.1 - - [04/Nov/2025 14:29:22] "POST /v1/chat/completions HTTP/1.1" 200 -
2025-11-04 14:29:22,474 - INFO - Prompt processing progress: 0/69
2025-11-04 14:29:23,633 - INFO - Prompt processing progress: 68/69
2025-11-04 14:29:23,671 - INFO - Prompt processing progress: 69/69
<|channel|>analysis<|message|>We need to respond. The user just says "Hello!" It's a greeting. We need to respond politely. Then maybe ask how can help. So produce friendly answer.<|end|><|start|>assistant<|channel|>final<|message|>Hello! 😊 How can I help you today?
🍏 > /bye

TomLucidor Nov 5, 2025
Author

Could you check Qwen3 and Kimi Linear just in case?

kush-gupt Nov 5, 2025

@TomLucidor So Kimi Linear just isn't in MLX yet unless you build off that PR because of how new it is. Once it's in MLX, you should be able to run uv tool upgrade mlx-lm and then the ramalama commands.

Here is Qwen3 Coder working just fine though:

❯ ramalama --runtime=mlx run hf://mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
INFO:ramalama:MLX runtime automatically uses --nocontainer mode
🍏 > Create an infinite python loop
Here are several ways to create infinite loops in Python:

## Method 1: Using `while True` (Most Common)
```python
while True:
    print("This loop will run forever")
    # Add your code here
    # Use break condition when needed

Method 2: Using `while 1` (Same as True)

while 1:
    print("This loop will run forever")
    # Add your code here

Method 3: Using `for` loop with `itertools.count()

import itertools

for i in itertools.count():
    print(f"Loop iteration: {i}")
    # Add your code here
    # Use break condition when needed

Method 4: Using `while True` with break condition

counter = 0
while True:
    print(f"Loop iteration: {counter}")
    counter += 1

    # Break condition - remove or modify as needed
    if counter >= 5:
        break

Method 5: Using a function with recursion (be careful!)

def infinite_loop():
    print("Infinite loop running")
    infinite_loop()  # Recursive call

# This will cause a RecursionError in Python
# infinite_loop()

Example with user input and break condition:

while True:
    user_input = input("Enter 'quit' to stop: ")

    if user_input.lower() == 'quit':
        print("Goodbye!")
        break
    else:
        print(f"You entered: {user_input}")

Example with a simple counter:

import time

count = 0
while True:
    print(f"Count: {count}")
    count += 1
    time.sleep(1)  # Wait 1 second between iterations

Important Notes:

Use Ctrl+C (or Ctrl+Z on Windows) to stop an infinite loop in the terminal
Always include a break condition in production code
Be careful with infinite loops in production applications as they can consume system resources
Use break statement to exit the loop when needed
Use continue to skip the current iteration and move to the next one

The most common and recommended approach is while True with proper break conditions.
🍏 > /bye

Answer selected by TomLucidor

TomLucidor Nov 10, 2025
Author

Merged 3 days ago (hype!) Please test that again, so that I can see if can work with coding agents. Could Be The Move.

How can this run MLX #2104

Uh oh!

TomLucidor Nov 4, 2025

Method 2: Using while 1 (Same as True)

Replies: 3 comments · 9 replies

Uh oh!

rhatdan Nov 4, 2025 Maintainer

Uh oh!

Uh oh!

engelmi Nov 4, 2025 Maintainer

Uh oh!

kush-gupt Nov 4, 2025

Uh oh!

engelmi Nov 5, 2025 Maintainer

Uh oh!

engelmi Nov 5, 2025 Maintainer

Uh oh!

engelmi Nov 7, 2025 Maintainer

Uh oh!

kush-gupt Nov 4, 2025

Uh oh!

kush-gupt Nov 4, 2025

Uh oh!

kush-gupt Nov 4, 2025

Uh oh!

TomLucidor Nov 5, 2025 Author

Uh oh!

kush-gupt Nov 5, 2025

Method 2: Using while 1 (Same as True)

Method 3: Using for loop with `itertools.count()

Method 4: Using while True with break condition

Method 5: Using a function with recursion (be careful!)

Example with user input and break condition:

Example with a simple counter:

Uh oh!

TomLucidor Nov 10, 2025 Author

TomLucidor
Nov 4, 2025

Method 2: Using `while 1` (Same as True)

Replies: 3 comments 9 replies

rhatdan
Nov 4, 2025
Maintainer

engelmi
Nov 4, 2025
Maintainer

engelmi Nov 5, 2025
Maintainer

engelmi Nov 5, 2025
Maintainer

engelmi Nov 7, 2025
Maintainer

kush-gupt
Nov 4, 2025

TomLucidor Nov 5, 2025
Author

Method 2: Using `while 1` (Same as True)

Method 3: Using `for` loop with `itertools.count()

Method 4: Using `while True` with break condition

TomLucidor Nov 10, 2025
Author