Skip to content

Added mapping of https to specific transport#2117

Merged
rhatdan merged 2 commits intocontainers:mainfrom
engelmi:map-https-to-transport
Nov 7, 2025
Merged

Added mapping of https to specific transport#2117
rhatdan merged 2 commits intocontainers:mainfrom
engelmi:map-https-to-transport

Conversation

@engelmi
Copy link
Member

@engelmi engelmi commented Nov 7, 2025

Relates to: #2104

In order to enable simply copying the huggingface and ollama URL from the website and pull the repository, the scheme is detected and then the prefix replaced by one of the specific transports. This also detects if a specific file, e.g. .gguf, should be pulled and doesn't perform the mapping in that case.

Example usage can be seen in the unit tests.
In essence, this enables to go to the huggingface and/or ollama website and simply copy the URL from the browser, e.g. https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit, and have all files in the repository being pulled instead of the .html file.
Note: This requires the hf cli at the moment for hf repos.

Summary by Sourcery

Add transport mapping for HTTPS model URLs and enhance CLI argument post-parsing to preserve and resolve model inputs consistently for both single and list cases.

New Features:

  • Support mapping HTTPS Huggingface and Ollama URLs to hf:// and ollama:// transports

Enhancements:

  • Introduce INITIAL_MODEL and UNRESOLVED_MODEL fields to retain original and pre-resolution MODEL inputs
  • Extend post_parse_setup to uniformly handle both string and list MODEL inputs when applying HTTPS mapping and shortname resolution

Tests:

  • Add parameterized tests for HTTPS mapping and shortname resolution verifying INITIAL_MODEL, UNRESOLVED_MODEL, and final MODEL values

@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Nov 7, 2025

Reviewer's Guide

This PR enhances the CLI argument setup by introducing automatic mapping of HTTPS model URLs to the appropriate transport schemes (hf:// or ollama://) while preserving direct file references, tracking the original and unresolved model inputs, and extending shortname resolution to both single and list inputs. Supporting unit tests were added to validate the end-to-end parsing behavior.

File-Level Changes

Change Details Files
Add HTTPS-to-transport mapping in post_parse_setup
  • Introduced map_https_to_transport to detect http(s) URLs and skip file-specific paths
  • Map huggingface and ollama domains to hf:/ and ollama:/ prefixes
  • Return None for URLs pointing to individual files to bypass mapping
ramalama/cli.py
Track INITIAL_MODEL and UNRESOLVED_MODEL during argument setup
  • Capture the raw MODEL value as INITIAL_MODEL before any translation
  • Store MODEL value before shortname resolution in UNRESOLVED_MODEL
  • Support both string and list types for MODEL throughout mapping and resolution
ramalama/cli.py
Extend shortname resolution to list inputs
  • Apply shortnames.resolve to each element when MODEL is a list
  • Preserve unresolved list in UNRESOLVED_MODEL for consistency
ramalama/cli.py
Add unit tests for URL mapping and model parsing
  • Created parameterized test covering direct file URLs, hf.co shortcuts, ollama.com URLs, and default shortnames
  • Assert correct INITIAL_MODEL, UNRESOLVED_MODEL, MODEL, and alias field assignments
test/unit/test_cli.py

Possibly linked issues

  • #unknown: The PR resolves the regression by adding HTTPS URL mapping for Hugging Face to enable correct model pulling.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @engelmi, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant quality-of-life improvement for ramalama users by enabling direct use of "https://" URLs from Hugging Face and Ollama for model inputs. The system now automatically detects and converts these URLs into their appropriate transport schemes ("hf://" or "ollama://"), simplifying the model pulling process. This enhancement includes smart detection to avoid converting direct file links, ensuring robust and intuitive behavior.

Highlights

  • Automatic URL Scheme Conversion: Implemented logic to automatically convert "https://" URLs from Hugging Face and Ollama domains into "hf://" and "ollama://" transport schemes, respectively.
  • Intelligent Path Handling: The conversion intelligently checks the URL path to ensure it's a repository path (e.g., "user/repo") and not a direct file link (e.g., ".gguf" file), preventing unintended mapping for specific files.
  • Enhanced Model Argument Processing: The post_parse_setup function now applies this URL scheme conversion to MODEL arguments, supporting both single string inputs and lists of model inputs, before the existing shortname resolution.
  • Comprehensive Unit Testing: New parameterized unit tests have been added to test_cli.py to thoroughly validate the new https to transport mapping and confirm that the existing shortname resolution continues to function correctly with the updated logic.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `ramalama/cli.py:331-340` </location>
<code_context>
+        return None
+
+    if getattr(args, "MODEL", None):
+        if isinstance(args.MODEL, str):
+            args.INITIAL_MODEL = args.MODEL
+            translated = map_https_to_transport(args.MODEL)
+            if translated:
+                args.MODEL = translated
+        if isinstance(args.MODEL, list):
+            args.INITIAL_MODEL = [m for m in args.MODEL]
+            for i in range(len(args.MODEL)):
+                translated = map_https_to_transport(args.MODEL[i])
+                if translated:
+                    args.MODEL[i] = translated
+
+    # Resolve shortnames of model input
     if getattr(args, "MODEL", None):
-        if args.subcommand != "rm":
+        if isinstance(args.MODEL, str):
+            args.UNRESOLVED_MODEL = args.MODEL
             resolved_model = shortnames.resolve(args.MODEL)
             if resolved_model:
</code_context>

<issue_to_address>
**suggestion:** Duplicated logic for str and list types could be refactored.

Refactor the repeated str and list handling for MODEL into a shared helper to simplify both transport mapping and shortname resolution.
</issue_to_address>

### Comment 2
<location> `test/unit/test_cli.py:181-190` </location>
<code_context>
+@pytest.mark.parametrize(
</code_context>

<issue_to_address>
**suggestion (testing):** Consider adding tests for list-type MODEL inputs and error conditions.

Add test cases for list-type MODEL inputs to verify correct handling, and include tests for error conditions like malformed URLs, unsupported domains, and missing MODEL attributes.

Suggested implementation:

```python
@pytest.mark.parametrize(
    "input_args, expected_initial, expected_unresolved, expected_resolved",
    [
        # Single MODEL string
        (
            Namespace(MODEL="https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf"),
            "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
            "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
            "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
        ),
        # List-type MODEL input
        (
            Namespace(MODEL=[
                "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
                "https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit"
            ]),
            [
                "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
                "https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit"
            ],
            [
                "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
                "https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit"
            ],
            [
                "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf",
                "https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit"
            ],
        ),
        # Malformed URL
        (
            Namespace(MODEL="not_a_url"),
            "not_a_url",
            "not_a_url",
            pytest.raises(ValueError),  # Expecting error handling
        ),
        # Unsupported domain
        (
            Namespace(MODEL="https://unsupported-domain.com/model.gguf"),
            "https://unsupported-domain.com/model.gguf",
            "https://unsupported-domain.com/model.gguf",
            pytest.raises(NoGGUFModelFileFound),  # Expecting error handling
        ),
        # Missing MODEL attribute
        (
            Namespace(),
            None,
            None,
            pytest.raises(AttributeError),  # Expecting error handling
        ),
        (
            Namespace(MODEL="https://hf.co/mlx-community/Kimi-Linear-48B-A3B-Instruct-4bit"),

```

You may need to adjust the test function to handle cases where `pytest.raises` is used as the expected output, i.e., wrap the call in a `with` block for those cases. Also, ensure that your code under test can handle list-type MODEL inputs and error conditions as expected.
</issue_to_address>

### Comment 3
<location> `test/unit/test_cli.py:202-204` </location>
<code_context>
+            "ollama://library/smollm:135m",
+            "ollama://library/smollm:135m",
+        ),
+        (Namespace(MODEL="tinyllama"), "tinyllama", "tinyllama", "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"),
+    ],
+)
+def test_post_parse_setup_model_input(
+    input_args: Namespace, expected_initial: str, expected_unresolved: str, expected_resolved: str
+):
+    input_args.debug = False
+    post_parse_setup(input_args)
+
+    assert hasattr(input_args, "INITIAL_MODEL"), "parsed arguments should always have INITIAL_MODEL field"
+    assert hasattr(input_args, "UNRESOLVED_MODEL"), "parsed arguments should always have RESOLVED_MODEL field"
+
+    assert input_args.INITIAL_MODEL == expected_initial
+    assert input_args.UNRESOLVED_MODEL == expected_unresolved
+    assert input_args.MODEL == expected_resolved
+    assert input_args.model == input_args.MODEL
</code_context>

<issue_to_address>
**suggestion (testing):** Consider adding a test for unsupported or unknown shortnames.

Please add a test case with an unknown shortname to confirm the code's behavior in this scenario, such as leaving MODEL unchanged or raising an error.

```suggestion
        (Namespace(MODEL="tinyllama"), "tinyllama", "tinyllama", "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"),
        # Test for unknown/unsupported shortname
        (Namespace(MODEL="unknownmodel"), "unknownmodel", "unknownmodel", "unknownmodel"),
    ],
)
```
</issue_to_address>

### Comment 4
<location> `test/unit/test_cli.py:217` </location>
<code_context>
+        (Namespace(MODEL="https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf"),
</code_context>

<issue_to_address>
**suggestion (testing):** Consider testing URLs with http (not https) and with additional path segments.

Please add tests for http URLs and for cases with varying numbers of path segments to fully cover the mapping logic.
</issue_to_address>

### Comment 5
<location> `ramalama/cli.py:314` </location>
<code_context>

 def post_parse_setup(args):
     """Perform additional setup after parsing arguments."""
+
</code_context>

<issue_to_address>
**issue (complexity):** Consider refactoring the model transformation logic to normalize input and apply both mapping and resolution in a single pass.

Consider normalizing `args.MODEL` to a list once, then running both transforms in one pass. For example:

```python
def _normalize_to_list(x):
    return x if isinstance(x, list) else [x]

def _apply_transforms(models: list[str]) -> list[str]:
    # 1) map https:// → hf:/ollama:
    # 2) resolve shortnames
    out = []
    for m in models:
        m2 = map_https_to_transport(m) or m
        out.append(shortnames.resolve(m2) or m2)
    return out

def post_parse_setup(args):
    # ...
    if not getattr(args, "MODEL", None):
        return

    raw = args.MODEL
    models = _normalize_to_list(raw)
    args.UNRESOLVED_MODEL = models.copy()
    transformed = _apply_transforms(models)

    # restore original shape
    args.MODEL = transformed if isinstance(raw, list) else transformed[0]

    if not hasattr(args, "model"):
        args.model = args.MODEL

    # ...
```

Benefits:
- removes duplicate str/list branches
- flattens nesting
- keeps `INITIAL_MODEL` or other tracking trivial if needed (`args.INITIAL_MODEL = raw` or `.copy()`)
</issue_to_address>

### Comment 6
<location> `ramalama/cli.py:337` </location>
<code_context>
def post_parse_setup(args):
    """Perform additional setup after parsing arguments."""

    # Resolve https:// model inputs to ollama or huggingface based on url domain
    def map_https_to_transport(input: str) -> str | None:
        if input.startswith("https://") or input.startswith("http://"):
            url = urlparse(input)
            # detect if the whole repo is defined or a specific file
            if url.path.count("/") != 2:
                return None
            if url.hostname in ["hf.co", "huggingface.co"]:
                return f"hf:/{url.path}"
            elif url.hostname in ["ollama.com"]:
                return f"ollama:/{url.path}"
        return None

    if getattr(args, "MODEL", None):
        if isinstance(args.MODEL, str):
            args.INITIAL_MODEL = args.MODEL
            translated = map_https_to_transport(args.MODEL)
            if translated:
                args.MODEL = translated
        if isinstance(args.MODEL, list):
            args.INITIAL_MODEL = [m for m in args.MODEL]
            for i in range(len(args.MODEL)):
                translated = map_https_to_transport(args.MODEL[i])
                if translated:
                    args.MODEL[i] = translated

    # Resolve shortnames of model input
    if getattr(args, "MODEL", None):
        if isinstance(args.MODEL, str):
            args.UNRESOLVED_MODEL = args.MODEL
            resolved_model = shortnames.resolve(args.MODEL)
            if resolved_model:
                args.MODEL = resolved_model
        if isinstance(args.MODEL, list):
            args.UNRESOLVED_MODEL = [m for m in args.MODEL]
            for i in range(len(args.MODEL)):
                resolved_model = shortnames.resolve(args.MODEL[i])
                if resolved_model:
                    args.MODEL[i] = resolved_model

        # TODO: This is a hack. Once we have more typing in place these special cases
        # _should_ be removed.
        if not hasattr(args, "model"):
            args.model = args.MODEL

    # Validate that --add-to-unit is only used with --generate and its format is <section>:<key>:<value>
    if hasattr(args, "add_to_unit") and (add_to_units := args.add_to_unit):
        if getattr(args, "generate", None) is None:
            parser = get_parser()
            parser.error("--add-to-unit can only be used with --generate")
        if not (all(len([value for value in unit_to_add.split(":", 2) if value]) == 3 for unit_to_add in add_to_units)):
            parser = get_parser()
            parser.error("--add-to-unit parameters must be of the form <section>:<key>:<value>")

    if hasattr(args, "runtime_args"):
        args.runtime_args = shlex.split(args.runtime_args)

    # MLX runtime automatically requires --nocontainer
    if getattr(args, "runtime", None) == "mlx":
        if getattr(args, "container", None) is True:
            logger.info("MLX runtime automatically uses --nocontainer mode")
        args.container = False

    if hasattr(args, 'pull'):
        args.pull = normalize_pull_arg(args.pull, getattr(args, 'engine', None))

    configure_logger("DEBUG" if args.debug else "WARNING")

</code_context>

<issue_to_address>
**issue (code-quality):** We've found these issues:

- Replace identity comprehension with call to collection constructor [×2] ([`identity-comprehension`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/identity-comprehension/))
- Invert any/all to simplify comparisons ([`invert-any-all`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/invert-any-all/))

<br/><details><summary>Explanation</summary>Convert list/set/tuple comprehensions that do not change the input elements into.

#### Before

```python
# List comprehensions
[item for item in coll]
[item for item in friends.names()]

# Dict comprehensions
{k: v for k, v in coll}
{k: v for k, v in coll.items()}  # Only if we know coll is a `dict`

# Unneeded call to `.items()`
dict(coll.items())  # Only if we know coll is a `dict`

# Set comprehensions
{item for item in coll}
```

#### After

```python
# List comprehensions
list(iter(coll))
list(iter(friends.names()))

# Dict comprehensions
dict(coll)
dict(coll)

# Unneeded call to `.items()`
dict(coll)

# Set comprehensions
set(coll)
```

All these comprehensions are just creating a copy of the original collection.
They can all be simplified by simply constructing a new collection directly. The
resulting code is easier to read and shows the intent more clearly.
Convert list/set/tuple comprehensions that do not change the input elements into.

#### Before

```python
# List comprehensions
[item for item in coll]
[item for item in friends.names()]

# Dict comprehensions
{k: v for k, v in coll}
{k: v for k, v in coll.items()}  # Only if we know coll is a `dict`

# Unneeded call to `.items()`
dict(coll.items())  # Only if we know coll is a `dict`

# Set comprehensions
{item for item in coll}
```

#### After

```python
# List comprehensions
list(iter(coll))
list(iter(friends.names()))

# Dict comprehensions
dict(coll)
dict(coll)

# Unneeded call to `.items()`
dict(coll)

# Set comprehensions
set(coll)
```

All these comprehensions are just creating a copy of the original collection.
They can all be simplified by simply constructing a new collection directly. The
resulting code is easier to read and shows the intent more clearly.
</details>
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request successfully introduces the functionality to map HTTPS URLs from Hugging Face and Ollama to their respective internal transport schemes, which is a valuable enhancement for user convenience. The new unit tests effectively validate this new behavior across various scenarios. My main feedback focuses on improving the maintainability of the post_parse_setup function by refactoring duplicated logic for model processing.

@engelmi engelmi force-pushed the map-https-to-transport branch from e5dcf32 to 992afa0 Compare November 7, 2025 09:25
Relates to: containers#2104

In order to enable simply copying the huggingface and ollama URL
from the website and pull the repository, the scheme is detected
and then the prefix replaced by one of the specific transports.
This also detects if a specific file, e.g. .gguf, should be pulled
and doesn't perform the mapping in that case.

Signed-off-by: Michael Engel <mengel@redhat.com>
Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi force-pushed the map-https-to-transport branch from b1509bf to 3135c70 Compare November 7, 2025 14:45
@rhatdan
Copy link
Member

rhatdan commented Nov 7, 2025

LGTM

@rhatdan rhatdan merged commit 4fecfb1 into containers:main Nov 7, 2025
39 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants