Fix `mlflow.spark.load_model` to handle Unity Catalog Volumes paths correctly by harupy · Pull Request #18672 · mlflow/mlflow

harupy · 2025-11-05T00:16:20Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18672/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18672/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/18672/merge

Related Issues/PRs

Fix #18668

What changes are proposed in this pull request?

This PR fixes a bug where mlflow.spark.load_model incorrectly prepends /dbfs/ to Unity Catalog Volumes paths when using dfs_tmpdir on Databricks clusters with Dedicated access mode.

The issue occurs because dbfs_hdfs_uri_to_fuse_path() was converting all paths to DBFS FUSE paths (e.g., /dbfs/...), but UC Volumes paths (e.g., /Volumes/...) should not be converted as they don't use the /dbfs prefix.

Changes:

Modified dbfs_hdfs_uri_to_fuse_path() in mlflow/utils/uri.py to detect UC Volumes paths and return them unchanged
Added type hints to the function signature for better code clarity
Updated the function's docstring to document the UC Volumes behavior

This brings load_model behavior in line with save_model, which already supports UC Volumes as dfs_tmpdir on Dedicated clusters.

How is this PR tested?

Existing unit/integration tests
Manual tests

The fix was manually tested on Databricks Runtime 17.3 LTS ML with a Dedicated cluster using UC Volumes as dfs_tmpdir.

Does this PR require documentation update?

No. You can skip the rest of this section.

Release Notes

Is this a user-facing change?

Yes. Give a description of this change to be included in the release notes for MLflow users.

Fixed mlflow.spark.load_model to properly handle Unity Catalog Volumes paths when using dfs_tmpdir on Databricks clusters with Dedicated access mode. Previously, the function incorrectly prepended /dbfs/ to UC Volumes paths, causing I/O errors.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

area/models: MLmodel format, model serialization/deserialization, flavors
area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
area/scoring: MLflow Model server, model deployment tools, Spark UDFs

How should the PR be classified in the release notes? Choose one:

rn/bug-fix - A user-facing bug fix worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes (this PR will be cherry-picked and included in the next patch release)

github-actions · 2025-11-05T00:29:39Z

Documentation preview for 87fd660 is available at:

https://pr-18672--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

…ction Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

mlflow/utils/uri.py

harupy · 2025-11-05T06:24:07Z

/resolve

🚀 Resolve workflow started

✅ Workflow completed successfully.

Claude Output

Summary

I've successfully resolved the PR review comment! Here's what was done:

PR Context

PR Fix mlflow.spark.load_model to handle Unity Catalog Volumes paths correctly #18672: Fix mlflow.spark.load_model to not prepend dbfs:/ for UC volumes
Repository: mlflow/mlflow

Review Comment Addressed

File: mlflow/utils/uri.py:414
Reviewer: harupy
Comment: "Update the Returns description."

Changes Made

Updated the docstring for the dbfs_hdfs_uri_to_fuse_path function to accurately reflect that UC Volumes paths are returned unchanged, rather than always returning a DBFS FUSE-style path.

Before:

Returns:
    A DBFS FUSE-style path, e.g. "/dbfs/my-directory"

After:

Returns:
    A DBFS FUSE-style path, e.g. "/dbfs/my-directory". For UC Volumes paths
    (e.g., "/Volumes/..."), returns the path unchanged.

Commit Details

✅ Committed locally with DCO sign-off (commit 87fd660)
✅ All pre-commit hooks passed
✅ Ready to push (not pushed per instructions)

The change clarifies the function's behavior for UC Volumes paths, which was added in this PR but not reflected in the documentation.

🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com>

WeichenXu123

LGTM!

…orrectly (mlflow#18672) Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…orrectly (#18672) Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…orrectly (mlflow#18672) Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…orrectly (#18672) Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

harupy marked this pull request as ready for review November 5, 2025 00:17

harupy mentioned this pull request Nov 5, 2025

[BUG] Not able to use a UC Volume as dfs_tmpdir in mlflow.spark.load_model when running on a Databricks Dedicated cluster #18668

Closed

11 tasks

Add type hint and handle UC Volumes in dbfs_hdfs_uri_to_fuse_path fun…

06a52c0

…ction Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy force-pushed the fix-dfs_tmpdir-spark branch from ae290ee to 06a52c0 Compare November 5, 2025 02:30

github-actions bot added the v3.6.0 label Nov 5, 2025

harupy changed the title ~~Fix mlflow.spark.load_model to not prepend dbfs:/~~ Fix mlflow.spark.load_model to not prepend dbfs:/ for UC volumes Nov 5, 2025

github-actions bot added the rn/bug-fix Mention under Bug Fixes in Changelogs. label Nov 5, 2025

harupy commented Nov 5, 2025

View reviewed changes

mlflow/utils/uri.py Outdated Show resolved Hide resolved

Address PR review comments

87fd660

🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mlflow-app[bot] <mlflow-app[bot]@users.noreply.github.com>

harupy changed the title ~~Fix mlflow.spark.load_model to not prepend dbfs:/ for UC volumes~~ Fix mlflow.spark.load_model to handle Unity Catalog Volumes paths correctly Nov 5, 2025

github-actions bot added area/model-registry Model registry, model registry APIs, and the fluent client calls for model registry area/models MLmodel format, model serialization/deserialization, flavors area/scoring MLflow Model server, model deployment tools, Spark UDFs labels Nov 5, 2025

harupy requested a review from WeichenXu123 November 5, 2025 13:14

WeichenXu123 approved these changes Nov 5, 2025

View reviewed changes

harupy added this pull request to the merge queue Nov 5, 2025

Merged via the queue into mlflow:master with commit 59c979f Nov 5, 2025
52 of 54 checks passed

harupy deleted the fix-dfs_tmpdir-spark branch November 5, 2025 13:47

github-actions bot added v3.6.1 and removed v3.6.0 labels Nov 8, 2025

B-Step62 added v3.6.0 and removed v3.6.1 labels Nov 11, 2025

serena-ruan mentioned this pull request Nov 27, 2025

Fix mlflow.spark.load_model to handle Unity Catalog Volumes paths correctly #19079

Closed

29 tasks

serena-ruan mentioned this pull request Dec 1, 2025

Fix mlflow.spark.load_model to handle Unity Catalog Volumes paths correctly #19131

Merged

29 tasks

B-Step62 mentioned this pull request Dec 5, 2025

Cherry pick 18672 into 2.22 #19233

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `mlflow.spark.load_model` to handle Unity Catalog Volumes paths correctly#18672

Fix `mlflow.spark.load_model` to handle Unity Catalog Volumes paths correctly#18672
harupy merged 2 commits intomlflow:masterfrom
harupy:fix-dfs_tmpdir-spark

harupy commented Nov 5, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

harupy commented Nov 5, 2025 •

edited by github-actions bot

Loading

Summary

PR Context

Review Comment Addressed

Changes Made

Commit Details

Uh oh!

WeichenXu123 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

harupy commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install mlflow from this PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

github-actions bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

harupy commented Nov 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

PR Context

Review Comment Addressed

Changes Made

Commit Details

Uh oh!

WeichenXu123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harupy commented Nov 5, 2025 •

edited

Loading

github-actions bot commented Nov 5, 2025 •

edited

Loading

harupy commented Nov 5, 2025 •

edited by github-actions bot

Loading