Skip to content

Add auth support for scorers#18699

Merged
BenWilson2 merged 6 commits intomlflow:masterfrom
BenWilson2:scorer-auth
Nov 17, 2025
Merged

Add auth support for scorers#18699
BenWilson2 merged 6 commits intomlflow:masterfrom
BenWilson2:scorer-auth

Conversation

@BenWilson2
Copy link
Member

@BenWilson2 BenWilson2 commented Nov 5, 2025 •
edited by github-actions bot
Loading

🛠 DevTools 🛠

Open in GitHub Codespaces

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18699/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18699/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/18699/merge

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Adds a new auth reference to handle permissions associated with scorers via MLflow's auth feature.
For future work (considering that API keys will be associated with scorers) it is critical to ensure there is a mechanism to restrict via RBAC any scorers that may be scheduled to an Experiment.

How is this PR tested?

  • Existing unit/integration tests
  • New unit/integration tests
  • Manual tests

Does this PR require documentation update?

  • No. You can skip the rest of this section.
  • Yes. I've updated:
    • Examples
    • API references
    • Instructions

Release Notes

Is this a user-facing change?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

  • area/tracking: Tracking Service, tracking client APIs, autologging
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/evaluation: MLflow model evaluation features, evaluation metrics, and evaluation workflows
  • area/gateway: MLflow AI Gateway client APIs, server, and third-party integrations
  • area/prompts: MLflow prompt engineering features, prompt templates, and prompt management
  • area/tracing: MLflow Tracing features, tracing APIs, and LLM tracing functionality
  • area/projects: MLproject format, project running backends
  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/build: Build and test infrastructure for MLflow
  • area/docs: MLflow documentation pages

How should the PR be classified in the release notes? Choose one:

  • rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
  • rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
  • rn/feature - A new user-facing feature worth mentioning in the release notes
  • rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
  • rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?
  • Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
    Bug fixes, doc updates and new features usually go into minor releases.
  • Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
    Bug fixes and doc updates usually go into patch releases.
  • Yes (this PR will be cherry-picked and included in the next patch release)
  • No (this PR will be included in the next minor release)

@BenWilson2
Add auth support for scorers
235a8c7 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
@BenWilson2
Docs and lint
f243ada 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
@github-actions github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/feature Mention under Features in Changelogs. labels Nov 5, 2025
@BenWilson2 BenWilson2 added the team-review Trigger a team review request label Nov 5, 2025
@BenWilson2
fix
62673e0 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
@BenWilson2
api inventory
212a0f7 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2025 •
edited
Loading

Documentation preview for 647712a is available at:

Changed Pages (1)
More info
  • Ignore this comment if this PR does not change the documentation.
  • The preview is updated when a new commit is pushed to this PR.
  • This comment was created by this workflow run.
  • The documentation was built by this workflow run.

WeichenXu123
WeichenXu123 reviewed Nov 13, 2025
View reviewed changes
mlflow/server/auth/sqlalchemy_store.py
perm = SqlScorerPermission(
experiment_id=experiment_id,
scorer_name=scorer_name,
user_id=user.id,
Copy link
Collaborator

@WeichenXu123 WeichenXu123 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q:
are we going to support granting permission to a group of users (e.g. I want to grant permission to all users for a scorer, but I don't want to add SqlScorerPermission item for every user, this way needs updates if new users are created.

Copy link
Member Author

@BenWilson2 BenWilson2 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RBAC sounds very cool, but our auth system doesn't support it for any existing entities. Do I think we should eventually support it? YES. But I think that should be part of the 'bring auth directly into the tracking server as a core feature' sort of work. I do think we need this. Just not for this PR.

WeichenXu123
WeichenXu123 reviewed Nov 13, 2025
View reviewed changes
mlflow/server/auth/routes.py Outdated
Comment on lines +19 to +22
CREATE_SCORER_PERMISSION = _get_rest_path("/mlflow/scorers/permissions/create")
GET_SCORER_PERMISSION = _get_rest_path("/mlflow/scorers/permissions/get")
UPDATE_SCORER_PERMISSION = _get_rest_path("/mlflow/scorers/permissions/update")
DELETE_SCORER_PERMISSION = _get_rest_path("/mlflow/scorers/permissions/delete")
Copy link
Collaborator

@WeichenXu123 WeichenXu123 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question:

Where are the permission validators for these endpoint path ?

These endpoint should only allow request from Administrator users, otherwise every user can grant permission to himself.

Copy link
Member Author

@BenWilson2 BenWilson2 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lines 727-730 have the permission endpoints that check validate_can_manage_scorer_permission so only admins and those with can_manage authority can modify.

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
docs/docs/self-hosting/security/basic-http-auth.mdx Outdated
<tbody>
<tr>
<td>Register Scorer</td>
<td>`2.0/mlflow/scorers/register`</td>
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>`2.0/mlflow/scorers/register`</td>
<td>`3.0/mlflow/scorers/register`</td>

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
docs/docs/self-hosting/security/basic-http-auth.mdx Outdated
</tr>
<tr>
<td>List Scorers</td>
<td>`2.0/mlflow/scorers/list`</td>
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>`2.0/mlflow/scorers/list`</td>
<td>`3.0/mlflow/scorers/list`</td>

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
docs/docs/self-hosting/security/basic-http-auth.mdx Outdated
</tr>
<tr>
<td>Get Scorer</td>
<td>`2.0/mlflow/scorers/get`</td>
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>`2.0/mlflow/scorers/get`</td>
<td>`3.0/mlflow/scorers/get`</td>

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
docs/docs/self-hosting/security/basic-http-auth.mdx Outdated
</tr>
<tr>
<td>Delete Scorer</td>
<td>`2.0/mlflow/scorers/delete`</td>
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>`2.0/mlflow/scorers/delete`</td>
<td>`3.0/mlflow/scorers/delete`</td>

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
docs/docs/self-hosting/security/basic-http-auth.mdx Outdated
</tr>
<tr>
<td>List Scorer Versions</td>
<td>`2.0/mlflow/scorers/list-versions`</td>
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<td>`2.0/mlflow/scorers/list-versions`</td>
<td>`3.0/mlflow/scorers/list-versions`</td>

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
mlflow/server/auth/__init__.py Outdated
)


def _get_permission_from_scorer_id() -> Permission:
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_get_permission_from_scorer_name?

TomeHirata
TomeHirata reviewed Nov 13, 2025
View reviewed changes
tests/server/auth/test_auth.py Outdated

_send_rest_tracking_post_request(
client.tracking_uri,
"/api/2.0/mlflow/scorers/permissions/create",
Copy link
Collaborator

@TomeHirata TomeHirata Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use 3.0 for new permission endpoints?

Copy link
Member Author

@BenWilson2 BenWilson2 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch - one day I'll actually remember to be consistent about this for new routes ;)

@BenWilson2
Merge branch 'master' into scorer-auth
62a4562 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
@BenWilson2
fix
647712a 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
TomeHirata
TomeHirata approved these changes Nov 14, 2025
View reviewed changes
Copy link
Collaborator

@TomeHirata TomeHirata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@BenWilson2 BenWilson2 added this pull request to the merge queue Nov 17, 2025
Merged via the queue into mlflow:master with commit c8adda9 Nov 17, 2025
46 of 48 checks passed
@BenWilson2 BenWilson2 deleted the scorer-auth branch November 17, 2025 17:05
mprahl pushed a commit to opendatahub-io/mlflow that referenced this pull request Nov 21, 2025
@BenWilson2 @mprahl
Add auth support for scorers (mlflow#18699)
ba46d9b 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
Tian-Sky-Lan pushed a commit to Tian-Sky-Lan/mlflow that referenced this pull request Nov 24, 2025
@BenWilson2 @Tian-Sky-Lan
Add auth support for scorers (mlflow#18699)
5ffcec4 
Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>
Signed-off-by: Tian Lan <sky.blue266000@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TomeHirata TomeHirata TomeHirata approved these changes

@B-Step62 B-Step62 Awaiting requested review from B-Step62

@daniellok-db daniellok-db Awaiting requested review from daniellok-db

@harupy harupy Awaiting requested review from harupy

@kevin-lyn kevin-lyn Awaiting requested review from kevin-lyn

@serena-ruan serena-ruan Awaiting requested review from serena-ruan

@xq-yin xq-yin Awaiting requested review from xq-yin

@WeichenXu123 WeichenXu123 Awaiting requested review from WeichenXu123

Assignees

No one assigned

Labels

area/tracking Tracking service, tracking client APIs, autologging rn/feature Mention under Features in Changelogs. team-review Trigger a team review request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BenWilson2 @WeichenXu123 @TomeHirata