Store gateway<>scorer binding correctly by TomeHirata · Pull Request #20176 · mlflow/mlflow

TomeHirata · 2026-01-21T06:20:00Z

Related Issues/PRs

n/a

What changes are proposed in this pull request?

This PR makes sure that the binding record is created between gateway and scorer when a gateway endpoint is used by a registered scorer.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

github-actions · 2026-01-21T06:20:21Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20176/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20176/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/20176/merge

TomeHirata · 2026-01-21T06:21:19Z

mlflow/entities/gateway_endpoint.py

 class GatewayResourceType(str, Enum):
    """Valid MLflow resource types that can use gateway endpoints."""

-    SCORER_JOB = "scorer_job"


renamed scorer_job to scorer since endpoint id is linked to a registered scorer instead of each scorer execution job

Copilot

Pull request overview

This PR corrects the resource type terminology for gateway-scorer bindings from "scorer_job" to "scorer" to better reflect the actual relationship being tracked. The change updates the enum value, tests, documentation, and UI labels across Python, TypeScript/JavaScript, and Java codebases. Additionally, new integration tests verify that endpoint bindings are correctly created when scorers are registered with gateway endpoints and properly cleaned up when scorers are deleted.

Changes:

Updated GatewayResourceType enum from SCORER_JOB = "scorer_job" to SCORER = "scorer"
Added endpoint binding creation logic in register_scorer() and cleanup logic in delete_scorer()
Updated all tests and UI components to use the new "scorer" resource type terminology
Added comprehensive integration tests for endpoint binding lifecycle

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
mlflow/entities/gateway_endpoint.py	Changed enum value from SCORER_JOB to SCORER
mlflow/store/tracking/sqlalchemy_store.py	Added binding creation/deletion logic and imported GatewayResourceType
mlflow/store/tracking/gateway/*.py	Updated documentation examples from "scorer_job" to "scorer"
tests/genai/scorers/test_scorer_CRUD.py	Added new integration tests for endpoint binding creation and deletion
tests/tracking/test_rest_tracking.py	Updated all test assertions to use new resource type
tests/store/tracking/test_rest_store.py	Updated mock responses and test data to use "scorer"
tests/store/tracking/test_gateway_sql_store.py	Updated all binding tests to use "scorer"
tests/entities/test_gateway_endpoint.py	Updated binding tests and enum assertions
mlflow/server/js/src/gateway/types.ts	Changed ResourceType from 'scorer_job' to 'scorer'
mlflow/server/js/src/gateway/components/*.tsx	Updated UI labels from "Scorer Job" to "Scorer"
mlflow/server/js/src/gateway/hooks/*.test.tsx	Updated test fixtures to use 'scorer'
mlflow/protos/service.proto	Updated documentation comments to use "scorer" example
mlflow/java/client/src/main/java/org/mlflow/api/proto/Service.java	Updated generated Java code with new documentation
mlflow/store/db_migrations/alembic.ini	Changed sqlalchemy.url from empty to hardcoded SQLite path

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mlflow/store/tracking/sqlalchemy_store.py

mlflow/store/db_migrations/alembic.ini

mlflow/store/tracking/sqlalchemy_store.py

Copilot · 2026-01-21T06:24:22Z

mlflow/store/tracking/sqlalchemy_store.py

+            # Verify the endpoint exists in the database before creating binding
+            if endpoint_id is not None:
+                endpoint_exists = (
+                    session.query(SqlGatewayEndpoint)
+                    .filter(SqlGatewayEndpoint.endpoint_id == endpoint_id)
+                    .first()
+                )
+                if endpoint_exists is not None:
+                    # Delete any existing binding for this scorer (in case of re-registration)
+                    session.query(SqlGatewayEndpointBinding).filter(
+                        SqlGatewayEndpointBinding.resource_type == GatewayResourceType.SCORER.value,
+                        SqlGatewayEndpointBinding.resource_id == name,
+                    ).delete()
+
+                    binding = SqlGatewayEndpointBinding(
+                        endpoint_id=endpoint_id,
+                        resource_type=GatewayResourceType.SCORER.value,
+                        resource_id=name,
+                        created_at=get_current_time_millis(),
+                        last_updated_at=get_current_time_millis(),
+                    )
+                    session.add(binding)
+


The endpoint existence check on line 2185-2189 is redundant since the endpoint was already retrieved on line 2134 via get_gateway_endpoint, which would have raised an exception if the endpoint didn't exist. This additional database query is unnecessary and could be removed to improve performance. If the intent is to handle the case where the endpoint was deleted between line 2134 and this point, the session transaction would prevent that scenario.

Suggested change

# Verify the endpoint exists in the database before creating binding

if endpoint_id is not None:

endpoint_exists = (

session.query(SqlGatewayEndpoint)

.filter(SqlGatewayEndpoint.endpoint_id == endpoint_id)

.first()

)

if endpoint_exists is not None:

# Delete any existing binding for this scorer (in case of re-registration)

session.query(SqlGatewayEndpointBinding).filter(

SqlGatewayEndpointBinding.resource_type == GatewayResourceType.SCORER.value,

SqlGatewayEndpointBinding.resource_id == name,

).delete()

binding = SqlGatewayEndpointBinding(

endpoint_id=endpoint_id,

resource_type=GatewayResourceType.SCORER.value,

resource_id=name,

created_at=get_current_time_millis(),

last_updated_at=get_current_time_millis(),

)

session.add(binding)

if endpoint_id is not None:

# Delete any existing binding for this scorer (in case of re-registration)

session.query(SqlGatewayEndpointBinding).filter(

SqlGatewayEndpointBinding.resource_type == GatewayResourceType.SCORER.value,

SqlGatewayEndpointBinding.resource_id == name,

).delete()

binding = SqlGatewayEndpointBinding(

endpoint_id=endpoint_id,

resource_type=GatewayResourceType.SCORER.value,

resource_id=name,

created_at=get_current_time_millis(),

last_updated_at=get_current_time_millis(),

)

session.add(binding)

mlflow/store/tracking/sqlalchemy_store.py

github-actions · 2026-01-21T06:28:16Z

Documentation preview for c84b2ab is available at:

https://pr-20176--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

- Use scorer_id instead of scorer name as resource_id for endpoint bindings to ensure globally unique identification across experiments - Add display_name field to endpoint bindings for showing human-readable scorer names in the UI (instead of UUIDs) - Update frontend to display display_name when available, falling back to resource_id for backwards compatibility 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

serena-ruan · 2026-01-22T08:53:28Z

mlflow/store/tracking/sqlalchemy_store.py

+                if endpoint_exists is not None:
+                    # Delete any existing binding for this scorer (in case of re-registration)
+                    # Use scorer_id for globally unique identification across experiments
+                    session.query(SqlGatewayEndpointBinding).filter(
+                        SqlGatewayEndpointBinding.resource_type == GatewayResourceType.SCORER.value,
+                        SqlGatewayEndpointBinding.resource_id == scorer.scorer_id,
+                    ).delete()


Why do we need this? The scorer can only be registered once right?

Scorers can have multiple versions and this is needed to clean up the binding of older versions.

serena-ruan

LGTM once https://github.com/mlflow/mlflow/pull/20176/files#r2715910365 is resolved

- Use scorer_id instead of scorer name as resource_id for endpoint bindings to ensure globally unique identification across experiments - Add display_name field to endpoint bindings for showing human-readable scorer names in the UI (instead of UUIDs) - Update frontend to display display_name when available, falling back to resource_id for backwards compatibility 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

…flow into fix/scorer/gateway-binding

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

Fix scorer binding

117df08

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

Copilot AI review requested due to automatic review settings January 21, 2026 06:20

TomeHirata added the team-review Trigger a team review request label Jan 21, 2026

TomeHirata requested a review from BenWilson2 January 21, 2026 06:20

github-actions bot added area/uiux Front-end, user experience, plotting, JavaScript, JavaScript dev server rn/bug-fix Mention under Bug Fixes in Changelogs. labels Jan 21, 2026

Copilot started reviewing on behalf of TomeHirata January 21, 2026 06:20 View session

github-actions bot requested review from daniellok-db and serena-ruan January 21, 2026 06:20

TomeHirata commented Jan 21, 2026

View reviewed changes

Copilot AI reviewed Jan 21, 2026

View reviewed changes

TomeHirata added the v3.9.0 label Jan 21, 2026

rename scorer job to scorer

25d640d

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

TomeHirata force-pushed the fix/scorer/gateway-binding branch from 006958f to 25d640d Compare January 21, 2026 06:50

serena-ruan reviewed Jan 22, 2026

View reviewed changes

serena-ruan approved these changes Jan 22, 2026

View reviewed changes

github-actions bot assigned serena-ruan Jan 22, 2026

TomeHirata and others added 6 commits January 22, 2026 18:47

Merge branch 'fix/scorer/gateway-binding' of github.com:TomeHirata/ml…

ddf89ae

…flow into fix/scorer/gateway-binding

Merge branch 'fix/scorer/gateway-binding' of github.com:TomeHirata/ml…

85e0e20

…flow into fix/scorer/gateway-binding

Merge branch 'fix/scorer/gateway-binding' of github.com:TomeHirata/ml…

428570d

…flow into fix/scorer/gateway-binding

version

6c0a96b

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

schema

c84b2ab

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

TomeHirata enabled auto-merge January 22, 2026 10:11

TomeHirata disabled auto-merge January 23, 2026 00:31

TomeHirata merged commit eb0c047 into mlflow:master Jan 23, 2026
54 of 56 checks passed

harupy pushed a commit to harupy/mlflow that referenced this pull request Jan 28, 2026

Store gateway<>scorer binding correctly (mlflow#20176)

0d3cd65

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

harupy pushed a commit to harupy/mlflow that referenced this pull request Jan 28, 2026

Store gateway<>scorer binding correctly (mlflow#20176)

32c9e12

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

harupy pushed a commit that referenced this pull request Jan 28, 2026

Store gateway<>scorer binding correctly (#20176)

c34ee57

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store gateway<>scorer binding correctly#20176

Store gateway<>scorer binding correctly#20176
TomeHirata merged 9 commits intomlflow:masterfrom
TomeHirata:fix/scorer/gateway-binding

TomeHirata commented Jan 21, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 21, 2026

Install mlflow from this PR

Uh oh!

TomeHirata Jan 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 21, 2026

Uh oh!

Uh oh!

github-actions bot commented Jan 21, 2026 •

edited

Loading

Uh oh!

serena-ruan Jan 22, 2026

Uh oh!

TomeHirata Jan 22, 2026 •

edited

Loading

Uh oh!

serena-ruan left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

TomeHirata commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

github-actions bot commented Jan 21, 2026

Install mlflow from this PR

Uh oh!

TomeHirata Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serena-ruan Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

TomeHirata Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serena-ruan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TomeHirata commented Jan 21, 2026 •

edited

Loading

github-actions bot commented Jan 21, 2026 •

edited

Loading

TomeHirata Jan 22, 2026 •

edited

Loading