[UI Telemetry] Update documentation page with details on UI telemetry by daniellok-db · Pull Request #19427 · mlflow/mlflow

daniellok-db · 2025-12-16T09:53:55Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/documentation [Files changed]
- stack/logging-for-tabs [Files changed]

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

This PR updates the usage-tracking page with information about the new UI telemetry.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

PR docs preview

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

Copilot

Pull request overview

This PR adds comprehensive UI telemetry functionality to MLflow, allowing the collection of usage data from user interactions with the MLflow UI. The implementation uses a SharedWorker architecture to batch and send telemetry events efficiently across multiple browser tabs.

Key Changes

Implements a SharedWorker-based telemetry client that batches UI interaction events and sends them to the server
Adds server-side handlers for receiving and processing UI telemetry data with configurable opt-in/opt-out controls
Adds a Settings page in the UI where users can control their telemetry preferences
Updates documentation to describe what UI telemetry data is collected and how to opt out

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`tests/telemetry/test_client.py`	Adds test for batch processing of telemetry records
`tests/server/test_handlers.py`	Adds comprehensive tests for UI telemetry GET/POST handlers
`tests/server/conftest.py`	Adds test fixtures for telemetry testing
`mlflow/telemetry/client.py`	Implements `add_records()` method for batching multiple telemetry records
`mlflow/server/handlers.py`	Adds GET/POST handlers for UI telemetry with caching and configuration management
`mlflow/server/__init__.py`	Registers new UI telemetry API endpoints
`mlflow/server/js/tsconfig.json`	Adds WebWorker library support for TypeScript
`mlflow/server/js/src/telemetry/worker/*.ts`	Implements SharedWorker for telemetry logging with queue management
`mlflow/server/js/src/telemetry/TelemetryClient.ts`	Client-side API for logging UI events to the worker
`mlflow/server/js/src/telemetry/TelemetryInfoAlert.tsx`	Alert component informing users about telemetry
`mlflow/server/js/src/telemetry/README.md`	Technical documentation for the telemetry architecture
`mlflow/server/js/src/settings/SettingsPage.tsx`	Settings UI for managing telemetry preferences
`mlflow/server/js/src/experiment-tracking/routes.ts`	Adds routing for the settings page
`mlflow/server/js/src/common/components/MlflowSidebar.tsx`	Adds Settings link to the navigation sidebar
`mlflow/server/js/src/app.tsx`	Integrates telemetry client with DesignSystemEventProvider
`mlflow/server/js/craco.config.js`	Configures webpack to bundle the telemetry worker separately
`mlflow/server/js/src/lang/default/en.json`	Adds translations for telemetry-related UI text
`mlflow/server/js/src/home/HomePage.tsx`	Displays telemetry info alert on the home page
`docs/docs/community/usage-tracking.mdx`	Documents UI telemetry data collection and opt-out procedures

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-18T00:55:59Z

docs/docs/community/usage-tracking.mdx

+    </tr>
+    <tr>
+      <td>Timestamp</td>
+      <td>The client-side timestam of when the interaction occurred</td>


Spelling error: "timestam" should be "timestamp"

Suggested change

<td>The client-side timestam of when the interaction occurred</td>

<td>The client-side timestamp of when the interaction occurred</td>

Copilot · 2025-12-18T00:55:59Z

mlflow/server/js/src/telemetry/README.md

+
+**LogQueue.ts**:
+
+- Simple class that batches logs and uploads them to the server every 15s.


The documentation inconsistency between the README and the actual LogQueue implementation. The README states logs are batched and uploaded "every 15s", but the actual implementation in LogQueue.ts uses FLUSH_INTERVAL_MS = 30000 (30 seconds). Update the documentation to match the actual implementation.

Suggested change

- Simple class that batches logs and uploads them to the server every 15s.

- Simple class that batches logs and uploads them to the server every 30s.

Copilot · 2025-12-18T00:55:59Z

mlflow/server/js/src/telemetry/worker/LogQueue.ts

+/**
+ * LogQueue for batching and uploading telemetry events
+ *
+ * Maintains a queue of telemetry records and flushes them every 15 seconds


The documentation comment incorrectly states the flush interval is 15 seconds, but the constant FLUSH_INTERVAL_MS is set to 30000 (30 seconds). Update the comment to accurately reflect the actual flush interval.

Suggested change

* Maintains a queue of telemetry records and flushes them every 15 seconds

* Maintains a queue of telemetry records and flushes them every 30 seconds

Copilot · 2025-12-18T00:56:00Z

mlflow/server/handlers.py



+# Cache for telemetry config with 3 hour TTL
+_telemetry_config_cache = TTLCache(maxsize=1, ttl=10800)


The cache TTL is hardcoded to 10800 seconds (3 hours). Consider extracting this as a named constant with a descriptive name like TELEMETRY_CONFIG_CACHE_TTL_SECONDS to improve maintainability and make it easier to adjust this value in the future.

Suggested change

_telemetry_config_cache = TTLCache(maxsize=1, ttl=10800)

TELEMETRY_CONFIG_CACHE_TTL_SECONDS = 10800

_telemetry_config_cache = TTLCache(maxsize=1, ttl=TELEMETRY_CONFIG_CACHE_TTL_SECONDS)

docs/docs/community/usage-tracking.mdx

github-actions · 2025-12-18T01:02:02Z

Documentation preview for 85a0c7a is available at:

https://pr-19427--mlflow-docs-preview.netlify.app/docs/latest/

Changed Pages (1)

community/usage-tracking (modified)

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

docs/docs/community/usage-tracking.mdx

serena-ruan · 2025-12-18T02:25:06Z

docs/docs/community/usage-tracking.mdx

+    <tr>
+      <td>Component ID of interactive UI elements</td>
+      <td>An ID string of an interactive element (e.g. button, switch, link, input field) in the UI. A log is generated upon clicking, typing, or otherwise interacting with such elements. A comprehensive list of component ID values can be found by [this search query](https://github.com/search?q=repo%3Amlflow%2Fmlflow%20componentId%3D&type=code).</td>
+      <td>`mlflow.prompts.list.create` (identifier for the "Create prompt" button on the prompts page)</td>
+      <td>To understand the usage patterns of different UI pages and features</td>
+    </tr>
+    <tr>
+      <td>Metadata associated with UI interactions</td>
+      <td>See [below table](#ui-interaction-metadata) for metadata associated with UI interaction logs.</td>
+      <td>`{ "isRemote": true, "browserFamily": "Chrome", "isMobile": false, "eventType": "onClick", "componentViewId": "88fc9edd-5e9e-4a17-abd2-c543f505b8eb", "componentId": "mlflow.prompts.list.create", "componentType": "button", timestamp_ns: 1765784028467000000 }`</td>
+      <td>To understand UI usage patterns in greater depth, and to inform prioritization of different platforms, browsers, and user flows.</td>
+    </tr>


I thought these are included in parameters field?

they are but i think it's not an important distinction (user probably doesn't care about the structure of the payload, just what data is contained inside), i also felt that including all those params into the existing table would be a bit noisy since they only apply to UI events

Can we include them in Tracked Events table?

docs/docs/community/usage-tracking.mdx

daniellok-db · 2025-12-18T02:40:42Z

docs/docs/community/usage-tracking.mdx

+Organizations can disable telemetry by blocking network access to `https://config.mlflow-telemetry.io`. When this endpoint is unreachable, MLflow automatically disables telemetry for the SDK.
+
+To opt out of UI telemetry, you can block network access to `https://d139nb52glx00z.cloudfront.net` from the MLflow server. Similar to above, if this endpoint is unreachable, UI telemetry will be disabled.


i wonder if we should make it easy and just say that you can disable all traffic to the mlflow-telemetry.io domain, since that covers both config and ingestion

Yes I think we don't need to mention this section especially https://d139nb52glx00z.cloudfront.net specifically. If we change it to be the same domain as sdk later then people won't check this page again to disable it, and for users who look at this section I feel like they may just opt out entire telemetry

done, simplified the section

daniellok-db · 2025-12-18T04:53:55Z

docs/docs/community/usage-tracking.mdx

+      <td>Browser family</td>
+      <td>An enumerated string describing the browser family</td>
+      <td>`Chrome`, `Safari`, `Firefox`, `Other`</td>
+    </tr>
+    <tr>
+      <td>Mobile status</td>
+      <td>A boolean indicating whether or not the event took place on a mobile device</td>
+      <td>`false`</td>
+    </tr>
+    <tr>


just to note: these do not actually exist in the logs right now but including it here so we can add them in the future without updating the docs

When do we plan to add them? we can update the doc later when adding them right, people may ask you about this data if seeing this :)

makes sense, removing for now then

serena-ruan

LGTM once #19427 (comment) is addressed!

…mlflow#19427)

…#19427)

daniellok-db changed the title ~~add docs~~ [UI Telemetry] Update documentation page with details on UI telemetry Dec 16, 2025

daniellok-db force-pushed the stack/documentation branch from db81eb3 to b264a48 Compare December 18, 2025 00:41

daniellok-db marked this pull request as ready for review December 18, 2025 00:53

Copilot AI review requested due to automatic review settings December 18, 2025 00:53

Copilot started reviewing on behalf of daniellok-db December 18, 2025 00:53 View session

Copilot AI reviewed Dec 18, 2025

View reviewed changes

serena-ruan reviewed Dec 18, 2025

View reviewed changes

docs/docs/community/usage-tracking.mdx Outdated Show resolved Hide resolved

serena-ruan reviewed Dec 18, 2025

View reviewed changes

docs/docs/community/usage-tracking.mdx Outdated Show resolved Hide resolved

daniellok-db commented Dec 18, 2025

View reviewed changes

daniellok-db force-pushed the stack/documentation branch from 2f0a68f to dd01b3d Compare December 18, 2025 03:31

github-actions bot assigned serena-ruan Dec 18, 2025

github-actions bot added v3.8.0 area/docs Documentation issues rn/none List under Small Changes in Changelogs. labels Dec 18, 2025

daniellok-db mentioned this pull request Dec 18, 2025

[UI Telemetry] Add rudimentary logging for navbar clicks #19480

Merged

29 tasks

daniellok-db commented Dec 18, 2025

View reviewed changes

daniellok-db added 6 commits December 18, 2025 15:42

add docs

d2dd18c

fix

914bdd1

comments and refine

47836a4

dnt

f8b351a

comments and simplify

67ae9cd

of

43f8e0c

daniellok-db force-pushed the stack/documentation branch from b2e82e6 to 43f8e0c Compare December 18, 2025 06:42

serena-ruan approved these changes Dec 18, 2025

View reviewed changes

remove

85a0c7a

daniellok-db added this pull request to the merge queue Dec 18, 2025

Merged via the queue into mlflow:master with commit 7787923 Dec 18, 2025
49 of 50 checks passed

daniellok-db deleted the stack/documentation branch December 18, 2025 08:27

WeichenXu123 pushed a commit to WeichenXu123/mlflow that referenced this pull request Dec 19, 2025

[UI Telemetry] Update documentation page with details on UI telemetry (…

49be808

…mlflow#19427)

WeichenXu123 pushed a commit that referenced this pull request Dec 19, 2025

[UI Telemetry] Update documentation page with details on UI telemetry (…

dbe087a

…#19427)

	<td>The client-side timestam of when the interaction occurred</td>
	<td>The client-side timestamp of when the interaction occurred</td>


		LogQueue.ts:

		- Simple class that batches logs and uploads them to the server every 15s.

	* Maintains a queue of telemetry records and flushes them every 15 seconds
	* Maintains a queue of telemetry records and flushes them every 30 seconds



		# Cache for telemetry config with 3 hour TTL
		_telemetry_config_cache = TTLCache(maxsize=1, ttl=10800)

	_telemetry_config_cache = TTLCache(maxsize=1, ttl=10800)
	TELEMETRY_CONFIG_CACHE_TTL_SECONDS = 10800
	_telemetry_config_cache = TTLCache(maxsize=1, ttl=TELEMETRY_CONFIG_CACHE_TTL_SECONDS)

		Organizations can disable telemetry by blocking network access to `https://config.mlflow-telemetry.io`. When this endpoint is unreachable, MLflow automatically disables telemetry for the SDK.

		To opt out of UI telemetry, you can block network access to `https://d139nb52glx00z.cloudfront.net` from the MLflow server. Similar to above, if this endpoint is unreachable, UI telemetry will be disabled.

Conversation

daniellok-db commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniellok-db Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serena-ruan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

daniellok-db commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 18, 2025 •

edited

Loading

daniellok-db Dec 18, 2025 •

edited

Loading