feat: Add gzip/deflate Content-Encoding support to OTLP traces endpoint by Miaoxiang-philips · Pull Request #19024 · mlflow/mlflow

Miaoxiang-philips · 2025-11-25T08:23:26Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19024/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19024/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/19024/merge

Related Issues/PRs

Related #18613

What changes are proposed in this pull request?

MLflow's OTLP/HTTP traces ingestion currently does not support any compression format.
OpenTelemetry Collector sends OTLP/HTTP requests using gzip compression by default, which causes MLflow to return:

error": "not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to http://host.docker.internal:5000/v1/traces responded with HTTP Status Code 400", "dropped_items": 1}

This PR adds support for the required OTLP/HTTP compression encodings:

gzip (required by OTLP specification)
deflate
identity (default)

The server now automatically decompresses the request body before parsing the protobuf payload. This makes MLflow compatible with all standard OTLP/HTTP exporters without requiring the user to disable compression.

This change does not affect any existing functionality and maintains full backward compatibility.

Reproduce steps:

prepare opentelemetry-collector.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp:
    endpoint: http://host.docker.internal:5000 # if running otel collector by docker, This can be connect to localhost:5000
    headers:
      x-mlflow-experiment-id: "0" # default id

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

start otel collector by docker

docker run --name otel-collector -d -p 4318:4318 -v $(pwd)/opentelemetry-collector.yaml:/etc/otel/collector/config.yam otel/opentelemetry-collector-contrib # latest tag or 0.139.0(in my environment)

send data with otlp

If you don't have a ready-made method, you can use telemetrygen to simulate data transmission, for example:

telemetrygen traces \
  --otlp-http http://localhost:4318 \
  --rate 10 \
  --duration 30s \
  --otlp-insecure

telemetrygen download link: telemetrygen package

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

mlflow/server/otel_api.py

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

- Extract Content-Encoding decompression logic into `decompress_otlp_body` helper in `mlflow/tracing/utils/otlp.py` - Support identity, gzip, and deflate encodings using pattern match (Python 3.10+) - Simplify handler by delegating decompression to helper - Improves maintainability and allows future reuse for other OTLP endpoints Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Miaoxiang-philips · 2025-11-26T03:21:23Z

Hi @harupy , Thank you for your comments. I have made some adjustments. Please review again

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

tests/tracing/test_otel_logging.py

tests/tracing/utils/test_otlp.py

github-actions · 2025-11-26T04:05:12Z

Documentation preview for 8f09079 is available at:

https://pr-19024--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Removed docstring comments from test functions for clarity. Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

- Parameterize compression tests to cover multiple encodings (identity, gzip, deflate-rfc, deflate-raw) - Parameterize error tests to cover invalid data for each encoding type - Add type hints to test functions for better code clarity - Use pytest 8.4+ check parameter for cleaner exception validation - Increase test coverage from 3 to 7 test cases 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy

LGTM!

harupy · 2025-11-26T05:55:35Z

@Miaoxiang-philips I've pushed a commit. Can you take a look and make sure everything looks ok?

Miaoxiang-philips · 2025-11-26T06:41:39Z

@Miaoxiang-philips I've pushed a commit. Can you take a look and make sure everything looks ok?

I've checked it, and it's fine for me.

harupy · 2025-11-26T09:29:25Z

@Miaoxiang-philips Thanks for the confirmation. I wonder if we could add a lightweight integration test (that can reproducer the error on master but works on this PR) using docker (or anything that could've detected the error)

Miaoxiang-philips · 2025-11-26T10:53:16Z

@Miaoxiang-philips Thanks for the confirmation. I wonder if we could add a lightweight integration test (that can reproducer the error on master but works on this PR) using docker (or anything that could've detected the error)

Sure, but I couldn't find anything related to integration testing in CONTRIBUTING.md. Could you give me some help information?

If you are only concerned about whether the change is reasonable, I can provide more information:

otel collector uses zip as compression by default when use otlphttp exporter, unless explicitly declared as none: otlphttpexporter

Generally, the /v1/traces endpoint needs to handle at least zip compression, which is quite normal

harupy · 2025-11-26T11:05:59Z

@Miaoxiang-philips Thanks for the reply! Could you share the steps to reproduce the error? I’ll turn them into a test case.

Miaoxiang-philips · 2025-11-27T03:02:00Z

Sure, I updated to the description of PR @harupy

harupy · 2025-11-27T08:42:00Z

@Miaoxiang-philips Thanks! I think I'm missing something. I ran uv run mlflow server (on master ) and follow the steps you shared but could not reproduce the error.

http://localhost:55679/debug/tracez:

Never mind, I was able to reproduce the error.

harupy · 2025-11-27T09:58:15Z

Awesome, manually confirmed it works!

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy · 2025-11-27T10:16:53Z

mlflow/tracing/utils/otlp.py

+    """
+    from fastapi import HTTPException, status
+
+    match content_encoding:


I removed identity. According to https://github.com/open-telemetry/opentelemetry-collector/blob/73f090cd0d59a3faa237168750d44a0036d8fd5c/config/configcompression/compressiontype.go#L23, there is no compression named identity.

I tested '' (empty string), 'none', and null. content_encoding is None for all of them.

receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 exporters: otlphttp: endpoint: http://host.docker.internal:5000 # if running otel collector by docker, This can be connect to localhost:5000 compression: ... 👈 headers: x-mlflow-experiment-id: "0" # default id service: pipelines: traces: receivers: [otlp] exporters: [otlphttp]

…nt (mlflow#19024) Signed-off-by: Miaoxiang <zs1030823169@gmail.com> Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Co-authored-by: miaoxiang.wang <miaoxiang.wang.extern@porsche.digital> Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Co-authored-by: harupy <17039389+harupy@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…nt (#19024) Signed-off-by: Miaoxiang <zs1030823169@gmail.com> Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> Co-authored-by: miaoxiang.wang <miaoxiang.wang.extern@porsche.digital> Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Co-authored-by: harupy <17039389+harupy@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

github-actions bot added v3.6.1 area/tracing MLflow Tracing and its integrations rn/bug-fix Mention under Bug Fixes in Changelogs. labels Nov 25, 2025

harupy reviewed Nov 25, 2025

View reviewed changes

mlflow/server/otel_api.py Outdated Show resolved Hide resolved

harupy reviewed Nov 25, 2025

View reviewed changes

mlflow/server/otel_api.py Outdated Show resolved Hide resolved

Miaoxiang-philips force-pushed the patch-2 branch from f8f0c84 to d254eeb Compare November 26, 2025 03:17

Miaoxiang-philips and others added 2 commits November 26, 2025 11:20

feat: Add gzip/deflate Content-Encoding support to OTLP traces endpoint

55fed99

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Miaoxiang-philips force-pushed the patch-2 branch from d254eeb to 03395bf Compare November 26, 2025 03:20

Miaoxiang-philips requested a review from harupy November 26, 2025 03:21

fix: remove needless package

885445a

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

harupy reviewed Nov 26, 2025

View reviewed changes

tests/tracing/test_otel_logging.py Outdated Show resolved Hide resolved

harupy reviewed Nov 26, 2025

View reviewed changes

tests/tracing/utils/test_otlp.py Outdated Show resolved Hide resolved

Miaoxiang-philips and others added 2 commits November 26, 2025 12:10

Update tests/tracing/test_otel_logging.py

d33c699

Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

fix: remove useless docstrings

eb34b97

Removed docstring comments from test functions for clarity. Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Miaoxiang-philips requested a review from harupy November 26, 2025 04:25

Miaoxiang-philips and others added 3 commits November 26, 2025 12:51

fix: lint issue

5e301e2

Signed-off-by: Miaoxiang <zs1030823169@gmail.com>

Lazy import

073d204

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy approved these changes Nov 26, 2025

View reviewed changes

fix: update OTLP trace export handling and improve decompression logic

5502102

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy reviewed Nov 27, 2025

View reviewed changes

harupy added v3.7.0 and removed v3.6.1 labels Nov 27, 2025

Merge branch 'master' into patch-2

8f09079

harupy enabled auto-merge November 27, 2025 11:19

harupy added this pull request to the merge queue Nov 27, 2025

Merged via the queue into mlflow:master with commit 4fab98a Nov 27, 2025
52 checks passed

B-Step62 mentioned this pull request Dec 3, 2025

Add doc for otlp compression #19181

Merged

29 tasks

B-Step62 mentioned this pull request Dec 11, 2025

[FR] Add support for compression when logging trace data files #16946

Closed

24 tasks

Conversation

Miaoxiang-philips commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install mlflow from this PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

Uh oh!

Uh oh!

Miaoxiang-philips commented Nov 26, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harupy left a comment

Choose a reason for hiding this comment

Uh oh!

harupy commented Nov 26, 2025

Uh oh!

Miaoxiang-philips commented Nov 26, 2025

Uh oh!

harupy commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Miaoxiang-philips commented Nov 26, 2025

Uh oh!

harupy commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Miaoxiang-philips commented Nov 27, 2025

Uh oh!

harupy commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harupy commented Nov 27, 2025

Uh oh!

harupy Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

harupy Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Miaoxiang-philips commented Nov 25, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

harupy commented Nov 26, 2025 •

edited

Loading

harupy commented Nov 26, 2025 •

edited

Loading

harupy commented Nov 27, 2025 •

edited

Loading

harupy Nov 27, 2025 •

edited

Loading