Plumb DLM error store through to DlmFrozenTransition classes by dakrone · Pull Request #145243 · elastic/elasticsearch

dakrone · 2026-03-30T22:12:38Z

This moves the DLM error store to be part of the PluginServices, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the DlmFrozenTransitionExecutor to handle uncaught exceptions currently.

Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself.

This moves the DLM error store to be part of the `PluginServices`, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the `DlmFrozenTransitionExecutor` to handle uncaught exceptions currently. Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself.

elasticsearchmachine · 2026-03-31T01:39:48Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

lukewhiting · 2026-03-31T09:19:37Z

@coderabbitai review

coderabbitai · 2026-03-31T09:19:47Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-03-31T09:26:09Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: f8a72c61-2199-4af4-af1b-d538dd1bb6fc

📥 Commits

Reviewing files that changed from the base of the PR and between d0db5d3 and 15d847d.

📒 Files selected for processing (23)

modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamsPlugin.java
modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java
modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/ErrorRecordingActionListener.java
modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/action/TransportExplainDataStreamLifecycleAction.java
modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/health/DataStreamLifecycleHealthInfoPublisher.java
modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleErrorStoreTests.java
modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleServiceTests.java
modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/action/TransportGetDataStreamLifecycleStatsActionTests.java
modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/health/DataStreamLifecycleHealthInfoPublisherTests.java
server/src/main/java/module-info.java
server/src/main/java/org/elasticsearch/dlm/DataStreamLifecycleErrorStore.java
server/src/main/java/org/elasticsearch/node/NodeConstruction.java
server/src/main/java/org/elasticsearch/node/PluginServiceInstances.java
server/src/main/java/org/elasticsearch/plugins/Plugin.java
x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DataStreamLifecycleConvertToFrozen.java
x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionExecutor.java
x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionPlugin.java
x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionRunnable.java
x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionService.java
x-pack/plugin/dlm-frozen-transition/src/test/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionExecutorTests.java
x-pack/plugin/dlm-frozen-transition/src/test/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionServiceTests.java
x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/integration/DataStreamLifecycleDownsamplingSecurityIT.java
x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/integration/DataStreamLifecycleServiceRuntimeSecurityIT.java

📝 Walkthrough

Walkthrough

The DataStreamLifecycleErrorStore class was relocated from the org.elasticsearch.datastreams.lifecycle package to org.elasticsearch.dlm package in the server module. Initialization was centralized in NodeConstruction and exposed through PluginServiceInstances and a new dlmErrorStore() method in the PluginServices interface. Components that previously created the error store locally (DataStreamsPlugin) now receive it via dependency injection. Import statements were updated across the data-streams, dlm-frozen-transition, and security test modules. The DlmFrozenTransitionRunnable interface was extended with a getProjectId() method, and related services now accept and utilize the error store dependency. The org.elasticsearch.dlm package is now exported by the server module.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

lukewhiting

A couple of very minor comments but nothing to block over :-) Thanks for taking this on!

lukewhiting · 2026-03-31T11:46:08Z

server/src/main/java/org/elasticsearch/dlm/DataStreamLifecycleErrorStore.java

 import org.apache.logging.log4j.Logger;
 import org.apache.logging.log4j.message.Message;


Should these use the ES wrapper's instead?

It would be nice if they did, but apparently the error store uses bits of the logger that our log manager does not support, so for now they have to remain using the log4j one.

lukewhiting · 2026-03-31T11:46:14Z

...transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionExecutor.java

+                    indexName,
+                    ex,
+                    Strings.format("Error executing transition for index [%s]", indexName),
+                    1


Do we want all errors logged at WARN or can we go with the same value as DATA_STREAM_SIGNALLING_ERROR_RETRY_INTERVAL_SETTING and stick to every 10th?

I was matching the behavior for now, I think we need to know whether we want this to be the one-stop-shop for error handling (in which case, we'll need to move that setting into core and add code to make it dynamic here) or whether we want to do more error handling in the converter. Let's discuss it.

…rics * upstream/main: (21 commits) Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {csv-spec:external-basic.topSnippetsFunction} elastic#145353 Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {csv-spec:external-basic.scoreFunction} elastic#145352 [DiskBBQ] Fix bug in NeighborQueue#popRawAndAddRaw (elastic#145324) Fix dense_vector default index options when using BFLOAT16 (elastic#145202) Use checked exceptions in entitlement constructor rules (elastic#145234) ESQL: DS: datasource file plugins should not return TEXT types (elastic#145334) Plumb DLM error store through to DlmFrozenTransition classes (elastic#145243) Make Settings.Builder.remove() fluent (elastic#145294) Add FLS tests for METRICS_INFO and TS_INFO (elastic#145211) Fix flaky SecurityFeatureResetTests (elastic#145063) [DOCS] Fix conflict markers in ESQL processing command list (elastic#145338) Skip certain metric assertions on Windows (elastic#144933) [ES|QL] Add schema reconciliation for multi-file external sources (elastic#145220) Simplify DiskBBQ dynamic visit ratio to linear (elastic#142784) ESQL: Disallow unmapped_fields=load with partial non-KEYWORD (elastic#144109) [Transform] Track Linked Projects (elastic#144399) Fix bulk scoring to process last batch instead of falling through to scalar tail (elastic#145316) Clean up TickerScheduleEngineTests (elastic#145303) [CI] ShardBulkInferenceActionFilterIT testRestart - Ensuring that secrets-inference index is available after full restart and unmuting test (elastic#145317) Add CRUD doc to the DistributedArchitectureGuide (elastic#144710) ...

…#145243) * Plumb DLM error store through to DlmFrozenTransition classes This moves the DLM error store to be part of the `PluginServices`, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the `DlmFrozenTransitionExecutor` to handle uncaught exceptions currently. Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself. # Conflicts: # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMConvertToFrozen.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionExecutor.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionPlugin.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionService.java # x-pack/plugin/dlm-frozen-transition/src/test/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionExecutorTests.java

…#145243) * Plumb DLM error store through to DlmFrozenTransition classes This moves the DLM error store to be part of the `PluginServices`, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the `DlmFrozenTransitionExecutor` to handle uncaught exceptions currently. Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself. # Conflicts: # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMConvertToFrozen.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionExecutor.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionPlugin.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionService.java # x-pack/plugin/dlm-frozen-transition/src/test/java/org/elasticsearch/xpack/dlm/frozen/DLMFrozenTransitionExecutorTests.java # Conflicts: # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DataStreamLifecycleConvertToFrozen.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionExecutor.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionPlugin.java # x-pack/plugin/dlm-frozen-transition/src/main/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionService.java # x-pack/plugin/dlm-frozen-transition/src/test/java/org/elasticsearch/xpack/dlm/frozen/DlmFrozenTransitionExecutorTests.java

This moves the `DATA_STREAM_SIGNALLING_ERROR_RETRY_INTERVAL_SETTING` setting into the core module so that it can be used by both the regular DLM service and the frozen conversion bits to set up the retry interval in the DLM error store. Relates to elastic#145243

…#145243) * Plumb DLM error store through to DlmFrozenTransition classes This moves the DLM error store to be part of the `PluginServices`, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the `DlmFrozenTransitionExecutor` to handle uncaught exceptions currently. Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself.

* Move DLM signalling error retry setting into core This moves the `DATA_STREAM_SIGNALLING_ERROR_RETRY_INTERVAL_SETTING` setting into the core module so that it can be used by both the regular DLM service and the frozen conversion bits to set up the retry interval in the DLM error store. Relates to #145243

…#145243) * Plumb DLM error store through to DlmFrozenTransition classes This moves the DLM error store to be part of the `PluginServices`, which allows us to share the same instance between the core and x-pack versions of the DLM code. It is used within the `DlmFrozenTransitionExecutor` to handle uncaught exceptions currently. Subsequent work will add this to the actual conversion task, so that errors can be recorded from the task itself.

* Move DLM signalling error retry setting into core This moves the `DATA_STREAM_SIGNALLING_ERROR_RETRY_INTERVAL_SETTING` setting into the core module so that it can be used by both the regular DLM service and the frozen conversion bits to set up the retry interval in the DLM error store. Relates to elastic#145243

dakrone requested review from lukewhiting and seanzatzdev March 30, 2026 22:12

dakrone added >non-issue :StorageEngine/Data streams Data streams and their lifecycles v9.4.0 labels Mar 30, 2026

dakrone removed request for lukewhiting and seanzatzdev March 30, 2026 22:12

elasticsearchmachine and others added 2 commits March 30, 2026 22:19

[CI] Auto commit changes from spotless

637a99d

Merge branch 'main' into dlm-plumb-error-store

15d847d

dakrone requested review from lukewhiting and seanzatzdev March 31, 2026 01:39

dakrone marked this pull request as ready for review March 31, 2026 01:39

dakrone requested a review from a team as a code owner March 31, 2026 01:39

elasticsearchmachine added the Team:StorageEngine label Mar 31, 2026

lukewhiting approved these changes Mar 31, 2026

View reviewed changes

dakrone enabled auto-merge (squash) March 31, 2026 15:19

dakrone merged commit 03a3120 into elastic:main Mar 31, 2026
35 checks passed

dakrone mentioned this pull request Mar 31, 2026

Move DLM signalling error retry setting into core #145371

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plumb DLM error store through to DlmFrozenTransition classes#145243

Plumb DLM error store through to DlmFrozenTransition classes#145243
dakrone merged 3 commits intoelastic:mainfrom
dakrone:dlm-plumb-error-store

dakrone commented Mar 30, 2026

Uh oh!

elasticsearchmachine commented Mar 31, 2026

Uh oh!

lukewhiting commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026

Walkthrough

Uh oh!

lukewhiting left a comment

Uh oh!

lukewhiting Mar 31, 2026

Uh oh!

dakrone Mar 31, 2026

Uh oh!

lukewhiting Mar 31, 2026

Uh oh!

dakrone Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		import org.apache.logging.log4j.Logger;
		import org.apache.logging.log4j.message.Message;

Conversation

dakrone commented Mar 30, 2026

Uh oh!

elasticsearchmachine commented Mar 31, 2026

Uh oh!

lukewhiting commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026

Walkthrough

Uh oh!

lukewhiting left a comment

Choose a reason for hiding this comment

Uh oh!

lukewhiting Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

dakrone Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

lukewhiting Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

dakrone Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants