Improve CPS cluster exclusion handling by richard-dennehy · Pull Request #143488 · elastic/elasticsearch

richard-dennehy · 2026-03-03T15:19:15Z

Allow expressions such as *:logs,-linked_project:*.

Assume that missing remote resolution results for a project alias are due to cluster exclusion, unless we have an exception

richard-dennehy · 2026-03-10T16:35:48Z

server/src/main/java/org/elasticsearch/action/admin/indices/resolve/ResolveIndexAction.java

                        terminalHandler.run();
                    }, failure -> {
                        logger.info("failed to resolve indices on remote cluster [" + clusterAlias + "]", failure);
+                        remoteExceptions.put(clusterAlias, failure);


Assuming we get at most one response per cluster alias

richard-dennehy · 2026-03-10T16:36:30Z

server/src/main/java/org/elasticsearch/action/fieldcaps/TransportFieldCapabilitiesAction.java

-        Map<String, ResolvedIndexExpressions.Builder> resolvedRemotelyBuilder = new ConcurrentHashMap<>();
-        for (String clusterAlias : remoteClusterIndices.keySet()) {
-            resolvedRemotelyBuilder.put(clusterAlias, ResolvedIndexExpressions.builder());
-        }


Pre-populating ResolvedIndexExpressions for each cluster like this conflicts with the error handling added in validate, as we expect missing entries for excluded clusters.

I haven't run into any reason why it needs to be pre-populated like this, beyond the single unit test that I changed

@piergm could you validate that this change is okay? Seems fine to me because the only consumers I'm aware of filter for SUCCESS anyhow but would be good to get a second opinion

Looks good to me (and nice simplification), plus we have enough tests to catch any corner cases in Field caps, ESQL, etc that I am confident that the change is ok if the CI is green!

richard-dennehy · 2026-03-10T16:37:58Z

server/src/main/java/org/elasticsearch/action/fieldcaps/TransportFieldCapabilitiesAction.java

-                Map<String, ResolvedIndexExpressions> resolvedRemotely = resolvedRemotelyBuilder.entrySet()
-                    .stream()
-                    .collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().build()));


there's a side effect of the expressions being cloned from the response to the map here - I assume we don't need that either

richard-dennehy · 2026-03-10T16:39:27Z

server/src/main/java/org/elasticsearch/action/search/TransportOpenPointInTimeAction.java

                        new ActionListenerResponseHandler<>(groupedListener.delegateResponse((l, failure) -> {
                            logger.info("Error occurred on remote cluster [" + clusterAlias + "]", failure);
-                            l.onResponse(Map.entry(clusterAlias, new SearchPlanningPhaseResolutionResult(null, failure)));
+                            delegate.onResponse(Map.entry(clusterAlias, new SearchPlanningPhaseResolutionResult(null, failure)));


I'm not sure if we need this change, I can't see that it makes a difference; I'm just semi-blindly copying it from the search changes

Right, this change is fine and still leads to correctly wrapping failure with a Map.entry instead of hard-failing. I want to simplify the listener handling here in a follow up there are more delegate.... calls in this flow than necessary.

richard-dennehy · 2026-03-10T16:42:40Z

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

        public final void onFailure(Exception e) {
            ShardSearchFailure f = new ShardSearchFailure(e);
            logCCSError(f, clusterAlias, skipOnFailure);
+            remoteExceptions.put(clusterAlias, e);


we need something like this to capture connection errors in MRT=false, but having to provide an empty HashMap in the MRT=true path is something of a code smell

can't come up with anything clean other than CCSActionListener sub-classes and something like an onInnerFailure method -- feels over-engineered though so lets leave as is.

richard-dennehy · 2026-03-10T17:07:01Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/index/IndexResolver.java

                    resolvedIndexExpressions,
-                    resolvedRemotely
+                    resolvedRemotely,
+                    Map.of()


Trying to add tests to cover this lead to the issues I raised with EQL and SQL; I've removed the EQL and SQL tests for now.

As I understand, this is here because the field caps call has lenient index options; I'd much prefer we passed the original index options (but with CPS rewriting disabled) so field caps fails when appropriate, as trying to gather remote exceptions here is potentially awkward.

Yeah lets defer to the analytics team to fix those

elasticsearchmachine · 2026-03-11T11:14:20Z

Pinging @elastic/es-security (Team:Security)

n1v0lg

LGTM -- nice work on this!

n1v0lg · 2026-03-11T13:10:18Z

server/src/main/java/org/elasticsearch/action/search/TransportOpenPointInTimeAction.java

                        new ActionListenerResponseHandler<>(groupedListener.delegateResponse((l, failure) -> {
                            logger.info("Error occurred on remote cluster [" + clusterAlias + "]", failure);
-                            l.onResponse(Map.entry(clusterAlias, new SearchPlanningPhaseResolutionResult(null, failure)));
+                            delegate.onResponse(Map.entry(clusterAlias, new SearchPlanningPhaseResolutionResult(null, failure)));


Right, this change is fine and still leads to correctly wrapping failure with a Map.entry instead of hard-failing. I want to simplify the listener handling here in a follow up there are more delegate.... calls in this flow than necessary.

n1v0lg · 2026-03-11T13:35:15Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/index/IndexResolver.java

                    resolvedIndexExpressions,
-                    resolvedRemotely
+                    resolvedRemotely,
+                    Map.of()


Yeah lets defer to the analytics team to fix those

n1v0lg · 2026-03-11T13:46:56Z

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

        public final void onFailure(Exception e) {
            ShardSearchFailure f = new ShardSearchFailure(e);
            logCCSError(f, clusterAlias, skipOnFailure);
+            remoteExceptions.put(clusterAlias, e);


can't come up with anything clean other than CCSActionListener sub-classes and something like an onInnerFailure method -- feels over-engineered though so lets leave as is.

# Conflicts: # server/src/main/java/org/elasticsearch/action/search/TransportOpenPointInTimeAction.java

richard-dennehy · 2026-03-12T16:40:36Z

CI is passing, but one of the jobs hasn't reported that to GH - I promise I'm merging on green CI 🙏

…elocations * upstream/main: (49 commits) CCS logging fixes (elastic#144070) Improve CPS cluster exclusion handling (elastic#143488) Remove snapshot condition now that node_reduce phase is in non-snapshot builds (elastic#144090) Drop deprecation warnings when updating a mapping in the cluster state applier (elastic#143884) (elastic#144040) Add ensureGreenAndNoInitializingShards helper (elastic#144044) Removed unnecessary applies_to blocks from deprecated query (elastic#144096) [CPS] Use single CrossProjectModeDecider instance (elastic#144030) Fix ESQL TS requests with LIMIT 0 (elastic#144031) ESQL: Remove `create` methods in aggs (elastic#144098) ES|QL: Refactor ChangeLimitOperator (elastic#144017) Add Paginated Hit Source Tests (elastic#142592) Fix test failure not preferred (elastic#144019) Remove serialization logic from EIS authorization response (elastic#144021) ESQL: CSV schema inference and parsing enhancements (elastic#144050) ESQL: Fix incorrectly optimized fork with nullify unmapped_fields (elastic#143030) Fix MMR release test using subqueries (elastic#144087) Refactoring `UserAgentPlugin` (elastic#140712) Drop non-finite samples in Prometheus remote write (elastic#144055) [TEST] Wait for internal inference indices to be created in authorization IT (elastic#143885) Disable ndjson datasource QA tests in release-tests (elastic#143992) ...

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

richard-dennehy added >non-issue :Security/Security Security issues without another label labels Mar 3, 2026

elasticsearchmachine added v9.4.0 serverless-linked Added by automation, don't add manually labels Mar 3, 2026

richard-dennehy force-pushed the remote-project-exclusion-4 branch 3 times, most recently from adb38fb to 29c88b2 Compare March 10, 2026 14:52

richard-dennehy changed the title ~~WIP handle CPS cluster exclusions~~ Improve CPS cluster exclusion handling Mar 10, 2026

richard-dennehy commented Mar 11, 2026

View reviewed changes

richard-dennehy marked this pull request as ready for review March 11, 2026 11:13

elasticsearchmachine added the Team:Security Meta label for security team label Mar 11, 2026

n1v0lg approved these changes Mar 11, 2026

View reviewed changes

richard-dennehy and others added 12 commits March 12, 2026 10:34

WIP pass remote project exceptions to validate

641606c

update javadoc

f0b927d

fix compile

e3cf122

fix remote connection failure in search

3ff57e5

# Conflicts: # server/src/main/java/org/elasticsearch/action/search/TransportOpenPointInTimeAction.java

handle remote connection errors in PIT

acd876a

[CI] Auto commit changes from spotless

f0c3e44

copy PIT connection fix

3a9537f

pass remote connection exceptions to validate when MRT=false

252d55f

update test assertions

01dab3c

alternative fix for remote connection errors in MRT=false

7440181

WIP update EQL index resolution validation

bf83a3f

remove EQL/SQL hacks

6ec48b0

richard-dennehy force-pushed the remote-project-exclusion-4 branch from ce165b0 to 6ec48b0 Compare March 12, 2026 12:15

richard-dennehy added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Mar 12, 2026

richard-dennehy merged commit 498aa28 into elastic:main Mar 12, 2026
41 of 42 checks passed

richard-dennehy mentioned this pull request Mar 13, 2026

restore dropped allowPartialSearchResults configuration #144186

Merged

michalborek pushed a commit to michalborek/elasticsearch that referenced this pull request Mar 23, 2026

Improve CPS cluster exclusion handling (elastic#143488)

96fdaa9

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

Conversation

richard-dennehy commented Mar 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n1v0lg Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Mar 11, 2026

Uh oh!

n1v0lg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richard-dennehy commented Mar 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

n1v0lg Mar 11, 2026 •

edited

Loading