Skip to content

Commit 6dfef31

Browse files
gsoldevilakibanamachine
gsoldevila
and
kibanamachine
authored
[CPS] Fix ESQL index validation for linked projects (#259141)
## Summary Fixes a bug in ESQL validation in Discover where indices that exist on a CPS-linked project are incorrectly flagged as unknown. Closes elastic/kibana-team#3107 ## Root cause The ESQL sources route (`/internal/esql/autocomplete/sources/{scope}`) always resolved indices against the local project, ignoring the project picker selection. Validation therefore failed for any index that only exists on a linked project. ## Fix ``` Browser Server ────────────────────────────── ────────────────────────────────────────── ESQL editor /internal/esql/autocomplete/sources/{scope} ├─ useObservable(getProjectRouting$()) └─ ?projectRouting=<value> │ re-renders + re-validates on change └─ EsqlService.getAllIndices(scope, projectRouting) ├─ SET project_routing (from query) └─ resolveIndex({ project_routing }) │ takes precedence over picker └─ CPS transport: inject if enabled, └─ effectiveRouting = SET ?? picker strip if not sent as ?projectRouting query param ``` The editor computes an `effectiveProjectRouting` value (`SET project_routing` statement takes precedence over the project picker selection) and sends it as an explicit `?projectRouting` query param to the sources route. The route forwards it directly to `resolveIndex()`. The existing CPS transport layer handles stripping it in non-CPS environments, so stateful and non-CPS Serverless deployments are unaffected. ## Changes ### ESQL sources route — explicit `projectRouting` query param ([`get_all_sources.ts`](src/platform/plugins/shared/esql/server/routes/get_all_sources.ts)) Adds an optional `?projectRouting` query param and passes it to `EsqlService.getAllIndices()`. ### `EsqlService.getAllIndices()` — direct injection ([`esql_service.ts`](src/platform/plugins/shared/esql/server/services/esql_service.ts)) Accepts an optional `projectRouting` and spreads `{ project_routing }` directly into both `resolveIndex` calls. The CPS transport layer handles the rest: - **CPS-enabled**: an explicit value takes precedence over the handler's default injection. - **CPS-disabled**: the transport strips `project_routing` unconditionally. ### ESQL editor — reactive picker + explicit param ([`esql_editor.tsx`](src/platform/packages/private/kbn-esql-editor/src/esql_editor.tsx), [`sources.ts`](src/platform/packages/shared/kbn-esql-utils/src/utils/callbacks/sources.ts)) - `useObservable(cps?.cpsManager?.getProjectRouting$() ?? EMPTY)` reads the picker value reactively; each change causes a re-render that busts the `memoizedSources` cache (triggering a fresh source fetch with the new routing). - `SET project_routing` extracted from the query overrides the picker: `effectiveProjectRouting = setProjectRouting ?? pickerProjectRouting`. - `effectiveProjectRouting` is sent as `?projectRouting` to the sources route. - A ref-guarded `useEffect([pickerProjectRouting])` calls `queryValidation` when the picker changes (skipping the initial mount to avoid a double-validation). The ESQL plugin's `ESQLLangEditor` wrapper passes `cps` (an optional plugin dep) through `KibanaContextProvider` so consumers need no changes. ## Also included: Core cleanup The PR also reverts some earlier exploratory additions that are no longer needed: - `'http-header'` option removed from `AsScopedOptions` (only `'space'` remains) - `getClient(opts)` removed from `ElasticsearchRequestHandlerContext` ## Testing - [ ] With CPS enabled, select a linked project in the project picker; write `FROM <index-on-that-project>` in Discover's ESQL editor — validation should pass (no "Unknown index" error). - [ ] Changing the project picker selection triggers a fresh validation. - [ ] Writing `SET project_routing=<other-project>; FROM <index-on-other-project>` uses the SET value, not the picker's value, for index resolution. - [ ] Without CPS (stateful / non-CPS Serverless), behaviour is unchanged. Made with [Cursor](https://cursor.com) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
1 parent 3402744 commit 6dfef31

15 files changed

Lines changed: 126 additions & 42 deletions

File tree

‎src/core/packages/elasticsearch/client-server-internal/src/cluster_client.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,17 +51,17 @@ interface CommonFactoryRoutingOpts {
5151
request?: ScopeableUrlRequest;
5252
}
5353

54-
interface SpaceFactoryRoutingOpts extends CommonFactoryRoutingOpts {
54+
interface ScopedFactoryRoutingOpts extends CommonFactoryRoutingOpts {
5555
projectRouting: 'space';
5656
request: ScopeableUrlRequest;
5757
}
5858

5959
/**
60-
* Discriminated union of routing options passed to {@link OnRequestHandlerFactory}.
61-
* Each variant carries exactly the data needed for that routing mode.
60+
* Union of routing options passed to {@link OnRequestHandlerFactory}.
61+
* The scoped variant carries the request so the factory can extract the space NPRE.
6262
* @internal
6363
*/
64-
export type FactoryRoutingOpts = CommonFactoryRoutingOpts | SpaceFactoryRoutingOpts;
64+
export type FactoryRoutingOpts = CommonFactoryRoutingOpts | ScopedFactoryRoutingOpts;
6565
/**
6666
* A factory that produces an {@link OnRequestHandler}, which can be bound to a request context.
6767
* @internal

‎src/core/packages/elasticsearch/server/src/client/cluster_client.ts‎

Lines changed: 24 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -14,31 +14,27 @@ import type { IScopedClusterClient } from './scoped_cluster_client';
1414
/**
1515
* Options for the `asScoped` method.
1616
*
17+
* **Background**: Cross-Project Search (CPS) is a Serverless feature that allows Kibana to
18+
* transparently orchestrate searches across multiple Elastic projects. Kibana itself does not
19+
* execute the cross-project logic - it forwards requests with the appropriate `project_routing`
20+
* parameter and Elasticsearch handles execution, security enforcement, and result aggregation.
21+
*
22+
* **Important**: These options only take effect in CPS-enabled Serverless environments. In all
23+
* other environments (stateful, non-CPS Serverless), any `project_routing` params are
24+
* stripped from requests to avoid Elasticsearch rejections and to preserve traditional
25+
* single-cluster routing behavior.
26+
*
1727
* @public
1828
*/
1929
export interface AsScopedOptions {
2030
/**
2131
* Controls how `project_routing` is automatically injected into Elasticsearch requests made
2232
* through the scoped client.
2333
*
24-
* **Background**: Cross-Project Search (CPS) is a Serverless feature that allows Kibana to
25-
* transparently orchestrate searches across multiple Elastic projects. Kibana itself does not
26-
* execute the cross-project logic - it forwards requests with the appropriate `project_routing`
27-
* header and Elasticsearch handles execution, security enforcement, and result aggregation.
28-
*
29-
* **Options**:
30-
* - `'space'`: Requests are routed to the Named Project Routing Expression (NPRE) configured
31-
* for the current Kibana space. Requires a {@link ScopeableUrlRequest} to be passed to
32-
* `asScoped` so that the space can be extracted from the URL pathname. Use this when the scope
33-
* of the query should match the data boundaries of the active space (e.g. alerting rules).
34-
*
35-
* When no options are passed to `asScoped`, requests are always routed to the origin project
36-
* (i.e. the Elasticsearch instance Kibana is directly connected to).
37-
*
38-
* **Important**: This option only takes effect in CPS-enabled Serverless environments. In all
39-
* other environments (stateful, non-CPS Serverless), any `project_routing` params are
40-
* stripped from requests to avoid Elasticsearch rejections and to preserve traditional
41-
* single-cluster routing behavior.
34+
* - `'space'`: Routes requests to the Named Project Routing Expression (NPRE) configured for
35+
* the current Kibana space. Requires a {@link ScopeableUrlRequest} to be passed to `asScoped`
36+
* so that the space can be extracted from the URL pathname. Use this when the scope of the
37+
* query should match the data boundaries of the active space (e.g. alerting rules).
4238
*/
4339
projectRouting: 'space';
4440
}
@@ -62,17 +58,21 @@ export interface IClusterClient {
6258
*/
6359
readonly asInternalUser: ElasticsearchClient;
6460

61+
/**
62+
* Creates a {@link IScopedClusterClient | scoped cluster client} bound to the given request,
63+
* forwarding the request's authentication headers to Elasticsearch, with CPS space routing.
64+
*
65+
* Requires a {@link ScopeableUrlRequest} so the space id can be extracted from the URL pathname.
66+
*
67+
* @param request - The incoming Kibana request.
68+
* @param opts - {@link AsScopedOptions} that configure CPS routing behavior.
69+
*/
6570
asScoped(request: ScopeableUrlRequest, opts: AsScopedOptions): IScopedClusterClient;
6671
/**
6772
* Creates a {@link IScopedClusterClient | scoped cluster client} bound to the given request,
68-
* forwarding the request's authentication headers to Elasticsearch.
73+
* forwarding the request's authentication headers to Elasticsearch with origin-only routing.
6974
*
7075
* @param request - The incoming request whose credentials authenticate Elasticsearch calls.
71-
* - {@link ScopeableRequest}: supports origin-only routing.
72-
* - {@link ScopeableUrlRequest}: additionally supports `'space'` routing (space id extracted from URL).
73-
* @param opts - Optional {@link AsScopedOptions} to configure CPS routing behavior.
74-
* - 'space': Routes the request to the NPRE configured for the current Kibana space.
75-
* The client will route the request to the origin project if no options are provided.
7676
*/
7777
asScoped(request: ScopeableRequest): IScopedClusterClient;
7878
}

‎src/core/packages/elasticsearch/server/src/request_handler_context.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,8 @@ import type { IScopedClusterClient } from './client';
1414
* @public
1515
*/
1616
export interface ElasticsearchRequestHandlerContext {
17+
/**
18+
* A pre-scoped {@link IScopedClusterClient} for the current request using origin-only routing.
19+
*/
1720
client: IScopedClusterClient;
1821
}

‎src/platform/packages/private/kbn-esql-editor/moon.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ dependsOn:
4848
- '@kbn/aiops-utils'
4949
- '@kbn/esql-resource-browser'
5050
- '@kbn/data-views-plugin'
51+
- '@kbn/cps'
5152
tags:
5253
- shared-browser
5354
- package

‎src/platform/packages/private/kbn-esql-editor/src/esql_editor.tsx‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import {
2020
import { i18n } from '@kbn/i18n';
2121
import moment from 'moment';
2222
import { isEqual, memoize } from 'lodash';
23+
import { EMPTY } from 'rxjs';
2324
import { Global, css } from '@emotion/react';
2425
import {
2526
getIndexPatternFromESQLQuery,
@@ -28,6 +29,7 @@ import {
2829
getJoinIndices,
2930
fixESQLQueryWithVariables,
3031
prettifyQuery,
32+
getProjectRoutingFromEsqlQuery,
3133
} from '@kbn/esql-utils';
3234
import type { CodeEditorProps } from '@kbn/code-editor';
3335
import { CodeEditor } from '@kbn/code-editor';
@@ -176,6 +178,7 @@ const ESQLEditorInternal = function ESQLEditor({
176178
const {
177179
application,
178180
core,
181+
cps,
179182
fieldsMetadata,
180183
uiSettings,
181184
uiActions,
@@ -195,6 +198,7 @@ const ESQLEditorInternal = function ESQLEditor({
195198
[core.http, core.userProfile, usageCollection]
196199
);
197200

201+
const pickerProjectRouting = useObservable(cps?.cpsManager?.getProjectRouting$() ?? EMPTY);
198202
const activeSolutionNavId = useObservable(core.chrome.getActiveSolutionNavId$());
199203
const activeSolutionId: ESQLRegistrySolutionId =
200204
(activeSolutionNavId as ESQLRegistrySolutionId) ?? ESQL_CLASSIC_SOLUTION_ID;
@@ -603,16 +607,22 @@ const ESQLEditorInternal = function ESQLEditor({
603607
return { cache: fn.cache, memoizedFieldsFromESQL: fn };
604608
}, []);
605609

610+
// `SET project_routing` in the query takes precedence over the project picker selection.
611+
const setProjectRouting = useMemo(() => getProjectRoutingFromEsqlQuery(code), [code]);
612+
const effectiveProjectRouting = setProjectRouting ?? pickerProjectRouting;
613+
606614
const { cache: dataSourcesCache, memoizedSources } = useMemo(() => {
615+
// Keying on effectiveProjectRouting ensures a fresh cache (and therefore a fresh fetch)
616+
// whenever either the SET statement or the picker selection changes.
607617
const fn = memoize(
608618
(...args: [CoreStart, (() => Promise<ILicense | undefined>) | undefined]) => ({
609619
timestamp: Date.now(),
610-
result: getESQLSources(...args),
620+
result: getESQLSources(...args, undefined, effectiveProjectRouting),
611621
})
612622
);
613623

614624
return { cache: fn.cache, memoizedSources: fn };
615-
}, []);
625+
}, [effectiveProjectRouting]);
616626

617627
const { cache: historyStarredItemsCache, memoizedHistoryStarredItems } = useMemo(() => {
618628
const fn = memoize(
@@ -903,6 +913,28 @@ const ESQLEditorInternal = function ESQLEditor({
903913
[dataSourcesCache, getJoinIndicesCallback, onQueryUpdate, queryValidation]
904914
);
905915

916+
// Re-validate when the project picker selection changes. useObservable causes a re-render
917+
// (and therefore a memoizedSources cache miss) automatically; this effect handles the
918+
// explicit re-validation trigger.
919+
//
920+
// queryValidationRef keeps the latest queryValidation without being listed as an effect
921+
// dependency: including queryValidation directly would cause the effect to fire on every
922+
// code edit (queryValidation's identity changes whenever `code` changes), doubling the
923+
// validation work and causing performance test timeouts.
924+
const queryValidationRef = useRef(queryValidation);
925+
useEffect(() => {
926+
queryValidationRef.current = queryValidation;
927+
}, [queryValidation]);
928+
929+
const isFirstPickerRenderRef = useRef(true);
930+
useEffect(() => {
931+
if (isFirstPickerRenderRef.current) {
932+
isFirstPickerRenderRef.current = false;
933+
return;
934+
}
935+
queryValidationRef.current({ active: true });
936+
}, [pickerProjectRouting]);
937+
906938
// Refresh the fields cache when a new field has been added to the lookup index
907939
const onNewFieldsAddedToLookupIndex = useCallback(async () => {
908940
esqlFieldsCache.clear?.();

‎src/platform/packages/private/kbn-esql-editor/src/types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import type { Storage } from '@kbn/kibana-utils-plugin/public';
1616
import type { KqlPluginStart } from '@kbn/kql/public';
1717
import type { UiActionsStart } from '@kbn/ui-actions-plugin/public';
1818
import type { DataPublicPluginStart } from '@kbn/data-plugin/public';
19+
import type { CPSPluginStart } from '@kbn/cps/public';
1920
import type {
2021
ESQLControlVariable,
2122
ESQLQueryStats,
@@ -113,6 +114,7 @@ export interface ESQLEditorDeps {
113114
kql: KqlPluginStart;
114115
fieldsMetadata?: FieldsMetadataPublicStart;
115116
usageCollection?: UsageCollectionStart;
117+
cps?: CPSPluginStart;
116118
esql?: EsqlPluginStartBase;
117119
}
118120

‎src/platform/packages/private/kbn-esql-editor/tsconfig.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,9 @@
4343
"@kbn/search-types",
4444
"@kbn/aiops-utils",
4545
"@kbn/esql-resource-browser",
46-
"@kbn/data-views-plugin" ],
46+
"@kbn/data-views-plugin",
47+
"@kbn/cps"
48+
],
4749
"exclude": [
4850
"target/**/*",
4951
]

‎src/platform/packages/shared/kbn-esql-utils/src/utils/callbacks/sources.ts‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,16 +29,23 @@ interface IntegrationsResponse {
2929
* @param core The core start contract to make HTTP requests.
3030
* @param areRemoteIndicesAvailable A boolean indicating if remote indices should be included.
3131
* @param signal Optional AbortSignal to cancel the request.
32+
* @param projectRouting Optional CPS project routing value forwarded to the server so that index
33+
* resolution reflects the project picker selection or an explicit `SET project_routing`
34+
* pre-statement. `SET project_routing` takes precedence over the picker value.
3235
* @returns A promise that resolves to an array of ESQLSourceResult objects.
3336
*/
3437
export const getIndicesList = async (
3538
core: Pick<CoreStart, 'http'>,
3639
areRemoteIndicesAvailable: boolean,
37-
signal?: AbortSignal
40+
signal?: AbortSignal,
41+
projectRouting?: string
3842
): Promise<ESQLSourceResult[]> => {
3943
const scope = areRemoteIndicesAvailable ? 'all' : 'local';
4044
const response = await core.http
41-
.get(`${SOURCES_AUTOCOMPLETE_ROUTE}${scope}`, { signal })
45+
.get(`${SOURCES_AUTOCOMPLETE_ROUTE}${scope}`, {
46+
signal,
47+
query: projectRouting ? { projectRouting } : undefined,
48+
})
4249
.catch((error) => {
4350
if (signal?.aborted) return [];
4451
// eslint-disable-next-line no-console
@@ -96,18 +103,22 @@ const getIntegrations = async (
96103
* @param core The core start contract to make HTTP requests and access application capabilities.
97104
* @param getLicense An optional function to retrieve the current license information.
98105
* @param signal Optional AbortSignal to cancel the request.
106+
* @param projectRouting Optional CPS project routing value forwarded to the server so that index
107+
* resolution reflects the project picker selection or an explicit `SET project_routing`
108+
* pre-statement. `SET project_routing` takes precedence over the picker value.
99109
* @returns A promise that resolves to an array of ESQLSourceResult objects.
100110
*/
101111
export const getESQLSources = async (
102112
core: Pick<CoreStart, 'application' | 'http'>,
103113
getLicense: (() => Promise<ILicense | undefined>) | undefined,
104-
signal?: AbortSignal
114+
signal?: AbortSignal,
115+
projectRouting?: string
105116
): Promise<ESQLSourceResult[]> => {
106117
const ls = await getLicense?.();
107118
const ccrFeature = ls?.getFeature('ccr');
108119
const areRemoteIndicesAvailable = ccrFeature?.isAvailable ?? false;
109120
const [allIndices, integrations] = await Promise.all([
110-
getIndicesList(core, areRemoteIndicesAvailable, signal),
121+
getIndicesList(core, areRemoteIndicesAvailable, signal, projectRouting),
111122
getIntegrations(core, signal),
112123
]);
113124
return [...allIndices, ...integrations];

‎src/platform/plugins/shared/esql/kibana.jsonc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
"server": true,
1010
"browser": true,
1111
"optionalPlugins": [
12+
"cps",
1213
"fieldsMetadata",
1314
"usageCollection",
1415
"licensing"

‎src/platform/plugins/shared/esql/moon.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ dependsOn:
4848
- '@kbn/discover-utils'
4949
- '@kbn/controls-schemas'
5050
- '@kbn/kql'
51+
- '@kbn/cps'
5152
- '@kbn/inspector-plugin'
5253
- '@kbn/inference-plugin'
5354
- '@kbn/actions-plugin'

0 commit comments

Comments
 (0)