ESQL: Views REST API by craigtaverner · Pull Request #137818 · elastic/elasticsearch

craigtaverner · 2025-11-10T11:22:06Z

This is extracted from the Views Prototype at #134995 and is originally based on the "ESQL: View" prototype by Nik but with the following changes:

No ES|QL language integration, only REST API for adding/removing and managing Views in cluster state
ViewMetadata now inside ProjectMetadata
NodeFeature to ensure metadata only exists in clusters fully upgraded
Wide CRUD support (add, update, delete, list)
View name validation
View count, length and depth restrictions

The REST API looks like:

PUT _query/view/critical_service_errors_v {"query":"FROM ..."}
DELETE _query/view/critical_service_errors_v
GET _query/view/critical_service_errors_v
GET _query/view

Checklist:

Extract REST API from views prototype
Remove list interface and generalise GET command to cover list use cases (more in-line with other APIs)
Verify that snapshot-only really blocks registration of REST API
Consider switching to use feature flags (instead of snapshot-only)
Consider dropping one of NodeFeature or TransportVersion since they cover the same requirement
Implement batched tasks, probably using SequentialAckingBatchedTaskExecutor
Stop using ProjectResolver and instead use the ProjectID as is done in Enrich REST API
Change ViewMetadata from a Map of name to View to instead be an array of View (which should contain the name)

elasticsearchmachine · 2025-11-10T11:22:32Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-11-10T11:22:32Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2025-11-10T11:22:55Z

Hi @craigtaverner, I've created a changelog YAML for you.

dakrone · 2025-11-10T15:37:14Z

Since these don't yet do anything, and will be part of multiple PRs before the feature is fully-fleshed out, I think it would be helpful for these to be behind a feature flag, so that they are not inadvertently released-and-then-changed.

craigtaverner · 2025-11-10T15:47:21Z

Since these don't yet do anything, and will be part of multiple PRs before the feature is fully-fleshed out, I think it would be helpful for these to be behind a feature flag, so that they are not inadvertently released-and-then-changed.

We've made this SNAPSHOT only, which is essentially the same as a feature flag, and is the approach normally taken in the ES|QL team for cases like this where we do not yet want the feature to be released. However, I will add more tests that explicitly assert that the feature is missing outside of SNAPSHOT builds.

dakrone · 2025-11-10T15:50:04Z

We've made this SNAPSHOT only, which is essentially the same as a feature flag, and is the approach normally taken in the ES|QL team for cases like this where we do not yet want the feature to be released. However, I will add more tests that explicitly assert that the feature is missing outside of SNAPSHOT builds.

As I read this, that is for the implementation itself, but does not prevent the REST APIs from being registered and available, even without the node feature. Unless I am misunderstanding it?

dakrone · 2025-11-10T15:52:11Z

(Another drive-by comment, but this is not a full review)

Why do we have both a "get" and "list" action/API for these? All of our other APIs that follow this model register a single get REST handler that takes an optional {name} parameter, returning all the <things> when the {name} is missing.

elasticsearchmachine · 2025-11-11T09:39:48Z

Hi @craigtaverner, I've created a changelog YAML for you.

This is extracted from the Views Prototype at elastic#134995 and is originally based on the "ESQL: View" prototype by Nik but with the following changes: * No ES|QL language integration, only REST API for adding/removing and managing Views in cluster state * ViewMetadata now inside ProjectMetadata * NodeFeature to ensure metadata only exists in clusters fully upgraded * Wide CRUD support (add, update, delete, list) * View name validation * View count, length and depth restrictions Co-authored-by: Nik Everett <nik9000@gmail.com>

…n on missing view

…der list structures

nielsbauman

Thanks for working on this, @craigtaverner! I scanned most parts of the code briefly and left a few comments, some of which are not a big deal.

...vileges-tests/src/javaRestTest/java/org/elasticsearch/xpack/security/operator/Constants.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/ViewMetadata.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/View.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/ClusterViewService.java

nielsbauman · 2025-11-12T17:39:54Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/view/InMemoryViewService.java

+ * Simple implementation of {@link ClusterViewService} that keeps the views in memory.
+ * This is useful for testing.
+ */
+public class InMemoryViewService extends ViewService {


I'm probably missing something, but why do we need this in-memory version exactly?

And, follow-up question: wouldn't it make more sense to put that in a test package somewhere, rather than having it in production code, if it's only useful for testing?

It is only moderately useful here, but very useful in the actual ES|QL views were we have semi-unit tests that run the full ES|QL stack, but outside of Elasticsearch, and then this becomes the main mechanism for enabling that. I could indeed move it.

In this PR it serves primarily as unit tests for the ViewService, without touching cluster state (so only tests the ViewService API internally). That still has value, so I'm happy to keep it here (although I moved to the test package, as you suggested).

nielsbauman · 2025-11-12T17:47:27Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/ClusterViewService.java

+        submitUnbatchedTask("update-esql-view-metadata", new ClusterStateUpdateTask() {
+            @Override
+            public ClusterState execute(ClusterState currentState) {
+                var project = getProjectMetadata(currentState);


getProjectMetadata uses the ProjectResolver, but the ProjectResolver is not meant to be used inside cluster state update tasks, as the thread context isn't copied over when running cluster state update tasks (which is what the project resolver relies on in multi-project mode). It's totally understandable that you would implement it like this; when the multi-project work gets picked up, we should put effort into making it clearer that the project resolver isn't supposed to work like this.

That means that you'll have to explicitly pass a ProjectId to this method. I would advise resolving the project ID in the transport actions, because:

I think having this getMetadata() method that relies on the project resolver can cause confusion and possible bugs, because it's not apparent at all that this method only works if the project ID is present in the thread context, which isn't always the case, such as here. I would advise doing something along the lines of:

protected ViewMetadata getMetadata(ProjectId projectId) { return getMetadata(clusterService.state().metadata().getProject(projectId)); } protected ViewMetadata getMetadata(ProjectMetadata projectMetadata) { return projectMetadata.custom(ViewMetadata.TYPE, ViewMetadata.EMPTY); }

Let me know what you think and if you have any questions/concerns.

I originally got this approach from the EnrichPolicyResolver, which uses ProjectResolver, but then also noticed that the EnrichPolicy CRUD used ProjectID instead, so it seems Enrich uses both approaches (but in different places, so probably correct). I can try switch to using ProjectID for CRUD as you suggest.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/TransportGetViewAction.java

… and test

nielsbauman

Left a few more minor comments, having reviewed everything now. Almost good to go from my side!

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/ViewService.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlFeatures.java

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/200_view.yml

nielsbauman · 2025-12-10T16:03:31Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/view/ViewRestTests.java

+
+import static org.elasticsearch.xpack.esql.plugin.EsqlFeatures.ESQL_VIEWS_FEATURE_FLAG;
+
+public class ViewRestTests extends AbstractViewTestCase {


Are all the changes in ‎x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/view only for this test class? Or are you planning on adding more tests in the future? The amount of plumbing/infra we do for this test is excessive and unnecessary; that's why I'm asking.

After this PR goes in work will start on further changes for it, so I will keep it at least for now. If in that work I end up not needing it more, I agree it would then be excessive and should go away.

nielsbauman

LGTM, thanks a lot for all the iterations both @craigtaverner and @dakrone!

This adds support for (get|put|delete)-view APIs to the specification Relates to elastic/elasticsearch#137818

* Add specification for ESQL Views CRUD This adds support for (get|put|delete)-view APIs to the specification Relates to elastic/elasticsearch#137818 * Update after make contrib * Apply suggestions from code review * Fix lint and run make contrib --------- Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>

This commit adds the `View` class as the fourth `IndexAbstraction`. This means that it can be "resolved" in the expression resolver, and moves us towards implementing the ES security model for views. It also includes code that prevents conflicting views, indices, aliases, and data streams with the same name. Additionally, we prevent indexing into a view or directly resolving it for a search (if someone searches for `*` for instance). Relates to elastic#137818

* Add ESQL View as an IndexAbstraction This commit adds the `View` class as the fourth `IndexAbstraction`. This means that it can be "resolved" in the expression resolver, and moves us towards implementing the ES security model for views. It also includes code that prevents conflicting views, indices, aliases, and data streams with the same name. Additionally, we prevent indexing into a view or directly resolving it for a search (if someone searches for `*` for instance). Relates to #137818 * Include views in comment * Add earlier filtering of views in IndexNameExpressionResolver * Use empty list of indices and null write index for View

* Add ESQL View as an IndexAbstraction This commit adds the `View` class as the fourth `IndexAbstraction`. This means that it can be "resolved" in the expression resolver, and moves us towards implementing the ES security model for views. It also includes code that prevents conflicting views, indices, aliases, and data streams with the same name. Additionally, we prevent indexing into a view or directly resolving it for a search (if someone searches for `*` for instance). Relates to elastic#137818 * Include views in comment * Add earlier filtering of views in IndexNameExpressionResolver * Use empty list of indices and null write index for View

Initial support for ES|QL Views. It is based on the underlying sub-query support added in ..., and builds on the Views REST API work done in #137818. Checklist: - [x] Core prototype - [x] Extract key parts of Nik's original prototype - [x] Write testing infrastructure to test views within csv-spec tests (single and multi-node tests) - [x] Expand REST API to cover get, list, create/update, delete views - [x] Add size and count limitations to stored view definitions - [x] Write testing infrastructure to test views in CsvTest (csv-spec unit testing) - [x] Expand scope to nested views - [x] Expand scope to multiple views and views+indexes - [x] Split into multiple PRs for easy review - [x] Multi-relation index resolution (gives CCS/CPS support) - [x] #136780 - [x] More resilient EsqlSpecTestCase - [x] #136781 - [x] View CRUD with REST - [x] #137818 - [x] Rebase prototype on the above to get cleaner, simple feature PR - [ ] Enhancements - [x] Prototype CCS and CPS - [x] Enforce view/index name uniqueness (done in separate PR) - [x] Add serialization tests to test source deserialization in re-written plans - [x] Ensure views are gated behind SNAPSHOT and/or feature flag

craigtaverner requested review from nielsbauman and nik9000 November 10, 2025 11:22

elasticsearchmachine added the v9.3.0 label Nov 10, 2025

craigtaverner force-pushed the views_rest_crud branch from 259d88e to 2b93fa5 Compare November 10, 2025 11:38

craigtaverner force-pushed the views_rest_crud branch 2 times, most recently from bb1ce21 to 89f766e Compare November 10, 2025 17:43

craigtaverner and others added 4 commits November 12, 2025 12:50

Update docs/changelog/137818.yaml

507bf1c

Refine changelog

b0ae25c

Expand get-views to cover use cases for listing views

73da7cd

craigtaverner force-pushed the views_rest_crud branch from f21d063 to 73da7cd Compare November 12, 2025 11:50

craigtaverner added 3 commits November 12, 2025 15:57

Fixed failing tests after changing behaviour of get to throw exceptio…

21ca8f0

…n on missing view

Fixed failing yaml-tests after changing behaviour of get to return ol…

050d89c

…der list structures

Remove list_views API (covered by get_view now)

f529f11

nielsbauman reviewed Nov 12, 2025

View reviewed changes

craigtaverner added 2 commits November 13, 2025 14:47

Simply fixes based on code review

2694cef

Merge remote-tracking branch 'origin/main' into views_rest_crud

d466dcd

dakrone added 4 commits December 10, 2025 08:26

Add no-op check to putView

ce285e4

Remove re-validating in the service (already validated at REST layer)…

742a3a2

… and test

Remove duplicated validation

aba2c07

Merge remote-tracking branch 'origin/main' into views_rest_crud

48c4b63

nielsbauman reviewed Dec 10, 2025

View reviewed changes

dakrone and others added 5 commits December 10, 2025 09:13

Use helper to resolve metadata

31cb0b5

Remove extra cluster feature requirements in yaml tests

ccff1de

Assert view length in yaml tests

4b6e807

Merge remote-tracking branch 'origin/main' into views_rest_crud

1650d1a

Use Capabilities API instead of cluster feature

f77b745

nielsbauman approved these changes Dec 10, 2025

View reviewed changes

dakrone added the test-release Trigger CI checks against release build label Dec 10, 2025

Merge remote-tracking branch 'origin/main' into views_rest_crud

0d7874d

dakrone added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Dec 10, 2025

dakrone added 2 commits December 10, 2025 14:28

Merge remote-tracking branch 'origin/main' into views_rest_crud

6c2da61

Merge remote-tracking branch 'origin/main' into views_rest_crud

9c01ec6

dakrone enabled auto-merge (squash) December 10, 2025 22:54

elasticsearchmachine merged commit d1f2980 into elastic:main Dec 10, 2025
35 checks passed

craigtaverner deleted the views_rest_crud branch December 10, 2025 23:25

dakrone added a commit to elastic/elasticsearch-specification that referenced this pull request Dec 11, 2025

Add specification for ESQL Views CRUD

2147d34

This adds support for (get|put|delete)-view APIs to the specification Relates to elastic/elasticsearch#137818

dakrone mentioned this pull request Dec 11, 2025

Add specification for ESQL Views CRUD elastic/elasticsearch-specification#5836

Merged

craigtaverner mentioned this pull request Dec 19, 2025

ES|QL Views #134995

Merged

23 tasks

dakrone mentioned this pull request Dec 30, 2025

Add ESQL View as an IndexAbstraction #139778

Merged

pquentin mentioned this pull request Jan 27, 2026

rest-api-spec: fix visibility for esql.get_views API #141354

Merged

adcoelho mentioned this pull request Mar 4, 2026

[ResponseOps] Update kibana system user with esql view permissions. #143568

Merged


		import static org.elasticsearch.xpack.esql.plugin.EsqlFeatures.ESQL_VIEWS_FEATURE_FLAG;

		public class ViewRestTests extends AbstractViewTestCase {

Conversation

craigtaverner commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 10, 2025

Uh oh!

elasticsearchmachine commented Nov 10, 2025

Uh oh!

elasticsearchmachine commented Nov 10, 2025

Uh oh!

dakrone commented Nov 10, 2025

Uh oh!

craigtaverner commented Nov 10, 2025

Uh oh!

dakrone commented Nov 10, 2025

Uh oh!

dakrone commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 11, 2025

Uh oh!

nielsbauman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nielsbauman Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

craigtaverner Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

craigtaverner Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

craigtaverner Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nielsbauman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nielsbauman Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

dakrone Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

craigtaverner commented Nov 10, 2025 •

edited

Loading

dakrone commented Nov 10, 2025 •

edited

Loading