Expose _tier metadata attribute in ESQL by idegtiarenko · Pull Request #139894 · elastic/elasticsearch

idegtiarenko · 2025-12-22T14:39:05Z

This change exposes _tier as an metadata attribute in ESQL.

Same as _search attribute emits top index tier preference, not current tier (in case if index is still relocating).
This attribute may be used in filter conditions, ~~but as for today this will not automatically eliminate entire index during can match phase~~ resulted condition is pushed into underlying QueryBuilder.

Related to #108264

elasticsearchmachine · 2025-12-22T14:39:45Z

Hi @idegtiarenko, I've created a changelog YAML for you.

...main/java/org/elasticsearch/xpack/cluster/routing/allocation/mapper/DataTierFieldMapper.java

nik9000

Code looks right to me.

elasticsearchmachine · 2025-12-22T15:20:32Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

github-actions · 2025-12-22T15:21:34Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/esql-metadata-fields.md

github-actions · 2025-12-22T15:21:36Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

idegtiarenko · 2025-12-23T08:50:48Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java


-    public void testMetadataFieldUnsupportedPrimitiveType() {
-        expectError("from test metadata _tier", "line 1:20: unsupported metadata field [_tier]");
-    }


This verifies the error is the same as below but for a different attribute.
It is safe to remove.

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

nik9000

For a followup I think:

We should have an integration test that proves that we filter out the index dispatch. I think Stas wrote one, but I don't know what it's called.

And maybe also one that proves that filters get pushed to the underlying index as MatchAll/MatchNone - maybe in PushQueriesIT or something.

I think both of these tests are fairly important, especially because the pushing process is fairly magical. And because the index dispatch bit is, like, a big hammer that we'd like to make sure works consistently. So a paranoid integration test all the way on the outside seems called for.

nik9000 · 2026-01-06T14:22:44Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/metadata.csv-spec

+
+_index:keyword
+employees
+;


I think we have this test above, right?

nik9000 · 2026-01-07T14:02:06Z

Thanks for adding the stuff to CanMatchIT. I hadn't realized it was there. But I'm happy we have it.

elasticsearchmachine · 2026-01-07T15:07:11Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.3	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 139894

(cherry picked from commit 272ca11)

(cherry picked from commit 272ca11) # Conflicts: # docs/reference/query-languages/esql/esql-metadata-fields.md # x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/CanMatchIT.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/TestPhysicalOperationProviders.java

* upstream/main: Add hook for blocking termination (elastic#133555) Delegate to ES93ScalarQuantizedVectorsFormat rather than copying behaviour (elastic#139834) Refactor compound block types (elastic#140219) Flush the rate buffer when the slice index changes (elastic#138856) Log linked project connection errors at debug during shutdown (elastic#140239) Periodic FIPS 140-3 buildkite pipelines (elastic#139909) ES|QL - Remove TERM function (elastic#139953) Fix name of started time field in shutdown status (elastic#139910) Drop `project_routing` from query params (elastic#140272) Fix flaky test: AllocationDecidersTests (elastic#140271) Add List Reindex API (elastic#140184) Expose _tier metadata attribute in ESQL (elastic#139894) Tweak TSDBRestEsqlIT#testTimeSeriesQuerying(...) (elastic#140210) Fix an OOM error when creating to many chained synonym graph token filter. (elastic#140026) Suppress Azure SDK error logs (elastic#139730) Rewritten integer sorts need to use SortedNumericSortField (elastic#139538) (elastic#139700) Adjust index versions for skippers for time series (elastic#139670) Fix host.name skippers index version range (elastic#139636) Remove BWC shim for a broken commit Fix index.mapping.use_doc_values_skipper defaults in serverless (elastic#139532)

This reverts commit 7dcb69e.

Expose _tier in ESQL

4d2e1ab

idegtiarenko requested a review from nik9000 December 22, 2025 14:39

idegtiarenko added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL v9.3.0 labels Dec 22, 2025

elasticsearchmachine added the v9.4.0 label Dec 22, 2025

Update docs/changelog/139894.yaml

1b57b9d

nik9000 reviewed Dec 22, 2025

View reviewed changes

...main/java/org/elasticsearch/xpack/cluster/routing/allocation/mapper/DataTierFieldMapper.java Show resolved Hide resolved

nik9000 reviewed Dec 22, 2025

View reviewed changes

idegtiarenko added auto-backport Automatically create backport pull requests when merged v9.3.1 v8.19.10 and removed v9.3.0 labels Dec 22, 2025

idegtiarenko added 3 commits December 22, 2025 16:07

fix org.elasticsearch.xpack.esql.CsvTests

2894fc6

align behavior with search

580dcde

update docs

56e38f3

idegtiarenko marked this pull request as ready for review December 22, 2025 15:20

idegtiarenko requested review from alex-spies and nik9000 December 22, 2025 15:22

idegtiarenko added 2 commits December 23, 2025 09:49

remove test that is no longer needed

e6691c4

Merge branch 'main' into support_tier_in_esql

03f2c5f

idegtiarenko commented Dec 23, 2025

View reviewed changes

Merge branch 'main' into support_tier_in_esql

9962835

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

nik9000 approved these changes Jan 6, 2026

View reviewed changes

MetadataTier optimizer tests

8a6d5cc

idegtiarenko added 3 commits January 7, 2026 13:56

add CanMatch integration test

44ec5df

Merge branch 'main' into support_tier_in_esql

dea6b09

remove empty line

aca04a1

nik9000 approved these changes Jan 7, 2026

View reviewed changes

idegtiarenko merged commit 272ca11 into elastic:main Jan 7, 2026
35 checks passed

idegtiarenko deleted the support_tier_in_esql branch January 7, 2026 15:05

idegtiarenko changed the title ~~Expose _tier in ESQL~~ Expose _tier metadata attribute in ESQL Jan 7, 2026

elasticsearchmachine added the backport pending label Jan 7, 2026

rjernst pushed a commit to rjernst/elasticsearch that referenced this pull request Jan 7, 2026

Expose _tier metadata attribute in ESQL (elastic#139894)

594438f

idegtiarenko added a commit to idegtiarenko/elasticsearch that referenced this pull request Jan 7, 2026

Expose _tier metadata attribute in ESQL (elastic#139894)

c97a2ee

(cherry picked from commit 272ca11)

This was referenced Jan 7, 2026

[9.3] Expose _tier metadata attribute in ESQL (#139894) #140277

Merged

[8.19] Expose _tier metadata attribute in ESQL (#139894) #140290

Merged

idegtiarenko removed the backport pending label Jan 7, 2026

elasticsearchmachine pushed a commit that referenced this pull request Jan 7, 2026

Expose _tier metadata attribute in ESQL (#139894) (#140277)

ec162ae

(cherry picked from commit 272ca11)

sidosera pushed a commit to sidosera/elasticsearch that referenced this pull request Jan 7, 2026

Expose _tier metadata attribute in ESQL (elastic#139894)

398ded4

idegtiarenko added a commit that referenced this pull request Jan 13, 2026

Revert "Expose _tier metadata attribute in ESQL (#139894) (#140290)"

ac03b39

This reverts commit 7dcb69e.

This was referenced Jan 13, 2026

[8.19] Revert _tier metadata attribute #140573

Merged

[9.3] Revert _tier metadata attribute #140574

Merged

idegtiarenko removed v8.19.10 v9.3.1 labels Jan 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose _tier metadata attribute in ESQL#139894

Expose _tier metadata attribute in ESQL#139894
idegtiarenko merged 12 commits intoelastic:mainfrom
idegtiarenko:support_tier_in_esql

idegtiarenko commented Dec 22, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

Uh oh!

nik9000 left a comment

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

github-actions bot commented Dec 22, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 22, 2025

When to use applies_to tags:

What NOT to do:

Uh oh!

idegtiarenko Dec 23, 2025

Uh oh!

nik9000 left a comment

Uh oh!

nik9000 Jan 6, 2026

Uh oh!

nik9000 commented Jan 7, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

idegtiarenko commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

Uh oh!

nik9000 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

github-actions bot commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Dec 22, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

idegtiarenko Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 left a comment

Choose a reason for hiding this comment

Uh oh!

nik9000 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Jan 7, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 7, 2026

💔 Backport failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

idegtiarenko commented Dec 22, 2025 •

edited

Loading

github-actions bot commented Dec 22, 2025 •

edited

Loading