TS_INFO information retrieval command by leontyevdv · Pull Request #142721 · elastic/elasticsearch

leontyevdv · 2026-02-19T16:56:34Z

Adds the TS_INFO command, a time-series metadata introspection command built on top of METRICS_INFO (#141667). While METRICS_INFO returns one row per distinct metric across all time series, TS_INFO returns one row per (metric, time-series) combination, providing finer-grained visibility into individual time series.

TS_INFO produces the same 6 columns as METRICS_INFO plus an additional dimensions column:

TS k8s | METRICS_INFO | SORT metric_name

metric_name	data_stream	unit	metric_type	field_type	dimension_fields
network.eth0.rx	k8s	packets	gauge	integer	[cluster, pod, region]
network.eth0.tx	k8s	packets	gauge	integer	[cluster, pod, region]
network.total_bytes_in	k8s	bytes	counter	long	[cluster, pod, region]
network.total_cost	k8s	usd	counter	double	[cluster, pod, region]

TS_INFO expands this to one row per time series, adding the dimensions column:

TS k8s | TS_INFO | KEEP metric_name, dimensions | WHERE metric_name == "network.eth0.rx" | SORT dimensions

metric_name	dimensions
network.eth0.rx	{"cluster": "prod", "pod": "one", "region": "[eu, us]"}
network.eth0.rx	{"cluster": "prod", "pod": "three", "region": "[eu, us]"}
network.eth0.rx	{"cluster": "prod", "pod": "two", "region": "[eu, us]"}
network.eth0.rx	{"cluster": "qa", "pod": "one"}
network.eth0.rx	{"cluster": "qa", "pod": "three"}
network.eth0.rx	{"cluster": "qa", "pod": "two"}
network.eth0.rx	{"cluster": "staging", "pod": "one", "region": "us"}
network.eth0.rx	{"cluster": "staging", "pod": "three", "region": "us"}
network.eth0.rx	{"cluster": "staging", "pod": "two", "region": "us"}

This PR contains generative tests for both METRICS_INFO and TS_INFO commands.

Closes #139296
Closes #141413

Add METRICS_INFO command to retrieve metrics-related metadata. Part of elastic#139296

Add mata.unit to the k8s-mappings.json Part of elastic#139296

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java

Regenerate ESQL parser to fix conflicts Part of elastic#139296

Change METRICS_INFO to scan documents with unique _tsid to retrieve metrics and dimensions metadata from them. Part of elastic#139296

Improve JavaDoc Part of elastic#139296

Polish code Part of elastic#139296

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Fix a compilation error occurred due to the merge with main. Part of elastic#139296

- Address review comments. - Optimize DistinctByOperator - Optimize collection of metrics and dimensions Part of elastic#139296

Change grouping by adding a concept of MetricSignature and 2 steps of grouping. Part of elastic#139296

Add MetricsInfoOperatorTests Part of elastic#139296

Ban PipelineBreakers before METRICS_INFO. Part of elastic#139296

Make MetricsInfo a PipelineBreaker. Add deduplication of information.

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Make MetricsInfo a PipelineBreaker. Add deduplication of information.

Change grouping to take into account a data stream name

Fix test

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseLexer.interp # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseLexer.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/PlanWritables.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java

Add INTERMEDIATE reduction

leontyevdv · 2026-02-25T13:23:49Z

Not sure if you're planning to add docs in this PR, but if yes there's some info in the README :)

Hey @leemthompo ! We are not going to add any documentation so far because it's going to be a command for internal use. Kibana is going to use it to build dashboards.

leemthompo · 2026-02-25T13:26:37Z

it's going to be a command for internal use. Kibana is going to use it to build dashboards.

Not sure if it makes sense to have a release note in that case?

leontyevdv · 2026-02-25T13:35:57Z

it's going to be a command for internal use. Kibana is going to use it to build dashboards.

Not sure if it makes sense to have a release note in that case?

Yes, I agree. It's generated automatically. I wonder if I can remove it and if I can remove the one that I merged for a similar METRICS_INFO command.

dnhatn

LGTM. Thanks Dima!

One main concern: from my perspective, the terms TS and Metrics are interchangeable, so TS_INFO and METRICS_INFO also seem interchangeable. Would it make sense to introduce a parameter to METRICS_INFO to distinguish the behavior instead of two commands?

@kkrik-es Are you okay with this?

kkrik-es · 2026-02-26T08:47:41Z

One main concern: from my perspective, the terms TS and Metrics are interchangeable, so TS_INFO and METRICS_INFO also seem interchangeable. Would it make sense to introduce a parameter to METRICS_INFO to distinguish the behavior instead of two commands?

We do use time series and metrics interchangeably, but this goes to the actual retrieved information: the former retrieves information per time series (tsid), while the latter does so per metric name (field). These two are fairly different - and we could have DIMENSIONS_INFO for retrieving dimension names along with associated metrics. Now, we could just use METRICS_INFO for all of them, passing args to differentiate what we retrieve. TS_INFO doesn't seem to add too much overhead, implementation-wise, and I like the clean semantics, but can be convinced either way. Let's also check with @felixbarny who proposed TS_INFO in the first place.

leontyevdv · 2026-02-26T10:38:21Z

I've done some benchmarking using the 240M-high-cardinality dataset. Here are the results:

TS metrics-hostmetricsreceiver.otel-default | METRICS_INFO | SORT metric_name
  
real    0m5.287s
user    0m0.001s
sys     0m0.007s

---

TS metrics-hostmetricsreceiver.otel-default | METRICS_INFO | KEEP metric_name, metric_type | SORT metric_name

real    0m3.995s
user    0m0.003s
sys     0m0.005s

---

TS metrics-hostmetricsreceiver.otel-default | METRICS_INFO | STATS metric_count = COUNT(*) BY metric_type | SORT metric_type
  
real    0m3.860s
user    0m0.005s
sys     0m0.002s

---

TS metrics-hostmetricsreceiver.otel-default | TS_INFO | SORT metric_name, dimensions

real    0m19.630s
user    0m0.007s
sys     0m0.013s

---

TS metrics-hostmetricsreceiver.otel-default | TS_INFO | STATS ts_count = COUNT(*) BY metric_name | SORT metric_name

real    0m15.327s
user    0m0.004s
sys     0m0.004s

fyi: @kkrik-es , @dnhatn , @felixbarny

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/generator/EsqlQueryGenerator.java

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

felixbarny · 2026-03-02T11:15:01Z

One main concern: from my perspective, the terms TS and Metrics are interchangeable, so TS_INFO and METRICS_INFO also seem interchangeable. Would it make sense to introduce a parameter to METRICS_INFO to distinguish the behavior instead of two commands?

Time series and metric are not interchangeable terms. A metric is defined by the metric name. A time series is defined by the combination of all dimensions and the metric name. See also https://opentelemetry.io/docs/specs/otel/metrics/data-model/#timeseries-model.

That's exactly how the two commands differ: METRICS_INFO returns a row per metric name while TS_INFO returns a row per unique metric name and _tsid.

sidosera · 2026-03-02T13:37:45Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/TsInfoOperator.java

+    private final MetricsInfoOperator.MetricFieldLookup fieldLookup;
+    private final int metadataSourceChannel;
+    private final int indexChannel;
+    /** FINAL-mode field: input channel indices for the 7 output columns. Null in INITIAL mode. */


Suggested change

/** FINAL-mode field: input channel indices for the 7 output columns. Null in INITIAL mode. */

/** FINAL-mode field: input channel indices for the 7 output columns. Nulls in INITIAL mode. */

Not really, the filed is an array of primitive types (int[]) so it cannot contain nulls. In this particular case, the null value is assigned to the field and not to the values inside the array.

Makes sense, thanks for the context @leontyevdv

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/generator/EsqlQueryGenerator.java

…locations * upstream/main: (94 commits) Mute org.elasticsearch.xpack.esql.qa.mixed.EsqlClientYamlIT test {p0=esql/40_tsdb/TS Command grouping on text field} elastic#142544 Mute org.elasticsearch.index.store.StoreDirectoryMetricsIT testDirectoryMetrics elastic#143419 Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test elastic#143023 TS_INFO information retrieval command (elastic#142721) ESQL: External source parallel execution and distribution (elastic#143349) Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=DOC_VALUES]} elastic#143414 Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=NONE]} elastic#143413 Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=STORED]} elastic#143412 Removing ingest random sampling (elastic#143289) Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT test elastic#143023 [Transform] Clean up internal tests (elastic#143246) Skip time series field type merge for non-TS agg queries (elastic#143262) Enable zero-copy SIMD vector scoring on searchable snapshots (frozen tier) (elastic#141718) Mute org.elasticsearch.xpack.search.CrossClusterAsyncSearchIT testCancelViaExpirationOnRemoteResultsWithMinimizeRoundtrips elastic#143407 Fix MemorySegmentUtilsTests (elastic#143391) Unmute testWorkflowsRestrictionAllowsAccess (elastic#143308) Cancel async query on expiry (elastic#143016) ESQL: Finish migrating error testing (elastic#143322) Reduce LuceneOperator.Status memory consumption with large QueryDSL queries (elastic#143175) ESQL: Generative testing with full text functions (elastic#142961) ...

Add a TS_INFO command - a time-series metadata introspection command built on top of METRICS_INFO (elastic#141667). While METRICS_INFO returns one row per distinct metric across all time series, TS_INFO returns one row per (metric, time-series) combination, providing finer-grained visibility into individual time series. TS_INFO produces the same 6 columns as METRICS_INFO plus an additional dimensions column. --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Liam Thompson <leemthompo@gmail.com>

leontyevdv and others added 30 commits February 2, 2026 16:25

feat: Add METRICS_INFO command

dea78d5

Add METRICS_INFO command to retrieve metrics-related metadata. Part of elastic#139296

feat: Add METRICS_INFO command

ed11965

Add mata.unit to the k8s-mappings.json Part of elastic#139296

feat: Add METRICS_INFO command

5b29c92

Regenerate ESQL parser to fix conflicts Part of elastic#139296

feat: Add METRICS_INFO command

fb83617

Change METRICS_INFO to scan documents with unique _tsid to retrieve metrics and dimensions metadata from them. Part of elastic#139296

feat: Add METRICS_INFO command

8701483

Improve JavaDoc Part of elastic#139296

feat: Add METRICS_INFO command

7b92561

Polish code Part of elastic#139296

feat: Add METRICS_INFO command

0142de5

Polish code Part of elastic#139296

Merge branch 'main' into feat/metrics-info-information-retrieval-command

0a82028

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Update docs/changelog/141667.yaml

c912907

feat: Add METRICS_INFO command

6547fcd

Fix a compilation error occurred due to the merge with main. Part of elastic#139296

feat: Add METRICS_INFO command

8a642e0

- Address review comments. - Optimize DistinctByOperator - Optimize collection of metrics and dimensions Part of elastic#139296

Merge branch 'main' into feat/metrics-info-information-retrieval-command

0cc2759

feat: Add METRICS_INFO command

f702cce

Change grouping by adding a concept of MetricSignature and 2 steps of grouping. Part of elastic#139296

Merge branch 'main' into feat/metrics-info-information-retrieval-command

2fd2cea

feat: Add METRICS_INFO command

9512d89

Add MetricsInfoOperatorTests Part of elastic#139296

Merge branch 'main' into feat/metrics-info-information-retrieval-command

27d3e04

feat: Add METRICS_INFO command

698a417

Ban PipelineBreakers before METRICS_INFO. Part of elastic#139296

[CI] Auto commit changes from spotless

6ffaf42

Address PR comments

2ddf789

Make MetricsInfo a PipelineBreaker. Add deduplication of information.

Merge branch 'main' into feat/metrics-info-information-retrieval-command

bd90f04

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Address PR comments

84ebc5e

Make MetricsInfo a PipelineBreaker. Add deduplication of information.

Address PR comments

9f5cb03

Change grouping to take into account a data stream name

Merge branch 'main' into feat/metrics-info-information-retrieval-command

61fae7d

Address PR comments

bc72a16

Fix test

Address PR comments

ad8516c

Fix test

Address PR comments

c84eae3

Fix test

Merge branch 'main' into feat/metrics-info-information-retrieval-command

7ac4384

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Address PR comments

f38d9a1

Add INTERMEDIATE reduction

leontyevdv requested a review from dnhatn February 25, 2026 14:00

Merge branch 'main' into feat/ts-info-information-retrieval-command

16dcb77

dnhatn approved these changes Feb 26, 2026

View reviewed changes

Merge branch 'main' into feat/ts-info-information-retrieval-command

e6f1198

leontyevdv added 9 commits February 26, 2026 15:01

Merge branch 'main' into feat/ts-info-information-retrieval-command

63265ea

Merge branch 'main' into feat/ts-info-information-retrieval-command

d30ded3

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Merge branch 'main' into feat/ts-info-information-retrieval-command

adcbbc1

Merge branch 'main' into feat/ts-info-information-retrieval-command

617e8a6

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/generator/EsqlQueryGenerator.java

Merge main

b571e56

Merge branch 'main' into feat/ts-info-information-retrieval-command

6db5fd2

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Fix serialization, add serialization tests

f6a6b35

Merge branch 'main' into feat/ts-info-information-retrieval-command

8017cda

Revert a "Fix serialization, add serialization tests"

7828c4c

Merge branch 'main' into feat/ts-info-information-retrieval-command

987f70f

sidosera reviewed Mar 2, 2026

View reviewed changes

leontyevdv added 4 commits March 2, 2026 16:53

Fix multi-node test for TS_INFO

b439e18

Merge branch 'main' into feat/ts-info-information-retrieval-command

6ed5a02

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/generator/EsqlQueryGenerator.java

Merge main

c0f3788

Merge branch 'main' into feat/ts-info-information-retrieval-command

fba5439

leontyevdv merged commit 4be64f2 into elastic:main Mar 2, 2026
35 checks passed

This was referenced Mar 6, 2026

[ML] Wait for cluster state in test #143767

Merged

[Transform] Disable PIT for CPS #143876

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TS_INFO information retrieval command#142721

TS_INFO information retrieval command#142721
leontyevdv merged 87 commits intoelastic:mainfrom
leontyevdv:feat/ts-info-information-retrieval-command

leontyevdv commented Feb 19, 2026 •

edited

Loading

Uh oh!

leontyevdv commented Feb 25, 2026

Uh oh!

leemthompo commented Feb 25, 2026

Uh oh!

leontyevdv commented Feb 25, 2026

Uh oh!

dnhatn left a comment

Uh oh!

kkrik-es commented Feb 26, 2026

Uh oh!

leontyevdv commented Feb 26, 2026 •

edited

Loading

Uh oh!

felixbarny commented Mar 2, 2026

Uh oh!

sidosera Mar 2, 2026

Uh oh!

leontyevdv Mar 2, 2026

Uh oh!

sidosera Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	/** FINAL-mode field: input channel indices for the 7 output columns. Null in INITIAL mode. */
	/** FINAL-mode field: input channel indices for the 7 output columns. Nulls in INITIAL mode. */

Conversation

leontyevdv commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leontyevdv commented Feb 25, 2026

Uh oh!

leemthompo commented Feb 25, 2026

Uh oh!

leontyevdv commented Feb 25, 2026

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

kkrik-es commented Feb 26, 2026

Uh oh!

leontyevdv commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixbarny commented Mar 2, 2026

Uh oh!

sidosera Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

leontyevdv Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

sidosera Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

leontyevdv commented Feb 19, 2026 •

edited

Loading

leontyevdv commented Feb 26, 2026 •

edited

Loading