Skip to content

PromQL Support in ESQL#137988

Merged
costin merged 74 commits intomainfrom
esql/promql
Nov 20, 2025
Merged

PromQL Support in ESQL#137988
costin merged 74 commits intomainfrom
esql/promql

Conversation

@costin
Copy link
Copy Markdown
Member

@costin costin commented Nov 12, 2025

This implementation adds native PromQL (Prometheus Query Language support to ESQL through a new PROMQL command, enabling users to query time-series data using familiar Prometheus syntax by leveraging the time-series/TS support available in ESQL.

The capability is available behind a feature flag as it's still wip yet mature enough to be merged into main.

  TS k8s
  | promql step 1h (
      avg by (pod) (avg_over_time(network.bytes_in{pod=~"host-0|host-1|host-2"}[1h]))
    )
  | LIMIT 1000

This query starts from the k8s time-series index, calculates the average of network.bytes_in over 1-hour windows for specific pods, aggregates by pod dimension with a 1-hour step, and limits results to 1000 rows.

Functionality overview

The current PROMQL command supports the following PromQL functionality:

  • Label-based selectors for filtering time-series (e.g., http_requests_total{job="apiserver", handler="/api/comments"})
  • Range vector selectors for time windows (e.g., http_requests_total[5m])
  • Rate calculations over counters: rate(), irate(), increase(), delta(), etc
  • Time-based aggregations: avg_over_time(), sum_over_time()`, etc..
  • Cross-series aggregations: sum by(), avg by(), etc...

Current Limitations

This initial implementation focuses on time-series aggregation queries. Advanced PromQL features like label manipulation, instant-vector transformations, mathematical functions, and certain query modifiers are not yet supported.

costin and others added 30 commits October 14, 2025 15:22
Add PromQL support in ESQL under the PROMQL command.
This commit includes the grammar, associated logical plan and basic
rules for performing translation, from filtering to stats transpilation.

* Grammar
Stand-alone grammar definition and parser, currently 'embedded' inside
ESQL (the current scenario).
Grammar include to pass the valid tests (~200 queries) from Prometheus
3.5; the invalid ones are disabled for now since the parser needs to
validate the syntax using

* Transpilation
Implement basic PromQL plan translation:
 Selector → Filter with label matcher conditions
 RangeSelector → Filter + Bucket for time bucketing
 WithinSeriesAggregate → ESQL function over selector
 AcrossSeriesAggregate → TimeSeriesAggregate with groupings
Extended PromQL grammar to support optional time-range parameters
before the query expression. Parameters are specified as
space-separated name-value pairs (e.g., "step 5m")
and can appear in any order.
Currently only literals and field references are allowed, expressions
such as (now() - 1h) are not supported.

The query itself is enclosed in parentheses to enable future assignment
syntax like "promql x=(...)" while avoiding ambiguity.

Implementation details:
- Introduced two-mode lexer architecture:
- PROMQL_PARAMS_MODE: Captures parameter name-value pairs using
  generic identifiers (PROMQL_UNQUOTED_IDENTIFIER or
  QUOTED_IDENTIFIER)
- PROMQL_QUERY_MODE: Captures query text with depth tracking for
  balanced parentheses. Skipping then ends up with ambiguitities
  error when dealing with nested queries or having promql inside
  explain
- Parser uses recursive rule (promqlQueryPart) to handle nested
  parentheses in PromQL expressions without limiting nesting depth
- Added depth tracking methods in LexerConfig
  (incPromqlDepth, decPromqlDepth, resetPromqlDepth, isPromqlQuery)
  to distinguish between query-closing and query-internal
  parentheses
- Query text extraction preserves original source verbatim using parse
  tree token positions
Examples:
promql step 5m (rate(http_requests[5m]))
promql time now (up {job="api"})
promql start ?start_time step ?interval (sum(network_bytes))
This is to align more closely with the query parameter names for the prometheus range_query API.
This is to avoid ambiguity with org.elasticsearch.xpack.esql.parser.LogicalPlanBuilder
Previously Vector operations were defined as Expressions which didn't
match the actual semantics not the rest of the parsing infra. This has
now been addressed by making the nodes LogicalPlans.
Remove GrammarTests in favor of PromqlAstTests to test not just the
grammar but also the semantic validation of the queries, especially
as the latter includes the former.

The invalid tests are currently passing, but there are still some valid
tests that are failing, likely due to incorrect AST assembly.
Refactor Subquery from Expression into LogicalPlan. Duration parsing is
wip.
Unify the logic for arithmetics between scalars alone and scalars and
time durations
To simplify date arithmetic and since the time unit is not necessary,
TimeValue has been replaced with java.time.Duration both in the parser
and throughout the AST
Parser now performs comparison folding for literals.
Paranthesized expressions are now allowed.
In the process refactored the folding code to make
it more compact.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stratoula Lexer for the nested promql ESQL command

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stratoula Parser for the nested promql ESQL command

@costin costin added the ES|QL-ui Impacts ES|QL UI label Nov 18, 2025
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/kibana-esql (ES|QL-ui)

@costin
Copy link
Copy Markdown
Member Author

costin commented Nov 19, 2025

@elasticsearchmachine run elasticsearch-ci/part-2

Copy link
Copy Markdown
Member

@kkrik-es kkrik-es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

@costin costin merged commit 1543ef9 into main Nov 20, 2025
35 checks passed
@costin costin deleted the esql/promql branch November 20, 2025 18:16
@costin
Copy link
Copy Markdown
Member Author

costin commented Nov 20, 2025

To review the PR individual history, check the log of the PR itself : git fetch origin pull/137988/head:pull/137988; git log pull/137988

not-napoleon added a commit that referenced this pull request Nov 24, 2025
Part of #137988
Implement #138349 for the t-digest field type.

This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
afoucret pushed a commit to afoucret/elasticsearch that referenced this pull request Nov 26, 2025
Part of elastic#137988
Implement elastic#138349 for the t-digest field type.

This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
ncordon pushed a commit to ncordon/elasticsearch that referenced this pull request Nov 26, 2025
* Introduce PromQL command

Add PromQL support in ESQL under the PROMQL command.
This commit includes the grammar, associated logical plan and basic
rules for performing translation, from filtering to stats transpilation.

* Grammar
Stand-alone grammar definition and parser, currently 'embedded' inside
ESQL (the current scenario).
Grammar include to pass the valid tests (~200 queries) from Prometheus
3.5; the invalid ones are disabled for now since the parser needs to
validate the syntax using

* Transpilation
Implement basic PromQL plan translation:
 Selector → Filter with label matcher conditions
 RangeSelector → Filter + Bucket for time bucketing
 WithinSeriesAggregate → ESQL function over selector
 AcrossSeriesAggregate → TimeSeriesAggregate with groupings

---------

Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
ncordon pushed a commit to ncordon/elasticsearch that referenced this pull request Nov 26, 2025
Part of elastic#137988
Implement elastic#138349 for the t-digest field type.

This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
ivancea added a commit that referenced this pull request Nov 27, 2025
Fixes #138426

The telemetry feature was added on #137988
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ES|QL-ui Impacts ES|QL UI >non-issue :StorageEngine/ES|QL Timeseries / metrics / PromQL / logsdb capabilities in ES|QL Team:StorageEngine v9.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants