Merged
Conversation
Add PromQL support in ESQL under the PROMQL command. This commit includes the grammar, associated logical plan and basic rules for performing translation, from filtering to stats transpilation. * Grammar Stand-alone grammar definition and parser, currently 'embedded' inside ESQL (the current scenario). Grammar include to pass the valid tests (~200 queries) from Prometheus 3.5; the invalid ones are disabled for now since the parser needs to validate the syntax using * Transpilation Implement basic PromQL plan translation: Selector → Filter with label matcher conditions RangeSelector → Filter + Bucket for time bucketing WithinSeriesAggregate → ESQL function over selector AcrossSeriesAggregate → TimeSeriesAggregate with groupings
Extended PromQL grammar to support optional time-range parameters
before the query expression. Parameters are specified as
space-separated name-value pairs (e.g., "step 5m")
and can appear in any order.
Currently only literals and field references are allowed, expressions
such as (now() - 1h) are not supported.
The query itself is enclosed in parentheses to enable future assignment
syntax like "promql x=(...)" while avoiding ambiguity.
Implementation details:
- Introduced two-mode lexer architecture:
- PROMQL_PARAMS_MODE: Captures parameter name-value pairs using
generic identifiers (PROMQL_UNQUOTED_IDENTIFIER or
QUOTED_IDENTIFIER)
- PROMQL_QUERY_MODE: Captures query text with depth tracking for
balanced parentheses. Skipping then ends up with ambiguitities
error when dealing with nested queries or having promql inside
explain
- Parser uses recursive rule (promqlQueryPart) to handle nested
parentheses in PromQL expressions without limiting nesting depth
- Added depth tracking methods in LexerConfig
(incPromqlDepth, decPromqlDepth, resetPromqlDepth, isPromqlQuery)
to distinguish between query-closing and query-internal
parentheses
- Query text extraction preserves original source verbatim using parse
tree token positions
Examples:
promql step 5m (rate(http_requests[5m]))
promql time now (up {job="api"})
promql start ?start_time step ?interval (sum(network_bytes))
This is to align more closely with the query parameter names for the prometheus range_query API.
This is to avoid ambiguity with org.elasticsearch.xpack.esql.parser.LogicalPlanBuilder
Previously Vector operations were defined as Expressions which didn't match the actual semantics not the rest of the parsing infra. This has now been addressed by making the nodes LogicalPlans.
Remove GrammarTests in favor of PromqlAstTests to test not just the grammar but also the semantic validation of the queries, especially as the latter includes the former. The invalid tests are currently passing, but there are still some valid tests that are failing, likely due to incorrect AST assembly.
Refactor Subquery from Expression into LogicalPlan. Duration parsing is wip.
Unify the logic for arithmetics between scalars alone and scalars and time durations
To simplify date arithmetic and since the time unit is not necessary, TimeValue has been replaced with java.time.Duration both in the parser and throughout the AST
Parser now performs comparison folding for literals. Paranthesized expressions are now allowed. In the process refactored the folding code to make it more compact.
costin
commented
Nov 18, 2025
Member
Author
There was a problem hiding this comment.
stratoula Lexer for the nested promql ESQL command
costin
commented
Nov 18, 2025
Member
Author
There was a problem hiding this comment.
stratoula Parser for the nested promql ESQL command
Collaborator
|
Pinging @elastic/kibana-esql (ES|QL-ui) |
Member
Author
|
@elasticsearchmachine run elasticsearch-ci/part-2 |
felixbarny
approved these changes
Nov 20, 2025
Member
Author
|
To review the PR individual history, check the log of the PR itself : |
not-napoleon
added a commit
that referenced
this pull request
Nov 24, 2025
Part of #137988 Implement #138349 for the t-digest field type. This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
afoucret
pushed a commit
to afoucret/elasticsearch
that referenced
this pull request
Nov 26, 2025
Part of elastic#137988 Implement elastic#138349 for the t-digest field type. This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
ncordon
pushed a commit
to ncordon/elasticsearch
that referenced
this pull request
Nov 26, 2025
* Introduce PromQL command Add PromQL support in ESQL under the PROMQL command. This commit includes the grammar, associated logical plan and basic rules for performing translation, from filtering to stats transpilation. * Grammar Stand-alone grammar definition and parser, currently 'embedded' inside ESQL (the current scenario). Grammar include to pass the valid tests (~200 queries) from Prometheus 3.5; the invalid ones are disabled for now since the parser needs to validate the syntax using * Transpilation Implement basic PromQL plan translation: Selector → Filter with label matcher conditions RangeSelector → Filter + Bucket for time bucketing WithinSeriesAggregate → ESQL function over selector AcrossSeriesAggregate → TimeSeriesAggregate with groupings --------- Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
ncordon
pushed a commit
to ncordon/elasticsearch
that referenced
this pull request
Nov 26, 2025
Part of elastic#137988 Implement elastic#138349 for the t-digest field type. This changes the behavior of the sum sub-field of t-digest when the digest is empty. Prior to this PR we treated it as 0, and this changes it to be null. This avoids a division by zero error in ESQL when trying to calculate the average of an empty histogram. We are adopting the same behavior for the exponential histogram field (see PR linked above), and this is important to keep the semantics of the two fields as close as possible.
ivancea
added a commit
that referenced
this pull request
Nov 27, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implementation adds native PromQL (Prometheus Query Language support to ESQL through a new PROMQL command, enabling users to query time-series data using familiar Prometheus syntax by leveraging the time-series/TS support available in ESQL.
The capability is available behind a feature flag as it's still wip yet mature enough to be merged into main.
This query starts from the k8s time-series index, calculates the average of network.bytes_in over 1-hour windows for specific pods, aggregates by pod dimension with a 1-hour step, and limits results to 1000 rows.
Functionality overview
The current PROMQL command supports the following PromQL functionality:
http_requests_total{job="apiserver", handler="/api/comments"})http_requests_total[5m])rate(),irate(),increase(),delta(), etcavg_over_time(), sum_over_time()`, etc..sum by(),avg by(), etc...Current Limitations
This initial implementation focuses on time-series aggregation queries. Advanced PromQL features like label manipulation, instant-vector transformations, mathematical functions, and certain query modifiers are not yet supported.