ES|QL: Configurable score, key and group by columns for FUSE by ioanatia · Pull Request #135079 · elastic/elasticsearch

ioanatia · 2025-09-19T11:51:45Z

tracked in #123389

FUSE uses:

a column for scoring (_score by default)
a column that designates the result group from which the row originates (_fork by default)
a list of columns to merge rows together (_id, _index by default)

We want to be able to configure these with non-default columns.
We extend the syntax to be able to support:

FUSE linear SCORE BY my_score GROUP BY my_fork KEY BY my_index,my_id WITH { "normalizer": "minmax" }

SCORE BY/GROUP BY/KEY BY/WITH can be used in any order.

ioanatia · 2025-09-19T12:55:43Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/fuse/Fuse.java

+
+    @Override
+    public void postAnalysisVerification(Failures failures) {
+        if (score.dataType() != DataType.DOUBLE) {


I made a note in #123389 that as a follow up, we might want to accept any numeric type for the score column.
I have not done this yet, since (1) the operator expects it to a double and (2) it would mean refactoring FuseScoreEval/FuseScoreEvalExec to be able to receive both an input score attribute and an output score attribute.

I think this is fine, users could convert it with TO_DOUBLE if needed

ioanatia · 2025-09-19T13:00:06Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/fuse.csv-spec

+;
+
+
+fuseWithRowAndRRF


I added some tests at the end that are not using FORK, we did not have any before.
For FUSE we always wanted to support the case where it's not used in combination with FORK.

The other aspect is that I wanted some tests where the scores are more predictable and we are not using full text search.

elasticsearchmachine · 2025-09-19T13:01:30Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

afoucret · 2025-09-19T13:26:25Z

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4


 fuseCommand
-    : DEV_FUSE (fuseType=identifier)? fuseOptions=commandNamedParameters
+    : DEV_FUSE (fuseType=identifier)? (SCORE BY score=qualifiedName)? (KEY BY key=fields)? (GROUP BY group=qualifiedName)? fuseOptions=commandNamedParameters


I am a little bit skeptical with the syntax here.

My first concern is that it requires a strict ordering of the optional clauses which is a little bit trappy for the user.
One could not understand why the following is not working:

| FUSE RRF KEY BY key column SCORE BY my_score WITH { ... custom options }

I wonder if those parameter should just be part of the commandNamedParameters:

| FUSE RRF WITH {"key_column": "my_key", "score_column": "my_score", ... ... custom options }\

Also we were asked that all the column creation are using the assignment pattern, so it should probably be something like:

| FUSE my_score=RRF WITH {"key_column": "my_key", ... ... custom options }

I agree with @afoucret . We may want to have the options in any order, and just fail the parsing if we're repeating an option.

I get the skepticism, but this is the syntax we agreed we will implement for FUSE with the larger group.
We discussed even the option of having the columns as part of MapExpression as you suggest.
In the end, we went with the syntax that you see implemented here.

We went through many iterations to get to this shape, this PR is simply implementing what we agreed with.

In regards with the syntax being a bit too rigid that it expects a certain order, I think we can follow up on it so we make SCORE BY/KEY BY/GROUP BY interchangeable.

carlosdelest

Looks good!

I think a follow up would be nice with grammar changes so KEY BY, GROUP BY can be used in any order.

carlosdelest · 2025-09-19T14:12:00Z

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4


 fuseCommand
-    : DEV_FUSE (fuseType=identifier)? fuseOptions=commandNamedParameters
+    : DEV_FUSE (fuseType=identifier)? (SCORE BY score=qualifiedName)? (KEY BY key=fields)? (GROUP BY group=qualifiedName)? fuseOptions=commandNamedParameters


I agree with @afoucret . We may want to have the options in any order, and just fail the parsing if we're repeating an option.

carlosdelest · 2025-09-19T14:14:55Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/fuse/Fuse.java

+
+    @Override
+    public void postAnalysisVerification(Failures failures) {
+        if (score.dataType() != DataType.DOUBLE) {


I think this is fine, users could convert it with TO_DOUBLE if needed

carlosdelest · 2025-09-19T14:15:46Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/fuse/Fuse.java

+            failures.add(
+                fail(
+                    discriminator,
+                    "expected GROUP BY field [{}] to be KEYWORD or TEXT, not {}",


Is it necessary that we group / key by text or keyword? Why not use any type? At least numeric types could make sense to me.

Of course we can do that in a follow up, just curious about your thinking

I think this is a restriction we have on the operator where we expect the discriminator values to be strings.
but I think we can lift this later.

carlosdelest · 2025-09-19T14:20:27Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

Should we add a test to ensure the parameters are passed down to the Fuse plan?

…ble_columns

elasticsearchmachine · 2025-09-22T10:14:53Z

Pinging @elastic/kibana-esql (ES|QL-ui)

ioanatia · 2025-09-22T12:39:33Z

@carlosdelest I added statement parser tests and made sure that GROUP BY/KEY BY/SCORE BY are interchangeable

carlosdelest

Thanks for adding support for unordered clauses! 💯

…#135079)

Configurable score,key and group by columns for FUSE

d8b80df

ioanatia added >non-issue Team:Search - Relevance The Search organization Search Relevance team v9.2.0 :Search Relevance/ES|QL Search functionality in ES|QL labels Sep 19, 2025

ioanatia mentioned this pull request Sep 19, 2025

ES|QL: Hybrid search with FUSE (GA) #123389

Open

17 tasks

ioanatia commented Sep 19, 2025

View reviewed changes

ioanatia requested review from afoucret and carlosdelest September 19, 2025 13:00

ioanatia marked this pull request as ready for review September 19, 2025 13:01

elasticsearchmachine added Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch and removed Team:Search - Relevance The Search organization Search Relevance team labels Sep 19, 2025

afoucret reviewed Sep 19, 2025

View reviewed changes

ioanatia requested a review from afoucret September 19, 2025 14:25

carlosdelest approved these changes Sep 19, 2025

View reviewed changes

ioanatia added 3 commits September 22, 2025 10:44

Merge remote-tracking branch 'elasticsearch/main' into fuse_configura…

a67776f

…ble_columns

Add statement parser tests

8c76d5d

Allow GROUP BY/KEY BY/SCORE BY/WITH to be in any order

4c9e26c

ioanatia added the ES|QL-ui Impacts ES|QL UI label Sep 22, 2025

fix test

dfaf6c4

ioanatia requested a review from carlosdelest September 22, 2025 12:38

carlosdelest approved these changes Sep 22, 2025

View reviewed changes

ioanatia merged commit fe4c41e into elastic:main Sep 22, 2025
34 checks passed

ioanatia deleted the fuse_configurable_columns branch September 22, 2025 15:35

gmjehovich pushed a commit to gmjehovich/elasticsearch that referenced this pull request Sep 22, 2025

ES|QL: Configurable score, key and group by columns for FUSE (elastic…

dfa9d8e

…#135079)

DonalEvans pushed a commit to DonalEvans/elasticsearch that referenced this pull request Sep 22, 2025

ES|QL: Configurable score, key and group by columns for FUSE (elastic…

b65b2ba

…#135079)

		;


		fuseWithRowAndRRF

Conversation

ioanatia commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Sep 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ioanatia Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Sep 22, 2025

Uh oh!

ioanatia commented Sep 22, 2025

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ioanatia commented Sep 19, 2025 •

edited

Loading

ioanatia Sep 19, 2025 •

edited

Loading