Skip to content

Pinned retriever#126401

Merged
mridula-s109 merged 100 commits intomainfrom
pinned-retriever
May 1, 2025
Merged

Pinned retriever#126401
mridula-s109 merged 100 commits intomainfrom
pinned-retriever

Conversation

@mridula-s109
Copy link
Copy Markdown
Contributor

@mridula-s109 mridula-s109 commented Apr 7, 2025

Pinned Retriever for Elasticsearch

This PR introduces a new pinned retriever for Elasticsearch, available under the search-business-rules plugin.

The pinned retriever allows users to explicitly pin specific documents (via ids or docs) so they always appear at the top of the search results, regardless of their organic relevance score. It is implemented as a compound retriever that rewrites internally to a PinnedQueryBuilder.


Key Features

  • Pin Documents by ID or Full Doc: Pin specific documents either by their _id or by providing inline doc content.
  • Custom Explanation Support: When used with "explain": true, the search response now includes a custom _explanation field for pinned results.
    Example:
    "description": "Pinned document by ids, original explanation:"
    

Example API Request

POST my-index/_search
{
  "explain": true,
  "retriever": {
    "pinned": {
      "ids": ["1", "2"],
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "title": "machine learning"
            }
          }
        }
      }
    }
  }
}

Sample Response

"_explanation": {
  "value": 1.7014124e+38,
  "description": "Pinned document, original explanation:",
  "details": [
    {
      "value": 1,
      "description": "doc [0] with an original score of [1.7014124E38] is at rank [1] from the following source queries.",
      "details": [
        {
          // additional explanation details...
        }
      ]
    }
  ]
}

@mridula-s109 mridula-s109 added >enhancement auto-backport Automatically create backport pull requests when merged :SearchOrg/Relevance Label for the Search (solution/org) Relevance team labels Apr 7, 2025
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @mridula-s109, I've created a changelog YAML for you.

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/search-eng (Team:SearchOrg)

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/search-relevance (Team:Search - Relevance)

@mridula-s109 mridula-s109 enabled auto-merge (squash) April 7, 2025 21:43
@mridula-s109 mridula-s109 requested review from a team and kderusso April 7, 2025 21:44
Copy link
Copy Markdown
Member

@kderusso kderusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick pass through, I'm surprised this works without registering the retriever?

@mridula-s109 mridula-s109 marked this pull request as draft April 8, 2025 09:44
auto-merge was automatically disabled April 8, 2025 09:44

Pull request was converted to draft

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @mridula-s109, I've created a changelog YAML for you.

kderusso and others added 2 commits April 30, 2025 14:17
* Fix compilation error

* Remove SearchPlugin from META-INF

* Remove duplicate FeatureSpecification

* Add service as test resource

* Move to test

* More file moving

* Remove from module-info, remove file

* Make search business rules plugin extensible

* fix ent search plugin

* Delete dup files

* [CI] Auto commit changes from spotless

* Add module info back in

---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Copy link
Copy Markdown
Member

@kderusso kderusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some hopefully final feedback! Thanks for your iterations on this.
As a best practice it would be good to request review from the Search Relevance team so everyone on the team is aware of this PR.

Awaiting CI and final changes but this is really close!

@mridula-s109
Copy link
Copy Markdown
Contributor Author

Some hopefully final feedback! Thanks for your iterations on this. As a best practice it would be good to request review from the Search Relevance team so everyone on the team is aware of this PR.

Awaiting CI and final changes but this is really close!

I have resolved all the comments @kderusso , hopefully the ci build passes through and everything works as intended.

Copy link
Copy Markdown
Member

@ioanatia ioanatia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I tested it locally.
Checked the comments from @kderusso from the last review and it looks to me that all have been addressed.

public static final TransportVersion INTRODUCE_FAILURES_LIFECYCLE = def(9_065_0_00);
public static final TransportVersion PROJECT_METADATA_SETTINGS = def(9_066_00_0);
public static final TransportVersion AGGREGATE_METRIC_DOUBLE_BLOCK = def(9_067_00_0);
public static final TransportVersion PINNED_RETRIEVER = def(9_068_0_00);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a side note - but when we backport this to 8.19, we need to pay extra attention on how we backport this transport version

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah @mridula-s109 - we will create a PINNED_RETRIEVER_8_19 transport version

Copy link
Copy Markdown
Member

@kderusso kderusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for all the iterations!

public static final TransportVersion INTRODUCE_FAILURES_LIFECYCLE = def(9_065_0_00);
public static final TransportVersion PROJECT_METADATA_SETTINGS = def(9_066_00_0);
public static final TransportVersion AGGREGATE_METRIC_DOUBLE_BLOCK = def(9_067_00_0);
public static final TransportVersion PINNED_RETRIEVER = def(9_068_0_00);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah @mridula-s109 - we will create a PINNED_RETRIEVER_8_19 transport version

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

💔 Backport failed

Status Branch Result
8.19 Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 126401

@mridula-s109
Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
8.19

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged backport pending >enhancement :SearchOrg/Relevance Label for the Search (solution/org) Relevance team v8.19.0 v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants