[Search Pipelines] Support ad hoc pipelines by msfroh · Pull Request #7253 · opensearch-project/OpenSearch

msfroh · 2023-04-19T22:24:27Z

Description

This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state.

The new search source field is gated behind the search pipelines feature flag. Also, serialization of the field is gated behind a 3.0 version check. I don't think we should backport to the 2.x branch before we release 2.7. I think we'll want to update that version check to 2.8.

Issues Resolved

#6717

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-04-19T22:46:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14348/
CommitID: 5d05e5c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-19T22:58:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14350/
CommitID: bbd7c90
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-21T17:07:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/14416/
CommitID: 9469e77
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

msfroh · 2023-05-03T17:24:55Z

@andrross, @reta -- you two have context on Search Pipelines. Would either of you be able to take a look at this?

server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state. Signed-off-by: Michael Froh <froh@amazon.com>

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java

CHANGELOG.md

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

github-actions · 2023-05-04T20:36:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/14911/
CommitID: c0bdd3d

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

codecov-commenter · 2023-05-04T20:40:19Z

Codecov Report

❌ Patch coverage is 80.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.58%. Comparing base (2646d40) to head (fbd5321).
⚠️ Report is 3342 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/search/builder/SearchSourceBuilder.java	60.00%	7 Missing and 1 partial ⚠️
...nsearch/search/pipeline/SearchPipelineService.java	80.95%	3 Missing and 1 partial ⚠️
...pensearch/search/RandomSearchRequestGenerator.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #7253      +/-   ##
============================================
- Coverage     70.60%   70.58%   -0.03%     
+ Complexity    59582    59564      -18     
============================================
  Files          4877     4878       +1     
  Lines        285806   285844      +38     
  Branches      41165    41175      +10     
============================================
- Hits         201787   201756      -31     
- Misses        67446    67516      +70     
+ Partials      16573    16572       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

1. Create all streams in try-with-resources. 2. Keep methods in Pipeline package-private. 3. Remove spurious character in CHANGELOG. Also, I remembered that in a future change, I'm going to call the no-op pipeline "_none" (like when bypassing an ingest pipeline) so I should name it "_none" from the start. Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-05-04T23:21:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/14925/
CommitID: 24be027

reta · 2023-05-05T12:39:01Z

@andrross mind to take a look? thank you! ;-)

...ine-common/src/yamlRestTest/resources/rest-api-spec/test/search_pipeline/30_filter_query.yml

server/src/main/java/org/opensearch/search/pipeline/PipelinedRequest.java

CHANGELOG.md

- Added opensearch.internal annotation to PipelinedRequest. - Made PipelinedRequest final. - Removed TODO from PipelinedRequest. - Moved changelog entry to 2.x. Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-05-06T01:18:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/14970/
CommitID: fbd5321

opensearch-trigger-bot · 2023-05-06T01:52:10Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7253-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c43d7138636023315605f8a4caf91cf407c8753a
# Push it to GitHub
git push --set-upstream origin backport/backport-7253-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7253-to-2.x.

@reta

* [Search Pipelines] Support ad hoc pipelines This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state. Signed-off-by: Michael Froh <froh@amazon.com> * Added changelog entry Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta Resolve + transform request into PipelinedRequest, then use that to transform the search response. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate more feedback from @reta 1. Create all streams in try-with-resources. 2. Keep methods in Pipeline package-private. 3. Remove spurious character in CHANGELOG. Also, I remembered that in a future change, I'm going to call the no-op pipeline "_none" (like when bypassing an ingest pipeline) so I should name it "_none" from the start. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @andrross - Added opensearch.internal annotation to PipelinedRequest. - Made PipelinedRequest final. - Removed TODO from PipelinedRequest. - Moved changelog entry to 2.x. Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit c43d713)

@reta

* [Search Pipelines] Support ad hoc pipelines This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state. Signed-off-by: Michael Froh <froh@amazon.com> * Added changelog entry Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta Resolve + transform request into PipelinedRequest, then use that to transform the search response. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate more feedback from @reta 1. Create all streams in try-with-resources. 2. Keep methods in Pipeline package-private. 3. Remove spurious character in CHANGELOG. Also, I remembered that in a future change, I'm going to call the no-op pipeline "_none" (like when bypassing an ingest pipeline) so I should name it "_none" from the start. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @andrross - Added opensearch.internal annotation to PipelinedRequest. - Made PipelinedRequest final. - Removed TODO from PipelinedRequest. - Moved changelog entry to 2.x. Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit c43d713)

@reta

* [Search Pipelines] Support ad hoc pipelines (#7253) * [Search Pipelines] Support ad hoc pipelines This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state. Signed-off-by: Michael Froh <froh@amazon.com> * Added changelog entry Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta Resolve + transform request into PipelinedRequest, then use that to transform the search response. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate more feedback from @reta 1. Create all streams in try-with-resources. 2. Keep methods in Pipeline package-private. 3. Remove spurious character in CHANGELOG. Also, I remembered that in a future change, I'm going to call the no-op pipeline "_none" (like when bypassing an ingest pipeline) so I should name it "_none" from the start. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @andrross - Added opensearch.internal annotation to PipelinedRequest. - Made PipelinedRequest final. - Removed TODO from PipelinedRequest. - Moved changelog entry to 2.x. Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit c43d713) * Lower back-compatibility check to 2.8.0 Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com>

@reta

* [Search Pipelines] Support ad hoc pipelines This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state. Signed-off-by: Michael Froh <froh@amazon.com> * Added changelog entry Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta Resolve + transform request into PipelinedRequest, then use that to transform the search response. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate more feedback from @reta 1. Create all streams in try-with-resources. 2. Keep methods in Pipeline package-private. 3. Remove spurious character in CHANGELOG. Also, I remembered that in a future change, I'm going to call the no-op pipeline "_none" (like when bypassing an ingest pipeline) so I should name it "_none" from the start. Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @andrross - Added opensearch.internal annotation to PipelinedRequest. - Made PipelinedRequest final. - Removed TODO from PipelinedRequest. - Moved changelog entry to 2.x. Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

msfroh requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, dblock, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja, tlfeng and xuezhou25 as code owners April 19, 2023 22:24

msfroh force-pushed the ad_hoc_search_pipeline branch from bbd7c90 to 9469e77 Compare April 21, 2023 16:32

msfroh mentioned this pull request Apr 24, 2023

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

Merged

6 tasks

noCharger self-requested a review May 1, 2023 17:20

reta reviewed May 3, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/search/TransportSearchAction.java Outdated Show resolved Hide resolved

msfroh force-pushed the ad_hoc_search_pipeline branch from 9469e77 to c0bdd3d Compare May 4, 2023 20:03

msfroh commented May 4, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Outdated Show resolved Hide resolved