Skip to content

Fix bulk API ignores ingest pipeline for upsert#12883

Merged
reta merged 7 commits intoopensearch-project:mainfrom
gaobinlong:fix_pipeline
Apr 3, 2024
Merged

Fix bulk API ignores ingest pipeline for upsert#12883
reta merged 7 commits intoopensearch-project:mainfrom
gaobinlong:fix_pipeline

Conversation

@gaobinlong
Copy link
Copy Markdown
Contributor

@gaobinlong gaobinlong commented Mar 24, 2024 •
edited
Loading

Description

When there's an ingest pipeline specified in the bulk update action with upsert, bulk API doesn't set the pipeline in upsertRequest correctly, so that the specified pipeline will not be executed. This PR fixes the bug and adds a yaml test case for it.

Related Issues

#12854

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gaobinlong
Fix bulk API ignores ingest pipeline for upsert
c2a7820 
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@gaobinlong
Modify change log
481070f 
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2024 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change b19c271

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/sql.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2024

❌ Gradle check result for 481070f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2024

❌ Gradle check result for c2a7820: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2024

✅ Gradle check result for a26af14: SUCCESS

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 25, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.37%. Comparing base (b15cb0c) to head (b19c271).
Report is 99 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #12883      +/-   ##
============================================
- Coverage     71.42%   71.37%   -0.05%     
- Complexity    59978    60233     +255     
============================================
  Files          4985     5015      +30     
  Lines        282275   283826    +1551     
  Branches      40946    41134     +188     
============================================
+ Hits         201603   202576     +973     
- Misses        63999    64436     +437     
- Partials      16673    16814     +141

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gaobinlong
Use existing pipeline instead
3a18815 
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2024

❕ Gradle check result for 3a18815: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testZeroLagOnCreateIndex

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@gaobinlong
merge main
60ef6a1 
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 26, 2024

❌ Gradle check result for 60ef6a1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 27, 2024

✅ Gradle check result for b19c271: SUCCESS

@gaobinlong
Copy link
Copy Markdown
Contributor Author

gaobinlong commented Apr 3, 2024

Could any maintainer help to review this PR? @reta, @andrross, @dblock , thank you!

@reta
Copy link
Copy Markdown
Contributor

reta commented Apr 3, 2024

Could any maintainer help to review this PR? @reta, @andrross, @dblock , thank you!

Looking, sorry about that @gaobinlong

@reta reta merged commit f2e8398 into opensearch-project:main Apr 3, 2024
@reta reta added bug Something isn't working backport 2.x Backport to 2.x branch labels Apr 3, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 3, 2024
@github-actions
Fix bulk API ignores ingest pipeline for upsert (#12883)
e6ebd8a 
* Fix bulk API ignores ingest pipeline for upsert

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Use existing pipeline instead

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
(cherry picked from commit f2e8398)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Apr 3, 2024
Merged
reta added a commit that referenced this pull request Apr 3, 2024
@opensearch-trigger-bot @github-actions @reta
[Backport 2.x] Fix bulk API ignores ingest pipeline for upsert (#13055)
90907d4 
* Fix bulk API ignores ingest pipeline for upsert (#12883)

* Fix bulk API ignores ingest pipeline for upsert

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Use existing pipeline instead

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
(cherry picked from commit f2e8398)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Update 70_bulk.yml

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
@gaobinlong gaobinlong mentioned this pull request Apr 25, 2024
Closed
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@gaobinlong @shiv0408
Fix bulk API ignores ingest pipeline for upsert (opensearch-project#1…
16f79b7 
…2883)

* Fix bulk API ignores ingest pipeline for upsert

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Use existing pipeline instead

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
@BrewTestBot BrewTestBot mentioned this pull request May 14, 2024
Merged
@reta reta mentioned this pull request Jul 17, 2024
Merged
3 tasks
@mgodwan mgodwan mentioned this pull request Mar 3, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@reta reta reta approved these changes

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@andrross andrross Awaiting requested review from andrross

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal

@mch2 mch2 Awaiting requested review from mch2

@msfroh msfroh Awaiting requested review from msfroh

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@sachinpkale sachinpkale Awaiting requested review from sachinpkale

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli

@shwetathareja shwetathareja Awaiting requested review from shwetathareja

@sohami sohami Awaiting requested review from sohami

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah

+1 more reviewer

@kkmr kkmr kkmr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gaobinlong @reta @kkmr