Support system ingest pipelines for bulk update operations by q-andy · Pull Request #18277 · opensearch-project/OpenSearch

q-andy · 2025-05-13T16:48:08Z

Description

Support system ingest pipelines for bulk update operations

Update Request Type Classification

Introduce a method to expose all child index requests associated with an update operation

Pipeline Resolution Enhancement

Use resolveSystemIngestPipeline to enable resolving only the system ingest pipeline while setting the others to NOOP
Based on update request fields, we extract the update request children and conditionally resolve ALL pipelines, resolve ONLY system ingest pipelines, or no pipelines at all.

Slot Management

Introduce childSlot to track individual child index requests within anupdate operation
Use innerslot to map pipeline execution results back to the correct child request using (slot, innerSlot) pairs
Maintain proper error handling and response mapping for both parent and child operations to their original bulk request slot

Related Issues

Resolves #18276
Partially addresses #17742

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-05-13T16:59:15Z

❌ Gradle check result for f8169f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-13T18:22:12Z

❌ Gradle check result for 94653c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-13T19:19:18Z

❌ Gradle check result for 959b59f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

server/src/main/java/org/opensearch/ingest/IngestService.java

github-actions · 2025-05-14T19:26:22Z

❌ Gradle check result for b2c9db0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T20:32:18Z

❌ Gradle check result for fbafb82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T21:14:47Z

❌ Gradle check result for 2341708: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T22:52:45Z

✅ Gradle check result for f5aa904: SUCCESS

codecov · 2025-05-14T22:53:10Z

Codecov Report

Attention: Patch coverage is 94.69697% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.74%. Comparing base (5255f5b) to head (d37361f).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...main/java/org/opensearch/ingest/IngestService.java	94.52%	1 Missing and 3 partials ⚠️
...rg/opensearch/action/bulk/TransportBulkAction.java	93.18%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18277      +/-   ##
============================================
+ Coverage     72.71%   72.74%   +0.02%     
- Complexity    68094    68136      +42     
============================================
  Files          5537     5537              
  Lines        313312   313375      +63     
  Branches      45460    45473      +13     
============================================
+ Hits         227813   227951     +138     
+ Misses        66967    66956      -11     
+ Partials      18532    18468      -64

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-06-02T23:40:56Z

❌ Gradle check result for 915a5a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

github-actions · 2025-06-04T21:51:18Z

❌ Gradle check result for 82d3c3f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andy Qin <qinandy@amazon.com>

github-actions · 2025-06-11T06:49:25Z

❌ Gradle check result for ff5d40d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-06-11T14:58:04Z

❌ Gradle check result for 3d53411: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

q-andy · 2025-06-11T15:21:44Z

flakey tests
#18485
#15840
#14407
#16658

github-actions · 2025-06-11T15:38:42Z

❌ Gradle check result for 3d53411: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…update-2

Signed-off-by: Andy Qin <qinandy@amazon.com>

q-andy · 2025-06-11T16:03:48Z

Added same fix for example mapping plugin yaml tests related to legacy templates

github-actions · 2025-06-11T17:00:01Z

✅ Gradle check result for d37361f: SUCCESS

…h-project#18277) * Support system ingest pipelines for bulk update operations Signed-off-by: Andy Qin <qinandy@amazon.com> * Spotless Signed-off-by: Andy Qin <qinandy@amazon.com> # Conflicts: # CHANGELOG.md * Refactor UpdateRequest type checking Signed-off-by: Andy Qin <qinandy@amazon.com> * Fix integ tests after rebase Signed-off-by: Andy Qin <qinandy@amazon.com> * Rename innerSlot to childSlot Signed-off-by: Andy Qin <qinandy@amazon.com> * Update yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update comments Signed-off-by: Andy Qin <qinandy@amazon.com> * Add allowed warning header for yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update warning headers Signed-off-by: Andy Qin <qinandy@amazon.com> --------- Signed-off-by: Andy Qin <qinandy@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

…h-project#18277) * Support system ingest pipelines for bulk update operations Signed-off-by: Andy Qin <qinandy@amazon.com> * Spotless Signed-off-by: Andy Qin <qinandy@amazon.com> # Conflicts: # CHANGELOG.md * Refactor UpdateRequest type checking Signed-off-by: Andy Qin <qinandy@amazon.com> * Fix integ tests after rebase Signed-off-by: Andy Qin <qinandy@amazon.com> * Rename innerSlot to childSlot Signed-off-by: Andy Qin <qinandy@amazon.com> * Update yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update comments Signed-off-by: Andy Qin <qinandy@amazon.com> * Add allowed warning header for yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update warning headers Signed-off-by: Andy Qin <qinandy@amazon.com> --------- Signed-off-by: Andy Qin <qinandy@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com>

…h-project#18277) * Support system ingest pipelines for bulk update operations Signed-off-by: Andy Qin <qinandy@amazon.com> * Spotless Signed-off-by: Andy Qin <qinandy@amazon.com> # Conflicts: # CHANGELOG.md * Refactor UpdateRequest type checking Signed-off-by: Andy Qin <qinandy@amazon.com> * Fix integ tests after rebase Signed-off-by: Andy Qin <qinandy@amazon.com> * Rename innerSlot to childSlot Signed-off-by: Andy Qin <qinandy@amazon.com> * Update yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update comments Signed-off-by: Andy Qin <qinandy@amazon.com> * Add allowed warning header for yaml tests Signed-off-by: Andy Qin <qinandy@amazon.com> * Update warning headers Signed-off-by: Andy Qin <qinandy@amazon.com> --------- Signed-off-by: Andy Qin <qinandy@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

q-andy requested a review from a team as a code owner May 13, 2025 16:48

github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing labels May 13, 2025

q-andy force-pushed the system-pipeline-bulk-update-2 branch from 94653c1 to 959b59f Compare May 13, 2025 18:41

opensearch-ci-bot mentioned this pull request May 13, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchServiceTests #18238

Closed

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java Outdated Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java Outdated Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/ingest/IngestService.java Outdated Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/ingest/IngestService.java Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/ingest/IngestService.java Outdated Show resolved Hide resolved

bzhangam reviewed May 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/ingest/IngestService.java Show resolved Hide resolved

opensearch-ci-bot mentioned this pull request May 14, 2025

[AUTOCUT] Gradle Check Flaky Test Report for QueueResizableOpenSearchThreadPoolExecutorTests #14297

Open

q-andy force-pushed the system-pipeline-bulk-update-2 branch from b2c9db0 to fbafb82 Compare May 14, 2025 20:21

q-andy force-pushed the system-pipeline-bulk-update-2 branch from fbafb82 to 2341708 Compare May 14, 2025 21:04

q-andy force-pushed the system-pipeline-bulk-update-2 branch from 2341708 to f5aa904 Compare May 14, 2025 21:39

opensearch-ci-bot mentioned this pull request May 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SimpleSecureNetty4TransportTests #18071

Open

q-andy force-pushed the system-pipeline-bulk-update-2 branch from f5aa904 to 915a5a2 Compare June 2, 2025 23:31

bzhangam reviewed Jun 4, 2025

View reviewed changes

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java Outdated Show resolved Hide resolved

q-andy force-pushed the system-pipeline-bulk-update-2 branch from 82d3c3f to 1d83672 Compare June 4, 2025 22:46

Add allowed warning header for yaml tests

ff5d40d

Signed-off-by: Andy Qin <qinandy@amazon.com>

opensearch-ci-bot mentioned this pull request Jun 11, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

q-andy closed this Jun 11, 2025

q-andy reopened this Jun 11, 2025

q-andy closed this Jun 11, 2025

q-andy reopened this Jun 11, 2025

q-andy added 2 commits June 11, 2025 08:53

Merge remote-tracking branch 'origin/main' into system-pipeline-bulk-…

58230c1

…update-2

Update warning headers

d37361f

Signed-off-by: Andy Qin <qinandy@amazon.com>

q-andy force-pushed the system-pipeline-bulk-update-2 branch from 3d53411 to d37361f Compare June 11, 2025 15:53

owaiskazi19 merged commit 8a59ca1 into opensearch-project:main Jun 11, 2025
30 checks passed

This was referenced Jun 11, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Open

[AUTOCUT] Gradle Check Flaky Test Report for ShardIndexingPressureIT #15830

Closed

[AUTOCUT] Gradle Check Flaky Test Report for InternalEngineTests #15838

Open

q-andy mentioned this pull request Jun 12, 2025

Fix ingest document wrapper constructor compilation errors in unit tests opensearch-project/neural-search#1383

Merged

5 tasks

BrewTestBot mentioned this pull request Jun 24, 2025

opensearch 3.1.0 Homebrew/homebrew-core#228059

Merged

opensearch-ci-bot mentioned this pull request Jun 27, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ClientYamlTestSuiteIT #14319

Open

This was referenced Aug 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for QueryPhaseTests #14551

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteIndexShardTests #16116

Open

This was referenced Jan 10, 2026

docs: add system-ingest-pipeline report for v3.1.0 tkykenmt/opensearch-feature-explorer#947

Merged

[feature] System Ingest Pipeline tkykenmt/opensearch-feature-explorer#912

Closed

Conversation

q-andy commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

codecov bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

Uh oh!

github-actions bot commented Jun 4, 2025

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

q-andy commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

q-andy commented Jun 11, 2025

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

q-andy commented May 13, 2025 •

edited

Loading

codecov bot commented May 14, 2025 •

edited

Loading

q-andy commented Jun 11, 2025 •

edited

Loading