Skip to content

[Backport 2.x] Change default batch size of bulk API to Integer.MAX_VALUE#14881

Merged
gaobinlong merged 2 commits into2.xfrom
backport/backport-14725-to-2.x
Jul 23, 2024
Merged

[Backport 2.x] Change default batch size of bulk API to Integer.MAX_VALUE#14881
gaobinlong merged 2 commits into2.xfrom
backport/backport-14725-to-2.x

Conversation

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

@opensearch-trigger-bot opensearch-trigger-bot Bot commented Jul 22, 2024

Backport 97f26cc from #14725.

@github-actions
Deprecate batch_size parameter on bulk API (#14725)
50d32ab 
By default the full _bulk payload will be passed to ingest processors as a
batch, with any sub batching logic to be implemented by each processor if
necessary.

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
(cherry picked from commit 97f26cc)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 22, 2024

❌ Gradle check result for 50d32ab: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 22, 2024

❌ Gradle check result for 50d32ab: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@dblock
Copy link
Copy Markdown
Member

dblock commented Jul 22, 2024 

2024-07-22T21:43:09.9282635Z /home/runner/work/OpenSearch/OpenSearch/server/src/test/java/org/opensearch/ingest/IngestServiceTests.java:1988: error: cannot find symbol
2024-07-22T21:43:09.9284554Z 
2024-07-22T21:43:09.9285829Z         final List<Integer> dropHandler = new ArrayList<>();
2024-07-22T21:43:09.9286997Z                                               ^
2024-07-22T21:43:09.9288157Z   symbol:   class ArrayList
2024-07-22T21:43:09.9289050Z   location: class IngestServiceTests
2024-07-22T21:43:09.9290046Z > Task :server:compileTestJava

@dblock dblock mentioned this pull request Jul 22, 2024
Merged
3 tasks
@dblock
Copy link
Copy Markdown
Member

dblock commented Jul 22, 2024

Doesn't look like a fluke, @chishui

@chishui
Copy link
Copy Markdown
Contributor

chishui commented Jul 23, 2024 •
edited
Loading

@dblock import java.util.ArrayList; is missing in 2.x but in main, but I don't have permission to update backport PR, is it possible that you can help add this line to the file or grant me permission to this PR?

@gaobinlong
Fix test failure
0a97145 
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 23, 2024

❌ Gradle check result for 0a97145: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 23, 2024

✅ Gradle check result for 0a97145: SUCCESS

@codecov
Copy link
Copy Markdown

codecov Bot commented Jul 23, 2024

Codecov Report

Attention: Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 71.56%. Comparing base (7e7e775) to head (0a97145).
Report is 8 commits behind head on 2.x.

Files Patch % Lines
...pensearch/rest/action/document/RestBulkAction.java 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##                2.x   #14881      +/-   ##
============================================
- Coverage     71.59%   71.56%   -0.04%     
+ Complexity    62621    62537      -84     
============================================
  Files          5110     5110              
  Lines        293685   293651      -34     
  Branches      42801    42791      -10     
============================================
- Hits         210265   210151     -114     
- Misses        65801    65851      +50     
- Partials      17619    17649      +30

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gaobinlong gaobinlong merged commit ed24a14 into 2.x Jul 23, 2024
@gaobinlong gaobinlong deleted the backport/backport-14725-to-2.x branch July 23, 2024 06:43
kkewwei pushed a commit to kkewwei/OpenSearch that referenced this pull request Jul 24, 2024
@opensearch-trigger-bot @github-actions
@gaobinlong @kkewwei
[Backport 2.x] Change default batch size of bulk API to Integer.MAX_V…
8f67cbb 
…ALUE (opensearch-project#14881)

* Deprecate batch_size parameter on bulk API (opensearch-project#14725)

By default the full _bulk payload will be passed to ingest processors as a
batch, with any sub batching logic to be implemented by each processor if
necessary.

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
(cherry picked from commit 97f26cc)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix test failure

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: kkewwei <kkewwei@163.com>
This was referenced Sep 6, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrross andrross andrross approved these changes

@gaobinlong gaobinlong gaobinlong approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dblock @chishui @andrross @gaobinlong