Fix remove ingest processor handing ignore_missing parameter not correctly#10089

Merged

msfroh merged 10 commits intoopensearch-project:mainfrom

gaobinlong:fix_remove_processor

Sep 25, 2023

Contributor

gaobinlong commented Sep 18, 2023 •

edited by msfroh

Loading

Description

Remove ingest processor does not handle ignore_missing parameter correctly, when the resolved filed path of template snippet like {{field}} is empty string and ignore_missing is true, the processor does not exit quietly, but throw an illegal_argument_exception, this should be a bug.

The main change of this PR:

Fix the bug
Add some yml test and unit test

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.


          Fix remove ingest processor handing ignore_missing parameter not corr…

…ectly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

gaobinlong requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, peternied, reta, ryanbogan, sachinpkale, saratvemulapalli, setiah, shwetathareja, sohami and tlfeng as code owners

September 18, 2023 02:14

Contributor

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25757/
CommitID: 6609905
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented Sep 18, 2023 •

edited

Loading

Compatibility status:

Checks if related components are compatible with change 2042c87

Incompatible components

Incompatible components: [https://github.com/opensearch-project/k-nn.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]


          modify change log

18940b5

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Contributor

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25790/
CommitID: 18940b5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta reviewed

View reviewed changes

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java Outdated

reta reviewed

View reviewed changes

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java Outdated

gaobinlong added 2 commits

September 19, 2023 11:32


          Fix test failure and optimize some code

fd07fdf

Signed-off-by: Gao Binlong <gbinlong@amazon.com>


          Merge remote-tracking branch 'upstream/main' into fix_remove_processor

eaa9366

Contributor

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25829/
CommitID: eaa9366
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

msfroh reviewed

View reviewed changes

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java Outdated

gaobinlong added 2 commits

September 22, 2023 22:27


          merge main

05b8b2f

Signed-off-by: Gao Binlong <gbinlong@amazon.com>


          Minor change

2ef6638

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Contributor

github-actions bot commented Sep 22, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26120/
CommitID: 2ef6638
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

gaobinlong added 2 commits

September 23, 2023 09:46


          Modify test code

e0fbcf6

Signed-off-by: Gao Binlong <gbinlong@amazon.com>


          Merge remote-tracking branch 'upstream/main' into fix_remove_processor

cebeb23

reta reviewed

View reviewed changes

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java

reta approved these changes

View reviewed changes

Contributor

github-actions bot commented Sep 23, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26171/
CommitID: cebeb23
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

gaobinlong added 2 commits

September 24, 2023 17:19


          Merge remote-tracking branch 'upstream/main' into fix_remove_processor

ba6946b


          Split a test method to two

2042c87

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Contributor

github-actions bot commented Sep 24, 2023

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore

URL: https://build.ci.opensearch.org/job/gradle-check/26192/
CommitID: 2042c87
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov bot commented Sep 24, 2023

Codecov Report

Merging #10089 (2042c87) into main (1dde018) will decrease coverage by 0.20%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #10089      +/-   ##
============================================
- Coverage     71.22%   71.02%   -0.20%     
+ Complexity    58240    58138     -102     
============================================
  Files          4828     4828              
  Lines        274419   274423       +4     
  Branches      39989    39991       +2     
============================================
- Hits         195443   194905     -538     
- Misses        62574    63195     +621     
+ Partials      16402    16323      -79

Files Changed	Coverage Δ
.../org/opensearch/ingest/common/RemoveProcessor.java	`96.96% <100.00%> (+7.31%)`	⬆️

... and 454 files with indirect coverage changes

msfroh merged commit c44337e into opensearch-project:main

reta added the backport 2.x label

opensearch-trigger-bot bot mentioned this pull request

[Backport 2.x] Fix remove ingest processor handing ignore_missing parameter not correctly #10218

Merged

opensearch-trigger-bot bot pushed a commit that referenced this pull request


          Fix remove ingest processor handing ignore_missing parameter not corr…

825c09c

…ectly (#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
(cherry picked from commit c44337e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

reta pushed a commit that referenced this pull request


          Fix remove ingest processor handing ignore_missing parameter not corr…

643a1bb

…ectly (#10089) (#10218)

* Fix remove ingest processor handing ignore_missing parameter not correctly



* modify change log



* Fix test failure and optimize some code



* Minor change



* Modify test code



* Split a test method to two



---------


(cherry picked from commit c44337e)

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

brusic pushed a commit to brusic/OpenSearch that referenced this pull request


          Fix remove ingest processor handing ignore_missing parameter not corr…

7de7dac

…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

gaobinlong mentioned this pull request

[BUG] Remove ingest processor doesn't handle ignore_missing parameter correctly #10066

Closed

vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request


          Fix remove ingest processor handing ignore_missing parameter not corr…

d643d1e

…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request


          Fix remove ingest processor handing ignore_missing parameter not corr…

ff080fa

…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

msfroh msfroh left review comments

reta reta approved these changes

anasalkouz Awaiting requested review from anasalkouz

andrross Awaiting requested review from andrross

Bukhtawar Awaiting requested review from Bukhtawar

CEHENKLE Awaiting requested review from CEHENKLE

dblock Awaiting requested review from dblock

gbbafna Awaiting requested review from gbbafna

setiah Awaiting requested review from setiah

kartg Awaiting requested review from kartg

kotwanikunal Awaiting requested review from kotwanikunal

mch2 Awaiting requested review from mch2

nknize Awaiting requested review from nknize

owaiskazi19 Awaiting requested review from owaiskazi19

peternied Awaiting requested review from peternied

Rishikesh1159 Awaiting requested review from Rishikesh1159

ryanbogan Awaiting requested review from ryanbogan

saratvemulapalli Awaiting requested review from saratvemulapalli

shwetathareja Awaiting requested review from shwetathareja

dreamer-89 Awaiting requested review from dreamer-89

tlfeng Awaiting requested review from tlfeng

VachaShah Awaiting requested review from VachaShah

dbwiddis Awaiting requested review from dbwiddis

sachinpkale Awaiting requested review from sachinpkale

sohami Awaiting requested review from sohami

Labels