Add copy ingest processor by gaobinlong · Pull Request #11870 · opensearch-project/OpenSearch

gaobinlong · 2024-01-12T13:57:59Z

Description

This PR adds a new ingest processor called copy processor which can copy the whole object from one existing field to another field, this is useful when users want to copy a nested field to the root level and then delete the original field, this cannot be achieved by set processor because it doesn't support copying object, only basic data types are supported. In addition, even though script processor can be used to copy object, but writing painless script is not easy for users. The copy processor provides an easy way to copy object.

The usage of copy processor are as follows:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "copy": {
          "source_field": "{{foo}}", 
          "target_field":"target",
          "ignore_missing":true,
          "override_target":true,
          "remove_source":true
        }
      }
    ]
  },
  "docs": [
    {
      "_version_type":"external_gte",
      "_version": 1,
      "_source": {
        "foo":"a", 
        "a": {
          "c":"1"
        },
        "target":1
      }
    }
  ]
}

, the result is:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "1",
        "_version_type": "external_gte",
        "_source": {
          "foo": "a",
          "target": {
            "c": "1"
          }
        },
        "_ingest": {
          "timestamp": "2024-01-12T13:44:27.287123Z"
        }
      }
    }
  ]
}

.
Both the source_field and target_field support template snippets, and there are three extra parameters:

ignore_missing: if true, exit quietly when source_field doesn't exist or has a empty field path, defaults to false
override_target: if true, override the value of target_field if it already exists, defaults to false
remove_source: if true, remove the source_field after the copy operation, defaults to false

Related Issues

#10134

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

github-actions · 2024-01-12T14:13:36Z

Compatibility status:

Checks if related components are compatible with change 441fde8

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git]

github-actions · 2024-01-12T14:19:22Z

❌ Gradle check result for d8edaf3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-12T14:26:30Z

❌ Gradle check result for 0595d7e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

github-actions · 2024-01-12T15:32:55Z

❕ Gradle check result for 441fde8: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-01-12T15:34:31Z

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (5c82ab8) 71.43% compared to head (441fde8) 71.45%.

Files	Patch %	Lines
...va/org/opensearch/ingest/common/CopyProcessor.java	93.61%	3 Missing ⚠️
...search/ingest/common/IngestCommonModulePlugin.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11870      +/-   ##
============================================
+ Coverage     71.43%   71.45%   +0.01%     
+ Complexity    59407    59376      -31     
============================================
  Files          4921     4922       +1     
  Lines        278989   279037      +48     
  Branches      40543    40550       +7     
============================================
+ Hits         199287   199374      +87     
+ Misses        63086    63045      -41     
- Partials      16616    16618       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

msfroh

@gaobinlong -- thanks for doing this! I really like this approach.

Can you link to the document issue for the new processor?

deshsidd · 2024-01-12T20:10:24Z

LGTM

gaobinlong · 2024-01-15T02:53:02Z

@gaobinlong -- thanks for doing this! I really like this approach.

Can you link to the document issue for the new processor?

Thanks, I've done that, and I'll open a document PR for this processor.

deshsidd · 2024-01-16T21:42:37Z

Anything remaining here? Can we merge?

dblock · 2024-01-17T17:18:25Z

Do you want this in 2.x? Backport?

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> (cherry picked from commit 6d2d4dd) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

--------- (cherry picked from commit 6d2d4dd) Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

gaobinlong · 2024-01-18T02:24:37Z

Do you want this in 2.x? Backport?

Yeah, see this PR has been backported to 2.x yet, thank you all @dblock @msfroh @deshsidd.

reta · 2024-01-22T19:58:46Z

New flaky tests #11974

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Add copy ingest processor

d8edaf3

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Modify changelog

0595d7e

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Fix test failure

441fde8

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

msfroh approved these changes Jan 12, 2024

View reviewed changes

deshsidd approved these changes Jan 12, 2024

View reviewed changes

gaobinlong mentioned this pull request Jan 15, 2024

[DOC] Add copy ingest processor opensearch-project/documentation-website#6147

Closed

4 tasks

msfroh merged commit 6d2d4dd into opensearch-project:main Jan 16, 2024

msfroh added the backport 2.x Backport to 2.x branch label Jan 17, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 17, 2024

[Backport 2.x] Add copy ingest processor #11907

Merged

This was referenced Jan 18, 2024

Set ingest processor supports copying from one field to another #10529

Closed

Set ingest processor can copy the whole object from one field to another #10134

Closed

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Mar 1, 2024

Add copy ingest processor (opensearch-project#11870)

7ae31d8

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>

rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024

Add copy ingest processor (opensearch-project#11870)

73c6a79

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>

shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024

Add copy ingest processor (opensearch-project#11870)

cd311c2

--------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

reta mentioned this pull request Jul 17, 2024

Add Gao Binlong as maintainer #14796

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add copy ingest processor#11870

Add copy ingest processor#11870
msfroh merged 3 commits intoopensearch-project:mainfrom
gaobinlong:copy

gaobinlong commented Jan 12, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Jan 12, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

codecov bot commented Jan 12, 2024

Uh oh!

msfroh left a comment

Uh oh!

deshsidd commented Jan 12, 2024

Uh oh!

gaobinlong commented Jan 15, 2024

Uh oh!

deshsidd commented Jan 16, 2024

Uh oh!

dblock commented Jan 17, 2024

Uh oh!

gaobinlong commented Jan 18, 2024

Uh oh!

reta commented Jan 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gaobinlong commented Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

github-actions bot commented Jan 12, 2024

Uh oh!

codecov bot commented Jan 12, 2024

Codecov Report

Uh oh!

msfroh left a comment

Choose a reason for hiding this comment

Uh oh!

deshsidd commented Jan 12, 2024

Uh oh!

gaobinlong commented Jan 15, 2024

Uh oh!

deshsidd commented Jan 16, 2024

Uh oh!

dblock commented Jan 17, 2024

Uh oh!

gaobinlong commented Jan 18, 2024

Uh oh!

reta commented Jan 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gaobinlong commented Jan 12, 2024 •

edited

Loading

github-actions bot commented Jan 12, 2024 •

edited

Loading