Skip to content

add copy ingest processor#56985

Closed
gaobinlong wants to merge 1 commit intoelastic:masterfrom
gaobinlong:add_copy_processor
Closed

add copy ingest processor#56985
gaobinlong wants to merge 1 commit intoelastic:masterfrom
gaobinlong:add_copy_processor

Conversation

@gaobinlong
Copy link
Copy Markdown
Contributor

@gaobinlong gaobinlong commented May 20, 2020

Relates to #55682 and #51046.
The main point of this PR is adding a copy ingest processor which can copy the value of one field to another and even the top level of the document. The supported data types which can be copied are boolean, number, array, object, string, date.

@astefan astefan added the :Distributed/Ingest Node Execution or management of Ingest Pipelines label May 20, 2020
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label May 20, 2020
@danhermann danhermann self-requested a review May 20, 2020 11:08
@danhermann
Copy link
Copy Markdown
Contributor

@gaobinlong, thank you for this contribution. It's useful functionality that we would like to incorporate into ingest pipelines. We are reconsidering how we've organized the processors and are trying to determine whether this fits best as part of the existing set processor or as its own separate processor. If you'll bear with us while we consider that, we will let you know and then work to get this PR refactored if necessary and then merged.

@gaobinlong
Copy link
Copy Markdown
Contributor Author

@danhermann, I have tried to use set processor to copy one field to another, but found that we cannot copy array,object and boolean, because the template snippet always produces string value. I'm eager to know your idea about that and glad to do more work.

@danhermann
Copy link
Copy Markdown
Contributor

@gaobinlong, apologies for the delay on this one. We decided that this is how the set processor should work instead of producing string values as you note that it does now. If you would like to continue this PR by modifying the set processor to correctly handle, I would be happy to review it.

@gaobinlong
Copy link
Copy Markdown
Contributor Author

@danhermann, I'm glad to continue doing that and will refractor the code soon.

@gaobinlong
Copy link
Copy Markdown
Contributor Author

@danhermann , I have refactored the code and opened a new PR #63540, this PR can be closed lately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Ingest Node Execution or management of Ingest Pipelines Team:Data Management (obsolete) DO NOT USE. This team no longer exists.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants