Skip to content

[Ingest] Remove meta processor and expose templates in other processors#15415

Merged
martijnvg merged 1 commit intoelastic:feature/ingestfrom
martijnvg:ingest_template_infra
Dec 18, 2015
Merged

[Ingest] Remove meta processor and expose templates in other processors#15415
martijnvg merged 1 commit intoelastic:feature/ingestfrom
martijnvg:ingest_template_infra

Conversation

@martijnvg
Copy link
Copy Markdown
Member

  • Added template infrastructure via FieldPathTemplate (adds template support for field like settings) and ValueSource (adds template support for value like settings.
  • Adjusted the set and remove processor to be able to use templates via the new infrastructure. The field and value settings are now templateble.
  • Removed the meta processor, since any processor can now access and modify meta fields.

PR for #14990

@martijnvg martijnvg added review :Distributed/Ingest Node Execution or management of Ingest Pipelines labels Dec 14, 2015
@rmuir
Copy link
Copy Markdown
Contributor

rmuir commented Dec 14, 2015

why can't this plugin use mustache via the scriptengine service? Sorry: I don't think we should have secret embedded script engines inside other plugins. I have found this before and it seriously pisses me off, we can't do this.

@martijnvg
Copy link
Copy Markdown
Member Author

@rmuir That is the plan. The ingest framework has been directly using mustache, because it can't directly depend on the script service. It needs to be ES agnostic. In a follow up PR, I plan to add a TemplateService (or something like that) that the ingest framework will use directly. The implementation in the ingest plugin will then use lang-mustache module via the script service.

@martijnvg martijnvg changed the title [Ingest] Add template infrastructure [Ingest] Remove meta processor and expose templates in other processors Dec 14, 2015
@rmuir
Copy link
Copy Markdown
Contributor

rmuir commented Dec 14, 2015

The ingest framework has been directly using mustache, because it can't directly depend on the script service. It needs to be ES agnostic

This makes no sense.

In a follow up PR, I plan to add a TemplateService (or something like that) that the ingest framework will use directly.

This makes no sense.

Please, lets just use the scriptengineservice, and lookup "mustache", it is really just that simple.

@uboness
Copy link
Copy Markdown
Contributor

uboness commented Dec 14, 2015

This makes no sense.

why? The processors themselves cannot depend on ES specific codebase. The reason for this is that we want to reuse the processors in LS. So in order to have processors use templates, we need to have some sort of a Template abstraction (I think that's what @martijnvg refers to with TemplateService) that is part of the ingest code (at least the code that will be shared with LS).

@rmuir
Copy link
Copy Markdown
Contributor

rmuir commented Dec 14, 2015

This does not justify forking scripting engines, sorry. It needs to go thru the script engine service.

@nik9000
Copy link
Copy Markdown
Member

nik9000 commented Dec 14, 2015

The processors themselves cannot depend on ES specific codebase.

I wonder if that is a good excuse to yank the some of the ES code base into a library you can share.

@rmuir
Copy link
Copy Markdown
Contributor

rmuir commented Dec 14, 2015

our script engine support is not something that needs to become a library, its out of the question. We do everything possible to contain their craziness.

@martijnvg
Copy link
Copy Markdown
Member Author

@rmuir @uboness I've updated the PR. Ingest no longer has a mustache compile time dependency and indirectly depends on the lang-mustache module. I've added a TemplateService interface to avoid that the ingest framework gets a compile dependency on ES. The ingest plugin provides an implementation that uses ES' script service.

I wonder if that is a good excuse to yank the some of the ES code base into a library you can share.

@nik9000 Indirectly using the script service in ingest avoids granting permissions in ingest that a template engine may need. Also it has already been granted in the lang-moustache module, so no need to grant twice.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

straightforward is a single word ;)

@martijnvg martijnvg force-pushed the ingest_template_infra branch from 3204db4 to 26357cd Compare December 17, 2015 15:24
@martijnvg
Copy link
Copy Markdown
Member Author

@javanna I've updated this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Ingest Node Execution or management of Ingest Pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants