Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions by mrodm · Pull Request #455 · elastic/package-spec

mrodm · 2022-12-16T12:27:43Z

What does this PR do?

This PR adds support in input packages to include under elasticsearch key dynamic templates and properties (mappings). This PR allows to define elasticsearch.index_template.* keys. It uses the same definition as the data_stream manifests in integration packages.

This PR also adds a new field under _dev/build/build.yml named import_common_dynamic_mappings (other options import_common_mappings?). The aim of this field is to allow developers to opt in to be added automatically by elastic-package
some common definitions needed.

This support would be introduced in package-spec versions >= 2.3.0.

Additional notes:

Updated .gitignore to just ignore build folder at the root directory. There are several build folders used in our specs, and git raises some errors because of this:

 $ git add spec/integration/_dev/build/build.spec.yml
The following paths are ignored by one of your .gitignore files:
spec/integration/_dev/build
hint: Use -f if you really want to add them.
hint: Turn this message off by running
hint: "git config advice.addIgnoredFile false"

According to the Jenkins outputs it would be needed to just ignore the one from the root level:

[2022-11-23T10:15:49.793Z] mkdir -p /var/lib/jenkins/workspace/gest-manager_package-spec_PR-452/src/github.com/elastic/package-spec/build/test-coverage/coverage-unit-report
[2022-11-23T10:15:49.793Z] mkdir -p /var/lib/jenkins/workspace/gest-manager_package-spec_PR-452/src/github.com/elastic/package-spec/build/test-results

Why is it important?

This would allow to elastic-package to add the most common mapping definitions needed to avoid conflicts in mappings for both input and integrations (data_strreams) packages.

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

Relates Temporarily import hard-coded list of common fields elastic-package#1018

…and properties from ECS Remove elasticsearch field in input manifest from <2.3.0 versions

elasticmachine · 2022-12-16T12:37:40Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-20T14:55:04.625+0000
Duration: 9 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	678
Skipped	0
Total	678

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-16T12:37:47Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`7/7`)	💚
Files	68.0% (`17/25`)	👍
Classes	76.471% (`26/34`)	👍
Methods	56.075% (`60/107`)	👍
Lines	42.229% (`538/1274`)	👍
Conditionals	100.0% (`0/0`)	💚

mrodm · 2022-12-20T10:28:24Z

code/go/pkg/validator/validator_test.go

 	}{
 		"good":                               {},
 		"good_v2":                            {},
+		"good_input":                         {},


Added a new good_input package to add these keys. It is copied from the sql_input test package.

mrodm · 2022-12-20T10:29:40Z

test/packages/good_v2/manifest.yml

@@ -1,4 +1,4 @@
-format_version: 2.0.0


Update version in good_v2 to allow adding the new field in _dev/build/build.yml

jsoriano · 2022-12-20T11:16:17Z

spec/integration/_dev/build/build.spec.yml

+  - before: 2.3.0
+    patch:
+      - op: remove
+        path: "/properties/dependencies/properties/ecs/properties/import_common_dynamic_mappings"


👍 this will encourage users of this to update the format of their packages 🙂

jsoriano · 2022-12-20T11:19:18Z

spec/integration/_dev/build/build.spec.yml

              type: string
              description: Source reference
              pattern: '^git@.+'
+            import_common_dynamic_mappings:


Nit. common looks redundant here.

Suggested change

import_common_dynamic_mappings:

import_dynamic_mappings:

And maybe dynamic too, this feature imports mappings for this schema, they happen to be mostly dynamic, but could be static too.

Suggested change

import_common_dynamic_mappings:

import_mappings:

Or:

Suggested change

import_common_dynamic_mappings:

import_template:

I was not totally sure about this name. Agreed that common does not provide any value to the name.
I will update this field to be import_mappings

Thanks!

spec/changelog.yml

hop-dev

Addition of the elasticsearch key at the top level of input packages looks good to me 👍 Thanks for this Mario!

mrodm added 4 commits December 16, 2022 12:40

Add definition for elasticsearch.index_template property

2adcc70

Add elasticsearch field into input manifest

c3b46bc

Add parameter to opt-in about adding or not common dynamic templates …

dc5b95a

…and properties from ECS Remove elasticsearch field in input manifest from <2.3.0 versions

Just ignore build folder at the root of the repo

5ae48af

mrodm self-assigned this Dec 16, 2022

Add changelog entry

2056f3c

mrodm added 5 commits December 16, 2022 14:11

Fix reference path

aedf843

Add json patch to remove import_common_dynamic_mappings before 2.3.0

291a635

Update test package good_v2

9a32db0

Add a reference good_input package

8c21116

Add good_input to be tested

0dcd837

mrodm mentioned this pull request Dec 20, 2022

Add common dynamic mappings and properties from ECS automatically elastic/elastic-package#1073

Merged

mrodm commented Dec 20, 2022

View reviewed changes

mrodm marked this pull request as ready for review December 20, 2022 10:30

mrodm requested a review from a team as a code owner December 20, 2022 10:30

mrodm requested a review from hop-dev December 20, 2022 10:30

jsoriano reviewed Dec 20, 2022

View reviewed changes

mrodm added 3 commits December 20, 2022 15:49

Add changelog entry for flag

a1bde88

Rename field for import mappings

70d4563

Update test packages

6f56801

mrodm requested a review from jsoriano December 20, 2022 18:03

jsoriano approved these changes Dec 21, 2022

View reviewed changes

mrodm mentioned this pull request Dec 22, 2022

Temporarily import hard-coded list of common fields elastic/elastic-package#1018

Closed

3 tasks

hop-dev approved these changes Dec 22, 2022

View reviewed changes

hop-dev mentioned this pull request Dec 22, 2022

[Fleet] Input only packages: Create index templates and pipelines at package policy creation time elastic/kibana#145529

Closed

mrodm merged commit 342bdf7 into elastic:main Dec 22, 2022

mrodm deleted the add_elsaticsearch_field_input branch December 22, 2022 13:23

github-actions bot mentioned this pull request Feb 1, 2026

Otel Input Packages should be time series elastic/integrations#17183

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions#455

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions#455
mrodm merged 13 commits intoelastic:mainfrom
mrodm:add_elsaticsearch_field_input

mrodm commented Dec 16, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Dec 16, 2022 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Dec 16, 2022

Uh oh!

mrodm Dec 20, 2022

Uh oh!

mrodm Dec 20, 2022

Uh oh!

jsoriano Dec 20, 2022

Uh oh!

jsoriano Dec 20, 2022

Uh oh!

jsoriano Dec 20, 2022

Uh oh!

mrodm Dec 20, 2022

Uh oh!

Uh oh!

hop-dev left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mrodm commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

elasticmachine commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Dec 16, 2022

🌐 Coverage report

Uh oh!

mrodm Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

mrodm Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

mrodm Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hop-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrodm commented Dec 16, 2022 •

edited

Loading

elasticmachine commented Dec 16, 2022 •

edited

Loading