Skip to content

[EPM] Add mapping field types to index template generation v2#60266

Merged
skh merged 15 commits intoelastic:masterfrom
skh:55865-index-template-types-2
Mar 18, 2020
Merged

[EPM] Add mapping field types to index template generation v2#60266
skh merged 15 commits intoelastic:masterfrom
skh:55865-index-template-types-2

Conversation

@skh
Copy link
Copy Markdown
Contributor

@skh skh commented Mar 16, 2020 •
edited
Loading

Summary

Mostly implements #55865

  • mostly follows the logic in https://github.com/elastic/beats/blob/master/libbeat/template/processor.go how to handle various data types
  • does not, however, care about backward compatibility with anything earlier than 7.6, which simplifies the logic quite a bit
  • flattens all fields (instead of expanding all fields, as beats does) after discussion with @ruflin
  • expands (un-flattens) and deduplicates all fields
  • silently removes alias fields with a path pointing to a non-existing field
  • still missing:
    • handling of object data type
    • integration tests
    • all other settings in the generated index templates need to be reviewed
  • there is some code duplication now between fields/field.ts and kibana/index_pattern/install.ts which I kept because it still might diverge

How to test this?

  • On a fresh elasticsearch setup (i.e. with system & base package not yet installed) open the IngestManager UI. The initial package installation, including index template generation, should succeed without error.
  • Installation of all other available packages should likewise succeed without error.
  • Inspect the mappings in the generated index templates in Kibana's dev console (GET _cat/templates, then GET /_template/$TEMPLATE_NAME). The relevant templates are logs-$PACKAGE.$DATASET and metrics-$PACKAGE.$DATASET, so e.g. metrics-system.process
  • Inspect server/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap, especially the snapshot for the system.yml test
  • server/services/epm/fields/__snapshots__/field.test.ts.snap might also be interesting, as it shows the intermediate data structure after processFields(), but before the transformation into the mappings structure.

Follow-up issues:

@skh skh mentioned this pull request Mar 16, 2020
Closed
@skh skh force-pushed the 55865-index-template-types-2 branch from 2e1d7cb to 36fdca0 Compare March 18, 2020 10:01
@skh skh marked this pull request as ready for review March 18, 2020 11:21
@skh skh requested a review from a team March 18, 2020 11:21
This was referenced Mar 18, 2020
Closed
Closed
@neptunian neptunian added v7.7.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project labels Mar 18, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 18, 2020

Pinging @elastic/ingest-management (Feature:EPM)

@skh skh mentioned this pull request Mar 18, 2020
Closed
@skh
Copy link
Copy Markdown
Contributor Author

skh commented Mar 18, 2020

@neptunian thank you for your comments on this PR and #59894 !

I've created a bunch of new issues for most of it, see the initial description of this PR. Changes to this PR:

  • array type fields should work now
  • don't throw errors on duplicate fields, just silently drop everything after the first occurence. This might need to be more sophisticated at some point
  • added one unit test (it's a start...)

This is ready for another look now.

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Mar 18, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@skh skh merged commit f93ec79 into elastic:master Mar 18, 2020
@skh skh mentioned this pull request Mar 18, 2020
Merged
jloleysens added a commit that referenced this pull request Mar 18, 2020
@jloleysens
Merge branch 'app/painless' of github.com:elastic/kibana into app/pai…
8733480 
…nless

* 'app/painless' of github.com:elastic/kibana: (64 commits)
  Fix filter scope in bool query (#60488)
  change index pattern id to be the same as index pattern title (#60436)
  [Endpoint] resolver v1 events (#59233)
  Branding fixes for dashboard, loader and space selector (#60073)
  skip flaky suite (#60535)
  [SIEM][Detection Engine] Fixes bug with timeline templates not working
  Fixed errors which are happening if switch between alert types (#60453)
  [EPM] Add mapping field types to index template generation v2 (#60266)
  [NP] Cutover ensureDefaultIndexPattern to kibana_utils (#59895)
  Closes #60265. Adds Beta badge to service map (#60482)
  [Visualize] Duplicated query filters in es request (#60106)
  [ML] Disable functional transform tests
  Fixes to service map single node banner (#60072)
  [Uptime] replace fetch with kibana http (#59881)
  Upgrade @types/node to match Node.js runtime (#60368)
  [License Management] NP migration (#60250)
  Fix create alert button from not showing in alerts list (#60444)
  [SIEM][Case] Update connector through flyout (#60307)
  add data-test-subj where possible on SO management table (#60226)
  Enforce `required` presence for value/key validation of `recordOf` and `mapOf`. (#60406)
  ...
skh added a commit that referenced this pull request Mar 19, 2020
@skh
[EPM] Add mapping field types to index template generation v2 (#60266) (
7f70a71 
#60533)

* Add properties needed for index templates to Field

* Add data type handling to template generation

* Adjust tests

* Update fields test snapshots

* Remove duplicate fields from test file

* Add test cases

* Enhance processFields

* move expand stage to expandFields
* fix expandFields
* add deduplication stage dedupFields

* Use processField() to preprocess fields

* Remove alias fields with invalid path

* Remove obsolete code.

* Fix documentation.

* Add unit tests for getField()

* Don't fail on invalid input for now.

* Validate array fields.

* Guard against invalid input.
jloleysens added a commit to jloleysens/kibana that referenced this pull request Mar 19, 2020
@jloleysens
Merge remote-tracking branch 'upstream/app/painless' into minor-refactor
3f00238 
* upstream/app/painless: (66 commits)
  Another i18n issue
  Fix i18n
  Fix filter scope in bool query (elastic#60488)
  change index pattern id to be the same as index pattern title (elastic#60436)
  [Endpoint] resolver v1 events (elastic#59233)
  Branding fixes for dashboard, loader and space selector (elastic#60073)
  skip flaky suite (elastic#60535)
  [SIEM][Detection Engine] Fixes bug with timeline templates not working
  Fixed errors which are happening if switch between alert types (elastic#60453)
  [EPM] Add mapping field types to index template generation v2 (elastic#60266)
  [NP] Cutover ensureDefaultIndexPattern to kibana_utils (elastic#59895)
  Closes elastic#60265. Adds Beta badge to service map (elastic#60482)
  [Visualize] Duplicated query filters in es request (elastic#60106)
  [ML] Disable functional transform tests
  Fixes to service map single node banner (elastic#60072)
  [Uptime] replace fetch with kibana http (elastic#59881)
  Upgrade @types/node to match Node.js runtime (elastic#60368)
  [License Management] NP migration (elastic#60250)
  Fix create alert button from not showing in alerts list (elastic#60444)
  [SIEM][Case] Update connector through flyout (elastic#60307)
  ...
@jen-huang jen-huang added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neptunian neptunian neptunian approved these changes

Assignees

@skh skh

Labels

Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.7.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@skh @elasticmachine @kibanamachine @neptunian @jen-huang