Add initial support to validate the mappings in system tests by mrodm · Pull Request #2214 · elastic/elastic-package

mrodm · 2024-11-08T11:57:11Z

Add validation of the mappings when running system tests.

This process compares the mappings installed by Fleet (preview mappings) with the ones obtained after ingesting new documents into Elasticsearch.

This validation for the time being is behind an environment variable as a technical preview:

# Default value and same behavior as now (validation of fields)
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=fields elastic-package test system -v

# Test just validation of mappings 
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings elastic-package test system -v

# Test both validation of mappings and fields
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=all elastic-package test system -v

Assumptions considered while working on this PR: #2206 (comment)

As part of this PR:

CI pipeline has been updated to remove steps related to running system tests with Elastic Agent.
- It will be kept one step to ensure that the feature is working.
CI pipeline has been updated to add steps to run mapping validation in system tests.

Author's checklist

Update CI steps to remove old stack agents
Update CI to add new steps for validating fields using mappings
Add environment variable to validate that fields are documented with different methods
Add new environment variables to docs.
Remove debug/testing log statements
Test with Elastic stack 8.x
- Some packages are failing like prometheus failing since some fields depend on dynamic templates (not implemented yet)
Test with Elastic stack 7.x (f5 package in integrations repository)
Test with integration and input packages (sql_input test package in elastic-package repo).
Test with Elastic stacks with LogsDB enabled (synthetics).
- Tested in CI with ti_anomali_logsdb and auth0_logsdb

mrodm · 2024-11-12T18:24:15Z

/test

mrodm · 2024-11-13T15:00:22Z

internal/fields/validate.go

+
+	dataStreamName string
+
+	LocalSchema []FieldDefinition


New variable to hold just the Field Definitions from the local directory (package).

It is needed to check if a given mapping is related to a field definition with type array.

Why not using Schema?

mrodm · 2024-11-13T15:00:42Z

internal/fields/validate.go


 	v.Schema = append(fields, v.Schema...)
+
+	v.LocalSchema = fields


Added here for completeness, but it is not used in this validator.

Should we remove it from here?

If not used I would say to remove it.

internal/fields/mappings.go

mrodm · 2024-12-09T18:25:22Z

test integrations

elastic-vault-github-plugin-prod · 2024-12-09T18:32:10Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

elasticmachine · 2024-12-11T15:07:35Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 1f3588b

Failed CI Steps

:go: Integration test: system

History

💚 Build #4412 succeeded ee755e9
💚 Build #4396 succeeded aa60ec4
💛 Build #4395 was flaky c0dbfbb
💚 Build #4388 succeeded 5097aee

cc @mrodm

mrodm · 2024-12-11T15:50:54Z

test integrations

elastic-vault-github-plugin-prod · 2024-12-11T15:59:15Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828

mrodm · 2024-12-11T17:48:43Z

Another package (tines) failing due to a missing mapping https://buildkite.com/elastic/integrations/builds/19362#0193b678-1607-4f12-8d46-c212e3cd4f1e

Should it be validated using dynamic templates?

Error:

test case failed: one or more errors found in mappings in logs-tines.audit_logs index template: [0] field "tines.audit_log.inputs.inputs.options" is undefined: missing definition for path (not in ECS)

Field definition in the package:

                - name: options
                  type: object
                  object_type: keyword
                  object_type_mapping_type: "*"

No mapping created in the preview, and this mapping is created in the actual mappings after ingesting some docs (no dynamic template present):

        "tines.audit_log.inputs.inputs.options": {
          "type": "keyword",
          "ignore_above": 1024
        },

cc @jsoriano

jsoriano · 2024-12-11T19:02:31Z

No mapping created in the preview

This is weird because the generated component template makes sense. In tines 1.13 subobjects: false` is moved from an object to the whole index, and this is the difference in the component template:

+++ /tmp/tines-subobjects-false-global/component_templates/logs-tines.audit_logs@package.json	2024-12-11 19:53:31.500743669 +0100
@@ -7,7 +7,7 @@
           "lifecycle": {
             "name": "logs"
           },
-          "default_pipeline": "logs-tines.audit_logs-1.12.2",
+          "default_pipeline": "logs-tines.audit_logs-1.13.0",
           "mapping": {
             "total_fields": {
               "limit": "1000"
@@ -113,8 +113,7 @@
                               "createFormEmptyState": {
                                 "type": "boolean"
                               }
-                            },
-                            "subobjects": false
+                            }
                           },
                           "eventName": {
                             "ignore_above": 1024,
@@ -347,7 +346,8 @@
               }
             }
           }
-        }
+        },
+        "subobjects": false
       }
     },
     "_meta": {

I can see mappings in the preview, the main difference is that when subobjects: false is used, as you mention, the properties are directly defined in the root, with dots in the names:

      ...
      "properties": {
        "@timestamp": {
          "type": "date",
          "ignore_malformed": false
        },
        "data_stream.dataset": {
          "type": "constant_keyword"
        },
        "data_stream.namespace": {
          "type": "constant_keyword"
        },
        "data_stream.type": {
          "type": "constant_keyword"
        },
        "event.agent_id_status": {
          "type": "keyword",
          "ignore_above": 1024
        },
        ...

With what version of the stack you are missing the mappings in the preview?

jsoriano · 2024-12-11T19:25:14Z

Ah, you mean that no mapping is generated in the preview for this dynamic mapping. Yes, this looks like some issue in Fleet, it is not generating the dynamic mappings for subobjects.

jsoriano · 2024-12-11T19:32:26Z

The problem is not related to subobjects, it can be also reproduced without setting subobjects: false. I think the problem is related to tines.audit_log.inputs.inputs.options not being empty, it also defines a property:

                - name: options.createFormEmptyState
                  type: boolean

This looks like an issue in Fleet, it could generate both mappings, the more specific static mapping for the boolean, and the dynamic template as catch all for the rest.

jsoriano

I think we can go on with this change and polish things in future PRs.

mrodm · 2024-12-12T09:26:01Z

Ah, you mean that no mapping is generated in the preview for this dynamic mapping.

Exactly, that specific field definition had no mapping nor dynamic mapping in the preview.

The problem is not related to subobjects, it can be also reproduced without setting subobjects: false. I think the problem is related to tines.audit_log.inputs.inputs.options not being empty, it also defines a property:
                - name: options.createFormEmptyState
                  type: boolean
This looks like an issue in Fleet, it could generate both mappings, the more specific static mapping for the boolean, and the dynamic template as catch all for the rest.

Thanks for checking this out!

No dynamic mapping was being generated for tines.audit_log.inputs.inputs.options.*, and this package uses the tines.audit_log.inputs.inputs.options field directly, without having any mapping for it or its sub-properties. The workaround ensures that there is a mapping for tines.audit_log.inputs.inputs.* that serves for tines.audit_log.inputs.inputs.options as well as for its subobjects. The configured dynamic mapping was not being generated due to some issue in Fleet that we are investigating. We detected this issue while refactoring field mappings tests in elastic-package, more about this in elastic/elastic-package#2214[1]. [1]elastic/elastic-package#2214 (comment) Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…ic#12082) No dynamic mapping was being generated for tines.audit_log.inputs.inputs.options.*, and this package uses the tines.audit_log.inputs.inputs.options field directly, without having any mapping for it or its sub-properties. The workaround ensures that there is a mapping for tines.audit_log.inputs.inputs.* that serves for tines.audit_log.inputs.inputs.options as well as for its subobjects. The configured dynamic mapping was not being generated due to some issue in Fleet that we are investigating. We detected this issue while refactoring field mappings tests in elastic-package, more about this in elastic/elastic-package#2214[1]. [1]elastic/elastic-package#2214 (comment) Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

mrodm added 9 commits November 7, 2024 16:49

Add validation using mappings

f8c82cf

Remove validation per setting/parameter

e211e34

Add environment variable

724329a

[CI] Keep just one step to test system tests with stack agents

a2a4c92

[CI] Add steps to test new validation method

4ee6da6

Add tests

14d7e2a

Add exception for empty objects - TBC

ff3d85d

Update label CI step

b5bed3a

Reuse same fields validator to sanitize docs

2ef29cf

mrodm self-assigned this Nov 8, 2024

mrodm added 14 commits November 8, 2024 15:31

Update field definition to type keyword

36d59fe

Update README test package nginx

6c7af00

Add skip for dynamic objects

4e12656

Add more checks about objects with dynamic true

a326b7f

Add docs for environment variables

8dedfc2

Add more tests

c36fc3b

Compare value constant_keyword just in one case

a48f52d

Restore array type

e94169c

Add validation for type array objects

ec0dd1c

[CI] Remove leftover in label step

7d7ad49

Revamp comments and examples

1949adf

Create private method for creating the mappings validator

ac612fd

Revamp comments

facee62

Include also the definition that contains multi_fields

e03dbab

mrodm added 2 commits November 13, 2024 10:55

Rename methods

1e05f97

Fix typo in README

20775d8

mrodm commented Nov 13, 2024

View reviewed changes

internal/fields/mappings.go Show resolved Hide resolved

mrodm added 3 commits December 9, 2024 18:37

Update test cases

399cb61

Reorder code in validateConstantKeywordField

ab616d9

Add missing fields in tests

1ec0670

mrodm added 3 commits December 11, 2024 12:44

Create list of fields to be considered as exception

0e1d3ef

Set external ECS in parseECSFieldsSchema and appendECSMappingMultifields

45e3fcc

Remove some debug messages

1f3588b

mrodm requested a review from jsoriano December 11, 2024 18:04

jsoriano approved these changes Dec 11, 2024

View reviewed changes

mrodm merged commit 6def317 into elastic:main Dec 12, 2024

mrodm deleted the validate-mappings-first branch December 12, 2024 11:02

This was referenced Dec 12, 2024

Workaround issue with missing dynamic mapping in tines package elastic/integrations#12082

Merged

Fix generation of dynamic mapping for object with specific subfield elastic/kibana#204104

Merged

mrodm mentioned this pull request Dec 13, 2024

Do not add as exception fields leaf objects #2277

Merged

This was referenced Dec 16, 2024

Calculate exception fields just with mappings validation #2280

Merged

Fix validation of properties and parameters based on mappings #2288

Merged

Validate fields based on their mappings and the dynamic templates set #2285

Merged


		v.Schema = append(fields, v.Schema...)

		v.LocalSchema = fields

Conversation

mrodm commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

Uh oh!

mrodm commented Nov 12, 2024

Uh oh!

mrodm Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

jsoriano Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

mrodm Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

jsoriano Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrodm commented Dec 9, 2024

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 9, 2024

Uh oh!

elasticmachine commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

History

Uh oh!

mrodm commented Dec 11, 2024

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Uh oh!

mrodm commented Dec 11, 2024

Uh oh!

jsoriano commented Dec 11, 2024

Uh oh!

jsoriano commented Dec 11, 2024

Uh oh!

jsoriano commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

mrodm commented Dec 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Nov 8, 2024 •

edited

Loading

elasticmachine commented Dec 11, 2024 •

edited

Loading

jsoriano commented Dec 11, 2024 •

edited

Loading