Add flattened field validation logic#177
Conversation
For flattened field types don't validate the leaf fields. Fixes elastic#176
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
internal/fields/validate.go
Outdated
| } | ||
| } | ||
| case map[string]interface{}: | ||
| if definition := findElementDefinition("", key, v.schema); definition != nil && definition.Type == "flattened" { |
There was a problem hiding this comment.
I see that you reused the same invocation as below (findElementDefinition("", key, v.schema)). I think it's a good moment to refactor it a bit:
findElementDefinition(key, v.schema) which calls findElementDefinitionForRoot("", key, v.schema)
and apply then same in validateScalarElement()
I would also wrap the entire line with IsFieldFlattened or similar.
There was a problem hiding this comment.
I asked for a small refactoring.
I really appreciate adding the test case. Could you please link the Github issue in the code comment? In general you can add a comment justifying why do we need to treat specially "flattened" fields.
EDIT:
I see that the PR build was aborted. We need a green status to merge the PR once all relevant changes are introduced.
mtojek
left a comment
There was a problem hiding this comment.
I left one nit-pick. If the PR build goes green, feel free to merge it. You have to bump up the dependency in the Integrations repository (can be done together with your PR).
|
@mtojek I think you'll need to trigger testing in order to get a green build.
|
|
jenkins run the tests please |
The tests revealed a few issues. There was an error in the pipeline for update-user-json.log because
serviceEventDetails was not present. This was the error
"error": {
"message": "Cannot invoke \\\"Object.getClass()\\\" because \\\"receiver\\\" is null"
}
The aws.cloudtrail.read_only field was mapped as keyword but was actual a JSON boolean.
I changed the type to boolean, but do not plan to backport this change to Filebeat.
And lastly some ECS user_agent fields were missing.
This depends on elastic/elastic-package#177 to make the flattened fields pass
test validation.
For flattened field types don't validate the leaf fields.
Fixes #176