Add support for dynamic fields#171
Merged
mtojek merged 13 commits intoelastic:masterfrom Nov 16, 2020
Merged
Conversation
Collaborator
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
Contributor
Author
|
A couple of updates here:
Sample failing output: ➜ nginx git:(169-ignore-fields) ✗ elastic-package test pipeline -v
2020/11/12 20:44:08 DEBUG Enable verbose logging
Run pipeline tests for the package
--- Test results for package: nginx - START ---
╭─────────┬────────────────────┬───────────┬────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼────────────────────┼───────────┼────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ nginx │ access │ pipeline │ test-access-event.json │ PASS │ 132.061159ms │
│ nginx │ access │ pipeline │ test-access-raw.log │ FAIL: test case failed: one or more problems with dynamic fields found in documents │ 127.889974ms │
│ nginx │ access │ pipeline │ test-access-single.log │ PASS │ 123.057553ms │
│ nginx │ error │ pipeline │ test-error-raw.log │ PASS │ 120.399112ms │
│ nginx │ ingress_controller │ pipeline │ test-ingest-raw.log │ PASS │ 127.86579ms │
╰─────────┴────────────────────┴───────────┴────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────┴──────────────╯
FAILURE DETAILS:
nginx/access test-access-raw.log:
[0] dynamic field "url.original" doesn't match the pattern (^fake.*$): /
[1] dynamic field "url.original" doesn't match the pattern (^fake.*$): /adsasd
[2] dynamic field "url.original" doesn't match the pattern (^fake.*$): /favicon.ico
[3] dynamic field "url.original" doesn't match the pattern (^fake.*$): /taga
[4] dynamic field "url.original" doesn't match the pattern (^fake.*$): /test
[5] dynamic field "url.original" doesn't match the pattern (^fake.*$): /test1
--- Test results for package: nginx - END ---
Done
Error: one or more test cases failed |
ycombinator
reviewed
Nov 16, 2020
docs/howto/pipeline_testing.md
Outdated
| }, | ||
| "ignored_fields": [ | ||
| "event.ingested" | ||
| ] |
Contributor
There was a problem hiding this comment.
I think this section needs to be updated?
ycombinator
reviewed
Nov 16, 2020
Contributor
Consider using |
added 2 commits
November 16, 2020 10:22
ycombinator
reviewed
Nov 16, 2020
|
|
||
| The `fields` section allows for customizing extra fields to be added to every read log entry (e.g. `@timestamp`, `ecs`). Use this property to extend your logs with data that can't be extracted from log content, but it's fine to have same field values for every record (e.g. timezone, hostname). | ||
|
|
||
| The `dynamic_fields` section allows for marking fields as dynamic (every time they have different non-static values), so that pattern matching instead of strict value check is applied. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #169
This PR modifies pipeline test runner to verify dynamic fields.
I'm opening this as draft as I need changes in elastic/package-spec#82 .