[Filebeat][httpjson] Fix date_cursor validation#21756
Merged
marc-gr merged 2 commits intoelastic:masterfrom Oct 15, 2020
Merged
[Filebeat][httpjson] Fix date_cursor validation#21756marc-gr merged 2 commits intoelastic:masterfrom
marc-gr merged 2 commits intoelastic:masterfrom
Conversation
83c85b7 to
b7e5ee4
Compare
Contributor
|
Pinging @elastic/siem (Team:SIEM) |
Contributor
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
andrewkroh
reviewed
Oct 13, 2020
Member
There was a problem hiding this comment.
Is there a requirement for the format to include seconds? Without seconds in the format this test won't pass, right?
Contributor
Author
There was a problem hiding this comment.
True, and the same applies for any other element of the date since we can't know the intent of the format. I changed it to just check for parse errors instead, just as a sanity check. Since now it is against a known date we can trigger a parse error from the tests.
42a0838 to
7b4df27
Compare
andrewkroh
approved these changes
Oct 14, 2020
Member
andrewkroh
left a comment
There was a problem hiding this comment.
LGTM. I think the docs may be missing some of the new cursor options for HTTP JSON.
2 tasks
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Oct 15, 2020
* Fix date_cursor validation * Only check for parse errors (cherry picked from commit 471df0b)
2 tasks
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Oct 15, 2020
* Fix date_cursor validation * Only check for parse errors (cherry picked from commit 471df0b)
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 15, 2020
* upstream/master: (29 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 15, 2020
* upstream/master: (216 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 15, 2020
* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 15, 2020
* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
marc-gr
added a commit
that referenced
this pull request
Oct 16, 2020
marc-gr
added a commit
that referenced
this pull request
Oct 16, 2020
Member
|
Hey @marc-gr, thanks for fixing this. These tests seem to be flaky in 7.9 branch too, could you please backport the change to this branch? |
2 tasks
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Oct 19, 2020
* Fix date_cursor validation * Only check for parse errors (cherry picked from commit 471df0b)
marc-gr
added a commit
that referenced
this pull request
Oct 19, 2020
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…tic#21982) * Fix date_cursor validation * Only check for parse errors (cherry picked from commit 15251d4)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
It fixes the
httpjsondate_cursorconfig validation by using a fixed date for format validation.Why is it important?
Before
nowwas used as the date to validate the format against, some formats (like the one used in the tests, which caused a new failing test recently #21748) depending on the current date, would fail or not, causing flaky validation outputs.Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature works- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues