Skip to content

[Filebeat][httpjson] Add recursive split to httpjson#21214

Merged
marc-gr merged 1 commit intoelastic:masterfrom
marc-gr:httpjson_recursive_split
Sep 23, 2020
Merged

[Filebeat][httpjson] Add recursive split to httpjson#21214
marc-gr merged 1 commit intoelastic:masterfrom
marc-gr:httpjson_recursive_split

Conversation

@marc-gr
Copy link
Copy Markdown
Contributor

@marc-gr marc-gr commented Sep 22, 2020

What does this PR do?

Adds ability to split recursively to have more than one level of split events by functionality.

NOTE: uses the .. notation instead of . to avoid breaking users using . notation currently.

Why is it important?

This will unblock the development of some modules.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

We do not want to add it to the changelog since is only for internal use and to be removed soon

@marc-gr marc-gr requested a review from a team September 22, 2020 11:18
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 22, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 22, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21214 opened]

  • Start Time: 2020-09-22T11:19:58.896+0000

  • Duration: 59 min 7 sec

Test stats 🧪

Test Results
Failed 0
Passed 2501
Skipped 388
Total 2889

leehinman
leehinman reviewed Sep 22, 2020
View reviewed changes
x-pack/filebeat/input/httpjson/requester.go
if r.splitEventsBy == "" || !hasSplitKey {
// NOTE: this notation is only used internally, not meant to be documented
// and will be removed in the next release
keys := strings.SplitN(splitKey, "..", 2)
Copy link
Copy Markdown
Contributor

@leehinman leehinman Sep 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is unlikely but this means what you split on can't have ".." in it. Could we maybe turn splitKey into an Array (here and in config). Then you could have any value, and it could be arbitrarily deep.

Copy link
Copy Markdown
Contributor

@leehinman leehinman Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be a breaking change, so I'm taking back the suggestion.

@marc-gr marc-gr merged commit 43ca900 into elastic:master Sep 23, 2020
@marc-gr marc-gr deleted the httpjson_recursive_split branch September 23, 2020 13:42
@marc-gr marc-gr mentioned this pull request Sep 23, 2020
Merged
3 tasks
marc-gr added a commit to marc-gr/beats that referenced this pull request Sep 23, 2020
@marc-gr
Add recursive split to httpjson (elastic#21214)
e69a81c 
(cherry picked from commit 43ca900)
marc-gr added a commit that referenced this pull request Sep 23, 2020
@marc-gr
Add recursive split to httpjson (#21214) (#21252)
f29b00f 
(cherry picked from commit 43ca900)
v1v added a commit to v1v/beats that referenced this pull request Sep 24, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
c59bffe 
…ne-2.0-arm

* upstream/master: (29 commits)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  Add acceptable event log keys to winlog (elastic#21205)
  Add elastic-agent to gitignore (elastic#21219)
  Add cloudfoundry tags to events (elastic#21177)
  [Ingest Manager] Agent includes pgp file (elastic#19480)
  Add compatibility note about ingress-controller-v0.34.1 (elastic#21209)
  [Ingest Manager] Support for UPGRADE_ACTION (elastic#21002)
  Fix libbeat.output.*.bytes metrics of Elasticsearch output (elastic#21197)
  [packaging] use docker.elastic.co/ubi8/ubi-minimal (elastic#21154)
  Add host inventory metrics to system module (elastic#20415)
  [Filebeat][Azure Module] Fixing event.outcome from result_type issue (elastic#20998)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 24, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
778c037 
…ne-2.0

* upstream/master: (33 commits)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  Add acceptable event log keys to winlog (elastic#21205)
  Add elastic-agent to gitignore (elastic#21219)
  Add cloudfoundry tags to events (elastic#21177)
  [Ingest Manager] Agent includes pgp file (elastic#19480)
  Add compatibility note about ingress-controller-v0.34.1 (elastic#21209)
  [Ingest Manager] Support for UPGRADE_ACTION (elastic#21002)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-2008
2cbe6d9 
* upstream/master: (417 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-7-32
a823b84 
* upstream/master: (399 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-10
a8cb7d8 
* upstream/master: (60 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman approved these changes

Assignees

No one assigned

Labels

enhancement Filebeat Filebeat v7.10.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@marc-gr @elasticmachine @leehinman