Skip to content

Cherry-pick #19246 to 7.x: [Filebeat][httpjson] Add split_events_by config setting#19482

Merged
marc-gr merged 1 commit intoelastic:7.xfrom
marc-gr:backport_19246_7.x
Jul 14, 2020
Merged

Cherry-pick #19246 to 7.x: [Filebeat][httpjson] Add split_events_by config setting#19482
marc-gr merged 1 commit intoelastic:7.xfrom
marc-gr:backport_19246_7.x

Conversation

@marc-gr
Copy link
Copy Markdown
Contributor

@marc-gr marc-gr commented Jun 29, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #19246 to 7.x branch. Original message:

What does this PR do?

Adds a split_events_by setting to the httpjson input to allow similar mechanics as the split filter for logstash.

Why is it important?

There are many use cases where a list of elements is passed as an API response, and we want to create each of them to a single event.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 29, 2020

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 29, 2020 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19482 updated]

  • Reason: Aborted from #10

  • Start Time: 2020-07-14T20:07:00.477+0000

  • Duration: 70 min 20 sec

  • Commit: 3c478d939afea3b2d4fcf6bd364414db0c9be491

Test stats 🧪

Test Results
Failed 0
Passed 2213
Skipped 385
Total 2598

Log output

Expand to view the last 100 lines of log output 

[2020-07-14T20:58:52.952Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-14T20:58:52.952Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-14T20:58:52.952Z] + '[' -f packetbeat/build/coverage/full.cov ']'
[2020-07-14T20:58:52.952Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-14T20:58:52.952Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-14T20:58:52.952Z] + '[' -f winlogbeat/build/coverage/full.cov ']'
[2020-07-14T20:58:52.952Z] + for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat
[2020-07-14T20:58:52.952Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-14T20:58:52.952Z] + '[' -f journalbeat/build/coverage/full.cov ']'
[2020-07-14T20:58:54.255Z] Post stage
[2020-07-14T20:58:54.328Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats
[2020-07-14T20:58:55.100Z] Starting "default"...
[2020-07-14T20:58:55.101Z] Machine "default" is already running.
[2020-07-14T20:58:56.578Z] Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.102:2376": dial tcp 192.168.99.102:2376: connect: connection refused
[2020-07-14T20:58:56.578Z] You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
[2020-07-14T20:58:56.578Z] Be advised that this will trigger a Docker daemon restart which might stop running containers.
[2020-07-14T20:58:56.578Z] 
[2020-07-14T20:58:56.578Z] Client:
[2020-07-14T20:58:56.578Z]  Version:           18.06.1-ce
[2020-07-14T20:58:56.578Z]  API version:       1.38
[2020-07-14T20:58:56.578Z]  Go version:        go1.10.3
[2020-07-14T20:58:56.578Z]  Git commit:        e68fc7a
[2020-07-14T20:58:56.578Z]  Built:             Tue Aug 21 17:21:31 2018
[2020-07-14T20:58:56.578Z]  OS/Arch:           darwin/amd64
[2020-07-14T20:58:56.578Z]  Experimental:      false
[2020-07-14T20:58:56.578Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-07-14T20:58:56.578Z] It requires Docker daemon to be installed and running
[2020-07-14T21:16:43.584Z] ...........Sending interrupt signal to process
[2020-07-14T21:16:47.383Z] Sending interrupt signal to process
[2020-07-14T21:16:54.241Z] failed to run compiled magefile: signal: terminated
[2020-07-14T21:16:54.251Z] script returned exit code 255
[2020-07-14T21:16:54.316Z] Recording test results
[2020-07-14T21:16:55.995Z] Stashed 3 file(s)
[2020-07-14T21:16:56.006Z] Archiving artifacts
[2020-07-14T21:16:56.799Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-14T21:16:57.063Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-14T21:16:57.063Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-14T21:16:57.063Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-14T21:16:58.381Z] Failed in branch Filebeat x-pack
[2020-07-14T21:16:58.573Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats
[2020-07-14T21:16:58.580Z] Click here to forcibly terminate running steps
[2020-07-14T21:16:58.886Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-14T21:16:58.899Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats/Lint
[2020-07-14T21:16:59.000Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-14T21:16:59.078Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-14T21:16:59.154Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-14T21:16:59.538Z] + cat
[2020-07-14T21:16:59.538Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-14T21:16:59.538Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-14T21:17:06.151Z] runbld>>> runbld started
[2020-07-14T21:17:06.151Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-14T21:17:08.068Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19482' in order of occurrence in the config (last value wins).
[2020-07-14T21:17:09.457Z] runbld>>> Debug logging enabled.
[2020-07-14T21:17:09.457Z] runbld>>> Storing result
[2020-07-14T21:17:09.718Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-14T21:17:09.718Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200714211709-A9C2AD55
[2020-07-14T21:17:09.718Z] runbld>>> Adding system facts.
[2020-07-14T21:17:10.664Z] runbld>>> Adding vcs info for the latest commit:  3c478d939afea3b2d4fcf6bd364414db0c9be491
[2020-07-14T21:17:10.664Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-14T21:17:10.664Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-14T21:17:10.664Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-14T21:17:10.664Z] Processing JUnit reports with runbld...
[2020-07-14T21:17:11.234Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-14T21:17:11.234Z] runbld>>> DURATION: 26ms
[2020-07-14T21:17:11.234Z] runbld>>> STDOUT: 40 bytes
[2020-07-14T21:17:11.234Z] runbld>>> STDERR: 49 bytes
[2020-07-14T21:17:11.234Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-14T21:17:11.234Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats
[2020-07-14T21:17:12.177Z] runbld>>> Storing build metadata: 
[2020-07-14T21:17:12.177Z] runbld>>> Adding test report.
[2020-07-14T21:17:12.177Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19482/src/github.com/elastic/beats
[2020-07-14T21:17:12.747Z] runbld>>> Found 7 test output files
[2020-07-14T21:17:13.691Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 2598 Skipped: 363
[2020-07-14T21:17:13.691Z] runbld>>> Storing result
[2020-07-14T21:17:13.691Z] runbld>>> FAILURES: 0
[2020-07-14T21:17:13.952Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-14T21:17:13.952Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200714211709-A9C2AD55
[2020-07-14T21:17:13.952Z] runbld>>> Email notification disabled by environment variable.
[2020-07-14T21:17:13.952Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-14T21:17:19.583Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19482
[2020-07-14T21:17:19.728Z] [INFO] getVaultSecret: Getting secrets
[2020-07-14T21:17:19.809Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-14T21:17:20.604Z] + chmod 755 generate-build-data.sh
[2020-07-14T21:17:20.604Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19482/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19482/runs/9 ABORTED 4219864
[2020-07-14T21:17:20.604Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19482/runs/9/steps/?limit=10000 -o steps-info.json
[2020-07-14T21:17:21.154Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19482/runs/9/tests/?status=FAILED -o tests-errors.json
[2020-07-14T21:17:21.405Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19482/runs/9/log/ -o pipeline-log.txt

@andrewkroh
Copy link
Copy Markdown
Member

andrewkroh commented Jul 6, 2020

Looks like there is a problem with the build:

[2020-07-03T08:21:39.971Z] # github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson
[2020-07-03T08:21:39.971Z] vet: input/httpjson/httpjson_test.go:633:8: undeclared name: createTestServer
[2020-07-03T08:21:39.971Z] Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 2

@marc-gr
Copy link
Copy Markdown
Contributor Author

marc-gr commented Jul 6, 2020

Looks like there is a problem with the build:

[2020-07-03T08:21:39.971Z] # github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson
[2020-07-03T08:21:39.971Z] vet: input/httpjson/httpjson_test.go:633:8: undeclared name: createTestServer
[2020-07-03T08:21:39.971Z] Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 2

yep it is pending this #19538 👍

@marc-gr marc-gr force-pushed the backport_19246_7.x branch 7 times, most recently from 28d5615 to 3c478d9 Compare July 14, 2020 20:06
@marc-gr marc-gr force-pushed the backport_19246_7.x branch from 3c478d9 to 21ac1cb Compare July 14, 2020 21:14
@marc-gr
Copy link
Copy Markdown
Contributor Author

marc-gr commented Jul 14, 2020

python test: Integration Testing are passing locally

@marc-gr marc-gr merged commit aaf95a2 into elastic:7.x Jul 14, 2020
@marc-gr marc-gr deleted the backport_19246_7.x branch July 14, 2020 21:54
@zube zube bot removed the [zube]: Done label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@adriansr adriansr adriansr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@marc-gr @elasticmachine @andrewkroh @adriansr