Skip to content

Add automatic retries and exponential backoff to Filebeat httpjson input#18956

Merged
alakahakai merged 9 commits intoelastic:masterfrom
alakahakai:retryablehttp-okta
Jun 19, 2020
Merged

Add automatic retries and exponential backoff to Filebeat httpjson input#18956
alakahakai merged 9 commits intoelastic:masterfrom
alakahakai:retryablehttp-okta

Conversation

@alakahakai
Copy link
Copy Markdown

@alakahakai alakahakai commented Jun 3, 2020 •
edited
Loading

Add automatic retries and exponential backoff to Filebeat httpjson input. If an HTTP error is returned by the client, e.g., connection errors, or if a 500-range response code is received (except 501), then a retry is automatically invoked after a defined wait period.

go test result:

=== RUN   TestProviderCanonical
--- PASS: TestProviderCanonical (0.00s)
=== RUN   TestGetProviderIsCanonical
--- PASS: TestGetProviderIsCanonical (0.00s)
=== RUN   TestIsEnabled
--- PASS: TestIsEnabled (0.00s)
=== RUN   TestGetTokenURL
--- PASS: TestGetTokenURL (0.00s)
=== RUN   TestGetTokenURLWithAzure
--- PASS: TestGetTokenURLWithAzure (0.00s)
=== RUN   TestGetEndpointParams
--- PASS: TestGetEndpointParams (0.00s)
=== RUN   TestGetEndpointParamsWithAzure
--- PASS: TestGetEndpointParamsWithAzure (0.00s)
=== RUN   TestConfigValidationCase1
--- PASS: TestConfigValidationCase1 (0.00s)
=== RUN   TestConfigValidationCase2
--- PASS: TestConfigValidationCase2 (0.00s)
=== RUN   TestConfigValidationCase3
--- PASS: TestConfigValidationCase3 (0.00s)
=== RUN   TestConfigValidationCase4
--- PASS: TestConfigValidationCase4 (0.00s)
=== RUN   TestConfigValidationCase5
--- PASS: TestConfigValidationCase5 (0.00s)
=== RUN   TestConfigValidationCase6
--- PASS: TestConfigValidationCase6 (0.00s)
=== RUN   TestConfigValidationCase7
--- PASS: TestConfigValidationCase7 (0.00s)
=== RUN   TestConfigOauth2Validation
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together
=== RUN   TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme
=== RUN   TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set
=== RUN   TestConfigOauth2Validation/must_fail_with_an_unknown_provider
=== RUN   TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url
=== RUN   TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id
=== RUN   TestConfigOauth2Validation/azure_must_have_client_credentials_set
=== RUN   TestConfigOauth2Validation/azure_config_is_valid
=== RUN   TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set
=== RUN   TestConfigOauth2Validation/google_must_fail_if_no_ADC_available
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found
=== RUN   TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set
=== RUN   TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON
=== RUN   TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON
--- PASS: TestConfigOauth2Validation (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together (0.00s)
    --- PASS: TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme (0.00s)
    --- PASS: TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set (0.00s)
    --- PASS: TestConfigOauth2Validation/must_fail_with_an_unknown_provider (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_config_is_valid (0.00s)
    --- PASS: TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_no_ADC_available (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON (0.00s)
=== RUN   TestGetNextLinkFromHeader
--- PASS: TestGetNextLinkFromHeader (0.00s)
=== RUN   TestCreateRequestInfoFromBody
--- PASS: TestCreateRequestInfoFromBody (0.00s)
=== RUN   TestGetRateLimitCase1
--- PASS: TestGetRateLimitCase1 (0.00s)
=== RUN   TestGetRateLimitCase2
--- PASS: TestGetRateLimitCase2 (0.00s)
=== RUN   TestGetRateLimitCase3
--- PASS: TestGetRateLimitCase3 (0.00s)
=== RUN   TestGET
2020-06-16T20:53:14.947-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53858"}
2020-06-16T20:53:14.947-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53858
--- PASS: TestGET (0.00s)
=== RUN   TestGetHTTPS
2020-06-16T20:53:14.948-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53860"}
2020-06-16T20:53:14.948-0700	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-16T20:53:14.948-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53860
--- PASS: TestGetHTTPS (0.00s)
=== RUN   TestRateLimitRetry
2020-06-16T20:53:14.949-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:325	HTTP request failed	{"url": "http://127.0.0.1:53862", "http.response.status_code": 429, "http.response.body": ""}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:262	Rate Limit: No need to apply rate limit.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
--- PASS: TestRateLimitRetry (0.00s)
=== RUN   TestErrorRetry
2020-06-16T20:53:14.950-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53865"}
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53865
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 506): retrying in 1s (5 left)
2020-06-16T20:53:15.951-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 508): retrying in 2s (4 left)
--- PASS: TestErrorRetry (3.01s)
=== RUN   TestArrayResponse
2020-06-16T20:53:17.955-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53900"}
2020-06-16T20:53:17.956-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53900
--- PASS: TestArrayResponse (0.00s)
=== RUN   TestPOST
2020-06-16T20:53:17.957-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53902"}
2020-06-16T20:53:17.957-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53902
--- PASS: TestPOST (0.00s)
=== RUN   TestRepeatedPOST
2020-06-16T20:53:17.958-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:17.958-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:20.963-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:20.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.962-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:23.962-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:435	Context done.	{"url": "http://127.0.0.1:53904"}
--- PASS: TestRepeatedPOST (6.01s)
=== RUN   TestRunStop
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:133	httpjson input worker has started.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53970
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:551	[ERR] GET http://127.0.0.1:53970 request failed: Get http://127.0.0.1:53970: context canceled
2020-06-16T20:53:23.963-0700	ERROR	[httpjson]	httpjson/input.go:138	failed to execute http client.Do: Get http://127.0.0.1:53970: context canceled	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:139	httpjson input worker has stopped.	{"url": "http://127.0.0.1:53970"}
--- PASS: TestRunStop (0.00s)
=== RUN   TestOAuth2
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53972"}
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53971
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53972
--- PASS: TestOAuth2 (0.00s)
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson	9.214s
(base)

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 3, 2020 •
edited
Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-18T15:15:42.525+0000

  • Duration: 124 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 8464
Skipped 1553
Total 10017

Steps errors

Expand to view the steps failures

  • Name: Make -C libbeat testsuite

    • Description: make -C libbeat testsuite

    • Duration: 6 min 7 sec

    • Start Time: 2020-06-18T15:39:31.209+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.cancel(TimeoutStepExecution.java:165)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.access$100(TimeoutStepExecution.java:38)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$1.run(TimeoutStepExecution.java:139)
[2020-06-18T17:18:49.067Z] 	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-06-18T17:18:49.067Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-06-18T17:18:49.067Z] 
[2020-06-18T17:18:49.083Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-18T17:18:49.267Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Lint
[2020-06-18T17:18:49.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-18T17:18:50.074Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-18T17:18:50.342Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-18T17:18:50.827Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-18T17:18:51.350Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-18T17:18:51.718Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-18T17:18:52.066Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-18T17:18:52.456Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-oss
[2020-06-18T17:18:52.742Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-18T17:18:53.092Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-18T17:18:53.590Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-18T17:18:53.996Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-18T17:18:54.467Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-18T17:18:54.841Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-18T17:18:55.109Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-18T17:18:55.339Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-18T17:18:55.661Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-18T17:18:55.908Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-18T17:18:56.187Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-18T17:18:56.551Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-18T17:18:56.775Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-18T17:18:56.956Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-18T17:18:57.239Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-18T17:18:57.441Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-18T17:18:57.659Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-18T17:18:57.848Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-oss
[2020-06-18T17:18:58.060Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-18T17:18:58.245Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-18T17:18:58.463Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-18T17:18:58.697Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-18T17:18:58.999Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-18T17:18:59.400Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-18T17:18:59.607Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-18T17:18:59.769Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-18T17:19:00.030Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-18T17:19:00.197Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-18T17:19:00.372Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-18T17:19:00.922Z] + cat
[2020-06-18T17:19:00.922Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-18T17:19:00.922Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-18T17:19:07.692Z] runbld>>> runbld started
[2020-06-18T17:19:07.692Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-06-18T17:19:08.663Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18956' in order of occurrence in the config (last value wins).
[2020-06-18T17:19:10.102Z] runbld>>> Debug logging enabled.
[2020-06-18T17:19:10.102Z] runbld>>> Storing result
[2020-06-18T17:19:10.102Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-18T17:19:10.102Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:10.102Z] runbld>>> Adding system facts.
[2020-06-18T17:19:10.692Z] runbld>>> Adding vcs info for the latest commit:  4657cbfc603647101b87550a45d0124e94ccbf92
[2020-06-18T17:19:10.692Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-18T17:19:10.971Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-18T17:19:10.971Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-18T17:19:10.971Z] Processing JUnit reports with runbld...
[2020-06-18T17:19:11.262Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-18T17:19:11.262Z] runbld>>> DURATION: 29ms
[2020-06-18T17:19:11.262Z] runbld>>> STDOUT: 40 bytes
[2020-06-18T17:19:11.262Z] runbld>>> STDERR: 49 bytes
[2020-06-18T17:19:11.262Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-18T17:19:11.262Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:12.691Z] runbld>>> Storing build metadata: 
[2020-06-18T17:19:12.691Z] runbld>>> Adding test report.
[2020-06-18T17:19:12.691Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:14.103Z] runbld>>> Found 105 test output files
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-18T17:19:16.148Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 9867 Skipped: 1316
[2020-06-18T17:19:16.148Z] runbld>>> Storing result
[2020-06-18T17:19:16.148Z] runbld>>> FAILURES: 0
[2020-06-18T17:19:16.415Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-18T17:19:16.415Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:16.415Z] runbld>>> Email notification disabled by environment variable.
[2020-06-18T17:19:16.415Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-18T17:19:22.124Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956
[2020-06-18T17:19:22.285Z] [INFO] getVaultSecret: Getting secrets
[2020-06-18T17:19:22.358Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-18T17:19:23.479Z] + chmod 755 generate-build-data.sh
[2020-06-18T17:19:23.479Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18 ABORTED 7420646
[2020-06-18T17:19:23.479Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/steps/?limit=10000 -o steps-info.json
[2020-06-18T17:19:25.962Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/tests/?status=FAILED -o tests-errors.json
[2020-06-18T17:19:27.305Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/log/ -o pipeline-log.txt

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 3, 2020

Pinging @elastic/siem (Team:SIEM)

@alakahakai
Copy link
Copy Markdown
Author

alakahakai commented Jun 8, 2020

jenkins, test this

andrewkroh
andrewkroh requested changes Jun 8, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please update the documentation for the httpjson input to include these new options.

@andrewkroh andrewkroh requested a review from marc-gr June 8, 2020 23:03
@alakahakai alakahakai requested a review from andrewkroh June 17, 2020 04:22
@alakahakai
Copy link
Copy Markdown
Author

alakahakai commented Jun 18, 2020

@andrewkroh Please review again. thanks

andrewkroh
andrewkroh approved these changes Jun 18, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alakahakai alakahakai merged commit 0dcb3df into elastic:master Jun 19, 2020
@andrewkroh andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Jun 30, 2020
@andrewkroh
Copy link
Copy Markdown
Member

andrewkroh commented Jun 30, 2020 •
edited
Loading

Please backport this to 7.8 and 7.x.

@marc-gr marc-gr mentioned this pull request Jul 1, 2020
Merged
@marc-gr marc-gr added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 1, 2020
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Jul 13, 2020
@marc-gr
Add automatic retries and exponential backoff to Filebeat httpjson in…
94a0bdf 
…put (elastic#18956)

- Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956)
- Add test cases
- Support the new OAuth2 functionality

(cherry picked from commit 0dcb3df)
marc-gr added a commit that referenced this pull request Jul 13, 2020
@marc-gr
Add automatic retries and exponential backoff to Filebeat httpjson in…
ce7ccb2 
…put (#18956) (#19538)

- Add automatic retries and exponential backoff to Filebeat httpjson input (#18956)
- Add test cases
- Support the new OAuth2 functionality

(cherry picked from commit 0dcb3df)

Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
@andrewkroh andrewkroh mentioned this pull request Aug 13, 2020
Closed
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@melchiormoulin
Add automatic retries and exponential backoff to Filebeat httpjson in…
96c079a 
…put (elastic#18956)

- Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956)
- Add test cases
- Support the new OAuth2 functionality
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@leehinman leehinman Awaiting requested review from leehinman

@marc-gr marc-gr Awaiting requested review from marc-gr

Assignees

No one assigned

Labels

enhancement Filebeat Filebeat review v7.9.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@alakahakai @elasticmachine @andrewkroh @marc-gr