Cherry-pick #15568 to 7.x: Fix panic: don't send events if client is nil by mtojek · Pull Request #15677 · elastic/beats

mtojek · 2020-01-20T13:33:21Z

Cherry-pick of PR #15568 to 7.x branch. Original message:

This PR fixes panic reported in https://travis-ci.org/elastic/beats/jobs/637053125?utm_medium=notification&utm_source=github_status

=== RUN   TestSendMultipleViaLogstashTLS
2020-01-14T20:25:29.189Z	DEBUG	[tls]	tlscommon/tls.go:155	successfully loaded CA certificate: ../../../testing/environments/docker/logstash/pki/tls/certs/logstash.crt
2020-01-14T20:25:29.189Z	DEBUG	[logstash]	logstash/async.go:111	connect
2020-01-14T20:25:30.363Z	INFO	elasticsearch/client.go:174	Elasticsearch url: http://elasticsearch:9200
2020-01-14T20:25:30.363Z	DEBUG	[elasticsearch]	elasticsearch/client.go:774	DELETE http://elasticsearch:9200/beat-logstash-int-multiple-tls-7756-2020.01.14  <nil>
2020-01-14T20:25:30.370Z	DEBUG	[elasticsearch]	elasticsearch/client.go:774	PUT http://elasticsearch:9200/beat-logstash-int-multiple-tls-7756-2020.01.14  {"settings":{"number_of_replicas":0,"number_of_shards":1}}
2020-01-14T20:25:30.679Z	DEBUG	[logstash]	logstash/async.go:159	1 events out of 1 events sent to logstash host logstash:5055. Continue sending
2020-01-14T20:25:30.680Z	DEBUG	[logstash]	logstash/async.go:159	1 events out of 1 events sent to logstash host logstash:5055. Continue sending
2020-01-14T20:25:53.276Z	DEBUG	[transport]	transport/client.go:218	handle error: read tcp 172.18.0.9:55626->172.18.0.8:5044: i/o timeout
2020-01-14T20:25:53.277Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:55626->172.18.0.8:5044: i/o timeout
2020-01-14T20:25:53.277Z	DEBUG	[transport]	transport/client.go:131	closing
2020-01-14T20:25:57.470Z	DEBUG	[transport]	transport/client.go:218	handle error: read tcp 172.18.0.9:58642->172.18.0.8:5055: i/o timeout
2020-01-14T20:25:57.470Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:58642->172.18.0.8:5055: i/o timeout
2020-01-14T20:25:57.470Z	DEBUG	[transport]	transport/client.go:131	closing
2020-01-14T20:25:58.733Z	DEBUG	[transport]	transport/client.go:218	handle error: read tcp 172.18.0.9:55676->172.18.0.8:5044: i/o timeout
2020-01-14T20:25:58.733Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:55676->172.18.0.8:5044: i/o timeout
2020-01-14T20:25:58.734Z	DEBUG	[transport]	transport/client.go:131	closing
2020-01-14T20:26:00.682Z	DEBUG	[transport]	transport/client.go:218	handle error: read tcp 172.18.0.9:58688->172.18.0.8:5055: i/o timeout
2020-01-14T20:26:00.682Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:58688->172.18.0.8:5055: i/o timeout
2020-01-14T20:26:00.682Z	DEBUG	[transport]	transport/client.go:131	closing
2020-01-14T20:26:00.682Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:58688->172.18.0.8:5055: i/o timeout
2020-01-14T20:26:00.682Z	DEBUG	[logstash]	logstash/async.go:159	1 events out of 1 events sent to logstash host logstash:5055. Continue sending
2020-01-14T20:26:00.682Z	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp 172.18.0.9:58688->172.18.0.8:5055: i/o timeout
2020-01-14T20:26:00.683Z	DEBUG	[logstash]	logstash/async.go:159	1 events out of 1 events sent to logstash host logstash:5055. Continue sending
2020-01-14T20:26:00.683Z	DEBUG	[logstash]	logstash/async.go:116	close connection
2020-01-14T20:26:00.683Z	DEBUG	[logstash]	logstash/async.go:116	close connection
2020-01-14T20:26:00.683Z	ERROR	logstash/async.go:256	Failed to publish events caused by: client is not connected
2020-01-14T20:26:02.155Z	DEBUG	[elasticsearch]	elasticsearch/client.go:774	DELETE http://elasticsearch:9200/beat-logstash-int-multiple-tls-7756-2020.01.14  <nil>
--- FAIL: TestSendMultipleViaLogstashTLS (33.07s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb1b8b6]
goroutine 129 [running]:
testing.tRunner.func1(0xc0001af000)
	/usr/local/go/src/testing/testing.go:874 +0x3a3
panic(0xc99bc0, 0x1221e80)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/elastic/beats/vendor/github.com/elastic/go-lumber/client/v2.(*AsyncClient).Send(0x0, 0xc000030040, 0xc000030030, 0x1, 0x1, 0x6fc5fa, 0xc0000e1c01)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-lumber/client/v2/async.go:103 +0x26
github.com/elastic/beats/libbeat/outputs/logstash.(*asyncClient).sendEvents(0xc00013cb40, 0xc00017a600, 0xc00011c050, 0x1, 0x1, 0xc0000e1da0, 0xbdae62)
	/go/src/github.com/elastic/beats/libbeat/outputs/logstash/async.go:205 +0x119
github.com/elastic/beats/libbeat/outputs/logstash.(*asyncClient).Publish(0xc00013cb40, 0xe28820, 0xc000020140, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/outputs/logstash/async.go:154 +0x46b
github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc00013ce40, 0xe28820, 0xc000020140, 0xc000020140, 0x7cc0e6a5061e3f7c)
	/go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b
github.com/elastic/beats/libbeat/outputs/logstash.(*testOutputer).PublishEvent(0xc00000e160, 0xbf7fa6d6894ae90a, 0x91b5bec13, 0x126b800, 0x0, 0xc0003620c0, 0x0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/outputs/logstash/logstash_integration_test.go:548 +0xb8
github.com/elastic/beats/libbeat/outputs/logstash.testSendMultipleViaLogstash(0xc0001af000, 0xd53915, 0xc, 0xe448268201)
	/go/src/github.com/elastic/beats/libbeat/outputs/logstash/logstash_integration_test.go:338 +0x155
github.com/elastic/beats/libbeat/outputs/logstash.TestSendMultipleViaLogstashTLS(0xc0001af000)
	/go/src/github.com/elastic/beats/libbeat/outputs/logstash/logstash_integration_test.go:323 +0x45
testing.tRunner(0xc0001af000, 0xd73ac8)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350
FAIL	github.com/elastic/beats/libbeat/outputs/logstash	39.240s
?   	github.com/elastic/beats/libbeat/outputs/outest	[no test files]

The panic occurs if the client connection has been closed, but there is a need to send some more events.

* Fix panic: don't send events if client is nil * Use mutex * Add CHANGELOG entry * Rename changelog entry * Fix: changelog * Temporarily use specific logstash release (cherry picked from commit 3e39fdf)

Fix panic: don't send events if client is nil (elastic#15568)

b5e70e2

* Fix panic: don't send events if client is nil * Use mutex * Add CHANGELOG entry * Rename changelog entry * Fix: changelog * Temporarily use specific logstash release (cherry picked from commit 3e39fdf)

mtojek added backport review labels Jan 20, 2020

Fix CHANGELOG

dda5d5d

mtojek requested review from a team and urso January 20, 2020 13:39

urso approved these changes Jan 20, 2020

View reviewed changes

mtojek merged commit 7ef21b1 into elastic:7.x Jan 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #15568 to 7.x: Fix panic: don't send events if client is nil#15677

Cherry-pick #15568 to 7.x: Fix panic: don't send events if client is nil#15677
mtojek merged 2 commits intoelastic:7.xfrom
mtojek:backport_15568_7.x

mtojek commented Jan 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mtojek commented Jan 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants