When the monitoring user does not have the required privileges to write into .monitoring-beats-* but does have other indexing privileges (e.g. to write to auditbeat-*), a Beat will not log any errors about the failed monitoring bulk index requests.
The reason is that while Elasticsearch will return 403 Forbidden when the user has no write privileges at all, it will return 200 for a _bulk request if a user has at least one write privilege - even if it's not for the monitoring index.
Instead, errors are going to be returned in the body, e.g. see the following (I had to add a debug statement into the code, this is not what a Beat actually prints):
2019-10-29T10:33:46.579Z DEBUG [elasticsearch] elasticsearch/client.go:835 POST http://localhost:9200/.monitoring-beats-7-2019.10.29/_bulk 200 {"took":0,"errors":true,"items":[{"index":{"_index":".monitoring-beats-7-2019.10.29","_id":null,"status":403,"error":{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [auditbeat]"}}}]} {}
For normal event publishing, Libbeat parses the return body in a function called bulkCollectPublishFails:
|
// bulkCollectPublishFails checks per item errors returning all events |
|
// to be tried again due to error code returned for that items. If indexing an |
|
// event failed due to some error in the event itself (e.g. does not respect mapping), |
|
// the event will be dropped. |
|
func bulkCollectPublishFails( |
But this does not happen for the body of monitoring data bulk requests.
I tried to fix this, but the two logics for sending collected vs. monitoring data are quite different so couldn't do it easily.
/cc @ycombinator - seems you've touched this code most recently
When the monitoring user does not have the required privileges to write into
.monitoring-beats-*but does have other indexing privileges (e.g. to write toauditbeat-*), a Beat will not log any errors about the failed monitoring bulk index requests.The reason is that while Elasticsearch will return
403 Forbiddenwhen the user has no write privileges at all, it will return200for a_bulkrequest if a user has at least one write privilege - even if it's not for the monitoring index.Instead, errors are going to be returned in the body, e.g. see the following (I had to add a debug statement into the code, this is not what a Beat actually prints):
For normal event publishing, Libbeat parses the return body in a function called
bulkCollectPublishFails:beats/libbeat/outputs/elasticsearch/client.go
Lines 471 to 475 in 9b96d62
But this does not happen for the body of monitoring data bulk requests.
I tried to fix this, but the two logics for sending collected vs. monitoring data are quite different so couldn't do it easily.
/cc @ycombinator - seems you've touched this code most recently