Add default_field option to fields.yml#14341
Conversation
dcd77f4 to
28a1e5b
Compare
|
I'm expecting a test failure for Filebeat because it has over 1024 fields. Then I'll push a fix to make it go green. |
438b8d1 to
aef5c0e
Compare
The number of fields in the Elasticsearch index template's `settings.index.query.default_field` option has grown over time, and is now greater than 1024 in Filebeat (Elastic licensed version). This causes queries to Elasticsearch to fail when a list of fields is not specified because there is a default limit of 1024 in Elasticsearch. This adds a new setting to fields.yml called `default_field` whose value can be true/false (defaults to true). When true the text/keyword fields are added to the `default_field` list (as was the behavior before this change). And when set to false the field is omitted from the default_field list. This adds a test for every beat to check if the default_field list contains more than 1000 fields. The limit is a little less than 1024 because `fields.*` is in the default_field list already and at query time that wildcard will be expanded and count toward the limit. Fixes elastic#14262
aef5c0e to
33c3573
Compare
|
I trimmed out the latest additions from the Zeek module and now x-pack/filebeat is showing 939 and queries in Kibana are working. |
ruflin
left a comment
There was a problem hiding this comment.
LGTM
Glad to have the main_test.go now for all Beats.
Can't approve the zeek changes, would be good to have someone from the team doing this.
| if !ok { | ||
| t.Fatalf("settings.index.query.default_field value has an unexpected type: %T", v) | ||
| } | ||
|
|
There was a problem hiding this comment.
What about also executing a elasticsearch query here in addition to catch potential other errors. Say for example ES decides to lower it to 512?
There was a problem hiding this comment.
That would need to be added somewhere else IMO. I put the test here so that it could be executed quickly as a unit test and not have an external dependency on ES.
This could be put into system test. Each beat should probably call setup to install its template and ILM policy, index a document, then perform a query.
There was a problem hiding this comment.
Agree, this would fit well here: https://github.com/elastic/beats/blob/master/filebeat/tests/system/test_base.py#L30 Perhaps someone in the team from @urso could follow up with this?
There was a problem hiding this comment.
@andrewkroh @ruflin I've created #14352 to make a change in libbeat to be more proactive to detect that error.
|
@ph I think that falls under SIEM. |
libbeat/template/processor.go
Outdated
| switch field.Type { | ||
| case "", "keyword", "text": | ||
| addToDefaultFields(&field) | ||
| if field.DefaultField == nil || *field.DefaultField { |
There was a problem hiding this comment.
I don't like that the nil check above means a DefaultField==nil behaves as an implicit default to true. It's my understanding that the check in line 59 prevents this from ever being nil, but still looks dangerous to me. I would turn the nil check into an error.
This really doesn't matter much unless we expect that some day the default for 'default_field' will be false, so feel free to ignore.
* Add default_field option to fields.yml The number of fields in the Elasticsearch index template's `settings.index.query.default_field` option has grown over time, and is now greater than 1024 in Filebeat (Elastic licensed version). This causes queries to Elasticsearch to fail when a list of fields is not specified because there is a default limit of 1024 in Elasticsearch. This adds a new setting to fields.yml called `default_field` whose value can be true/false (defaults to true). When true the text/keyword fields are added to the `default_field` list (as was the behavior before this change). And when set to false the field is omitted from the default_field list. This adds a test for every beat to check if the default_field list contains more than 1000 fields. The limit is a little less than 1024 because `fields.*` is in the default_field list already and at query time that wildcard will be expanded and count toward the limit. Fixes elastic#14262 * Exclude new zeek datasets from default_field list (cherry picked from commit 9f21b96)
* Add default_field option to fields.yml The number of fields in the Elasticsearch index template's `settings.index.query.default_field` option has grown over time, and is now greater than 1024 in Filebeat (Elastic licensed version). This causes queries to Elasticsearch to fail when a list of fields is not specified because there is a default limit of 1024 in Elasticsearch. This adds a new setting to fields.yml called `default_field` whose value can be true/false (defaults to true). When true the text/keyword fields are added to the `default_field` list (as was the behavior before this change). And when set to false the field is omitted from the default_field list. This adds a test for every beat to check if the default_field list contains more than 1000 fields. The limit is a little less than 1024 because `fields.*` is in the default_field list already and at query time that wildcard will be expanded and count toward the limit. Fixes elastic#14262 * Exclude new zeek datasets from default_field list
The number of fields in the Elasticsearch index template's
settings.index.query.default_fieldoption has grown over time, and is now greater than 1024 in Filebeat (Elastic licensed version). This causes queries to Elasticsearch to fail when a list of fields is not specified because there is a default limit of 1024 in Elasticsearch.This adds a new setting to fields.yml called
default_fieldwhose value can be true/false (defaults to true). When true the text/keyword fields are added to thedefault_fieldlist (as was the behavior before this change). And when set to false the field is omitted from the default_field list.This adds a test for every beat to check if the default_field list contains more than 1000 fields. The limit is a little less than 1024 because
fields.*is in the default_field list already and at query time that wildcard will be expanded and count toward the limit.Fixes #14262