Skip to content

Add option to adjust _source options + improvements#4317

Merged
ruflin merged 1 commit intoelastic:masterfrom
tsg:template_source_settings
May 16, 2017
Merged

Add option to adjust _source options + improvements#4317
ruflin merged 1 commit intoelastic:masterfrom
tsg:template_source_settings

Conversation

@tsg
Copy link
Copy Markdown
Contributor

@tsg tsg commented May 15, 2017 •
edited
Loading

This PR adds/fixes:

  • Ability to disable _source, or set other _source related options
  • Fixes the overwrite logic (was using the wrong template name on the check)
  • Fixes error handling
  • Integration tests for overwritting

The source can be disabled like this:

setup.template:
  settings._source.enabled: false

And _source can be also partially enabled via "includes" and "excludes", for example:

setup.template:
  settings._source.excludes: ["metricset", "system.process.cmdline"]

Adding the discuss label for naming. I hesitated between using _source or source_settings for these options, but I like the look of _source.enabled: false.

Part of #3654 and #4112.

Remaining TODOs:

  • Update docs
  • Changelog

@tsg tsg added discuss Issue needs further discussion. in progress Pull request is currently in progress. review labels May 15, 2017
@tsg tsg force-pushed the template_source_settings branch from eb48b26 to a9fc7d1 Compare May 15, 2017 11:56
@tsg tsg mentioned this pull request May 15, 2017
Closed
9 tasks
ruflin
ruflin reviewed May 15, 2017
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

libbeat/template/config.go Outdated
Copy link
Copy Markdown
Contributor

@ruflin ruflin May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first I was also sceptical about using _source but I think for ES users it must be quite intuitive to use it like this.

libbeat/template/template.go Outdated
Copy link
Copy Markdown
Contributor

@ruflin ruflin May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing both names I was thinking if it could be settings._source and settings.x but I would not know how to call x :-(

Is there are stuff we potentially going to add here?

Copy link
Copy Markdown
Contributor Author

@tsg tsg May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can imagine there could be more yes (e.g. _all), although I don't plan more at the moment. We could maybe use index for x, so you'd have: settings.index.number_of_shards and settings._source.enabled: false. Trying to figure out if there can be any settings that do not fall under index...

@tsg
Copy link
Copy Markdown
Contributor Author

tsg commented May 15, 2017

I've updated the code to have settings.index for the global index settings and settings._source for the _source settings. Elasticsearch is a bit lenient when it comes to the index settings in the template. Both of these work:

"settings": {
  "number_of_shards": 1
}

and

"settings"; {
  "index": {
    "number_of_shards": 1
  }
}

With this change, we are can only use the second version. I think that's good anyway, as we don't want two ways of accomplishing the same thing.

elasticmachine
elasticmachine reviewed May 15, 2017
View reviewed changes
libbeat/template/config.go Outdated
Copy link
Copy Markdown
Contributor

@elasticmachine elasticmachine May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golint] reported by reviewdog 🐶
exported type TemplateSettings should have comment or be unexported

libbeat/template/config.go Outdated
Copy link
Copy Markdown
Contributor

@elasticmachine elasticmachine May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golint] reported by reviewdog 🐶
type name will be used as template.TemplateSettings by other packages, and that stutters; consider calling this Settings

@tsg tsg removed in progress Pull request is currently in progress. discuss Issue needs further discussion. labels May 15, 2017
Add option to adjust _source options + improvements
3072a79 
This adds/fixes:

* Ability to disable _source, or set other _source related options
* Moves template index settings under `settings.index`
* Fixes the overwrite logic (was using the wrong template name on the check)
* Fixes error handling
* Integration tests for overwritting

Part of elastic#3654 and elastic#4112.
@tsg tsg force-pushed the template_source_settings branch from 289450b to 3072a79 Compare May 15, 2017 21:01
ruflin
ruflin approved these changes May 16, 2017
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the new structure with the index prefix of the settings. Left a minor docs comment.

libbeat/docs/template-config.asciidoc
@@ -35,3 +35,17 @@ setup.template.settings:
index.number_of_shards: 1
Copy link
Copy Markdown
Contributor

@ruflin ruflin May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be updated to put index on the line above?

Copy link
Copy Markdown
Contributor Author

@tsg tsg May 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of like it like it is now because it matches the Elasticsearch docs where they usually talk about index.* as the name of the options: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

@ruflin ruflin merged commit b2ccdcb into elastic:master May 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@elasticmachine elasticmachine elasticmachine left review comments

@ruflin ruflin ruflin approved these changes

Assignees

No one assigned

Labels

review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tsg @ruflin @elasticmachine