Improve reference docs that describe how to set options dynamically by dedemorton · Pull Request #8290 · elastic/beats

dedemorton · 2018-09-11T21:33:26Z

Closes issue #3227

There's been some confusion about how some of the settings under indices (like mappings and default) work in the ES output. Since other settings (like pipelines) use the same data structures, I've applied my updates to those descriptions, too.

Please suggest a good example to use for the topics section.

Also let me know if the examples I've added are too contrived. I'm open to adding better examples, but mainly wanted to clarify how the settings work.

…settings

exekias · 2018-09-12T09:59:39Z

libbeat/docs/outputconfig.asciidoc

+------------------------------------------------------------------------------
+output.elasticsearch:
+  hosts: ["http://localhost:9200"]
+  index: "%\{[fields.log_type]\}-%\{[beat.version]\}-%\{+yyyy.MM.dd}\"


Perhaps it would be nice to explain that it's always recommended to include beat.version here to avoid mapping issues on upgrades?

exekias

This is looking good, I left a comment on index overriding

dedemorton · 2018-09-15T03:08:29Z

@exekias Can you review the change I pushed and let me know if anyone else needs to review this. Thanks!

exekias

PR looks good to me, I'm afraid I'm not familiar enough with this to provide a meaninful example for kafka output.

Maybe @ph can chime in?

ph

thanks @exekias for the ping, I've looked at the implementation, and we are using mappings in the code, the examples were using the right key but the asciidoc were using mapping.

Everything else look good.

ph · 2018-09-18T14:57:00Z

libbeat/docs/outputconfig.asciidoc

+    - index: "%{[fields.log_type]}"
+      mappings:
+        "critical": "sev1"
+        "normal": "sev2"


There is no need to have quotes for mappings.

critical: "sev1" normal: "sev2"

ph · 2018-09-18T14:57:51Z

libbeat/docs/outputconfig.asciidoc

+    - pipeline: "%{[fields.log_type]}"
+      mappings:
+        "critical": "sev1_pipeline"
+        "normal": "sev2_pipeline"


same comment as above

ph · 2018-09-18T15:03:41Z

libbeat/docs/outputconfig.asciidoc

+references, such as `%{[fields.name]}`, the fields must exist, or the rule fails. 

-*`mapping`*: Dictionary mapping index names to new names
+*`mapping`*:: A dictionary that takes the value returned by `index` and maps it


s/mapping/mappings/

Good catch! I fixed it in the code, but not the descriptions.

ph · 2018-09-18T15:04:03Z

libbeat/docs/outputconfig.asciidoc

+references, such as `%{[fields.name]}`, the fields must exist, or the rule
+fails.  
+
+*`mapping`*:: A dictionary that takes the value returned by `pipeline` and maps


s/mapping/mappings/

ph · 2018-09-18T15:04:15Z

libbeat/docs/outputconfig.asciidoc

+references, such as `%{[fields.name]}`, the fields must exist, or the rule
+fails. 
+
+*`mapping`*:: A dictionary that takes the value returned by `topic` and maps it


s/mapping/mappings/

ph · 2018-09-18T15:04:24Z

libbeat/docs/outputconfig.asciidoc

+fails.

-*`mapping`*: Dictionary mapping key values to new names
+*`mapping`*: A dictionary that takes the value returned by `key` and maps it to


s/mapping/mappings/

dedemorton · 2018-09-19T23:28:11Z

@ph I've fixed the issues that you noticed (see 3a33035). Do you have any ideas for a Kafka example that I can add here: https://github.com/elastic/beats/pull/8290/files#diff-f1a89e6c3a1ce6f296f47ad83b989ff7R872

ph · 2018-09-24T12:31:45Z

@dedemorton LGTM

…lastic#8290) * Improve Elasticsearch output docs about indices, pipelines, and keys settings * Updates from review * Change setting name from mapping to mappings * Remove note to reviewer

…8478) (#8529) * Clarify support for ssl options for modules (#7967) * Clarify support for ssl options for modules * Change example to show http module * Update Elasticsearch module examples to show http in the URL (#8226) * Improve reference docs that describe how to set options dynamically (#8290) * Improve Elasticsearch output docs about indices, pipelines, and keys settings * Updates from review * Change setting name from mapping to mappings * Remove note to reviewer * Fix conditional coding (#8446) * Suppress dashboard info when dashboards aren't available (#8395) * Clarify add_docker_metadata docs (#8478)

…lastic#8290) * Improve Elasticsearch output docs about indices, pipelines, and keys settings * Updates from review * Change setting name from mapping to mappings * Remove note to reviewer

…8478) (#8528) * Clarify support for ssl options for modules (#7967) * Clarify support for ssl options for modules * Change example to show http module * Update Elasticsearch module examples to show http in the URL (#8226) * Improve reference docs that describe how to set options dynamically (#8290) * Improve Elasticsearch output docs about indices, pipelines, and keys settings * Updates from review * Change setting name from mapping to mappings * Remove note to reviewer * Fix conditional coding (#8446) * Suppress dashboard info when dashboards aren't available (#8395) * Clarify add_docker_metadata docs (#8478)

…tic#8290 elastic#8395 elastic#8446 elastic#8478) (elastic#8528) * Clarify support for ssl options for modules (elastic#7967) * Clarify support for ssl options for modules * Change example to show http module * Update Elasticsearch module examples to show http in the URL (elastic#8226) * Improve reference docs that describe how to set options dynamically (elastic#8290) * Improve Elasticsearch output docs about indices, pipelines, and keys settings * Updates from review * Change setting name from mapping to mappings * Remove note to reviewer * Fix conditional coding (elastic#8446) * Suppress dashboard info when dashboards aren't available (elastic#8395) * Clarify add_docker_metadata docs (elastic#8478)

Improve Elasticsearch output docs about indices, pipelines, and keys …

0024887

…settings

dedemorton added docs review needs_backport PR is waiting to be backported to other branches. labels Sep 11, 2018

exekias reviewed Sep 12, 2018

View reviewed changes

Updates from review

77a37aa

exekias approved these changes Sep 17, 2018

View reviewed changes

ph suggested changes Sep 18, 2018

View reviewed changes

Change setting name from mapping to mappings

3a33035

ph approved these changes Sep 24, 2018

View reviewed changes

Remove note to reviewer

e0d1b53

dedemorton mentioned this pull request Sep 24, 2018

Kafka output config docs need config example under "topics" setting #8423

Closed

dedemorton merged commit eb6ab40 into elastic:master Sep 25, 2018

dedemorton mentioned this pull request Sep 29, 2018

Improve documentation about the indices #3227

Closed

dedemorton mentioned this pull request Oct 1, 2018

Backport to 6.4: mutliple doc changes (#7967 #8226 #8290 #8395 #8446 #8478) #8528

Merged

dedemorton mentioned this pull request Oct 1, 2018

Backport to 6.x: mutliple doc changes (#7967 #8226 #8290 #8395 #8446 #8478) #8529

Merged

dedemorton deleted the issue#3227 branch October 1, 2018 23:34

dedemorton removed the needs_backport PR is waiting to be backported to other branches. label Oct 1, 2018

Conversation

dedemorton commented Sep 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

exekias left a comment

Choose a reason for hiding this comment

Uh oh!

dedemorton commented Sep 15, 2018

Uh oh!

exekias left a comment

Choose a reason for hiding this comment

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dedemorton commented Sep 19, 2018

Uh oh!

ph commented Sep 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants