Clarify when to define multiple prospectors#1091
Conversation
|
@ruflin can you have a look, too |
There was a problem hiding this comment.
another use-case is adding 'fields' to a prospector.
There was a problem hiding this comment.
I would actually add fields as the first thing to mention. I think currently this is the most common reason people use multiple prospectors as the assign fields to the files
There was a problem hiding this comment.
one nice use-case I've seen was:
fields:
json: true
fields_under_root: true
Then in logstash you can filter and do json parsing like:
if [json] {
json {...}
}
|
@dedemorton I was thinking about this again. We should definitively have the page you created here. The part I'm not sure about is how we can make sure, people don't get the impression only the configurations mentioned above are reasons to create and additional prospector. Every single configuration in the prospector is a reason for it. Just to mention a few: document_type, scan_frequency, multiline, force_close_files. Should we perhaps show all existing config options (without comments) to make sure people understand that each of this config option they can set per prospector? |
17a83ea to
85236c7
Compare
|
@ruflin I don't think we should add all config options because we would need to worry about updating the list every time we add a new option. Plus it might actually make the text harder to follow. Instead, I've tried to make the text a bit clearer so users understand that there are a bunch of prospector-specific config options. @urso I like the use case of adding fields to the output, so I've used that in the example instead of include_lines and exclude_lines. |
There was a problem hiding this comment.
not sure I like this phrase, as it pretty much sounds like magic:
(each prospector begins with a
-)
maybe we should stress the fact config files are YAML and prospectors: requires a YAML array which normally is written by having every element beginning with -
@ruflin changed configs to have input_type: log right after the - (no empty new line). The docs should do the same (for some reason people like to remove the - itself)
There was a problem hiding this comment.
@urso OK, I'll make this sound less magical. :-) But I think the change for input_type to come after the dash is for master, right, and not 1.1? If so, I'll change the example after I've cherry-picked the commit into master.
|
LGTM. @dedemorton Ready to be merged? We can still improve small things later. |
85236c7 to
b0f9727
Compare
There was a problem hiding this comment.
change 'json' to 'apache'. Setting 'json: true' is kinda unrealistic for apache logs
b0f9727 to
9b69295
Compare
9b69295 to
e4481e1
Compare
|
Changed json to apache, as urso requested |
Clarify when to define multiple prospectors
Remaining changes to close #685
@urso Can you review this and check the example? I took the patterns in the example from the doc, but I am not sure if they are realistic for the files that you might find in the apache2 directory. If they aren't, please suggest a different pattern...I just think we need to show more configuration for the example to demonstrate why someone might want to specify multiple prospectors.