Skip to content

multiline processing for tail input plugin#5603

Closed
javicrespo wants to merge 5 commits intoinfluxdata:masterfrom
javicrespo:master
Closed

multiline processing for tail input plugin#5603
javicrespo wants to merge 5 commits intoinfluxdata:masterfrom
javicrespo:master

Conversation

@javicrespo
Copy link
Copy Markdown
Contributor

@javicrespo javicrespo commented Mar 18, 2019

Implements #1818/#3228

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

@javicrespo javicrespo marked this pull request as ready for review March 18, 2019 22:37
@glinton glinton added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin area/logging labels Mar 19, 2019
@0UserName
Copy link
Copy Markdown

When you plan to merge this PR?

@lsitzmann
Copy link
Copy Markdown

Hi.
I have a short question about this PR. In the docs you reference https://www.elastic.co/guide/en/logstash/2.4/plugins-filters-multiline.html and there they say that this multiline-implementation isn't thread-safe.
Does this also apply for this implementation in telegraf? Is it safely possible to tail multiple streams with multiline-support using this implementation?

Thanks, Lars

@javicrespo
Copy link
Copy Markdown
Contributor Author

@lsitzmann I referenced the logstash doc because the configuration options are a heavily inspired (or rather a copy) by the logstash plugin.
With regards to thread safety, the tailer implementation is based on goroutines so it should be thread safe (take this with a grain of salt cause I'm nowhere near being a Golang expert).
Each file is individually tailed and the multi line buffering is also managed in a separate and dedicated golang channel.

@lsitzmann
Copy link
Copy Markdown

@danielnelson : Are there any chances, that this PR will be merged soon? Currently we have to manually copy log files from several production servers because the data collected with telegraf isn't complete.
We are really looking foreward for this feature.

@lijingwei9060
Copy link
Copy Markdown

@javicrespo , i think channelOpen(telegraf/plugins/inputs/tail/tail.go line 212) should be initiated with true, otherwise i will get "Error in plugin: E! Error tailing file /var/log/bingocloud/cc2.log, Error: tomb: still alive"

@javicrespo
Copy link
Copy Markdown
Contributor Author

@lijingwei9060 Should be fixed with the last commit. I've also rewritten the timeout unit test to repro the bug.

@lsitzmann
Copy link
Copy Markdown

Hi again.
We are using this PR in a production environment for about one month now. We parsed about 500k lines of logfiles with it and it seems to work very reliably - at least in our configuration.
About 1% of the log-entries are multi-lines.

We'd really appreciate to switch back to an officially released version of telegraf so once again my question: is it foreseeable when this PR will make it into an release??

@danielnelson danielnelson added this to the 1.12.0 milestone Jun 5, 2019
@danielnelson
Copy link
Copy Markdown
Contributor

I've added this to the 1.12 milestone, sorry I know it is frustrating to wait for pull requests to be merged but I feel code review is required in order to keep our bug count at a reasonable level.

Copy link
Copy Markdown
Contributor

@glinton glinton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sjwang90 sjwang90 modified the milestones: 1.13.0, 1.14.0 Nov 15, 2019
@dsinang
Copy link
Copy Markdown

dsinang commented Nov 26, 2019

Hello, would really like to see this PR merged and released.

Any idea on when that may be ?

@DSpeichert
Copy link
Copy Markdown
Contributor

@javicrespo I think this PR needs rebasing at this point.

@danielnelson danielnelson removed this from the 1.14.0 milestone Mar 18, 2020
@danielnelson danielnelson added this to the 1.15.0 milestone Mar 18, 2020
@zibuyu1995
Copy link
Copy Markdown

this PR will make it into an release?

@ssoroka
Copy link
Copy Markdown
Contributor

ssoroka commented Apr 9, 2020

waiting on @javicrespo to implement on top of the latest master (the merge from master might have confused things there). If not interested, we or someone else could implement this feature.

@javicrespo
Copy link
Copy Markdown
Contributor Author

Considering that I don’t need this feature anymore and the effort to retrofit it to the latest version in master seems non-trivial to me, I’m going to pass. Feel free to pick it up 🙂

@lsitzmann
Copy link
Copy Markdown

We‘d really appreciate if anyone would do the integration of this feature.
We use it on a lot of systems- and now we are stuck to and old and self-build version of telegraf.

@ssoroka
Copy link
Copy Markdown
Contributor

ssoroka commented Apr 13, 2020

Closing this in favor of #7309

@ssoroka ssoroka closed this Apr 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/logging feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.