Skip to content

Skipping unparsable lines in docker input#12268

Merged
exekias merged 1 commit intoelastic:masterfrom
marqc:docker_input_skip_unparsable_lines
May 27, 2019
Merged

Skipping unparsable lines in docker input#12268
exekias merged 1 commit intoelastic:masterfrom
marqc:docker_input_skip_unparsable_lines

Conversation

@marqc
Copy link
Copy Markdown
Contributor

@marqc marqc commented May 24, 2019

fix proposal for problem referenced in https://discuss.elastic.co/t/input-docker-stuck-when-bad-offset-is-saved-in-registry-file-bug/181828 where docker input is unparsable (either input itself is corrupted, or offset in registry is corrupted)

@marqc marqc requested a review from a team as a code owner May 24, 2019 11:03
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 24, 2019

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

houndci-bot
houndci-bot reviewed May 24, 2019
View reviewed changes
@ph ph added Filebeat Filebeat review labels May 24, 2019
@kvch
Copy link
Copy Markdown
Contributor

kvch commented May 27, 2019

jenkins test this

kvch
kvch reviewed May 27, 2019
View reviewed changes
filebeat/input/log/harvester.go Outdated
Copy link
Copy Markdown
Contributor

@kvch kvch May 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think this log message should be Info, as we have already emitted an error message from DockerJSON with the error message. To me this is rather a notification telling the user that a line was skipped due to an error before.

Copy link
Copy Markdown
Contributor Author

@marqc marqc May 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to info

Copy link
Copy Markdown
Contributor

@kvch kvch May 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you have changed the wrong line for the case default.

Copy link
Copy Markdown
Contributor Author

@marqc marqc May 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this time, I changed what i was supposed to change ;)

@marqc marqc force-pushed the docker_input_skip_unparsable_lines branch from cbaafbf to eda4a25 Compare May 27, 2019 08:04
houndci-bot
houndci-bot reviewed May 27, 2019
View reviewed changes
@marqc marqc force-pushed the docker_input_skip_unparsable_lines branch 2 times, most recently from 7789894 to 5c3ab65 Compare May 27, 2019 08:56
@exekias
Copy link
Copy Markdown
Contributor

exekias commented May 27, 2019

Thank you for opening, this is defensive code that should help when this problems arises. Could you add an entry in CHANGELOG.next.asciidoc?

@exekias
Copy link
Copy Markdown
Contributor

exekias commented May 27, 2019

would love to see some unit tests in docker_json_test.go

@marqc marqc force-pushed the docker_input_skip_unparsable_lines branch from 5c3ab65 to 281f5b9 Compare May 27, 2019 13:46
@marqc
Copy link
Copy Markdown
Contributor Author

marqc commented May 27, 2019

@exekias changelog entry added, unit tests for docker_json changed to cover error message verification

@exekias
Copy link
Copy Markdown
Contributor

exekias commented May 27, 2019

jenkins, test this please

@exekias exekias added bug needs_backport PR is waiting to be backported to other branches. labels May 27, 2019
@exekias exekias merged commit 3451d06 into elastic:master May 27, 2019
This was referenced Jun 19, 2019
Closed
Merged
@exekias exekias added v7.2.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jun 19, 2019
exekias pushed a commit to exekias/beats that referenced this pull request Jun 19, 2019
@marqc
Skipping unparsable lines in docker input (elastic#12268)
101a641 
(cherry picked from commit 3451d06)
@adawalli
Copy link
Copy Markdown

adawalli commented Jun 20, 2019

Will this be backported to 6.x? Really hitting this with a bunch of our containers

exekias pushed a commit that referenced this pull request Jul 8, 2019
Cherry-pick #12268 to 7.2: Skipping unparsable lines in docker input (#…
2a43b3e 
…12608)

* Skipping unparsable lines in docker input (#12268)

(cherry picked from commit 3451d06)
@exekias exekias mentioned this pull request Jul 8, 2019
Merged
exekias pushed a commit to exekias/beats that referenced this pull request Jul 8, 2019
@marqc
Skipping unparsable lines in docker input (elastic#12268)
f650f0b 
(cherry picked from commit 3451d06)
@exekias exekias added the v6.8.2 label Jul 8, 2019
@exekias
Copy link
Copy Markdown
Contributor

exekias commented Jul 8, 2019

I've opened a backport to 6.8 branch: #12813

exekias pushed a commit that referenced this pull request Jul 11, 2019
Cherry-pick #12268 to 6.8: Skipping unparsable lines in docker input (#…
13b1738 
…12813)

* Skipping unparsable lines in docker input (#12268)

(cherry picked from commit 3451d06)
@detro
Copy link
Copy Markdown

detro commented Jul 16, 2019

Hello.
I have a question about this fix: why is it tagged with 7.2.0 but it seems like it will be out only in 7.2.1? Was this just a hickup or am I misunderstanding the meaning of the tag?

Regardless, thanks for this: looking forward to update Filebeat and get rid of this issue. We have had long running containers that logged nothing for the last 4 days because of the issue that this code fixes.

@detro detro mentioned this pull request Jul 16, 2019
Closed
@detro
Copy link
Copy Markdown

detro commented Jul 22, 2019 •
edited
Loading

Hello. Any update about the release of the 7.2.1 Docker image?
Alternatively, can I ask for a pointer (a doc?) so I can build and package a Docker image with Filebeat 7.2.1?

Thanks in advance :)

@ChrsMark ChrsMark mentioned this pull request Nov 20, 2020
Merged
This was referenced Apr 1, 2021
Merged
Merged
Merged
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
Cherry-pick elastic#12268 to 7.2: Skipping unparsable lines in docker…
86044a2 
… input (elastic#12608)

* Skipping unparsable lines in docker input (elastic#12268)

(cherry picked from commit 0cc16fd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers

@kvch kvch kvch approved these changes

@houndci-bot houndci-bot houndci-bot left review comments

@exekias exekias exekias approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@exekias exekias

Labels

bug Filebeat Filebeat review v6.8.2 v7.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@marqc @elasticmachine @kvch @exekias @adawalli @detro @houndci-bot @ph