Skip to content

Cherry-pick #19459 to 7.x: [Filebeat] Fix reference leak in TCP and Unix socket inputs#19501

Merged
andrewkroh merged 1 commit intoelastic:7.xfrom
andrewkroh:backport_19459_7.x
Jul 13, 2020
Merged

Cherry-pick #19459 to 7.x: [Filebeat] Fix reference leak in TCP and Unix socket inputs#19501
andrewkroh merged 1 commit intoelastic:7.xfrom
andrewkroh:backport_19459_7.x

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Jun 29, 2020

Cherry-pick of PR #19459 to 7.x branch. Original message:

What does this PR do?

The tcp and unix input sources were leaking references causing a memory leak.
When an accepted connection ended inputsource/common.Closer was
supposed to delete the pointer that it held to the connection, but due to a code
error delete was being called on the wrong map.

Why is it important?

This fixes a memory leak with the tcp and unix inputs.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@andrewkroh andrewkroh added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Jun 29, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 29, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19501 updated]

  • Start Time: 2020-07-13T16:24:23.785+0000

  • Duration: 126 min 15 sec

Test stats 🧪

Test Results
Failed 0
Passed 3678
Skipped 677
Total 4355

Log output

Expand to view the last 100 lines of log output

[2020-07-13T18:29:30.592Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-13T18:29:30.592Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-13T18:29:30.592Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-13T18:29:30.592Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-13T18:29:31.184Z] java.lang.InterruptedException: no matches found within 10000
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2826)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2686)
[2020-07-13T18:29:31.184Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3073)
[2020-07-13T18:29:31.184Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1594658995265943986.c.elastic-ci-prod.internal/10.224.0.212:41184
[2020-07-13T18:29:31.184Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-07-13T18:29:31.185Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-07-13T18:29:31.185Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.act(FilePath.java:1069)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.act(FilePath.java:1058)
[2020-07-13T18:29:31.185Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-13T18:29:31.185Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-13T18:29:31.185Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-07-13T18:29:31.185Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-07-13T18:29:31.185Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-13T18:29:31.185Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-13T18:29:31.185Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T18:29:31.185Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T18:29:31.185Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-07-13T18:29:31.185Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-07-13T18:29:31.581Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-13T18:29:31.960Z] Failed in branch Filebeat x-pack
[2020-07-13T18:30:15.995Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-07-13T18:30:16.169Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:16.480Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-13T18:30:16.492Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Lint
[2020-07-13T18:30:16.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-13T18:30:16.641Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-13T18:30:16.718Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-13T18:30:16.789Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-13T18:30:16.859Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-13T18:30:16.928Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats/Filebeat-oss
[2020-07-13T18:30:17.282Z] + cat
[2020-07-13T18:30:17.282Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-13T18:30:17.282Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-13T18:30:23.881Z] runbld>>> runbld started
[2020-07-13T18:30:23.881Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-13T18:30:25.800Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19501' in order of occurrence in the config (last value wins).
[2020-07-13T18:30:27.198Z] runbld>>> Debug logging enabled.
[2020-07-13T18:30:27.198Z] runbld>>> Storing result
[2020-07-13T18:30:27.198Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-13T18:30:27.198Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713183026-F9B62998
[2020-07-13T18:30:27.198Z] runbld>>> Adding system facts.
[2020-07-13T18:30:28.599Z] runbld>>> Adding vcs info for the latest commit:  bcf4477704df8cc26fbae8b97a0bd59ed9cf24c6
[2020-07-13T18:30:28.599Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-13T18:30:28.599Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-13T18:30:28.599Z] Processing JUnit reports with runbld...
[2020-07-13T18:30:28.599Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-13T18:30:28.863Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-13T18:30:28.863Z] runbld>>> DURATION: 53ms
[2020-07-13T18:30:28.863Z] runbld>>> STDOUT: 40 bytes
[2020-07-13T18:30:28.863Z] runbld>>> STDERR: 49 bytes
[2020-07-13T18:30:28.863Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-13T18:30:28.863Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:29.811Z] runbld>>> Storing build metadata: 
[2020-07-13T18:30:29.811Z] runbld>>> Adding test report.
[2020-07-13T18:30:29.811Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19501/src/github.com/elastic/beats
[2020-07-13T18:30:30.756Z] runbld>>> Found 11 test output files
[2020-07-13T18:30:31.699Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4355 Skipped: 653
[2020-07-13T18:30:31.960Z] runbld>>> Storing result
[2020-07-13T18:30:31.960Z] runbld>>> FAILURES: 0
[2020-07-13T18:30:31.960Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-13T18:30:31.960Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713183026-F9B62998
[2020-07-13T18:30:32.221Z] runbld>>> Email notification disabled by environment variable.
[2020-07-13T18:30:32.221Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-13T18:30:38.066Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19501
[2020-07-13T18:30:38.276Z] [INFO] getVaultSecret: Getting secrets
[2020-07-13T18:30:38.351Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-13T18:30:39.076Z] + chmod 755 generate-build-data.sh
[2020-07-13T18:30:39.076Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3 ABORTED 7575033
[2020-07-13T18:30:39.076Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3/steps/?limit=10000 -o steps-info.json
[2020-07-13T18:30:39.627Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19501/runs/3/tests/?status=FAILED -o tests-errors.json

@andrewkroh andrewkroh force-pushed the backport_19459_7.x branch from ba9ed57 to d73b07f Compare July 13, 2020 13:16
…19459)

The tcp and unix input sources were leaking references causing a memory leak.
When an accepted connection ended inputsource/common.Closer was
supposed to delete the pointer that it held to the connection, but due to a code
error `delete` was being called on the wrong map.

Instead of modifying the common.Closer I replaced it with a cancellable context.Context which
is designed to propagate signals from parent to children and requires less code.

(cherry picked from commit 61f4846)
@andrewkroh andrewkroh force-pushed the backport_19459_7.x branch from d73b07f to bcf4477 Compare July 13, 2020 16:17
@andrewkroh andrewkroh merged commit 03b4227 into elastic:7.x Jul 13, 2020
@zube zube bot removed the [zube]: Done label Oct 13, 2020
@andrewkroh andrewkroh deleted the backport_19459_7.x branch January 14, 2022 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Services (Deprecated) Label for the former Integrations-Services team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants