Skip to content

[Ingest Manager] Thread safe sorted set#21290

Merged
michalpristas merged 3 commits intoelastic:masterfrom
michalpristas:agent-sorted-set
Sep 25, 2020
Merged

[Ingest Manager] Thread safe sorted set#21290
michalpristas merged 3 commits intoelastic:masterfrom
michalpristas:agent-sorted-set

Conversation

@michalpristas
Copy link
Copy Markdown
Contributor

What does this PR do?

I ran into panic complaining about concurrent map writes so making this thread safe

Why is it important?

Less panics

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@michalpristas michalpristas added bug Team:Ingest Management Ingest Management:beta2 Group issues for ingest management beta2 labels Sep 24, 2020
@michalpristas michalpristas requested a review from ph September 24, 2020 10:45
@michalpristas michalpristas self-assigned this Sep 24, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 24, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 24, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21290 updated]

  • Start Time: 2020-09-24T17:04:23.469+0000

  • Duration: 64 min 14 sec

Test stats 🧪

Test Results
Failed 1
Passed 15602
Skipped 1269
Total 16872

Test errors

Expand to view the tests failures

  • Name: Build&Test / libbeat-build / TestClientPublishEventKerberosAware – elasticsearch

    • Age: 1
    • Duration: 4.44
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: mage build test

    • Description: mage build test

    • Duration: 21 min 28 sec

    • Start Time: 2020-09-24T17:31:36.238+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-24T17:53:20.137+0000

    • log

  • Name: Terraform Apply on x-pack/metricbeat/module/aws

    • Description:

    • Duration: 0 min 2 sec

    • Start Time: 2020-09-24T17:31:22.363+0000

    • log

  • Name: Terraform Apply on x-pack/metricbeat/module/aws

    • Description:

    • Duration: 0 min 3 sec

    • Start Time: 2020-09-24T17:31:30.478+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/winlogbeat-windows-windows-2019/winlogbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/packetbeat-windows-windows-2019/packetbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/packetbeat-windows-windows-2019/packetbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/metricbeat-windows-windows-2019/metricbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/metricbeat-windows-windows-2019/metricbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019/x-pack/auditbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019/x-pack/auditbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019/x-pack/metricbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019/x-pack/metricbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-windows-windows-2019/heartbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-windows-windows-2019/heartbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.882Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-dockerlogbeat-build/x-pack/dockerlogbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-integration.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/metricbeat-pythonIntegTest/metricbeat/build/TEST-python-integration.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-python-unit.xml
[2020-09-24T18:08:14.883Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-go-unit.xml
[2020-09-24T18:08:14.883Z] + cat
[2020-09-24T18:08:14.883Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-24T18:08:14.883Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-24T18:08:21.499Z] runbld>>> runbld started
[2020-09-24T18:08:21.499Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-24T18:08:22.887Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-24T18:08:22.887Z] runbld>>> Matches in the system config:
[2020-09-24T18:08:22.887Z] runbld>>> - Matched ^elastic\+beats
[2020-09-24T18:08:22.887Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-24T18:08:24.385Z] runbld>>> Debug logging enabled.
[2020-09-24T18:08:24.385Z] runbld>>> Storing result
[2020-09-24T18:08:24.385Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-24T18:08:24.385Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924180823-B4D746A5
[2020-09-24T18:08:24.385Z] runbld>>> Adding system facts.
[2020-09-24T18:08:25.338Z] runbld>>> Adding vcs info for the latest commit:  6ed3f84f4b7f0d57fb072fedb70eb7524cde4059
[2020-09-24T18:08:25.338Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-24T18:08:25.338Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-24T18:08:25.338Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-24T18:08:25.338Z] Processing JUnit reports with runbld...
[2020-09-24T18:08:25.912Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-24T18:08:25.912Z] runbld>>> DURATION: 20ms
[2020-09-24T18:08:25.912Z] runbld>>> STDOUT: 40 bytes
[2020-09-24T18:08:25.912Z] runbld>>> STDERR: 49 bytes
[2020-09-24T18:08:25.912Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-24T18:08:25.912Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21290
[2020-09-24T18:08:26.859Z] runbld>>> Storing build metadata: 
[2020-09-24T18:08:26.859Z] runbld>>> Adding test report.
[2020-09-24T18:08:26.859Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21290/src/github.com/elastic/beats
[2020-09-24T18:08:27.432Z] runbld>>> Found 99 test output files
[2020-09-24T18:08:28.007Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21290/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-24T18:08:28.007Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21290/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-24T18:08:30.558Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 16872 Skipped: 1050
[2020-09-24T18:08:30.820Z] runbld>>> Storing result
[2020-09-24T18:08:30.820Z] runbld>>> FAILURES: 1
[2020-09-24T18:08:31.081Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-24T18:08:31.081Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924180823-B4D746A5
[2020-09-24T18:08:31.342Z] runbld>>> Email notification disabled by environment variable.
[2020-09-24T18:08:31.342Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-24T18:08:36.909Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21290
[2020-09-24T18:08:36.979Z] [INFO] getVaultSecret: Getting secrets
[2020-09-24T18:08:37.058Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-24T18:08:37.632Z] + chmod 755 generate-build-data.sh
[2020-09-24T18:08:37.632Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21290/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21290/runs/3 FAILURE 3853903
[2020-09-24T18:08:37.632Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21290/runs/3/steps/?limit=10000 -o steps-info.json

Copy link
Copy Markdown
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ph
Copy link
Copy Markdown
Contributor

ph commented Sep 24, 2020

Changelog?

@michalpristas michalpristas merged commit 1a80170 into elastic:master Sep 25, 2020
@michalpristas michalpristas added the needs_backport PR is waiting to be backported to other branches. label Sep 25, 2020
michalpristas added a commit to michalpristas/beats that referenced this pull request Sep 25, 2020
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
…peline-2.0

* upstream/master:
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
* upstream/master: (417 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
* upstream/master: (399 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
v1v added a commit to v1v/beats that referenced this pull request Sep 28, 2020
* upstream/master: (60 commits)
  libbeat/cmd/instance: report cgroup stats (elastic#21113)
  Configurable index template loading (elastic#21212)
  [Ingest Manager] Thread safe sorted set (elastic#21290)
  Change mirror of kafka download (elastic#19645)
  [Ingest manager] Copy Action store on upgrade (elastic#21298)
  [CI] Pipeline 2.0 for monorepos (elastic#20104)
  Stop running agent container as root by default (elastic#21213)
  Stop running auditbeat container as root by default (elastic#21202)
  Fix autodiscover flaky tests (elastic#21242)
  [Ingest Manager] Enabled dev builds (elastic#21241)
  Fix librpm installation in auditbeat build (elastic#21239)
  Fix prometheus default config (elastic#21253)
  Fix dev guide test command (elastic#21254)
  Move aws lambda metricset to GA (elastic#21255)
  [Docs] Typo in table syntax (elastic#20227)
  [ECS] Adds related.hosts to capture all hostnames and host identifiers on an event. (elastic#21160)
  Add recursive split to httpjson (elastic#21214)
  [DOCS] Add beat specific start widgets (elastic#21217)
  Fix timestamp handling in remote_write (elastic#21166)
  Fix aws, azure and googlecloud compute dashboards (elastic#21098)
  ...
michalpristas added a commit that referenced this pull request Sep 30, 2020
Cherry-pick #21290 to 7.x: Thread safe sorted set  (#21331)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Ingest Management:beta2 Group issues for ingest management beta2 needs_backport PR is waiting to be backported to other branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants