Skip to content

Update Kafka version parsing / supported range#27720

Merged
faec merged 8 commits intoelastic:masterfrom
faec:kafka-version
Sep 3, 2021
Merged

Update Kafka version parsing / supported range#27720
faec merged 8 commits intoelastic:masterfrom
faec:kafka-version

Conversation

@faec
Copy link
Copy Markdown
Contributor

@faec faec commented Sep 2, 2021 •
edited
Loading

What does this PR do?

Clean up the Kafka version parsing to use Sarama's internal constants and helpers, update the documentation to the current supported version range, and add a test to detect changes to the supported range.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@faec faec added enhancement cleanup Team:Elastic-Agent Label for the Agent team backport-v7.16.0 Automated backport with mergify labels Sep 2, 2021
@faec faec self-assigned this Sep 2, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 2, 2021

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 2, 2021
@faec faec changed the title Kafka version Update Kafka version parsing / supported range Sep 2, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 2, 2021 •
edited by jenkins-beats-ci bot
Loading

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-03T15:00:41.696+0000

  • Duration: 112 min 14 sec

  • Commit: 1589522

Test stats 🧪

Test Results
Failed 0
Passed 53674
Skipped 5323
Total 58997

Trends 🧪

Image of Build Times

Image of Tests

Log output

Expand to view the last 100 lines of log output 

[2021-09-03T16:47:07.714Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-03T16:47:08.032Z] + gcloud auth activate-service-account --key-file ****
[2021-09-03T16:47:08.600Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-03T16:47:08.930Z] + gsutil -m -q cp eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDE1ODk1MjJmZDkxNTQ3YzJiZmRjMzY2NmIyYmRhNGJjYmQxODg2MjY gs://beats-ci-temp/ci/cache/
[2021-09-03T16:47:48.793Z] tests\system\test_processors.py ........                                 [ 13%]
[2021-09-03T16:48:19.178Z] tests\system\test_reload.py ...                                          [ 14%]
[2021-09-03T16:48:19.178Z] tests\system\test_template.py s                                          [ 15%]
[2021-09-03T16:48:27.322Z] tests\system\test_timeseries.py .                                        [ 15%]
[2021-09-03T16:48:27.322Z] module\aerospike\test_aerospike.py s                                     [ 16%]
[2021-09-03T16:48:27.322Z] module\apache\test_apache.py ss                                          [ 17%]
[2021-09-03T16:48:27.593Z] module\ceph\test_ceph.py ssssssssssssssssssss                            [ 25%]
[2021-09-03T16:48:27.593Z] module\consul\test_consul.py ss                                          [ 25%]
[2021-09-03T16:48:27.593Z] module\couchbase\test_couchbase.py sss                                   [ 27%]
[2021-09-03T16:48:27.593Z] module\couchdb\test_couchdb.py s                                         [ 27%]
[2021-09-03T16:48:27.593Z] module\docker\test_docker.py sssssssss                                   [ 31%]
[2021-09-03T16:48:27.593Z] module\dropwizard\test_dropwizard.py sss                                 [ 32%]
[2021-09-03T16:48:27.593Z] module\elasticsearch\test_elasticsearch.py sssssssssss                   [ 36%]
[2021-09-03T16:48:27.593Z] module\envoyproxy\test_envoyproxy.py ss                                  [ 37%]
[2021-09-03T16:48:27.593Z] module\etcd\test_etcd.py ssssssss                                        [ 40%]
[2021-09-03T16:48:27.593Z] module\golang\test_golang.py s                                           [ 41%]
[2021-09-03T16:48:27.593Z] module\haproxy\test_haproxy.py sssssssssssssssssssssssss                 [ 51%]
[2021-09-03T16:48:37.595Z] module\http\test_http.py s.                                              [ 52%]
[2021-09-03T16:48:37.595Z] module\jolokia\test_jolokia.py ssss                                      [ 53%]
[2021-09-03T16:48:37.595Z] module\kafka\test_kafka.py ssssssssssssssssssss                          [ 61%]
[2021-09-03T16:48:37.595Z] module\kibana\test_kibana.py ss                                          [ 62%]
[2021-09-03T16:48:37.595Z] module\logstash\test_logstash.py sss                                     [ 63%]
[2021-09-03T16:48:37.595Z] module\memcached\test_memcached.py s                                     [ 64%]
[2021-09-03T16:48:37.595Z] module\mongodb\test_mongodb.py s                                         [ 64%]
[2021-09-03T16:48:37.595Z] module\munin\test_munin.py s                                             [ 65%]
[2021-09-03T16:48:37.595Z] module\mysql\test_mysql.py sssssss                                       [ 68%]
[2021-09-03T16:48:37.595Z] module\nats\test_nats.py ssssssssssssssssss                              [ 75%]
[2021-09-03T16:48:37.595Z] module\openmetrics\test_openmetrics.py s                                 [ 75%]
[2021-09-03T16:48:37.595Z] module\php_fpm\test_phpfpm.py s                                          [ 76%]
[2021-09-03T16:48:37.595Z] module\postgresql\test_postgresql.py ssssssssssssssssssss                [ 84%]
[2021-09-03T16:48:37.595Z] module\prometheus\test_prometheus.py sss                                 [ 85%]
[2021-09-03T16:48:37.595Z] module\redis\test_redis.py ssssssssssss                                  [ 90%]
[2021-09-03T16:49:02.889Z] module\system\test_system.py Cannot contact beats-ci-immutable-windows-10-1630686334251819478: java.lang.InterruptedException
[2021-09-03T16:50:35.556Z] ........s..s...s.s.                            [ 96%]
[2021-09-03T16:50:35.556Z] module\traefik\test_traefik.py s                                         [ 97%]
[2021-09-03T16:50:35.556Z] module\uwsgi\test_uwsgi.py ss                                            [ 97%]
[2021-09-03T16:50:35.556Z] module\vsphere\test_vsphere.py sss                                       [ 99%]
[2021-09-03T16:50:35.556Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-09-03T16:50:35.556Z] 
[2021-09-03T16:50:35.556Z] ============================== warnings summary ===============================
[2021-09-03T16:50:35.556Z] tests\system\test_lightmodules.py:57
[2021-09-03T16:50:35.556Z]   C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-09-03T16:50:35.556Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-09-03T16:50:35.556Z] 
[2021-09-03T16:50:35.556Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-09-03T16:50:35.556Z] - generated xml file: C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-09-03T16:50:35.556Z] ============================ slowest 20 durations =============================
[2021-09-03T16:50:35.556Z] 17.04s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-09-03T16:50:35.556Z] 16.99s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-09-03T16:50:35.556Z] 16.97s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-09-03T16:50:35.556Z] 12.59s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-09-03T16:50:35.556Z] 12.46s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-09-03T16:50:35.556Z] 11.11s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-09-03T16:50:35.556Z] 10.87s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-09-03T16:50:35.556Z] 10.72s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-09-03T16:50:35.556Z] 10.06s call     metricbeat/module/system/test_system.py::Test::test_fsstat
[2021-09-03T16:50:35.556Z] 9.88s call     metricbeat/module/system/test_system.py::Test::test_filesystem
[2021-09-03T16:50:35.556Z] 9.64s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-09-03T16:50:35.556Z] 9.26s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_complex_condition
[2021-09-03T16:50:35.556Z] 9.11s call     metricbeat/tests/system/test_processors.py::Test::test_multiple_actions
[2021-09-03T16:50:35.556Z] 9.11s call     metricbeat/tests/system/test_processors.py::Test::test_include_fields
[2021-09-03T16:50:35.556Z] 9.09s call     metricbeat/tests/system/test_processors.py::Test::test_contradictory_multiple_actions
[2021-09-03T16:50:35.556Z] 9.08s call     metricbeat/tests/system/test_processors.py::Test::test_dropfields_with_condition
[2021-09-03T16:50:35.556Z] 9.04s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-09-03T16:50:35.556Z] 9.03s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-09-03T16:50:35.556Z] 8.96s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-09-03T16:50:35.556Z] 8.93s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-09-03T16:50:35.556Z] =========== 40 passed, 207 skipped, 1 warning in 405.16s (0:06:45) ============
[2021-09-03T16:50:35.556Z] >> python test: Unit Testing Complete
[2021-09-03T16:50:35.947Z] 
[2021-09-03T16:50:35.948Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-09-03T16:50:38.184Z] 
[2021-09-03T16:50:38.185Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-09-03T16:50:40.096Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-09-03T16:50:40.096Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-09-03T16:50:40.117Z] Running in C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats\build
[2021-09-03T16:50:40.137Z] Recording test results
[2021-09-03T16:51:40.492Z] [Checks API] No suitable checks publisher found.
[2021-09-03T16:51:40.891Z] 
[2021-09-03T16:51:40.891Z] C:\Users\jenkins\workspace\PR-27720-4-079606a8-6b84-4422-8501-27658b32b427\src\github.com\elastic\beats>go clean -modcache 
[2021-09-03T16:52:50.185Z] + gsutil --version
[2021-09-03T16:52:51.610Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-03T16:52:51.934Z] + gcloud auth activate-service-account --key-file ****
[2021-09-03T16:52:52.503Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-03T16:52:52.827Z] + gsutil -m -q cp bWV0cmljYmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTAxNTg5NTIyZmQ5MTU0N2MyYmZkYzM2NjZiMmJkYTRiY2JkMTg4NjI2 gs://beats-ci-temp/ci/cache/
[2021-09-03T16:52:54.420Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-09-03T16:52:54.470Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-09-03T16:52:54.549Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27720/src/github.com/elastic/beats
[2021-09-03T16:52:54.920Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-27720
[2021-09-03T16:52:54.982Z] [INFO] getVaultSecret: Getting secrets
[2021-09-03T16:52:55.039Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-03T16:52:57.275Z] + chmod 755 generate-build-data.sh
[2021-09-03T16:52:57.275Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4 FAILURE 6734174
[2021-09-03T16:52:57.275Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/steps/?limit=10000 -o steps-info.json
[2021-09-03T16:53:08.259Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/tests/?status=FAILED -o tests-errors.json
[2021-09-03T16:53:11.400Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27720/runs/4/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53674
Skipped 5323
Total 58997

@faec faec requested a review from kvch September 2, 2021 19:10
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 3, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-version upstream/kafka-version
git merge upstream/master
git push upstream kafka-version

@faec faec merged commit f083957 into elastic:master Sep 3, 2021
@faec faec deleted the kafka-version branch September 3, 2021 19:16
mergify bot pushed a commit that referenced this pull request Sep 3, 2021
@faec
Update Kafka version parsing / supported range (#27720)
cfd5df6 
(cherry picked from commit f083957)
@mergify mergify bot mentioned this pull request Sep 3, 2021
Merged
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Sep 9, 2021
@mdelapenya
Merge branch 'master' into 27608-fix-elastic-agent-build
16ae989 
* master: (39 commits)
  [Heartbeat] Move JSON tests from python->go (elastic#27816)
  docs: simplify permissions for Dockerfile COPY (elastic#27754)
  Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789)
  [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774)
  warn log entry and no validation failure when both queue_url and buck… (elastic#27612)
  libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178)
  [Heartbeat] Add httpcommon options to ZipURL (elastic#27699)
  Add a header round tripper option to httpcommon (elastic#27509)
  [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779)
  Rename dashboards according to module.yml files for master (elastic#27749)
  Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726)
  Accept syslog dates with leading 0 (elastic#27775)
  [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727)
  [Filebeat] Threatintel compatibility updates (elastic#27323)
  Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707)
  [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491)
  Fix flakyness of TestFilestreamEmptyLine (elastic#27705)
  [Filebeat] kafka v2 using parsers (elastic#27335)
  Update Kafka version parsing / supported range (elastic#27720)
  Update Sarama to 1.29.1 (elastic#27717)
  ...
faec added a commit that referenced this pull request Sep 14, 2021
@mergify @faec
Update Kafka version parsing / supported range (#27720) (#27753)
fc6029f 
(cherry picked from commit f083957)

Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
@faec
Update Kafka version parsing / supported range (elastic#27720)
7058097
@rdner rdner mentioned this pull request Mar 22, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kvch kvch kvch approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@faec faec

Labels

backport-v7.16.0 Automated backport with mergify cleanup enhancement Team:Elastic-Agent Label for the Agent team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@faec @elasticmachine @kvch