Skip to content

Make beats module work for Stack Monitoring without xpack.enabled flag#21272

Merged
sayden merged 10 commits intoelastic:feature-stack-monitoring-mb-ecsfrom
sayden:feature/mb/elasticsearch/beats_state_xpack_flag
Mar 30, 2021
Merged

Make beats module work for Stack Monitoring without xpack.enabled flag#21272
sayden merged 10 commits intoelastic:feature-stack-monitoring-mb-ecsfrom
sayden:feature/mb/elasticsearch/beats_state_xpack_flag

Conversation

@sayden
Copy link
Copy Markdown
Contributor

@sayden sayden commented Sep 23, 2020 •
edited
Loading

Ready to test in Kibana. data.json is included to help troubleshooting and it has been generating testing the Metricbeat binary directly with Elasticsearch.

Both metricsets from Beats module are included in this PR

@sayden sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Sep 23, 2020
@sayden sayden self-assigned this Sep 23, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 23, 2020
@sayden sayden changed the base branch from master to feature-stack-monitoring-mb-ecs September 28, 2020 14:21
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 28, 2020 •
edited by jenkins-beats-ci bot
Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #21272 updated

  • Start Time: 2021-03-30T15:40:49.130+0000

  • Duration: 54 min 16 sec

  • Commit: cd88596

Test stats 🧪

Test Results
Failed 27
Passed 8346
Skipped 581
Total 8954

Trends 🧪

Image of Build Times

Image of Tests

Test errors 27

Expand to view the tests failures

> Show only the first 10 test failures

Build&Test / metricbeat-unitTest / TestReportErrorForMissingField – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestReportErrorForMissingField
    elastic_test.go:90: 
        	Error Trace:	elastic_test.go:90
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
    elastic_test.go:91: 
        	Error Trace:	elastic_test.go:91
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
--- FAIL: TestReportErrorForMissingField (0.00s)

Build&Test / metricbeat-unitTest / TestConfigureModule/xpack_enabled – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule/xpack_enabled
    elastic_test.go:209: 
        	Error Trace:	elastic_test.go:209
        	Error:      	Not equal: 
        	            	expected: elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"baz", "qux", "foo"}, XPackEnabled:true}
        	            	actual  : elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"foo", "bar"}, XPackEnabled:true}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,6 +2,5 @@
        	            	  Module: (string) (len=11) "test_module",
        	            	- MetricSets: ([]string) (len=3) {
        	            	-  (string) (len=3) "baz",
        	            	-  (string) (len=3) "qux",
        	            	-  (string) (len=3) "foo"
        	            	+ MetricSets: ([]string) (len=2) {
        	            	+  (string) (len=3) "foo",
        	            	+  (string) (len=3) "bar"
        	            	  },
        	Test:       	TestConfigureModule/xpack_enabled
    --- FAIL: TestConfigureModule/xpack_enabled (0.00s)

Build&Test / metricbeat-unitTest / TestConfigureModule – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule
--- FAIL: TestConfigureModule (0.00s)

Build&Test / metricbeat-windows-2008-windows-2008-r2 / TestReportErrorForMissingField – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestReportErrorForMissingField
    elastic_test.go:90: 
        	Error Trace:	elastic_test.go:90
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
    elastic_test.go:91: 
        	Error Trace:	elastic_test.go:91
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
--- FAIL: TestReportErrorForMissingField (0.00s)

Build&Test / metricbeat-windows-2008-windows-2008-r2 / TestConfigureModule/xpack_enabled – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule/xpack_enabled
    elastic_test.go:209: 
        	Error Trace:	elastic_test.go:209
        	Error:      	Not equal: 
        	            	expected: elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"baz", "qux", "foo"}, XPackEnabled:true}
        	            	actual  : elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"foo", "bar"}, XPackEnabled:true}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,6 +2,5 @@
        	            	  Module: (string) (len=11) "test_module",
        	            	- MetricSets: ([]string) (len=3) {
        	            	-  (string) (len=3) "baz",
        	            	-  (string) (len=3) "qux",
        	            	-  (string) (len=3) "foo"
        	            	+ MetricSets: ([]string) (len=2) {
        	            	+  (string) (len=3) "foo",
        	            	+  (string) (len=3) "bar"
        	            	  },
        	Test:       	TestConfigureModule/xpack_enabled
    --- FAIL: TestConfigureModule/xpack_enabled (0.00s)

Build&Test / metricbeat-windows-2008-windows-2008-r2 / TestConfigureModule – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule
--- FAIL: TestConfigureModule (0.01s)

Build&Test / metricbeat-windows-windows-2019 / TestReportErrorForMissingField – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestReportErrorForMissingField
    elastic_test.go:90: 
        	Error Trace:	elastic_test.go:90
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
    elastic_test.go:91: 
        	Error Trace:	elastic_test.go:91
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
--- FAIL: TestReportErrorForMissingField (0.00s)

Build&Test / metricbeat-windows-windows-2019 / TestConfigureModule/xpack_enabled – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule/xpack_enabled
    elastic_test.go:209: 
        	Error Trace:	elastic_test.go:209
        	Error:      	Not equal: 
        	            	expected: elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"baz", "qux", "foo"}, XPackEnabled:true}
        	            	actual  : elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"foo", "bar"}, XPackEnabled:true}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,6 +2,5 @@
        	            	  Module: (string) (len=11) "test_module",
        	            	- MetricSets: ([]string) (len=3) {
        	            	-  (string) (len=3) "baz",
        	            	-  (string) (len=3) "qux",
        	            	-  (string) (len=3) "foo"
        	            	+ MetricSets: ([]string) (len=2) {
        	            	+  (string) (len=3) "foo",
        	            	+  (string) (len=3) "bar"
        	            	  },
        	Test:       	TestConfigureModule/xpack_enabled
    --- FAIL: TestConfigureModule/xpack_enabled (0.00s)

Build&Test / metricbeat-windows-windows-2019 / TestConfigureModule – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule
--- FAIL: TestConfigureModule (0.00s)

Build&Test / metricbeat-windows-2012-windows-2012-r2 / TestReportErrorForMissingField – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestReportErrorForMissingField
    elastic_test.go:90: 
        	Error Trace:	elastic_test.go:90
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
    elastic_test.go:91: 
        	Error Trace:	elastic_test.go:91
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
--- FAIL: TestReportErrorForMissingField (0.00s)

Steps errors 14

Expand to view the steps failures

Show only the first 10 steps failures

metricbeat-windows-2012-windows-2012-r2 - mage build unitTest
  • Took 3 min 35 sec . View more details on here
  • Description: mage build unitTest
metricbeat-windows-10-windows-10 - mage build unitTest
  • Took 5 min 50 sec . View more details on here
  • Description: mage build unitTest
metricbeat-windows-8-windows-8 - mage build unitTest
  • Took 2 min 30 sec . View more details on here
  • Description: mage build unitTest
metricbeat-windows-2008-windows-2008-r2 - mage build unitTest
  • Took 2 min 23 sec . View more details on here
  • Description: mage build unitTest
metricbeat-windows-7-windows-7 - mage build unitTest
  • Took 3 min 42 sec . View more details on here
  • Description: mage build unitTest
metricbeat-windows-7-32-windows-7-32-bit - mage build unitTest
  • Took 3 min 14 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-Lint - make -C x-pack/metricbeat check;make -C x-pack/metricbeat update;make -C
  • Took 3 min 44 sec . View more details on here
  • Description: make -C x-pack/metricbeat check;make -C x-pack/metricbeat update;make -C metricbeat check;make -C metricbeat update;make check-no-changes;
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

Log output

Expand to view the last 100 lines of log output 

[2021-03-30T16:34:00.385Z] + FOUND_DOCKER_COMPOSE_VERSION=1.25.5
[2021-03-30T16:34:00.385Z] + '[' 1.25.5 == 1.21.0 ']'
[2021-03-30T16:34:00.385Z] + echo 'UNMET DEP: Installing docker-compose'
[2021-03-30T16:34:00.385Z] UNMET DEP: Installing docker-compose
[2021-03-30T16:34:00.385Z] + DC_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/docker-compose
[2021-03-30T16:34:00.385Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin
[2021-03-30T16:34:00.385Z] ++ uname -s
[2021-03-30T16:34:00.385Z] ++ uname -m
[2021-03-30T16:34:00.385Z] + curl -sSLo /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/docker-compose https://github.com/docker/compose/releases/download/1.21.0/docker-compose-Linux-x86_64
[2021-03-30T16:34:01.327Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/docker-compose
[2021-03-30T16:34:01.327Z] + .ci/scripts/install-terraform.sh
[2021-03-30T16:34:01.327Z] + MSG='environment variable missing.'
[2021-03-30T16:34:01.327Z] + TERRAFORM_VERSION=0.12.24
[2021-03-30T16:34:01.327Z] + HOME=/var/lib/jenkins/workspace/Beats_beats_PR-21272
[2021-03-30T16:34:01.327Z] + TERRAFORM_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform
[2021-03-30T16:34:01.327Z] ++ uname -s
[2021-03-30T16:34:01.327Z] ++ tr '[:upper:]' '[:lower:]'
[2021-03-30T16:34:01.327Z] + OS=linux
[2021-03-30T16:34:01.327Z] + command -v terraform
[2021-03-30T16:34:01.327Z] /usr/local/bin/terraform
[2021-03-30T16:34:01.328Z] + echo 'Found Terraform. Checking version..'
[2021-03-30T16:34:01.328Z] Found Terraform. Checking version..
[2021-03-30T16:34:01.328Z] ++ terraform --version
[2021-03-30T16:34:01.328Z] ++ awk '{print $2}'
[2021-03-30T16:34:01.328Z] ++ sed s/v//
[2021-03-30T16:34:01.589Z] + FOUND_TERRAFORM_VERSION='0.12.30
[2021-03-30T16:34:01.589Z] 
[2021-03-30T16:34:01.589Z] ersion
[2021-03-30T16:34:01.589Z] 0.14.9.'
[2021-03-30T16:34:01.589Z] + '[' 0.12.30 ersion 0.14.9. == 0.12.24 ']'
[2021-03-30T16:34:01.589Z] .ci/scripts/install-terraform.sh: line 16: [: too many arguments
[2021-03-30T16:34:01.589Z] + echo 'UNMET DEP: Installing Terraform'
[2021-03-30T16:34:01.589Z] UNMET DEP: Installing Terraform
[2021-03-30T16:34:01.589Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin
[2021-03-30T16:34:01.589Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2021-03-30T16:34:02.162Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform
[2021-03-30T16:34:02.162Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin
[2021-03-30T16:34:02.162Z] Archive:  /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform.zip
[2021-03-30T16:34:02.735Z]   inflating: /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform  
[2021-03-30T16:34:02.735Z] + rm /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform.zip
[2021-03-30T16:34:02.735Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-21272/bin/terraform
[2021-03-30T16:34:02.735Z] + make mage
[2021-03-30T16:34:02.735Z] Installing mage v1.11.0.
[2021-03-30T16:34:04.122Z] go: downloading github.com/magefile/mage v1.11.0
[2021-03-30T16:34:05.065Z] /var/lib/jenkins/workspace/Beats_beats_PR-21272/.magefile cleaned
[2021-03-30T16:34:05.425Z] + git config --get user.email
[2021-03-30T16:34:05.425Z] + [ -z  ]
[2021-03-30T16:34:05.425Z] + git config --global user.email beatsmachine@users.noreply.github.com
[2021-03-30T16:34:05.425Z] + git config --global user.name beatsmachine
[2021-03-30T16:34:05.730Z] + go mod download
[2021-03-30T16:34:21.396Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2021-03-30T16:34:21.396Z] + DIRECTORY=x-pack/metricbeat
[2021-03-30T16:34:21.396Z] + FAILED=0
[2021-03-30T16:34:21.396Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2021-03-30T16:34:21.396Z] + for tfstate in $(find $DIRECTORY -name terraform.tfstate)
[2021-03-30T16:34:21.396Z] ++ dirname x-pack/metricbeat/module/aws/terraform.tfstate
[2021-03-30T16:34:21.396Z] + cd x-pack/metricbeat/module/aws
[2021-03-30T16:34:21.396Z] + terraform destroy -auto-approve
[2021-03-30T16:34:25.608Z] 
[2021-03-30T16:34:25.608Z] Destroy complete! Resources: 0 destroyed.
[2021-03-30T16:34:25.608Z] + cd -
[2021-03-30T16:34:25.608Z] /var/lib/jenkins/workspace/Beats_beats_PR-21272/src/github.com/elastic/beats/src/github.com/elastic/beats
[2021-03-30T16:34:25.608Z] + exit 0
[2021-03-30T16:34:25.961Z] Client: Docker Engine - Community
[2021-03-30T16:34:25.961Z]  Version:           20.10.3
[2021-03-30T16:34:25.962Z]  API version:       1.41
[2021-03-30T16:34:25.962Z]  Go version:        go1.13.15
[2021-03-30T16:34:25.962Z]  Git commit:        48d30b5
[2021-03-30T16:34:25.962Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-30T16:34:25.962Z]  OS/Arch:           linux/amd64
[2021-03-30T16:34:25.962Z]  Context:           default
[2021-03-30T16:34:25.962Z]  Experimental:      true
[2021-03-30T16:34:25.962Z] 
[2021-03-30T16:34:25.962Z] Server: Docker Engine - Community
[2021-03-30T16:34:25.962Z]  Engine:
[2021-03-30T16:34:25.962Z]   Version:          20.10.3
[2021-03-30T16:34:25.962Z]   API version:      1.41 (minimum version 1.12)
[2021-03-30T16:34:25.962Z]   Go version:       go1.13.15
[2021-03-30T16:34:25.962Z]   Git commit:       46229ca
[2021-03-30T16:34:25.962Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-30T16:34:25.962Z]   OS/Arch:          linux/amd64
[2021-03-30T16:34:25.962Z]   Experimental:     false
[2021-03-30T16:34:25.962Z]  containerd:
[2021-03-30T16:34:25.962Z]   Version:          1.4.4
[2021-03-30T16:34:25.962Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-03-30T16:34:25.962Z]  runc:
[2021-03-30T16:34:25.962Z]   Version:          1.0.0-rc93
[2021-03-30T16:34:25.962Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-03-30T16:34:25.962Z]  docker-init:
[2021-03-30T16:34:25.962Z]   Version:          0.19.0
[2021-03-30T16:34:25.962Z]   GitCommit:        de40ad0
[2021-03-30T16:34:54.625Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-30T16:34:54.700Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21272/src/github.com/elastic/beats
[2021-03-30T16:35:04.527Z] Running on beats-ci-immutable-ubuntu-1804-1617121372977584408 in /var/lib/jenkins/workspace/Beats_beats_PR-21272
[2021-03-30T16:35:04.591Z] [INFO] getVaultSecret: Getting secrets
[2021-03-30T16:35:04.713Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-30T16:35:05.472Z] + chmod 755 generate-build-data.sh
[2021-03-30T16:35:05.472Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/5 FAILURE 3256042
[2021-03-30T16:35:05.472Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/5/steps/?limit=10000 -o steps-info.json
[2021-03-30T16:35:06.857Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/5/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 27
Passed 8346
Skipped 581
Total 8954

@sayden sayden marked this pull request as ready for review October 22, 2020 10:24
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 22, 2020

Pinging @elastic/stack-monitoring (Stack monitoring)

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 22, 2020

Pinging @elastic/integrations-services (Team:Services)

@chrisronline
Copy link
Copy Markdown
Contributor

chrisronline commented Oct 29, 2020

We actually don't use this dataset within the Stack Monitoring UI directly; however, it is used by the telemetry code, but I'm unable to fully test that until the beats_stats set is also converted.

Maybe we should create a single PR with both state and stats for beats and I can test that? Feel free to use them as separate PRs for the beats PR process but maybe merge them both into an another feature branch?

@sayden
Copy link
Copy Markdown
Contributor Author

sayden commented Nov 3, 2020

Sounds good! I'll close both and open the feature branch with their contents.

@sayden sayden closed this Nov 3, 2020
@sayden sayden mentioned this pull request Nov 3, 2020
Closed
@sayden sayden reopened this Dec 18, 2020
@sayden sayden requested a review from ycombinator December 28, 2020 17:02
@sayden sayden changed the title Make beats/state metricset work for Stack Monitoring without xpack.enabled flag Make beats module work for Stack Monitoring without xpack.enabled flag Dec 28, 2020
ycombinator
ycombinator requested changes Dec 28, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm seeing a lot of new fields being included in the event that weren't there before. I would suggest removing these until we need them. It's easy to add new fields later rather than remove them once they are part of the public API.

Also, shouldn't there be a module-level fields.yml that creates field aliases like we have in other Stack Monitoring modules' PRs?

@sayden sayden merged commit faa424f into elastic:feature-stack-monitoring-mb-ecs Mar 30, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 30, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #21272 updated

  • Start Time: 2021-03-30T17:03:55.745+0000

  • Duration: 37 min 38 sec

  • Commit: e609dcc

Test stats 🧪

Test Results
Failed 35
Passed 7882
Skipped 519
Total 8436

Trends 🧪

Image of Build Times

Image of Tests

Test errors 35

Expand to view the tests failures

> Show only the first 10 test failures

Build&Test / x-pack/metricbeat-unitTest / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module/statsd/test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../libbeat/tests/system/beat/beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
../../libbeat/tests/system/beat/beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
../../libbeat/tests/system/beat/beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
../../libbeat/tests/system/beat/beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

../../libbeat/tests/system/beat/beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-2012-windows-2012-r2 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-8-windows-8 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-2008-windows-2008-r2 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-7-windows-7 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-windows-2019 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-2016-windows-2016 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / x-pack/metricbeat-windows-10-windows-10 / test_server – x-pack.metricbeat.module.statsd.test_statsd.Test
    Expand to view the error details

     KeyError: 'fields'

    Expand to view the stacktrace

     self = <test_statsd.Test testMethod=test_server>

    def test_server(self):
        """
        statsd server metricset test
        """
    
        # Start the application
        self.render_config_template(modules=[{
            "name": "statsd",
            "metricsets": ["server"],
            "period": "5s",
            "host": STATSD_HOST,
            "port": STATSD_PORT,
        }])
        proc = self.start_beat()
        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
    
        # Send UDP packet with metric
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
        sock.close()
    
        self.wait_until(lambda: self.output_lines() > 0)
        proc.check_kill_and_wait()
        self.assert_no_logged_warnings(replace='use of closed network connection')
    
        # Verify output
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        evt = output[0]
        assert evt["statsd"]["metric1"]["value"] == 777
>       self.assert_fields_are_documented(evt)

module\statsd\test_statsd.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
    subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
    subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
name = 'beat.stats'

    def extract_fields(doc_list, name):
        fields = []
        dictfields = []
        aliases = []
    
        if doc_list is None:
            return fields, dictfields, aliases
    
        for field in doc_list:
    
            # Skip fields without name entry
            if "name" not in field:
                continue
    
            # Chain together names. Names in group `base` are top-level.
            if name != "" and name != "base":
                newName = name + "." + field["name"]
            else:
                newName = field["name"]
    
            if field.get("type") == "group":
>               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
E               KeyError: 'fields'

..\..\libbeat\tests\system\beat\beat.py:580: KeyError

Build&Test / metricbeat-windows-2008-windows-2008-r2 / TestReportErrorForMissingField – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestReportErrorForMissingField
    elastic_test.go:90: 
        	Error Trace:	elastic_test.go:90
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
    elastic_test.go:91: 
        	Error Trace:	elastic_test.go:91
        	Error:      	Not equal: 
        	            	expected: &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch stats API response"}
        	            	actual  : &errors.errorString{s:"Could not find field 'some.missing.field' in Elasticsearch API response"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,3 @@
        	            	 (*errors.errorString)({
        	            	- s: (string) (len=77) "Could not find field 'some.missing.field' in Elasticsearch stats API response"
        	            	+ s: (string) (len=71) "Could not find field 'some.missing.field' in Elasticsearch API response"
        	            	 })
        	Test:       	TestReportErrorForMissingField
--- FAIL: TestReportErrorForMissingField (0.00s)

Build&Test / metricbeat-windows-2008-windows-2008-r2 / TestConfigureModule/xpack_enabled – github.com/elastic/beats/v7/metricbeat/helper/elastic
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestConfigureModule/xpack_enabled
    elastic_test.go:209: 
        	Error Trace:	elastic_test.go:209
        	Error:      	Not equal: 
        	            	expected: elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"baz", "qux", "foo"}, XPackEnabled:true}
        	            	actual  : elastic.metricSetConfig{Module:"test_module", MetricSets:[]string{"foo", "bar"}, XPackEnabled:true}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,6 +2,5 @@
        	            	  Module: (string) (len=11) "test_module",
        	            	- MetricSets: ([]string) (len=3) {
        	            	-  (string) (len=3) "baz",
        	            	-  (string) (len=3) "qux",
        	            	-  (string) (len=3) "foo"
        	            	+ MetricSets: ([]string) (len=2) {
        	            	+  (string) (len=3) "foo",
        	            	+  (string) (len=3) "bar"
        	            	  },
        	Test:       	TestConfigureModule/xpack_enabled
    --- FAIL: TestConfigureModule/xpack_enabled (0.00s)

Steps errors 21

Expand to view the steps failures

Show only the first 10 steps failures

x-pack/metricbeat-unitTest - mage build unitTest
  • Took 4 min 30 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-windows-2019 - mage build unitTest
  • Took 7 min 9 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-2016-windows-2016 - mage build unitTest
  • Took 7 min 3 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-2012-windows-2012-r2 - mage build unitTest
  • Took 6 min 57 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-10-windows-10 - mage build unitTest
  • Took 8 min 46 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-2008-windows-2008-r2 - mage build unitTest
  • Took 7 min 51 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-8-windows-8 - mage build unitTest
  • Took 6 min 28 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-7-windows-7 - mage build unitTest
  • Took 7 min 59 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-7-32-windows-7-32-bit - mage build unitTest
  • Took 2 min 16 sec . View more details on here
  • Description: mage build unitTest
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

Log output

Expand to view the last 100 lines of log output 

[2021-03-30T17:36:38.543Z]     
[2021-03-30T17:36:38.543Z]         # Verify output
[2021-03-30T17:36:38.543Z]         output = self.read_output_json()
[2021-03-30T17:36:38.543Z]         self.assertGreater(len(output), 0)
[2021-03-30T17:36:38.543Z]         evt = output[0]
[2021-03-30T17:36:38.543Z]         assert evt["statsd"]["metric1"]["value"] == 777
[2021-03-30T17:36:38.543Z] >       self.assert_fields_are_documented(evt)
[2021-03-30T17:36:38.543Z] 
[2021-03-30T17:36:38.543Z] module\statsd\test_statsd.py:44: 
[2021-03-30T17:36:38.543Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2021-03-30T17:36:38.543Z] ..\..\libbeat\tests\system\beat\beat.py:701: in assert_fields_are_documented
[2021-03-30T17:36:38.543Z]     expected_fields, dict_fields, aliases = self.load_fields()
[2021-03-30T17:36:38.543Z] ..\..\libbeat\tests\system\beat\beat.py:624: in load_fields
[2021-03-30T17:36:38.543Z]     subfields, subdictfields, subaliases = extract_fields(item["fields"], "")
[2021-03-30T17:36:38.543Z] ..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
[2021-03-30T17:36:38.543Z]     subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
[2021-03-30T17:36:38.543Z] ..\..\libbeat\tests\system\beat\beat.py:580: in extract_fields
[2021-03-30T17:36:38.543Z]     subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
[2021-03-30T17:36:38.543Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2021-03-30T17:36:38.543Z] 
[2021-03-30T17:36:38.543Z] doc_list = [{'fields': [{'name': 'gc_next', 'type': 'long'}, {'name': 'memory.alloc', 'type': 'long'}, {'name': 'memory.total', '...': [...], 'name': 'write', 'type': 'group'}], 'name': 'output', 'type': 'group'}], 'name': 'libbeat', 'type': 'group'}]
[2021-03-30T17:36:38.543Z] name = 'beat.stats'
[2021-03-30T17:36:38.543Z] 
[2021-03-30T17:36:38.543Z]     def extract_fields(doc_list, name):
[2021-03-30T17:36:38.544Z]         fields = []
[2021-03-30T17:36:38.544Z]         dictfields = []
[2021-03-30T17:36:38.544Z]         aliases = []
[2021-03-30T17:36:38.544Z]     
[2021-03-30T17:36:38.544Z]         if doc_list is None:
[2021-03-30T17:36:38.544Z]             return fields, dictfields, aliases
[2021-03-30T17:36:38.544Z]     
[2021-03-30T17:36:38.544Z]         for field in doc_list:
[2021-03-30T17:36:38.544Z]     
[2021-03-30T17:36:38.544Z]             # Skip fields without name entry
[2021-03-30T17:36:38.544Z]             if "name" not in field:
[2021-03-30T17:36:38.544Z]                 continue
[2021-03-30T17:36:38.544Z]     
[2021-03-30T17:36:38.544Z]             # Chain together names. Names in group `base` are top-level.
[2021-03-30T17:36:38.544Z]             if name != "" and name != "base":
[2021-03-30T17:36:38.544Z]                 newName = name + "." + field["name"]
[2021-03-30T17:36:38.544Z]             else:
[2021-03-30T17:36:38.544Z]                 newName = field["name"]
[2021-03-30T17:36:38.544Z]     
[2021-03-30T17:36:38.544Z]             if field.get("type") == "group":
[2021-03-30T17:36:38.544Z] >               subfields, subdictfields, subaliases = extract_fields(field["fields"], newName)
[2021-03-30T17:36:38.544Z] E               KeyError: 'fields'
[2021-03-30T17:36:38.544Z] 
[2021-03-30T17:36:38.544Z] ..\..\libbeat\tests\system\beat\beat.py:580: KeyError
[2021-03-30T17:36:38.544Z] - generated xml file: C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats\x-pack\metricbeat\build\TEST-python-unit.xml -
[2021-03-30T17:36:38.544Z] ============================ slowest 20 durations =============================
[2021-03-30T17:36:38.544Z] 9.79s call     x-pack/metricbeat/module/statsd/test_statsd.py::Test::test_server
[2021-03-30T17:36:38.544Z] 1.57s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_export_index_pattern
[2021-03-30T17:36:38.544Z] 1.56s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_export_index_pattern_migration
[2021-03-30T17:36:38.544Z] 1.43s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_export_template
[2021-03-30T17:36:38.544Z] 1.26s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_export_config
[2021-03-30T17:36:38.544Z] 1.11s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_start_stop
[2021-03-30T17:36:38.544Z] 1.08s call     x-pack/metricbeat/tests/system/test_xpack_base.py::Test::test_export_ilm_policy
[2021-03-30T17:36:38.544Z] 
[2021-03-30T17:36:38.544Z] (13 durations < 0.005s hidden.  Use -vv to show these durations.)
[2021-03-30T17:36:38.544Z] =========================== short test summary info ===========================
[2021-03-30T17:36:38.544Z] FAILED module\statsd\test_statsd.py::Test::test_server - KeyError: 'fields'
[2021-03-30T17:36:38.544Z] ================== 1 failed, 6 passed, 22 skipped in 18.97s ===================
[2021-03-30T17:36:38.544Z] >> python test: Unit Testing Complete
[2021-03-30T17:36:38.544Z] Error: running "null\build\ve\windows\Scripts\pytest.exe --timeout=90 --durations=20 --junit-xml=build/TEST-python-unit.xml tests\system\test_xpack_base.py module\activemq\test_activemq.py module\appsearch\test_appsearch.py module\cockroachdb\test_cockroachdb.py module\coredns\test_coredns.py module\ibmmq\test_ibmmq.py module\mssql\test_mssql.py module\openmetrics\test_openmetrics.py module\redisenterprise\test_redisenterprise.py module\sql\test_sql.py module\stan\test_stan.py module\statsd\test_statsd.py" failed with exit code 1
[2021-03-30T17:36:38.931Z] 
[2021-03-30T17:36:38.931Z] C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-03-30T17:36:53.801Z] Copy .\x-pack\metricbeat\build into build\x-pack\metricbeat\build
[2021-03-30T17:36:53.801Z] Copy .\x-pack\metricbeat\null\build into build\x-pack\metricbeat\null\build
[2021-03-30T17:36:53.801Z] Copy .\x-pack\metricbeat\null\build\ve\windows\Lib\site-packages\pip\_internal\operations\build into build\x-pack\metricbeat\null\build\ve\windows\Lib\site-packages\pip\_internal\operations\build
[2021-03-30T17:36:53.815Z] Running in C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats\build
[2021-03-30T17:36:53.842Z] [INFO] Delete folders that are causing exceptions (See JENKINS-58421) is disabled for Windows.
[2021-03-30T17:36:53.851Z] Recording test results
[2021-03-30T17:36:54.892Z] [Checks API] No suitable checks publisher found.
[2021-03-30T17:36:55.259Z] 
[2021-03-30T17:36:55.259Z] C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats\build>tar --version  1>NUL 
[2021-03-30T17:36:55.633Z] 
[2021-03-30T17:36:55.633Z] C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats\build>tar --exclude=test-build-artifacts-x-pack/metricbeat-windows-10-windows-10.tgz -czf test-build-artifacts-x-pack/metricbeat-windows-10-windows-10.tgz . 
[2021-03-30T17:37:42.397Z] [INFO] Override default googleStorageUpload with some sleep
[2021-03-30T17:37:42.409Z] Sleeping for 1 min 24 sec
[2021-03-30T17:39:06.429Z] [Google Cloud Storage Plugin] Found 1 files to upload from pattern: test-build-artifacts-x-pack/metricbeat-windows-10-windows-10.tgz
[2021-03-30T17:39:06.645Z] [Google Cloud Storage Plugin] Uploading: test-build-artifacts-x-pack/metricbeat-windows-10-windows-10.tgz
[2021-03-30T17:39:08.626Z] [INFO] system-tests='build\x-pack\metricbeat\build\system-tests'. If no empty then let's create a tarball
[2021-03-30T17:39:08.974Z] 
[2021-03-30T17:39:08.974Z] C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats>tar --version  1>NUL 
[2021-03-30T17:39:09.326Z] 
[2021-03-30T17:39:09.326Z] C:\Users\jenkins\workspace\Beats_beats_PR-21272\src\github.com\elastic\beats>tar --exclude=x-pack-metricbeat--system-tests-windows.tgz -czf x-pack-metricbeat--system-tests-windows.tgz build\x-pack\metricbeat\build\system-tests 
[2021-03-30T17:39:09.635Z] [INFO] Override default googleStorageUpload with some sleep
[2021-03-30T17:39:09.648Z] Sleeping for 56 sec
[2021-03-30T17:40:05.666Z] [Google Cloud Storage Plugin] Found 1 files to upload from pattern: x-pack-metricbeat--system-tests-windows.tgz
[2021-03-30T17:40:05.504Z] [Google Cloud Storage Plugin] Uploading: x-pack-metricbeat--system-tests-windows.tgz
[2021-03-30T17:40:29.247Z] Failed in branch x-pack/metricbeat-windows-10-windows-10
[2021-03-30T17:40:29.364Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-30T17:40:29.429Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21272/src/github.com/elastic/beats
[2021-03-30T17:40:33.312Z] Running on beats-ci-immutable-ubuntu-1804-1617125388581396292 in /var/lib/jenkins/workspace/Beats_beats_PR-21272
[2021-03-30T17:40:33.405Z] [INFO] getVaultSecret: Getting secrets
[2021-03-30T17:40:33.492Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-30T17:40:34.798Z] + chmod 755 generate-build-data.sh
[2021-03-30T17:40:34.798Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/6 FAILURE 2198221
[2021-03-30T17:40:34.798Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/6/steps/?limit=10000 -o steps-info.json
[2021-03-30T17:40:36.181Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21272/runs/6/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 35
Passed 7882
Skipped 519
Total 8436

@sayden sayden mentioned this pull request Apr 27, 2021
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator ycombinator requested changes

Assignees

@sayden sayden

Labels

Feature:Stack Monitoring Metricbeat Metricbeat Team:Services (Deprecated) Label for the former Integrations-Services team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sayden @elasticmachine @chrisronline @ycombinator