Skip to content

Cherry-pick #16715 to 7.x: Center more functionality around RunnerFactory#18085

Merged
urso merged 1 commit intoelastic:7.xfrom
urso:backport_16715_7.x
Apr 29, 2020
Merged

Cherry-pick #16715 to 7.x: Center more functionality around RunnerFactory#18085
urso merged 1 commit intoelastic:7.xfrom
urso:backport_16715_7.x

Conversation

@urso
Copy link
Copy Markdown

@urso urso commented Apr 29, 2020

Cherry-pick of PR #16715 to 7.x branch. Original message:

  • Refactoring

What does this PR do?

Remove CheckableRunnerFactory and require RunnerFactory to implement
CheckConfig. CheckableRunnerFactory more and more superseded
RunnerFactory. As we want more config validation support in the future
as well I combined the two into RunnerFactory

Remove autodiscover.Adapter. The adapter did inherit from
CheckableRunnerFactory, giving us some inheritance chain between
RunnerFactory, CheckableRunnerFactory, and Adapter for autodiscovery.
By removig the Adapter and CheckableRunnerFactory we have one common
type (RunnerFactory) to integrate with config file reloading, static
input/module setup, and autodiscovery.

Add selectors for autodiscovery event selection that are used as
additional parameters when creating a new Autodiscover instance. This
gives us some more composability, yet I wonder if we can even remove those,
as every instance of NewAutodiscover did look for events with a 'config'
field of type []*common.Config. @exekias WDYT?

Why is it important?

The new input API will have a compatiblity layer creating a RunnerFactory. By having only one interface to integrate with, the integration will be more straight forward.

The change also makes it more clear how to integrate with autodiscovery and config file reloading, as it standardizes the two based on RunnerFactory only.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Developer Docs

The types autodiscovery.Adapter and cfgfile.CheckableRunnerFactory have been removed. Helpers for creating autodiscovery.Adapter have been removed as well. Instead cfgfile.RunnerFactory requires CheckConfig to be implemented now.

cfgfile.CheckableRunnerFactory and cfgfile.Runner factory have been combined, whereas the Adapter is split into cfgfile.RunnerFactory and autodiscover.EventConfigurer. The autodiscover.NewAutodiscover call now needs both, the RunnerFactory and EventConfigurer parameters.

The (cfgfile.RunnerFactory).Create method now accepts a beat.PipelineConnector instead of a beat.Pipeline as first parameter.

For example:

adapter := autodiscover.NewFactoryAdapter(bt.dynamicFactory)
autodiscover.NewAutodiscover("heartbeat", b.Publisher, adapter, bt.config.Autodiscover)

has become:

autodiscover.NewAutodiscover("heartbeat", b.Publisher, bt.dynamicFactory, autodiscover.QueryConfig(), bt.config.Autodiscover)

It is assumed that CheckConfig returns an error if the configuration does not apply. In case the factory did not support CheckConfig before, the functionality needs to be implemented like this:

func (f *Factory) CheckConfig(c *common.Config) error {
	_, err := f.Create(pubpipeline.NewNilPipeline(), c, nil)
	return err
}

* Center more functionality around RunnerFactory

Remove CheckableRunnerFactory and require RunnerFactory to implement
CheckConfig. CheckableRunnerFactory more and more superseded
RunnerFactory. As we want more config validation support in the future
as well I combined the two into RunnerFactory

Remove autodiscover.Adapter. The adapter did inherit from
CheckableRunnerFactory, giving us some inheritance chain between
RunnerFactory, CheckableRunnerFactory, and Adapter for autodiscovery.
By removig the Adapter and CheckableRunnerFactory we have one common
type (RunnerFactory) to integrate with config file reloading, static
input/module setup, and autodiscovery.

Add selectors for autodiscovery event selection that are used as
additional parameters when creating a new Autodiscover instance. This
gives us some more composability, yet I wonder if we can even remove those,
as every instance of NewAutodiscover did look for events with a 'config'
field of type []*common.Config.

(cherry picked from commit 7a1b524)
@urso urso requested a review from a team as a code owner April 29, 2020 11:29
@urso urso added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Apr 29, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations-services (Team:Services)

Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport lgtm!

@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 0
Passed 5823
Skipped 904
Total 6727

Steps errors

Expand to view the steps failures

  • Name: Make -C generator/_templates/metricbeat test
    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 3 min 11 sec<

    • Start Time: 2020-04-29T12:04:56.354+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T12:42:18.163Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T12:42:18.163Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T12:42:18.163Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T12:42:18.163Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T12:42:18.163Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T12:42:18.163Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T12:42:18.163Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T12:42:18.163Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T12:42:30.733Z] 
Creating activemq_8d5c646f5226f6d_activemq_1     ... done
...Stopping activemq_8d5c646f5226f6d_activemq_1     ... 
[2020-04-29T12:42:32.666Z] 
Stopping activemq_8d5c646f5226f6d_activemq_1     ... done
Removing activemq_8d5c646f5226f6d_activemq_1     ... 
[2020-04-29T12:42:40.987Z] 
Removing activemq_8d5c646f5226f6d_activemq_1     ... done
Creating activemq_bd10878642293355_activemq_1    ... 
[2020-04-29T12:43:05.515Z] 
Creating activemq_bd10878642293355_activemq_1    ... done
...Stopping activemq_bd10878642293355_activemq_1    ... 
[2020-04-29T12:43:06.912Z] 
Stopping activemq_bd10878642293355_activemq_1    ... done
Removing activemq_bd10878642293355_activemq_1    ... 
[2020-04-29T12:43:07.929Z] 
Removing activemq_bd10878642293355_activemq_1    ... done
Creating appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T12:43:09.902Z] 
Creating appsearch_7918a246f6f8_elasticsearch_1  ... done
Creating appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T12:48:49.009Z] 
Creating appsearch_7918a246f6f8_appsearch_1      ... done
.Stopping appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T12:48:49.009Z] Stopping appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T12:48:49.010Z] 
Stopping appsearch_7918a246f6f8_appsearch_1      ... done

Stopping appsearch_7918a246f6f8_elasticsearch_1  ... done
Removing appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T12:48:49.010Z] Removing appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T12:48:49.690Z] 
Removing appsearch_7918a246f6f8_elasticsearch_1  ... done

Removing appsearch_7918a246f6f8_appsearch_1      ... done
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T12:48:57.646Z] 
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... done
.Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T12:48:57.646Z] 
Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T12:48:58.305Z] 
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Creating coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T12:49:08.225Z] 
Creating coredns_7918a246f6f8_coredns_1          ... done
.Stopping coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T12:49:08.861Z] 
Stopping coredns_7918a246f6f8_coredns_1          ... done
Removing coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T12:49:09.542Z] 
Removing coredns_7918a246f6f8_coredns_1          ... done
Creating ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T12:49:39.386Z] 
Creating ibmmq_7918a246f6f8_ibmmq_1              ... done
.Stopping ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T12:49:39.386Z] 
Stopping ibmmq_7918a246f6f8_ibmmq_1              ... done
Removing ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T12:49:39.386Z] 
Removing ibmmq_7918a246f6f8_ibmmq_1              ... done
Creating mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T12:49:59.455Z] 
Creating mssql_7918a246f6f8_mssql_1              ... done
..Stopping mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T12:50:10.036Z] 
Stopping mssql_7918a246f6f8_mssql_1              ... done
Removing mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T12:50:14.595Z] 
Removing mssql_7918a246f6f8_mssql_1              ... done
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T12:50:21.860Z] 
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
.Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T12:50:22.234Z] 
Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T12:50:23.309Z] 
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Creating redisenterprise_403421ecf0675546_redisenterprise_1   ... 
[2020-04-29T12:53:07.211Z] 
Creating redisenterprise_403421ecf0675546_redisenterprise_1   ... done
..Stopping redisenterprise_403421ecf0675546_redisenterprise_1   ... 
[2020-04-29T12:53:12.487Z] 
Stopping redisenterprise_403421ecf0675546_redisenterprise_1   ... done
Removing redisenterprise_403421ecf0675546_redisenterprise_1   ... 
[2020-04-29T12:53:13.299Z] 
Removing redisenterprise_403421ecf0675546_redisenterprise_1   ... done
Creating sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T12:53:31.838Z] 
Creating sql_7918a246f6f8_mysql_1                             ... done
.Stopping sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T12:53:32.610Z] 
Stopping sql_7918a246f6f8_mysql_1                             ... done
Removing sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T12:53:33.935Z] 
Removing sql_7918a246f6f8_mysql_1                             ... done
Creating stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T12:53:49.070Z] 
Creating stan_7918a246f6f8_stan_1                             ... done
...Stopping stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T12:54:00.220Z] 
Stopping stan_7918a246f6f8_stan_1                             ... done
Removing stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T12:54:06.528Z] 
Removing stan_7918a246f6f8_stan_1                             ... done
.
[2020-04-29T12:54:06.528Z] [success] 59.14% test_xpack_base.Test.test_dashboards: 93.3063s
[2020-04-29T12:54:06.528Z] [success] 5.27% test_activemq.ActiveMqTest_0.test_broker_metrics_collected: 8.3209s
[2020-04-29T12:54:06.528Z] [success] 4.01% test_activemq.ActiveMqTest_1.test_queue_metrics_collected: 6.3314s
[2020-04-29T12:54:06.528Z] [success] 3.94% test_statsd.Test.test_server: 6.2117s
[2020-04-29T12:54:06.528Z] [success] 3.12% test_xpack_base.Test.test_migration: 4.9151s
[2020-04-29T12:54:06.528Z] [success] 2.55% test_xpack_base.Test.test_template: 4.0278s
[2020-04-29T12:54:06.529Z] [success] 2.14% test_stan.TestStan.test_metricset_2_subscriptions: 3.3829s
[2020-04-29T12:54:06.529Z] [success] 2.05% test_sql.Test.test_query: 3.2334s
[2020-04-29T12:54:06.529Z] [success] 1.90% test_openmetrics.Test.test_openmetrics: 3.0033s
[2020-04-29T12:54:06.529Z] [success] 1.82% test_cockroachdb.Test.test_status: 2.8728s
[2020-04-29T12:54:06.529Z] [success] 1.52% test_appsearch.Test.test_stats: 2.3966s
[2020-04-29T12:54:06.529Z] [success] 1.50% test_stan.TestStan.test_metricset_1_channels: 2.3706s
[2020-04-29T12:54:06.529Z] [success] 1.24% test_redisenterprise.Test_0.test_metricset_0_node: 1.9531s
[2020-04-29T12:54:06.529Z] [success] 1.21% test_redisenterprise.Test_0.test_metricset_1_proxy: 1.9092s
[2020-04-29T12:54:06.529Z] [success] 1.01% test_ibmmq.Test.test_qmgr: 1.5875s
[2020-04-29T12:54:06.529Z] [success] 0.94% test_activemq.ActiveMqTest_0.test_topic_metrics_collected: 1.4886s
[2020-04-29T12:54:06.529Z] [success] 0.93% test_activemq.ActiveMqTest_0.test_queue_metrics_collected: 1.4657s
[2020-04-29T12:54:06.529Z] [success] 0.87% test_coredns.Test.test_stats: 1.3783s
[2020-04-29T12:54:06.529Z] [success] 0.87% test_stan.TestStan.test_metricset_0_stats: 1.3773s
[2020-04-29T12:54:06.529Z] [success] 0.87% test_activemq.ActiveMqTest_1.test_broker_metrics_collected: 1.3766s
[2020-04-29T12:54:06.529Z] [success] 0.84% test_activemq.ActiveMqTest_1.test_topic_metrics_collected: 1.3199s
[2020-04-29T12:54:06.529Z] [success] 0.81% test_mssql.Test.test_status: 1.2815s
[2020-04-29T12:54:06.529Z] [success] 0.73% test_xpack_base.Test.test_start_stop: 1.1514s
[2020-04-29T12:54:06.529Z] [success] 0.71% test_mssql.Test.test_performance: 1.1212s
[2020-04-29T12:54:06.529Z] ----------------------------------------------------------------------
[2020-04-29T12:54:06.529Z] Ran 24 tests in 933.322s
[2020-04-29T12:54:06.529Z] 
[2020-04-29T12:54:06.529Z] OK
[2020-04-29T12:54:06.529Z] >> python test: Integration Testing Complete
[2020-04-29T12:54:06.529Z] >> Stopping Docker test environment...
[2020-04-29T12:54:11.974Z] Recording test results
[2020-04-29T12:54:31.181Z] Archiving artifacts
[2020-04-29T12:54:33.403Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T12:54:34.162Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T12:54:34.162Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.162Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T12:54:34.162Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.162Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T12:54:34.162Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.162Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T12:54:34.163Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.163Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T12:54:34.163Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.163Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T12:54:34.163Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.163Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T12:54:34.163Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T12:54:34.163Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T12:54:34.163Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T12:54:39.753Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18085
[2020-04-29T12:54:40.160Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T12:54:40.386Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T12:54:42.028Z] + chmod 755 generate-build-data.sh
[2020-04-29T12:54:42.028Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18085/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18085/runs/1 FAILURE 5057394
[2020-04-29T12:54:42.939Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18085/runs/1/steps/?limit=10000 -o steps-info.json
[2020-04-29T12:54:44.283Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18085/runs/1/tests/?status=FAILED -o tests-errors.json

@urso urso merged commit ef259cf into elastic:7.x Apr 29, 2020
@urso urso deleted the backport_16715_7.x branch April 29, 2020 12:59
@andresrc andresrc removed the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: Done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants