Input v2 compatibility layer by urso · Pull Request #19401 · elastic/beats

urso · 2020-06-25T18:59:21Z

Enhancement

What does this PR do?

This change introduces the input/v2/compat package, that provides a
compatibility layer for integration the input v2 API into existing beats
functionality.
The package provides a function for wrapping v2 inputs into a
cfgfile.RunnerFactory (used by autodiscovery, filebet module loading,
config file reloading, ....).

The current state of the full implementation can be seen here and sample inputs based on the new API.

The full list of changes will include:

Introduce v2 API interfaces
Introduce compatibility layer to integrate API with existing functionality
Introduce helpers for writing stateless inputs.
Introduce helpers for writing inputs that store a state between restarts.
Integrate new API with existing inputs and modules in filebeat.

Test coverage report:

gocov test ./... | gocov report
ok  	github.com/elastic/beats/v7/filebeat/input/v2/compat	0.009s	coverage: 94.6% of statements

github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go composeFactory.Create		 100.00% (11/11)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 factory.CheckConfig		 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 factory.Create			 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go composeFactory.CheckConfig	 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.Stop			 100.00% (3/3)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.Start			 100.00% (3/3)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 RunnerFactory			 100.00% (1/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go Combine			 100.00% (1/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 @99:5				 80.00% (4/5)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.String			 0.00% (0/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat		 --------------------------	 94.59% (35/37)

Why is it important?

This PR only introduces the compatiblity layer which will allow use to use new input with existing Beats functionality, but also run new and old architecture in parallel.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Related issues

Relates Filebeat input v2 API #15324

elasticmachine · 2020-06-25T18:59:23Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-06-25T19:51:28Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [urso commented: jenkins run the tests please]
Start Time: 2020-06-29T18:34:22.514+0000
Duration: 70 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	543
Skipped	127
Total	670

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 2 min 22 sec
- Start Time: 2020-06-29T19:30:14.657+0000
- log

urso · 2020-06-29T18:33:46Z

jenkins run the tests please

kvch · 2020-06-30T11:09:06Z

filebeat/input/v2/compat/compat.go

Are you going to resolve this TODO in an upcoming PR?

Answer: hmmm.... 😄

There is no common way of how IDs are defined/generated. Let me update this by computing a hash from the configuration object (some parts in libbeat do so).

The cursor based input manager allows you to configure an id setting per input that allows you to have 2 similar inputs with different ID. I think I will move the setting into this layer (will update this PR), so it can be applied to each input (ID will be added to structured logs + can be set by agent in the future).

sayden · 2020-06-30T11:28:44Z

filebeat/input/v2/compat/composed.go

What is this for? At least it does not clearly come to my mind so it may require a short comment to clarify.

Ah, this can be removed. I used to export the factory, but added the constructor that returns the expected interface type later. When implementing and exporting a concrete type you still might want the compiler to check that this concrete type implements the supposed interface correctly. Common pattern to do so is:

var _ <interface> = <struct type>{}

or (for interfaces on pointers):

var _ <interface> = (*<struct type>)(nil)

This declares an unused variable, but still triggers the type checker.

An example in the stdlib coming to mind is json.RawMessage.

The idea is to create the compiler error for the author modifying the type, and not the person using the type.

sayden · 2020-06-30T11:29:48Z

filebeat/input/v2/compat/composed.go

Nit: XXX? Also typo in compining

elasticmachine · 2020-06-30T12:35:06Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19401 updated]
Start Time: 2020-06-30T13:10:03.841+0000
Duration: 30 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	543
Skipped	127
Total	670

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 27 sec
- Start Time: 2020-06-30T13:35:46.457+0000
- log

urso · 2020-06-30T12:36:13Z

jenkins run the tests please

This change introduces the input/v2/compat package, that provides a compatibility layer for integration the input v2 API into existing beats functionality. The package provides a function for wrapping v2 inputs into a cfgfile.RunnerFactory (used by autodiscovery, filebet module loading, config file reloading, ....).

- fix typo - remove superfluous declaration - set input ID based on config hash or from setting if presen

urso added review Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Project:Filebeat-Input-v2 Team:Services (Deprecated) Label for the former Integrations-Services team labels Jun 25, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 25, 2020

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Jun 28, 2020

zube bot added [zube]: Inbox [zube]: In Review and removed [zube]: In Review [zube]: Inbox labels Jun 28, 2020

zube bot added [zube]: In Review [zube]: Inbox and removed [zube]: Inbox [zube]: In Review labels Jun 28, 2020

kvch reviewed Jun 30, 2020

View reviewed changes

sayden reviewed Jun 30, 2020

View reviewed changes

sayden approved these changes Jun 30, 2020

View reviewed changes

kvch approved these changes Jun 30, 2020

View reviewed changes

urso added 3 commits June 30, 2020 15:07

Review

cdf4fb1

- fix typo - remove superfluous declaration - set input ID based on config hash or from setting if presen

fix import order

54698d6

urso mentioned this pull request Jul 8, 2020

Cherry-pick #19401 to 7.x: Input v2 compatibility layer #19728

Merged

3 tasks

Conversation

urso commented Jun 25, 2020 • edited by andresrc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

elasticmachine commented Jun 25, 2020

Uh oh!

elasticmachine commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Uh oh!

urso commented Jun 29, 2020

Uh oh!

kvch Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

sayden Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

urso Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

sayden Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

urso Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

sayden Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Uh oh!

urso commented Jun 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

urso commented Jun 25, 2020 •

edited by andresrc

Loading

elasticmachine commented Jun 25, 2020 •

edited

Loading

elasticmachine commented Jun 30, 2020 •

edited

Loading