Skip to content

[WIP] Beatless initial beat (local only)#8170

Closed
ph wants to merge 29 commits intoelastic:feature-beatlessfrom
ph:fix/beatless-stdin-provided
Closed

[WIP] Beatless initial beat (local only)#8170
ph wants to merge 29 commits intoelastic:feature-beatlessfrom
ph:fix/beatless-stdin-provided

Conversation

@ph
Copy link
Copy Markdown
Contributor

@ph ph commented Aug 30, 2018

NOT READY FOR REVIEW

  • Initial registry for provider and function
  • local provider + stdin function for easy local testing.
  • License checker integration.
  • Sync client integration.

Testing configuration.

beatless.provider.local:
  functions:
    - type: stdin
      enabled: true

output.elasticsearch:
  hosts: ["localhost:9200"]

@ph ph added in progress Pull request is currently in progress. Functionbeat labels Aug 30, 2018
@ph ph self-assigned this Aug 30, 2018
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Runnable should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type FunctionFactory should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method DefaultProvider.Name should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method DefaultProvider.Functions should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function NewDefaultProvider should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type DefaultProvider should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Config should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type name will be used as feature.FeatureRegistry by other packages, and that stutters; consider calling this Registry

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported type FeatureRegistry should be of the form "FeatureRegistry ..." (with optional leading article)

@ph
Copy link
Copy Markdown
Contributor Author

ph commented Aug 30, 2018

I am adding test followed by the AWS provider :)

@ph ph changed the title [WIP] Beatless initial structure [WIP] Beatless initial beat (local only) Aug 30, 2018
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/close/closed

@ph
Copy link
Copy Markdown
Contributor Author

ph commented Aug 30, 2018

I will fix the package build before merging my other branches.

@ph ph closed this Aug 30, 2018
@ph ph changed the base branch from feature/beatless to feature-beatless August 30, 2018 20:15
@ph ph reopened this Aug 30, 2018
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method DefaultProvider.CreateFunctions should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method DefaultProvider.Name should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function NewDefaultProvider should have comment or be unexported

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Config should have comment or be unexported

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I have restored my laptop I forgot to add 1 to my global gitignore :(

ph added 16 commits September 10, 2018 15:34
* Beatless initial PR

Add a new beat inside the x-pack folder under the Elastic License,
minimal requirement changes to have a build and a test running.

Main makefile exclude ASL2 for x-pack but check for Elastic.
Beats can override the license in their Makefile.
In beat each data collector need to initialize his own beat.Client to
have access to the pipeline. The current pipeline implementation is
completely asynchronous, meaning when you publish something to the
queue, you don't know if it will be send or when it will be send.

Some system like aws lambda requires to be in sync, when the method
return we expect the events to be send. This PR allow to change the
behavior to have a sync publish that leverage the pipeline callbacks.

Notes: it also changes the client interface, since publish and publishAll can
return an error.

Usage:

```
sc, err := NewSyncClient(pipeline, beat.ClientConfig{})
if !assert.NoError(t, err) {
  return
}

err := sc.PublishAll()
if err != nil {
...
}

sc.Wait() // block until the publish is done.
defer sc.Close() // this call will also block
```
* License manager

Implements a License manager inside beats, as we development more
features that depends on the licensing and the capabilities of a remote
cluster we need a unique way to access that information. This commit
implements the following:

Add a License manager that can be started at the beginning of the beats
instance initialization. The manager takes a fetcher, currently we only
support Elasticsearch as the license backend but we could add support
for an Logstash endpoint that could proxy the license.

Notes:

- By default when the manager is started, no license is available,
calling `Get()` on the manager will return a license not found.

- The manager will periodically retrieve the license from the fetcher.

- When an error occurs on the periodic check, the license wont be
invalidated right away but will enter a grace period, after this period
the license will be invalidated and will replaced by the OSS license.

- License and capabilities and be retrieved by calling `Get()` or
registering a type implementing the `Watcher` interface.
@ph
Copy link
Copy Markdown
Contributor Author

ph commented Sep 10, 2018

I will take a look at the license failing.

Addition:
- This commit allows to skip files during the generating of the license
  notice file. In the current specific case the aws-lambda-go contains a
  `LICENSE-SUMMARY` file explaining what license cover what code.
- Add the MIT-0 zero content to the license check
- Add the updated license file.

Notes:
The lambda functions are licensed under a modified MIT License (MIT-0[1])
The other code is licensed under an Apache License v2.0

[1]:https://github.com/aws/mit-0
@ph ph force-pushed the feature-beatless branch from 35ec7fe to fd0a33e Compare September 25, 2018 17:39
@ph ph closed this Sep 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Functionbeat in progress Pull request is currently in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants