Add nomad processor by jorgemarey · Pull Request #14500 · elastic/beats

jorgemarey · 2019-11-13T12:45:25Z

This adds a nomad proccessor that annotates the logs with properties. Adds the following to each log:

region
namespace
dc
job
group
allocation
task
alloc_index
stream (stderr / stdour)
meta (those that start with a defined prefix)

The following is a configuration example:

filebeat.inputs:
- type: log
  paths:
    - /opt/nomad/alloc/*/alloc/logs/*
  exclude_files: ['\.fifo$']

processors:
- add_nomad_metadata:
    when:
      regexp:
        log.file.path: "/opt/nomad/alloc/.*"

The tests are missing. I'm working on them (I decided to do the PR first to do the needed changes for this to be aproved).

elasticmachine · 2019-11-13T12:45:27Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2019-11-13T12:45:27Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

libbeat/processors/add_nomad_metadata/nomad.go

libbeat/processors/add_nomad_metadata/config.go

libbeat/processors/add_nomad_metadata/cache.go

libbeat/processors/actions/extract_regexp.go

libbeat/common/nomad/util.go

libbeat/common/nomad/client.go

libbeat/processors/add_nomad_metadata/nomad.go

Previously DefaultConfig() was not being called so default environment vaiables weren't used. This also renames the library import from nomadbeat to nomadlib.

libbeat/processors/actions/decode_semaas.go

kvch · 2019-11-19T15:01:46Z

libbeat/processors/add_nomad_metadata/nomad.go

+		return nil, err
+	}
+
+	go watcher.Start()


Which function calls watcher.Stop? How do you make sure that the goroutine is stopped when it has to?

It's not called anywhere. I tried to find a way to call this, but the Processor interface only defines the Run and String methods. Anyway, in filebeat, the processor is defined on the configuration and applied until the program is stopped, so watcher.Stop would only be called at the end of the program, and in that case the program will exit, so no need to clean goroutines.

I was looking at other processors (docker, k8s) and the Stop functions of them are not called on the provider.

What should I do here?

libbeat/processors/add_nomad_metadata/nomad.go

libbeat/common/nomad/client.go

libbeat/common/nomad/util.go

kvch · 2019-11-19T15:10:30Z

Great addition, thank you! I added a few more questions to the PR. Could you please provide documentation or at least a description for the processors so we can add documentation for the processors?
Also, ping me when you are ready with the tests.

kvch · 2019-11-19T15:28:19Z

I have run make crosscompile in the root of Filebeat. Unfortunately, I am getting the following errors:

n@sleipnir ~/go/src/github.com/elastic/beats/filebeat                                                                                               [15:56:40]
> $ make crosscompile                                                                                                             ⬡ 10.14.1 [±feature/nomad ✓]
go get github.com/mitchellh/gox
mkdir -p /home/n/go/src/github.com/elastic/beats/filebeat/build/bin
gox -output="/home/n/go/src/github.com/elastic/beats/filebeat/build/bin/{{.Dir}}-{{.OS}}-{{.Arch}}" -os="linux darwin windows freebsd netbsd openbsd" -osarch="!darwin/arm !darwin/arm64 !darwin/386" -arch="amd64 386 arm ppc64 ppc64le"
Number of parallel builds: 3

-->     openbsd/386: github.com/elastic/beats/filebeat
-->     linux/ppc64: github.com/elastic/beats/filebeat
-->     windows/386: github.com/elastic/beats/filebeat
-->    netbsd/amd64: github.com/elastic/beats/filebeat
-->   freebsd/amd64: github.com/elastic/beats/filebeat
-->     freebsd/386: github.com/elastic/beats/filebeat
-->     freebsd/arm: github.com/elastic/beats/filebeat
-->    darwin/amd64: github.com/elastic/beats/filebeat
-->   linux/ppc64le: github.com/elastic/beats/filebeat
-->   windows/amd64: github.com/elastic/beats/filebeat
-->      netbsd/arm: github.com/elastic/beats/filebeat
-->      netbsd/386: github.com/elastic/beats/filebeat
-->       linux/386: github.com/elastic/beats/filebeat
-->     linux/amd64: github.com/elastic/beats/filebeat
-->       linux/arm: github.com/elastic/beats/filebeat
-->   openbsd/amd64: github.com/elastic/beats/filebeat

8 errors occurred:
--> openbsd/386 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> netbsd/amd64 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> freebsd/amd64 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> freebsd/386 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> freebsd/arm error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> netbsd/arm error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> netbsd/386 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

--> openbsd/amd64 error: exit status 1
Stderr: ../libbeat/processors/add_nomad_metadata/nomad.go:28:2: build constraints exclude all Go files in /home/n/go/src/github.com/elastic/beats/libbeat/common/nomad

../libbeat/scripts/Makefile:118: recipe for target 'crosscompile' failed
make: *** [crosscompile] Error 1

Could you please resolve this issue? I assume you need to adjust the build tags.

libbeat/processors/add_nomad_metadata/add_nomad_metadata.go

jorgemarey

I changed the build tags to fix the compiling problems.
I'm working on the tests. Sorry for the delay, I'm a little busy this days.
Where should I add the documentation?

kvch · 2019-11-26T13:34:22Z

Documentation should be added to the docs folder of the processor: libbeat/processors/add_nomad_metadata/docs and named as add_nomad_metadata.asciidoc. Then add a reference to it to the file libbeat/docs/processors-list.asciidoc. See the doc of add_docker_metadata: https://github.com/elastic/beats/blob/master/libbeat/processors/add_docker_metadata/docs/add_docker_metadata.asciidoc

kvch · 2019-12-09T08:12:55Z

libbeat/processors/add_nomad_metadata/add_nomad_metadata.go

+	jsprocessor.RegisterPlugin("AddNomadMetadata", New)
+}
+
+// New constructs a new add_kubernetes_metadata processor.


Please correct the comment.

libbeat/processors/add_nomad_metadata/add_nomad_metadata.go

kvch · 2019-12-09T08:16:07Z

libbeat/processors/add_nomad_metadata/docs/add_nomad_metadata.asciidoc

@@ -0,0 +1,79 @@
+[[add-nomad-metadata]]
+=== Add Nomad metadata


Please also mark the docs beta by adding the following line:

beta[]

kvch

LGTM, waiting for marking it beta.

houndci-bot · 2019-12-10T10:43:40Z

libbeat/processors/add_nomad_metadata/add_nomad_metadata_test.go

@@ -0,0 +1 @@
+package add_nomad_metadata


don't use an underscore in package name

kvch · 2019-12-16T13:15:21Z

Handing it over to @elastic/integrations

jsoriano · 2020-03-20T20:42:45Z

Hey @jorgemarey, thanks a lot for your efforts, they are very appreciated, and sorry for the delay giving feedback!

As commented in #16853, let's put this PR on hold by now and let's try to move #14954 forward. #14954 includes a nomad processor, but also an autodiscover provider.
I think that in any case there are good ideas here that could be added to #14954, so it would be great if you have some time to take a look there.

Add nomad processor

822e9a8

houndci-bot reviewed Nov 13, 2019

View reviewed changes

jorgemarey added 4 commits November 13, 2019 15:23

Add license

38b7a58

Fix struct tag & add comments

72a2353

Fmt files

d86d690

Update NOTICE

149a322

kvch added :Processors in progress Pull request is currently in progress. libbeat review labels Nov 15, 2019

kvch reviewed Nov 15, 2019

View reviewed changes

libbeat/processors/add_nomad_metadata/nomad.go Outdated Show resolved Hide resolved

kvch reviewed Nov 15, 2019

View reviewed changes

libbeat/processors/add_nomad_metadata/nomad.go Show resolved Hide resolved

Fix environmental variables not being used.

c20e0c9

Previously DefaultConfig() was not being called so default environment vaiables weren't used. This also renames the library import from nomadbeat to nomadlib.

houndci-bot reviewed Nov 16, 2019

View reviewed changes

libbeat/processors/actions/decode_semaas.go Outdated Show resolved Hide resolved

libbeat/processors/actions/decode_semaas.go Outdated Show resolved Hide resolved

Fix extract regexp name and add comment on usage

95b72c9

jorgemarey force-pushed the feature/nomad branch from 159f3fb to 95b72c9 Compare November 16, 2019 15:39

kvch reviewed Nov 19, 2019

View reviewed changes

libbeat/processors/add_nomad_metadata/nomad.go Show resolved Hide resolved

kvch reviewed Nov 19, 2019

View reviewed changes

libbeat/common/nomad/client.go Outdated Show resolved Hide resolved

kvch reviewed Nov 19, 2019

View reviewed changes

libbeat/common/nomad/util.go Outdated Show resolved Hide resolved

kvch added the needs_docs label Nov 19, 2019

jorgemarey added 2 commits November 25, 2019 16:51

Fix formatting problems

2e26ae6

Rename file

a8076b8

houndci-bot reviewed Nov 25, 2019

View reviewed changes

libbeat/processors/add_nomad_metadata/add_nomad_metadata.go Show resolved Hide resolved

jorgemarey commented Nov 25, 2019

View reviewed changes

kvch added the Team:Beats label Nov 26, 2019

kvch self-assigned this Nov 26, 2019

Add add_nomad_metadata processor documentation

1f2c600

kvch reviewed Dec 9, 2019

View reviewed changes

libbeat/processors/add_nomad_metadata/add_nomad_metadata.go Show resolved Hide resolved

kvch reviewed Dec 9, 2019

View reviewed changes

jorgemarey added 2 commits December 10, 2019 11:38

Mark nomad processor as beta

e1c415c

Add some test for extract regexp processor

4fdd1c1

houndci-bot reviewed Dec 10, 2019

View reviewed changes

kvch added Team:Integrations Label for the Integrations team and removed Team:Beats labels Dec 16, 2019

kvch removed their assignment Dec 23, 2019

urso added the [zube]: Inbox label Feb 4, 2020

andresrc added Team:Platforms Label for the Integrations - Platforms team [zube]: In Progress and removed libbeat [zube]: Inbox labels Feb 5, 2020

jsoriano mentioned this pull request Mar 5, 2020

Add support for Nomad #16853

Closed

7 tasks

jsoriano mentioned this pull request Mar 20, 2020

Autodiscover provider for Nomad #14954

Merged

4 tasks

jsoriano closed this Mar 20, 2020

zube bot added [zube]: Done and removed [zube]: In Progress labels Mar 20, 2020

andresrc removed the [zube]: Done label Mar 31, 2020

		@@ -0,0 +1,79 @@
		[[add-nomad-metadata]]
		=== Add Nomad metadata

Conversation

jorgemarey commented Nov 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 13, 2019

Uh oh!

elasticmachine commented Nov 13, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kvch Nov 19, 2019

Choose a reason for hiding this comment

Uh oh!

jorgemarey Nov 25, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kvch commented Nov 19, 2019

Uh oh!

kvch commented Nov 19, 2019

Uh oh!

Uh oh!

jorgemarey left a comment

Choose a reason for hiding this comment

Uh oh!

kvch commented Nov 26, 2019

Uh oh!

kvch Dec 9, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kvch Dec 9, 2019

Choose a reason for hiding this comment

Uh oh!

kvch left a comment

Choose a reason for hiding this comment

Uh oh!

houndci-bot Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

kvch commented Dec 16, 2019

Uh oh!

jsoriano commented Mar 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jorgemarey commented Nov 13, 2019 •

edited

Loading