Skip to content

Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support #32673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fearful-symmetry merged 45 commits into from
Sep 6, 2022
Merged

Update Metricbeat, Filebeat, libbeat with elastic-agent V2 support #32673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
561ade0
basic framework
fearful-symmetry Jun 30, 2022
a1f3e0f
continued tinkering
fearful-symmetry Jul 12, 2022
47b24ec
move away from ast code, use a struct
fearful-symmetry Jul 25, 2022
46e88f0
get metricbeat working, starting on filebeat
fearful-symmetry Aug 1, 2022
fa57a12
Merge remote-tracking branch 'upstream/main' into cm-v2-switch
fearful-symmetry Aug 2, 2022
570d267
add notice update
fearful-symmetry Aug 2, 2022
0a24a5a
Merge remote-tracking branch 'upstream/main' into cm-v2-switch
fearful-symmetry Aug 3, 2022
98cbbab
add basic config register
fearful-symmetry Aug 4, 2022
774608c
move over processors to individual beats
fearful-symmetry Aug 4, 2022
34dac6a
remove comments
fearful-symmetry Aug 4, 2022
0e9d755
start to integrate V2 client changes
fearful-symmetry Aug 4, 2022
0c6548f
finishing touches
fearful-symmetry Aug 12, 2022
20cb202
lint
fearful-symmetry Aug 12, 2022
cca151c
Merge remote-tracking branch 'upstream/main' into cm-v2-switch
fearful-symmetry Aug 12, 2022
d7c8b06
cleanup merge
fearful-symmetry Aug 12, 2022
4e1aabd
remove V1 controller
fearful-symmetry Aug 12, 2022
8eb9e7c
stil tinkering with linter
fearful-symmetry Aug 12, 2022
cea5f11
still fixing linter
fearful-symmetry Aug 12, 2022
45a8304
plz linter
fearful-symmetry Aug 12, 2022
50ef898
fmt x-pack files
fearful-symmetry Aug 12, 2022
a7b2345
notice update
fearful-symmetry Aug 12, 2022
01c422e
fix output test
fearful-symmetry Aug 15, 2022
8660a24
Merge remote-tracking branch 'upstream/main' into cm-v2-switch
fearful-symmetry Aug 15, 2022
2f45c54
refactor stop functions, refactor tests, some misc cleanup
fearful-symmetry Aug 16, 2022
890ce6c
fix client version string
fearful-symmetry Aug 16, 2022
43191e2
add devguide
fearful-symmetry Aug 16, 2022
d4a626c
Merge remote-tracking branch 'upstream/main' into cm-v2-switch
fearful-symmetry Aug 16, 2022
6446d36
linter
fearful-symmetry Aug 16, 2022
8ad939a
expand filebeat test
fearful-symmetry Aug 17, 2022
43b5c46
cleanup test
fearful-symmetry Aug 17, 2022
54e23ef
fix docs, add tests, debuggin
fearful-symmetry Aug 18, 2022
a43d00e
add signal handler
fearful-symmetry Aug 18, 2022
64218ba
fix mutex issue in register
fearful-symmetry Aug 22, 2022
2a5ec3f
Fix osquerybeat configuration for V2
aleksmaus Aug 23, 2022
44844e8
clean up component registration
fearful-symmetry Aug 25, 2022
44110d7
Merge remote-tracking branch 'origin/cm-v2-switch' into cm-v2-switch
fearful-symmetry Aug 25, 2022
b0d2580
spelling
fearful-symmetry Aug 25, 2022
8d8cf16
Merge remote-tracking branch 'upstream/feature-arch-v2' into cm-v2-sw…
fearful-symmetry Aug 30, 2022
a176bd9
remove workaround for filebeat types
fearful-symmetry Aug 30, 2022
3191e7c
try to fix filebeat tests
fearful-symmetry Aug 30, 2022
c942273
add nil checks, fix test, fix unit stop
fearful-symmetry Aug 31, 2022
b160c98
continue tinkering with nil type checks
fearful-symmetry Aug 31, 2022
d9626fa
add test for missing config datastreams, clean up nil handling
fearful-symmetry Aug 31, 2022
965404b
change nil protections, use getter methods
fearful-symmetry Sep 1, 2022
f038471
fix config access in output code
fearful-symmetry Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions NOTICE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9625,11 +9625,11 @@ Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-a

--------------------------------------------------------------------------------
Dependency : github.com/elastic/elastic-agent-client/v7
Version: v7.0.0-20210727140539-f0905d9377f6
Version: v7.0.0-20220804181728-b0328d2fe484
Licence type (autodetected): Elastic
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-client/v7@v7.0.0-20210727140539-f0905d9377f6/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-client/v7@v7.0.0-20220804181728-b0328d2fe484/LICENSE.txt:

ELASTIC LICENSE AGREEMENT

Expand Down Expand Up @@ -21297,11 +21297,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : golang.org/x/net
Version: v0.0.0-20220225172249-27dd8689420f
Version: v0.0.0-20220425223048-2871e0cb64e4
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/LICENSE:
Contents of probable licence file $GOMODCACHE/golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/LICENSE:

Copyright (c) 2009 The Go Authors. All rights reserved.

Expand Down Expand Up @@ -21593,11 +21593,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : google.golang.org/genproto
Version: v0.0.0-20220329172620-7be39ac1afc7
Version: v0.0.0-20220426171045-31bebdecfb46
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/google.golang.org/genproto@v0.0.0-20220329172620-7be39ac1afc7/LICENSE:
Contents of probable licence file $GOMODCACHE/google.golang.org/genproto@v0.0.0-20220426171045-31bebdecfb46/LICENSE:


Apache License
Expand Down Expand Up @@ -21805,11 +21805,11 @@ Contents of probable licence file $GOMODCACHE/google.golang.org/genproto@v0.0.0-

--------------------------------------------------------------------------------
Dependency : google.golang.org/grpc
Version: v1.45.0
Version: v1.46.0
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/google.golang.org/grpc@v1.45.0/LICENSE:
Contents of probable licence file $GOMODCACHE/google.golang.org/grpc@v1.46.0/LICENSE:


Apache License
Expand Down Expand Up @@ -32500,11 +32500,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : github.com/envoyproxy/go-control-plane
Version: v0.10.1
Version: v0.10.2-0.20220325020618-49ff273808a1
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/envoyproxy/go-control-plane@v0.10.1/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/envoyproxy/go-control-plane@v0.10.2-0.20220325020618-49ff273808a1/LICENSE:

Apache License
Version 2.0, January 2004
Expand Down
3 changes: 1 addition & 2 deletions filebeat/beater/filebeat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,10 +353,9 @@ func (fb *Filebeat) Run(b *beat.Beat) error {

// Register reloadable list of inputs and modules
inputs := cfgfile.NewRunnerList(management.DebugK, inputLoader, fb.pipeline)
reload.Register.MustRegisterList("filebeat.inputs", inputs)
reload.RegisterV2.MustRegisterInput(inputs)

modules := cfgfile.NewRunnerList(management.DebugK, moduleLoader, fb.pipeline)
reload.Register.MustRegisterList("filebeat.modules", modules)

var adiscover *autodiscover.Autodiscover
if fb.config.Autodiscover != nil {
Expand Down
10 changes: 5 additions & 5 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ require (
github.com/dustin/go-humanize v1.0.0
github.com/eapache/go-resiliency v1.2.0
github.com/eclipse/paho.mqtt.golang v1.3.5
github.com/elastic/elastic-agent-client/v7 v7.0.0-20210727140539-f0905d9377f6
github.com/elastic/elastic-agent-client/v7 v7.0.0-20220804181728-b0328d2fe484
github.com/elastic/go-concert v0.2.0
github.com/elastic/go-libaudit/v2 v2.3.2
github.com/elastic/go-licenser v0.4.0
Expand Down Expand Up @@ -168,16 +168,16 @@ require (
golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
golang.org/x/mod v0.5.1
golang.org/x/net v0.0.0-20220225172249-27dd8689420f
golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
golang.org/x/sync v0.0.0-20220513210516-0976fa681c29
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10
golang.org/x/text v0.3.7
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
golang.org/x/tools v0.1.9
google.golang.org/api v0.62.0
google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7
google.golang.org/grpc v1.45.0
google.golang.org/genproto v0.0.0-20220426171045-31bebdecfb46
google.golang.org/grpc v1.46.0
google.golang.org/protobuf v1.28.0
gopkg.in/inf.v0 v0.9.1
gopkg.in/jcmturner/aescts.v1 v1.0.1 // indirect
Expand Down Expand Up @@ -256,7 +256,7 @@ require (
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/elastic/elastic-transport-go/v8 v8.1.0 // indirect
github.com/envoyproxy/go-control-plane v0.10.1 // indirect
github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1 // indirect
github.com/envoyproxy/protoc-gen-validate v0.6.2 // indirect
github.com/evanphx/json-patch v4.12.0+incompatible // indirect
github.com/fearful-symmetry/gomsr v0.0.1 // indirect
Expand Down
18 changes: 11 additions & 7 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -596,8 +596,8 @@ github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 h1:lnDkqiRFKm0rxdljqr
github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3/go.mod h1:aPqzac6AYkipvp4hufTyMj5PDIphF3+At8zr7r51xjY=
github.com/elastic/elastic-agent-autodiscover v0.2.1 h1:Nbeayh3vq2FNm6xaFo34mhUdOu0EVlpj53CqCsbU0E4=
github.com/elastic/elastic-agent-autodiscover v0.2.1/go.mod h1:gPnzzfdYNdgznAb+iG9eyyXaQXBbAMHa+Y6Z8hXfcGY=
github.com/elastic/elastic-agent-client/v7 v7.0.0-20210727140539-f0905d9377f6 h1:nFvXHBjYK3e9+xF0WKDeAKK4aOO51uC28s+L9rBmilo=
github.com/elastic/elastic-agent-client/v7 v7.0.0-20210727140539-f0905d9377f6/go.mod h1:uh/Gj9a0XEbYoM4NYz4LvaBVARz3QXLmlNjsrKY9fTc=
github.com/elastic/elastic-agent-client/v7 v7.0.0-20220804181728-b0328d2fe484 h1:uJIMfLgCenJvxsVmEjBjYGxt0JddCgw2IxgoNfcIXOk=
github.com/elastic/elastic-agent-client/v7 v7.0.0-20220804181728-b0328d2fe484/go.mod h1:fkvyUfFwyAG5OnMF0h+FV9sC0Xn9YLITwQpSuwungQs=
github.com/elastic/elastic-agent-libs v0.2.2/go.mod h1:1xDLBhIqBIjhJ7lr2s+xRFFkQHpitSp8q2zzv1Dqg+s=
github.com/elastic/elastic-agent-libs v0.2.5/go.mod h1:chO3rtcLyGlKi9S0iGVZhYCzDfdDsAQYBc+ui588AFE=
github.com/elastic/elastic-agent-libs v0.2.7/go.mod h1:chO3rtcLyGlKi9S0iGVZhYCzDfdDsAQYBc+ui588AFE=
Expand Down Expand Up @@ -666,8 +666,9 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/go-control-plane v0.10.1 h1:cgDRLG7bs59Zd+apAWuzLQL95obVYAymNJek76W3mgw=
github.com/envoyproxy/go-control-plane v0.10.1/go.mod h1:AY7fTTXNdv/aJ2O5jwpxAPOWUZ7hQAEvzN5Pf27BkQQ=
github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1 h1:xvqufLtNVwAhN8NMyWklVgxnWohi+wtMGQMhtxexlm0=
github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/envoyproxy/protoc-gen-validate v0.6.2 h1:JiO+kJTpmYGjEodY7O1Zk8oZcNz1+f30UtwtXoFUPzE=
github.com/envoyproxy/protoc-gen-validate v0.6.2/go.mod h1:2t7qjJNvHPx8IjnBOzl9E9/baC+qXE/TeeyBRzgJDws=
Expand Down Expand Up @@ -1999,8 +2000,9 @@ golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d/go.mod h1:9nx3DQGgdP8bBQD5qx
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4 h1:HVyaeDAYux4pnY+D/SiwmLOR36ewZ4iGQIIrtnuCjFA=
golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190130055435-99b60b757ec1/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
Expand Down Expand Up @@ -2170,6 +2172,7 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220405052023-b1e9470b6e64/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220422013727-9388b58f7150/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 h1:WIoqL4EROvwiPdUtaip4VcDdpZ4kha7wBWZrbVKCIZg=
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down Expand Up @@ -2419,8 +2422,8 @@ google.golang.org/genproto v0.0.0-20211129164237-f09f9a12af12/go.mod h1:5CzLGKJ6
google.golang.org/genproto v0.0.0-20211203200212-54befc351ae9/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20211206160659-862468c7d6e0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7 h1:HOL66YCI20JvN2hVk6o2YIp9i/3RvzVUz82PqNr7fXw=
google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
google.golang.org/genproto v0.0.0-20220426171045-31bebdecfb46 h1:G1IeWbjrqEq9ChWxEuRPJu6laA67+XgTFHVSAvepr38=
google.golang.org/genproto v0.0.0-20220426171045-31bebdecfb46/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
google.golang.org/grpc v0.0.0-20160317175043-d3ddb4469d5a/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
Expand Down Expand Up @@ -2456,8 +2459,9 @@ google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnD
google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.40.1/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M=
google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
google.golang.org/grpc v1.46.0 h1:oCjezcn6g6A75TGoKYBPgKmVBLexhYLM6MebdrPApP8=
google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
Expand Down
4 changes: 1 addition & 3 deletions heartbeat/beater/heartbeat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,10 +182,8 @@ func (bt *Heartbeat) RunStaticMonitors(b *beat.Beat) (stop func(), err error) {

// RunCentralMgmtMonitors loads any central management configured configs.
func (bt *Heartbeat) RunCentralMgmtMonitors(b *beat.Beat) {
mons := cfgfile.NewRunnerList(management.DebugK, bt.dynamicFactory, b.Publisher)
reload.Register.MustRegisterList(b.Info.Beat+".monitors", mons)
inputs := cfgfile.NewRunnerList(management.DebugK, bt.dynamicFactory, b.Publisher)
reload.Register.MustRegisterList("inputs", inputs)
reload.RegisterV2.MustRegisterInput(inputs)
}

// RunReloadableMonitors runs the `heartbeat.config.monitors` portion of the yaml config if present.
Expand Down
48 changes: 29 additions & 19 deletions libbeat/cfgfile/cfgfile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"path/filepath"

"github.com/elastic/beats/v7/libbeat/common"
"github.com/elastic/beats/v7/libbeat/common/fleetmode"
"github.com/elastic/elastic-agent-libs/config"
"github.com/elastic/elastic-agent-libs/logp"
)
Expand Down Expand Up @@ -101,13 +102,13 @@ func HandleFlags() error {
home, err := filepath.Abs(filepath.Dir(os.Args[0]))
if err != nil {
if *homePath == "" {
return fmt.Errorf("The absolute path to %s could not be obtained. %v",
return fmt.Errorf("The absolute path to %s could not be obtained. %w",
os.Args[0], err)
}
home = *homePath
}

defaults.SetString("path.home", -1, home)
_ = defaults.SetString("path.home", -1, home)

if len(overwrites.GetFields()) > 0 {
common.PrintConfigDebugf(overwrites, "CLI setting overwrites (-E flag):")
Expand All @@ -133,30 +134,36 @@ func Read(out interface{}, path string) error {
// Load reads the configuration from a YAML file structure. If path is empty
// this method reads from the configuration file specified by the '-c' command
// line flag.
// This function cares about the underlying fleet setting, and if beats is running with
// the management.enabled flag, Load() will bypass reading a config file, and merely merge any overrides.
func Load(path string, beatOverrides []ConditionalOverride) (*config.C, error) {
var c *config.C
var err error

cfgpath := GetPathConfig()

if path == "" {
list := []string{}
for _, cfg := range configfiles.List() {
if !filepath.IsAbs(cfg) {
list = append(list, filepath.Join(cfgpath, cfg))
} else {
list = append(list, cfg)
if !fleetmode.Enabled() {
if path == "" {
list := []string{}
for _, cfg := range configfiles.List() {
if !filepath.IsAbs(cfg) {
list = append(list, filepath.Join(cfgpath, cfg))
} else {
list = append(list, cfg)
}
}
c, err = common.LoadFiles(list...)
} else {
if !filepath.IsAbs(path) {
path = filepath.Join(cfgpath, path)
}
c, err = common.LoadFile(path)
}
c, err = common.LoadFiles(list...)
} else {
if !filepath.IsAbs(path) {
path = filepath.Join(cfgpath, path)
if err != nil {
return nil, err
}
c, err = common.LoadFile(path)
}
if err != nil {
return nil, err
} else {
c = config.NewConfig()
}

if beatOverrides != nil {
Expand All @@ -183,6 +190,9 @@ func Load(path string, beatOverrides []ConditionalOverride) (*config.C, error) {
c,
overwrites,
)
if err != nil {
return nil, err
}
}

common.PrintConfigDebugf(c, "Complete configuration loaded:")
Expand All @@ -194,13 +204,13 @@ func LoadList(file string) ([]*config.C, error) {
logp.Debug("cfgfile", "Load config from file: %s", file)
rawConfig, err := common.LoadFile(file)
if err != nil {
return nil, fmt.Errorf("invalid config: %s", err)
return nil, fmt.Errorf("invalid config: %w", err)
}

var c []*config.C
err = rawConfig.Unpack(&c)
if err != nil {
return nil, fmt.Errorf("error reading configuration from file %s: %s", file, err)
return nil, fmt.Errorf("error reading configuration from file %s: %w", file, err)
}

return c, nil
Expand Down
4 changes: 2 additions & 2 deletions libbeat/cmd/instance/beat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ func (b *Beat) createBeater(bt beat.Creator) (beat.Beater, error) {
return nil, fmt.Errorf("error initializing publisher: %w", err)
}

reload.Register.MustRegister("output", b.makeOutputReloader(publisher.OutputReloader()))
reload.RegisterV2.MustRegisterOutput(b.makeOutputReloader(publisher.OutputReloader()))

// TODO: some beats race on shutdown with publisher.Stop -> do not call Stop yet,
// but refine publisher to disconnect clients on stop automatically
Expand Down Expand Up @@ -710,7 +710,7 @@ func (b *Beat) configure(settings Settings) error {
logp.Info("Beat ID: %v", b.Info.ID)

// initialize config manager
b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.Register, b.Beat.Info.ID)
b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.RegisterV2, b.Beat.Info.ID)
if err != nil {
return err
}
Expand Down
41 changes: 36 additions & 5 deletions libbeat/common/reload/reload.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,14 @@ import (
"github.com/elastic/elastic-agent-libs/mapstr"
)

// Register holds a registry of reloadable objects
var Register = NewRegistry()
// RegisterV2 is the special registry used for the V2 controller
var RegisterV2 = NewRegistry()

// InputRegName is the registation name for V2 inputs
const InputRegName = "input"

// OutputRegName is the registation name for V2 Outputs
const OutputRegName = "output"

// ConfigWithMeta holds a pair of config.C and optional metadata for it
type ConfigWithMeta struct {
Expand Down Expand Up @@ -106,13 +112,38 @@ func (r *Registry) MustRegister(name string, obj Reloadable) {
}
}

// MustRegisterList declares a reloadable object list
func (r *Registry) MustRegisterList(name string, list ReloadableList) {
if err := r.RegisterList(name, list); err != nil {
// MustRegisterOutput is a V2-specific registration function
// That declares a reloadable output
func (r *Registry) MustRegisterOutput(obj Reloadable) {
if err := r.Register(OutputRegName, obj); err != nil {
panic(err)
}
}

// MustRegisterInput is a V2-specific registration function
// that declares a reloadable object list for a beat input
func (r *Registry) MustRegisterInput(list ReloadableList) {
if err := r.RegisterList(InputRegName, list); err != nil {
panic(err)
}
}

// GetInputList is a V2-specific function
// That returns the reloadable list created for an input
func (r *Registry) GetInputList() ReloadableList {
r.RLock()
defer r.RUnlock()
return r.confsLists[InputRegName]
}

// GetReloadableOutput is a V2-specific function
// That returns the reloader for the registered output
func (r *Registry) GetReloadableOutput() Reloadable {
r.RLock()
defer r.RUnlock()
return r.confs[OutputRegName]
}

// GetRegisteredNames returns the list of names registered
func (r *Registry) GetRegisteredNames() []string {
r.RLock()
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/beater/metricbeat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ func (bt *Metricbeat) Run(b *beat.Beat) error {
// Centrally managed modules
factory := module.NewFactory(b.Info, bt.moduleOptions...)
modules := cfgfile.NewRunnerList(management.DebugK, factory, b.Publisher)
reload.Register.MustRegisterList(b.Info.Beat+".modules", modules)
reload.RegisterV2.MustRegisterInput(modules)
wg.Add(1)
go func() {
defer wg.Done()
Expand Down
2 changes: 1 addition & 1 deletion packetbeat/beater/packetbeat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ func (pb *packetbeat) runStatic(b *beat.Beat, factory *processorFactory) error {
// the runner by starting the beat's manager. It returns on the first fatal error.
func (pb *packetbeat) runManaged(b *beat.Beat, factory *processorFactory) error {
runner := newReloader(management.DebugK, factory, b.Publisher)
reload.Register.MustRegisterList("inputs", runner)
reload.RegisterV2.MustRegisterInput(runner)
logp.Debug("main", "Waiting for the runner to finish")

// Start the manager after all the hooks are registered and terminates when
Expand Down
Loading