Skip to content

Commit 87ab96f

Browse files
andrewkrohandrewkroh
committed
[Filebeat] Enable journald input in releases (#27351)
Use the main-debian8 variant of golang-crossbuilder to build Filebeat packages for linux/amd64 and linux/368. Previously a debian7 image was used, but debian7 does not have the systemd libraries needed for journald. Debian 8 uses glibc 2.19 and debian 7 uses 2.13. We used debian7 because the binaries produced worked on RHEL 6 which has glibc 2.12. To ensure that the Filebeat binaries for linux/amd64 and linux/386 continue to work on RHEL 6 the build process will check the glibc requirement to ensure it is <=2.12. To keep the linked glibc version compatible with RHEL 6 I had to upgrade to github.com/dgraph-io/badger/v3 from v2. v3 removed a bunch of cgo usages which reduced the need on various libc functions. badger adopted golang/glog for logging. This library is adds global CLI flags that conflict with Beats own logging CLI flags (like '-v'). This is a common problem for glog (k8s encountered this). To address the problem I forked golang/glog and made the flag.FlagSet used by the library configurable.
1 parent 4f081e6 commit 87ab96f

19 files changed

Lines changed: 554 additions & 106 deletions

File tree

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -491,6 +491,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
491491
- Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]
492492
- Enrich events of `state_service` metricset with kubernetes services' metadata. {pull}23730[23730]
493493
- Check fields are documented in aws metricsets. {pull}23887[23887]
494+
- Enable `journald` input type in Filebeat. {issue}7955[7955] {pull}27351[27351]
494495
- Move openmetrics module to oss. {pull}26561[26561]
495496
- Fix release state of kubernetes metricsets. {pull}26864[26864]
496497
- Add `gke` metricset collection to `gcp` module {pull}26824[26824]

‎NOTICE.txt‎

Lines changed: 19 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -4440,12 +4440,12 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
44404440

44414441

44424442
--------------------------------------------------------------------------------
4443-
Dependency : github.com/dgraph-io/badger/v2
4444-
Version: v2.2007.3-0.20201012072640-f5a7e0a1c83b
4443+
Dependency : github.com/dgraph-io/badger/v3
4444+
Version: v3.2103.1
44454445
Licence type (autodetected): Apache-2.0
44464446
--------------------------------------------------------------------------------
44474447

4448-
Contents of probable licence file $GOMODCACHE/github.com/dgraph-io/badger/v2@v2.2007.3-0.20201012072640-f5a7e0a1c83b/LICENSE:
4448+
Contents of probable licence file $GOMODCACHE/github.com/dgraph-io/badger/v3@v3.2103.1/LICENSE:
44494449

44504450
Apache License
44514451
Version 2.0, January 2004
@@ -9792,11 +9792,11 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
97929792

97939793
--------------------------------------------------------------------------------
97949794
Dependency : github.com/gogo/protobuf
9795-
Version: v1.3.1
9795+
Version: v1.3.2
97969796
Licence type (autodetected): BSD-3-Clause
97979797
--------------------------------------------------------------------------------
97989798

9799-
Contents of probable licence file $GOMODCACHE/github.com/gogo/protobuf@v1.3.1/LICENSE:
9799+
Contents of probable licence file $GOMODCACHE/github.com/gogo/protobuf@v1.3.2/LICENSE:
98009800

98019801
Copyright (c) 2013, The GoGo Authors. All rights reserved.
98029802

@@ -10311,11 +10311,11 @@ Contents of probable licence file $GOMODCACHE/github.com/gomodule/redigo@v1.8.3/
1031110311

1031210312
--------------------------------------------------------------------------------
1031310313
Dependency : github.com/google/flatbuffers
10314-
Version: v1.7.2-0.20170925184458-7a6b2bf521e9
10314+
Version: v1.12.0
1031510315
Licence type (autodetected): Apache-2.0
1031610316
--------------------------------------------------------------------------------
1031710317

10318-
Contents of probable licence file $GOMODCACHE/github.com/google/flatbuffers@v1.7.2-0.20170925184458-7a6b2bf521e9/LICENSE.txt:
10318+
Contents of probable licence file $GOMODCACHE/github.com/google/flatbuffers@v1.12.0/LICENSE.txt:
1031910319

1032010320

1032110321
Apache License
@@ -10506,7 +10506,7 @@ Contents of probable licence file $GOMODCACHE/github.com/google/flatbuffers@v1.7
1050610506
same "printed page" as the copyright notice for easier
1050710507
identification within third-party archives.
1050810508

10509-
Copyright 2014 Google Inc.
10509+
Copyright [yyyy] [name of copyright owner]
1051010510

1051110511
Licensed under the Apache License, Version 2.0 (the "License");
1051210512
you may not use this file except in compliance with the License.
@@ -21798,43 +21798,6 @@ Contents of probable licence file $GOMODCACHE/github.com/!burnt!sushi/xgb@v0.0.0
2179821798
// such litigation is filed.
2179921799

2180021800

21801-
--------------------------------------------------------------------------------
21802-
Dependency : github.com/DataDog/zstd
21803-
Version: v1.4.1
21804-
Licence type (autodetected): BSD-3-Clause
21805-
--------------------------------------------------------------------------------
21806-
21807-
Contents of probable licence file $GOMODCACHE/github.com/!data!dog/zstd@v1.4.1/LICENSE:
21808-
21809-
Simplified BSD License
21810-
21811-
Copyright (c) 2016, Datadog <info@datadoghq.com>
21812-
All rights reserved.
21813-
21814-
Redistribution and use in source and binary forms, with or without
21815-
modification, are permitted provided that the following conditions are met:
21816-
21817-
* Redistributions of source code must retain the above copyright notice,
21818-
this list of conditions and the following disclaimer.
21819-
* Redistributions in binary form must reproduce the above copyright notice,
21820-
this list of conditions and the following disclaimer in the documentation
21821-
and/or other materials provided with the distribution.
21822-
* Neither the name of the copyright holder nor the names of its contributors
21823-
may be used to endorse or promote products derived from this software
21824-
without specific prior written permission.
21825-
21826-
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21827-
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21828-
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21829-
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
21830-
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
21831-
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
21832-
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
21833-
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
21834-
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
21835-
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
21836-
21837-
2183821801
--------------------------------------------------------------------------------
2183921802
Dependency : github.com/Microsoft/hcsshim
2184021803
Version: v0.8.7
@@ -26435,11 +26398,11 @@ SOFTWARE.
2643526398

2643626399
--------------------------------------------------------------------------------
2643726400
Dependency : github.com/dgraph-io/ristretto
26438-
Version: v0.0.3-0.20200630154024-f66de99634de
26401+
Version: v0.1.0
2643926402
Licence type (autodetected): Apache-2.0
2644026403
--------------------------------------------------------------------------------
2644126404

26442-
Contents of probable licence file $GOMODCACHE/github.com/dgraph-io/ristretto@v0.0.3-0.20200630154024-f66de99634de/LICENSE:
26405+
Contents of probable licence file $GOMODCACHE/github.com/dgraph-io/ristretto@v0.1.0/LICENSE:
2644326406

2644426407
Apache License
2644526408
Version 2.0, January 2004
@@ -29856,12 +29819,12 @@ Contents of probable licence file $GOMODCACHE/github.com/golang-sql/civil@v0.0.0
2985629819
limitations under the License.
2985729820

2985829821
--------------------------------------------------------------------------------
29859-
Dependency : github.com/golang/glog
29860-
Version: v0.0.0-20160126235308-23def4e6c14b
29822+
Dependency : github.com/elastic/glog
29823+
Version: v1.0.1-0.20210831205241-7d8b5c89dfc4
2986129824
Licence type (autodetected): Apache-2.0
2986229825
--------------------------------------------------------------------------------
2986329826

29864-
Contents of probable licence file $GOMODCACHE/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/LICENSE:
29827+
Contents of probable licence file $GOMODCACHE/github.com/elastic/glog@v1.0.1-0.20210831205241-7d8b5c89dfc4/LICENSE:
2986529828

2986629829
Apache License
2986729830
Version 2.0, January 2004
@@ -35929,11 +35892,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
3592935892

3593035893
--------------------------------------------------------------------------------
3593135894
Dependency : github.com/kisielk/errcheck
35932-
Version: v1.2.0
35895+
Version: v1.5.0
3593335896
Licence type (autodetected): MIT
3593435897
--------------------------------------------------------------------------------
3593535898

35936-
Contents of probable licence file $GOMODCACHE/github.com/kisielk/errcheck@v1.2.0/LICENSE:
35899+
Contents of probable licence file $GOMODCACHE/github.com/kisielk/errcheck@v1.5.0/LICENSE:
3593735900

3593835901
Copyright (c) 2013 Kamil Kisiel
3593935902

@@ -36003,11 +35966,11 @@ match.go, match_test.go:
3600335966

3600435967
--------------------------------------------------------------------------------
3600535968
Dependency : github.com/klauspost/compress
36006-
Version: v1.12.2
35969+
Version: v1.12.3
3600735970
Licence type (autodetected): BSD-3-Clause
3600835971
--------------------------------------------------------------------------------
3600935972

36010-
Contents of probable licence file $GOMODCACHE/github.com/klauspost/compress@v1.12.2/LICENSE:
35973+
Contents of probable licence file $GOMODCACHE/github.com/klauspost/compress@v1.12.3/LICENSE:
3601135974

3601235975
Copyright (c) 2012 The Go Authors. All rights reserved.
3601335976
Copyright (c) 2019 Klaus Post. All rights reserved.
@@ -41653,11 +41616,11 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
4165341616

4165441617
--------------------------------------------------------------------------------
4165541618
Dependency : go.opencensus.io
41656-
Version: v0.22.2
41619+
Version: v0.22.5
4165741620
Licence type (autodetected): Apache-2.0
4165841621
--------------------------------------------------------------------------------
4165941622

41660-
Contents of probable licence file $GOMODCACHE/go.opencensus.io@v0.22.2/LICENSE:
41623+
Contents of probable licence file $GOMODCACHE/go.opencensus.io@v0.22.5/LICENSE:
4166141624

4166241625

4166341626
Apache License

‎dev-tools/mage/common.go‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import (
2626
"context"
2727
"crypto/sha256"
2828
"crypto/sha512"
29+
"debug/elf"
2930
"encoding/hex"
3031
"encoding/json"
3132
"fmt"
@@ -38,6 +39,7 @@ import (
3839
"path/filepath"
3940
"regexp"
4041
"runtime"
42+
"sort"
4143
"strconv"
4244
"strings"
4345
"sync"
@@ -913,3 +915,48 @@ func IntegrationTestEnvVars() []string {
913915
}
914916
return vars
915917
}
918+
919+
// ReadGLIBCRequirement returns the required glibc version for a dynamically
920+
// linked ELF binary. The target machine must have a version equal to or
921+
// greater than (newer) the returned value.
922+
func ReadGLIBCRequirement(elfFile string) (*SemanticVersion, error) {
923+
e, err := elf.Open(elfFile)
924+
if err != nil {
925+
return nil, err
926+
}
927+
928+
symbols, err := e.DynamicSymbols()
929+
if err != nil {
930+
return nil, err
931+
}
932+
933+
versionSet := map[SemanticVersion]struct{}{}
934+
for _, sym := range symbols {
935+
if strings.HasPrefix(sym.Version, "GLIBC_") {
936+
semver, err := NewSemanticVersion(strings.TrimPrefix(sym.Version, "GLIBC_"))
937+
if err != nil {
938+
continue
939+
}
940+
941+
versionSet[*semver] = struct{}{}
942+
}
943+
}
944+
945+
if len(versionSet) == 0 {
946+
return nil, errors.New("no GLIBC symbols found in binary (is this a static binary?)")
947+
}
948+
949+
var versions []SemanticVersion
950+
for ver := range versionSet {
951+
versions = append(versions, ver)
952+
}
953+
954+
sort.Slice(versions, func(i, j int) bool {
955+
a := versions[i]
956+
b := versions[j]
957+
return a.LessThan(&b)
958+
})
959+
960+
max := versions[len(versions)-1]
961+
return &max, nil
962+
}

‎dev-tools/mage/crossbuild.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ type crossBuildParams struct {
129129

130130
// CrossBuild executes a given build target once for each target platform.
131131
func CrossBuild(options ...CrossBuildOption) error {
132-
params := crossBuildParams{Platforms: Platforms, Target: defaultCrossBuildTarget, ImageSelector: crossBuildImage}
132+
params := crossBuildParams{Platforms: Platforms, Target: defaultCrossBuildTarget, ImageSelector: CrossBuildImage}
133133
for _, opt := range options {
134134
opt(&params)
135135
}
@@ -193,7 +193,7 @@ func buildMage() error {
193193
"-compile", CreateDir(filepath.Join("build", "mage-linux-"+arch)))
194194
}
195195

196-
func crossBuildImage(platform string) (string, error) {
196+
func CrossBuildImage(platform string) (string, error) {
197197
tagSuffix := "main"
198198

199199
switch {

‎dev-tools/mage/pkg.go‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121
"fmt"
2222
"log"
2323
"os"
24+
"path/filepath"
2425
"runtime"
2526
"strconv"
2627

@@ -242,3 +243,38 @@ func TestPackages(options ...TestPackagesOption) error {
242243

243244
return nil
244245
}
246+
247+
// TestLinuxForCentosGLIBC checks the GLIBC requirements of linux/amd64 and
248+
// linux/386 binaries to ensure they meet the requirements for RHEL 6 which has
249+
// glibc 2.12.
250+
func TestLinuxForCentosGLIBC() error {
251+
switch Platform.Name {
252+
case "linux/amd64", "linux/386":
253+
return TestBinaryGLIBCVersion(filepath.Join("build/golang-crossbuild", BeatName+"-linux-"+Platform.GOARCH), "2.12")
254+
default:
255+
return nil
256+
}
257+
}
258+
259+
func TestBinaryGLIBCVersion(elfPath, maxGlibcVersion string) error {
260+
requiredGlibc, err := ReadGLIBCRequirement(elfPath)
261+
if err != nil {
262+
if errors.Is(err, os.ErrNotExist) {
263+
return nil
264+
}
265+
return err
266+
}
267+
268+
upperBound, err := NewSemanticVersion(maxGlibcVersion)
269+
if err != nil {
270+
return err
271+
}
272+
273+
if !requiredGlibc.LessThanOrEqual(upperBound) {
274+
return fmt.Errorf("dynamically linked binary %q requires glibc "+
275+
"%v, but maximum allowed glibc is %v",
276+
elfPath, requiredGlibc, upperBound)
277+
}
278+
fmt.Printf(">> testBinaryGLIBCVersion: %q requires glibc %v or greater\n", elfPath, requiredGlibc)
279+
return nil
280+
}

‎dev-tools/mage/semver.go‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
// Licensed to Elasticsearch B.V. under one or more contributor
2+
// license agreements. See the NOTICE file distributed with
3+
// this work for additional information regarding copyright
4+
// ownership. Elasticsearch B.V. licenses this file to you under
5+
// the Apache License, Version 2.0 (the "License"); you may
6+
// not use this file except in compliance with the License.
7+
// You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package mage
19+
20+
import (
21+
"fmt"
22+
"regexp"
23+
"strconv"
24+
)
25+
26+
var SemanticVersionRegex = regexp.MustCompile(`(?m)^(\d+)\.(\d+)(?:\.(\d+))?`)
27+
28+
type SemanticVersion struct {
29+
Major, Minor, Patch int
30+
}
31+
32+
// NewSemanticVersion return a new SemanticVersion parsed from string in the
33+
// format of 'x.y' or 'x.y.z'.
34+
func NewSemanticVersion(s string) (*SemanticVersion, error) {
35+
matches := SemanticVersionRegex.FindStringSubmatch(s)
36+
if len(matches) < 4 {
37+
return nil, fmt.Errorf("invalid version format %q", s)
38+
}
39+
40+
major, _ := strconv.Atoi(matches[1])
41+
Minor, _ := strconv.Atoi(matches[2])
42+
Patch, _ := strconv.Atoi(matches[3])
43+
return &SemanticVersion{major, Minor, Patch}, nil
44+
}
45+
46+
// LessThan return true iff s is less than x.
47+
func (s *SemanticVersion) LessThan(x *SemanticVersion) bool {
48+
if s.Major != x.Major {
49+
return s.Major < x.Major
50+
}
51+
if s.Minor != x.Minor {
52+
return s.Minor < x.Minor
53+
}
54+
return s.Patch < x.Patch
55+
}
56+
57+
// LessThanOrEqual return true iff s is less than or equal to x.
58+
func (s *SemanticVersion) LessThanOrEqual(x *SemanticVersion) bool {
59+
if s.LessThan(x) {
60+
return true
61+
}
62+
return !x.LessThan(s)
63+
}
64+
65+
func (s SemanticVersion) String() string {
66+
return fmt.Sprintf("%d.%d.%d", s.Major, s.Minor, s.Patch)
67+
}

‎filebeat/docs/filebeat-options.asciidoc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ You can configure {beatname_uc} to use the following inputs:
7272
* <<{beatname_lc}-input-gcp-pubsub>>
7373
* <<{beatname_lc}-input-http_endpoint>>
7474
* <<{beatname_lc}-input-httpjson>>
75+
* <<{beatname_lc}-input-journald>>
7576
* <<{beatname_lc}-input-kafka>>
7677
* <<{beatname_lc}-input-log>>
7778
* <<{beatname_lc}-input-mqtt>>
@@ -106,6 +107,8 @@ include::../../x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc[]
106107

107108
include::../../x-pack/filebeat/docs/inputs/input-httpjson.asciidoc[]
108109

110+
include::inputs/input-journald.asciidoc[]
111+
109112
include::inputs/input-kafka.asciidoc[]
110113

111114
include::inputs/input-log.asciidoc[]

0 commit comments

Comments
 (0)