Skip to content

Cherry-pick #21086 to 7.x: [Filebeat] Add support for Cloudtrail digest files#21088

Merged
leehinman merged 1 commit intoelastic:7.xfrom
leehinman:backport_21086_7.x
Sep 15, 2020
Merged

Cherry-pick #21086 to 7.x: [Filebeat] Add support for Cloudtrail digest files#21088
leehinman merged 1 commit intoelastic:7.xfrom
leehinman:backport_21086_7.x

Conversation

@leehinman
Copy link
Copy Markdown
Contributor

@leehinman leehinman commented Sep 15, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #21086 to 7.x branch. Original message:

What does this PR do?

Adds support for Cloudtrail Digest Logs and Cloudtrail Insight logs.
Also adds options to ignore cloudtrail, cloudtrail-digest or
cloudtrail-insight logs if necessary.

Why is it important?

  • Cloudtrail digest files are used to determine integrity of Cloudtrail
    logs.
  • Users need the ability to ignore the digest or insight logs if necessary.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=aws TESTING_FILEBEAT_FILESETS=cloudtrail mage -v pythonIntegTest

Related issues

@leehinman
[Filebeat] Add support for Cloudtrail digest files (elastic#21086)
ad82d30 
* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes elastic#20943

(cherry picked from commit c9f7a99)
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 15, 2020

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 15, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 15, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21088 opened]

  • Start Time: 2020-09-15T01:44:17.543+0000

  • Duration: 56 min 20 sec

Test stats 🧪

Test Results
Failed 1
Passed 5568
Skipped 833
Total 6402

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat oss / test_default_settings – filebeat.tests.system.test_autodiscover.TestAutodiscover

    • Age: 1
    • Duration: 90.003
    • Error Details: Failed: Timeout >90.0s

Steps errors

Expand to view the steps failures

  • Name: Mage build test

    • Description: mage build test

    • Duration: 29 min 21 sec

    • Start Time: 2020-09-15T02:09:33.165+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-09-15T02:39:09.186Z]  Built:             Mon Jun 22 15:45:36 2020
[2020-09-15T02:39:09.186Z]  OS/Arch:           linux/amd64
[2020-09-15T02:39:09.186Z]  Experimental:      false
[2020-09-15T02:39:09.186Z] 
[2020-09-15T02:39:09.186Z] Server: Docker Engine - Community
[2020-09-15T02:39:09.186Z]  Engine:
[2020-09-15T02:39:09.186Z]   Version:          19.03.12
[2020-09-15T02:39:09.186Z]   API version:      1.40 (minimum version 1.12)
[2020-09-15T02:39:09.186Z]   Go version:       go1.13.10
[2020-09-15T02:39:09.186Z]   Git commit:       48a66213fe
[2020-09-15T02:39:09.186Z]   Built:            Mon Jun 22 15:44:07 2020
[2020-09-15T02:39:09.186Z]   OS/Arch:          linux/amd64
[2020-09-15T02:39:09.186Z]   Experimental:     false
[2020-09-15T02:39:09.186Z]  containerd:
[2020-09-15T02:39:09.186Z]   Version:          1.2.13
[2020-09-15T02:39:09.186Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-09-15T02:39:09.186Z]  runc:
[2020-09-15T02:39:09.186Z]   Version:          1.0.0-rc10
[2020-09-15T02:39:09.187Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-09-15T02:39:09.187Z]  docker-init:
[2020-09-15T02:39:09.187Z]   Version:          0.18.0
[2020-09-15T02:39:09.187Z]   GitCommit:        fec3683
[2020-09-15T02:39:09.187Z] Unable to find image 'alpine:3.4' locally
[2020-09-15T02:39:09.754Z] 3.4: Pulling from library/alpine
[2020-09-15T02:39:09.755Z] c1e54eec4b57: Pulling fs layer
[2020-09-15T02:39:10.014Z] c1e54eec4b57: Download complete
[2020-09-15T02:39:10.274Z] c1e54eec4b57: Pull complete
[2020-09-15T02:39:10.274Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-09-15T02:39:10.274Z] Status: Downloaded newer image for alpine:3.4
[2020-09-15T02:39:10.581Z] Failed in branch Filebeat oss
[2020-09-15T02:39:12.490Z] + python .ci/scripts/pre_archive_test.py
[2020-09-15T02:39:13.870Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2020-09-15T02:39:13.897Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/build
[2020-09-15T02:39:13.953Z] Recording test results
[2020-09-15T02:39:15.596Z] Stashed 4 file(s)
[2020-09-15T02:39:15.619Z] Archiving artifacts
[2020-09-15T02:39:16.423Z] + python .ci/scripts/search_system_tests.py
[2020-09-15T02:39:16.457Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2020-09-15T02:39:16.858Z] + tar --version
[2020-09-15T02:39:17.237Z] + tar --exclude=x-pack-filebeat--system-tests-linux.tgz -czf x-pack-filebeat--system-tests-linux.tgz build/x-pack/filebeat/build/system-tests
[2020-09-15T02:39:32.276Z] Archiving artifacts
[2020-09-15T02:39:52.350Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-09-15T02:39:52.394Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-09-15T02:39:52.467Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21088-1/source/source.tgz
[2020-09-15T02:39:52.487Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21088-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21088/source.tgz
[2020-09-15T02:40:03.199Z] + tar --version
[2020-09-15T02:40:03.525Z] + tar -xpf source.tgz
[2020-09-15T02:40:13.900Z] + rm source.tgz
[2020-09-15T02:40:13.925Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats
[2020-09-15T02:40:13.973Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Lint
[2020-09-15T02:40:14.139Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-15T02:40:14.301Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-15T02:40:14.460Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-15T02:40:14.621Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-15T02:40:14.780Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-oss
[2020-09-15T02:40:14.942Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-15T02:40:15.462Z] + cat
[2020-09-15T02:40:15.462Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-15T02:40:15.462Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-15T02:40:22.045Z] runbld>>> runbld started
[2020-09-15T02:40:22.045Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-15T02:40:23.430Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-15T02:40:23.430Z] runbld>>> Matches in the system config:
[2020-09-15T02:40:23.430Z] runbld>>> - Matched ^elastic\+beats
[2020-09-15T02:40:23.430Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-15T02:40:24.373Z] runbld>>> Debug logging enabled.
[2020-09-15T02:40:24.373Z] runbld>>> Storing result
[2020-09-15T02:40:24.634Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-15T02:40:24.634Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200915024024-3E31AED6
[2020-09-15T02:40:24.634Z] runbld>>> Adding system facts.
[2020-09-15T02:40:25.579Z] runbld>>> Adding vcs info for the latest commit:  ad82d3041340fc4e6b6950897c30653a428df17e
[2020-09-15T02:40:25.579Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-15T02:40:25.579Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-15T02:40:25.840Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-15T02:40:25.840Z] Processing JUnit reports with runbld...
[2020-09-15T02:40:26.100Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-15T02:40:26.101Z] runbld>>> DURATION: 36ms
[2020-09-15T02:40:26.101Z] runbld>>> STDOUT: 40 bytes
[2020-09-15T02:40:26.101Z] runbld>>> STDERR: 49 bytes
[2020-09-15T02:40:26.101Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-15T02:40:26.101Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21088
[2020-09-15T02:40:27.043Z] runbld>>> Storing build metadata: 
[2020-09-15T02:40:27.043Z] runbld>>> Adding test report.
[2020-09-15T02:40:27.043Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21088/src/github.com/elastic/beats
[2020-09-15T02:40:27.985Z] runbld>>> Found 16 test output files
[2020-09-15T02:40:29.374Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 6402 Skipped: 807
[2020-09-15T02:40:29.374Z] runbld>>> Storing result
[2020-09-15T02:40:29.374Z] runbld>>> FAILURES: 1
[2020-09-15T02:40:29.946Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-15T02:40:29.946Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200915024024-3E31AED6
[2020-09-15T02:40:29.946Z] runbld>>> Email notification disabled by environment variable.
[2020-09-15T02:40:29.946Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-15T02:40:35.930Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21088
[2020-09-15T02:40:36.235Z] [INFO] getVaultSecret: Getting secrets
[2020-09-15T02:40:36.326Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-15T02:40:37.407Z] + chmod 755 generate-build-data.sh
[2020-09-15T02:40:37.407Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21088/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21088/runs/1 FAILURE 3379591
[2020-09-15T02:40:37.407Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21088/runs/1/steps/?limit=10000 -o steps-info.json
[2020-09-15T02:40:38.751Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21088/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-09-15T02:40:39.302Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21088/runs/1/log/ -o pipeline-log.txt

@leehinman leehinman merged commit 7407ad1 into elastic:7.x Sep 15, 2020
@leehinman leehinman deleted the backport_21086_7.x branch October 5, 2020 19:06
@zube zube bot removed the [zube]: Done label Dec 14, 2020
@borrell borrell mentioned this pull request Dec 14, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@leehinman @elasticmachine @kaiyan-sheng