Update zoom module to ECS 1.8 by adriansr · Pull Request #23904 · elastic/beats

adriansr · 2021-02-08T11:11:03Z

What does this PR do?

Updates zoom to support ECS 1.8.

Why is it important?

Maps more user information to better understand the events.
Fixes a couple of issues with the existing module:
- user events: missing mapping for event.category (wrongly mapped to event.type).
- chat_channel events: fixed an error in the pipeline that caused some events to be dropped on ingestion.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-02-08T11:11:06Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-02-08T19:32:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23904 updated
Start Time: 2021-02-11T18:11:42.846+0000
Duration: 48 min 48 sec
Commit: 5ccd483

Test stats 🧪

Test	Results
Failed	0
Passed	6823
Skipped	1065
Total	7888

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6823
Skipped	1065
Total	7888

Most events that represent actions include operator_id and operator (email) fields to identify the user performing the action.

There's no zoom.user in zoomroom events

elasticmachine · 2021-02-12T17:40:13Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23904 updated
Start Time: 2021-02-12T17:37:41.564+0000
Duration: 2 min 20 sec
Commit: 62759e4

Trends 🧪

Steps errors

Expand to view the steps failures

`Git fetch`

Took 0 min 1 sec . View more details on here
Description: git fetch https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/* > fetch.log 2>&1

`Archive the artifacts`

Took 0 min 0 sec . View more details on here
Description: fetch.log

Log output

Expand to view the last 100 lines of log output

[2021-02-12T17:38:10.759Z] Fetching upstream changes from origin
[2021-02-12T17:38:10.759Z]  > git --version # timeout=10
[2021-02-12T17:38:10.764Z]  > git --version # 'git version 2.17.1'
[2021-02-12T17:38:10.764Z]  > git config --get remote.origin.url # timeout=10
[2021-02-12T17:38:10.770Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-12T17:38:10.775Z]  > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-12T17:38:11.330Z]  > git rev-parse current^{commit} # timeout=10
[2021-02-12T17:38:11.336Z]  > git branch -a -v --no-abbrev --contains dcbd74e89167d9e860b4232a3a4a39f943cf659e # timeout=10
[2021-02-12T17:38:11.358Z] Selected match: master revision dcbd74e89167d9e860b4232a3a4a39f943cf659e
[2021-02-12T17:38:11.358Z] The recommended git tool is: git
[2021-02-12T17:38:11.359Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-02-12T17:38:11.416Z]  > git rev-parse --is-inside-work-tree # timeout=10
[2021-02-12T17:38:11.421Z] Fetching changes from the remote Git repository
[2021-02-12T17:38:11.421Z]  > git config remote.origin.url git@github.com:elastic/apm-pipeline-library.git # timeout=10
[2021-02-12T17:38:11.427Z] Fetching without tags
[2021-02-12T17:38:11.427Z] Fetching upstream changes from git@github.com:elastic/apm-pipeline-library.git
[2021-02-12T17:38:11.427Z]  > git --version # timeout=10
[2021-02-12T17:38:11.432Z]  > git --version # 'git version 2.17.1'
[2021-02-12T17:38:11.432Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-12T17:38:11.437Z]  > git fetch --no-tags --progress -- git@github.com:elastic/apm-pipeline-library.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-12T17:38:12.001Z] Checking out Revision dcbd74e89167d9e860b4232a3a4a39f943cf659e (master)
[2021-02-12T17:38:12.001Z]  > git config core.sparsecheckout # timeout=10
[2021-02-12T17:38:12.006Z]  > git checkout -f dcbd74e89167d9e860b4232a3a4a39f943cf659e # timeout=10
[2021-02-12T17:38:12.025Z] Commit message: "docs: update CHANGELOG.md"
[2021-02-12T17:38:12.759Z] Excluding src/test/ from checkout of git git@github.com:elastic/apm-pipeline-library.git so that shared library test code cannot be accessed by Pipelines.
[2021-02-12T17:38:12.759Z] To remove this log message, move the test code outside of src/. To restore the previous behavior that allowed access to files in src/test/, pass -Dorg.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.INCLUDE_SRC_TEST_IN_LIBRARIES=true to the java command used to start Jenkins.
[2021-02-12T17:38:38.943Z] Still waiting to schedule task
[2021-02-12T17:38:38.943Z] ‘beats-ci-immutable-ubuntu-1804-1613060083115589585’ is offline
[2021-02-12T17:39:08.360Z] Running on beats-ci-immutable-ubuntu-1804-1613151504856377577 in /var/lib/jenkins/workspace/Beats_beats_PR-23904
[2021-02-12T17:39:08.464Z] �[39;49m[INFO] Override default checkout�[0m
[2021-02-12T17:39:08.508Z] Sleeping for 10 sec
[2021-02-12T17:39:18.590Z] The recommended git tool is: git
[2021-02-12T17:39:20.347Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-02-12T17:39:20.399Z] Wiping out workspace first.
[2021-02-12T17:39:20.419Z] Cloning the remote Git repository
[2021-02-12T17:39:20.419Z] Using shallow clone with depth 10
[2021-02-12T17:39:20.419Z] Avoid fetching tags
[2021-02-12T17:39:20.439Z] Cloning repository git@github.com:elastic/beats.git
[2021-02-12T17:39:20.482Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-23904 # timeout=10
[2021-02-12T17:39:20.527Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-02-12T17:39:20.527Z]  > git --version # timeout=10
[2021-02-12T17:39:20.531Z]  > git --version # 'git version 2.17.1'
[2021-02-12T17:39:20.532Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-12T17:39:20.551Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-02-12T17:39:39.837Z] Cleaning workspace
[2021-02-12T17:39:39.850Z] Using shallow fetch with depth 10
[2021-02-12T17:39:39.850Z] Pruning obsolete local branches
[2021-02-12T17:39:39.810Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-02-12T17:39:39.819Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-12T17:39:39.830Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-02-12T17:39:39.838Z]  > git rev-parse --verify HEAD # timeout=10
[2021-02-12T17:39:39.842Z] No valid HEAD. Skipping the resetting
[2021-02-12T17:39:39.842Z]  > git clean -fdx # timeout=10
[2021-02-12T17:39:39.853Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-02-12T17:39:39.853Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-12T17:39:39.857Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/23904/head:refs/remotes/origin/PR-23904 +refs/heads/feature-ecs-1.8:refs/remotes/origin/feature-ecs-1.8 # timeout=15
[2021-02-12T17:39:40.905Z] Merging remotes/origin/feature-ecs-1.8 commit c4b6fd2caa984cb15935b6289a6789a4e58e5ea3 into PR head commit 62759e491175d61a1cec570927655ddb99212f3f
[2021-02-12T17:39:40.911Z]  > git config core.sparsecheckout # timeout=10
[2021-02-12T17:39:40.915Z]  > git checkout -f 62759e491175d61a1cec570927655ddb99212f3f # timeout=15
[2021-02-12T17:39:43.181Z] Merge succeeded, producing 11509c9f694d05fcc2b471b2192650fa6b09cd01
[2021-02-12T17:39:43.182Z] Checking out Revision 11509c9f694d05fcc2b471b2192650fa6b09cd01 (PR-23904)
[2021-02-12T17:39:42.431Z]  > git remote # timeout=10
[2021-02-12T17:39:42.435Z]  > git config --get remote.origin.url # timeout=10
[2021-02-12T17:39:42.438Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-12T17:39:42.446Z]  > git merge c4b6fd2caa984cb15935b6289a6789a4e58e5ea3 # timeout=10
[2021-02-12T17:39:43.176Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-02-12T17:39:43.183Z]  > git config core.sparsecheckout # timeout=10
[2021-02-12T17:39:43.186Z]  > git checkout -f 11509c9f694d05fcc2b471b2192650fa6b09cd01 # timeout=15
[2021-02-12T17:39:46.683Z] Commit message: "Merge commit 'c4b6fd2caa984cb15935b6289a6789a4e58e5ea3' into HEAD"
[2021-02-12T17:39:46.685Z]  > git rev-list --no-walk 5ccd483ee1869c4e162242bb183d396e4c726a03 # timeout=10
[2021-02-12T17:39:46.707Z] Cleaning workspace
[2021-02-12T17:39:46.708Z]  > git rev-parse --verify HEAD # timeout=10
[2021-02-12T17:39:46.716Z] Resetting working tree
[2021-02-12T17:39:46.717Z]  > git reset --hard # timeout=10
[2021-02-12T17:39:46.795Z]  > git clean -fdx # timeout=10
[2021-02-12T17:39:47.548Z] Timeout set to expire in 3 hr 0 min
[2021-02-12T17:39:47.557Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-02-12T17:39:47.749Z] [INFO] Number of builds to be searched 10
[2021-02-12T17:39:48.432Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-02-12T17:39:48.442Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23904/src/github.com/elastic/beats
[2021-02-12T17:39:48.454Z] [INFO] gitCheckout: Checkout SCM PR-23904 with default customisation from the Item.
[2021-02-12T17:39:48.473Z] [INFO] Override default checkout
[2021-02-12T17:39:48.503Z] Sleeping for 10 sec
[2021-02-12T17:39:58.921Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-02-12T17:39:59.542Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-02-12T17:39:59.586Z] [WARN] gitCmd failed, further details in the archived file 'fetch.log'
[2021-02-12T17:39:59.644Z] Archiving artifacts
[2021-02-12T17:39:59.895Z] Stage "Lint" skipped due to earlier failure(s)
[2021-02-12T17:39:59.920Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-02-12T17:39:59.947Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-02-12T17:39:59.990Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23904/src/github.com/elastic/beats
[2021-02-12T17:40:00.553Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-23904
[2021-02-12T17:40:00.735Z] [INFO] getVaultSecret: Getting secrets
[2021-02-12T17:40:00.819Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-12T17:40:01.632Z] + chmod 755 generate-build-data.sh
[2021-02-12T17:40:01.632Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23904/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23904/runs/5 FAILURE 139807
[2021-02-12T17:40:01.882Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23904/runs/5/steps/?limit=10000 -o steps-info.json
[2021-02-12T17:40:02.433Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23904/runs/5/tests/?status=FAILED -o tests-errors.json
[2021-02-12T17:40:02.433Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-02-12T17:40:03.344Z] Retry 2/3 exited 22, retrying in 2 seconds...

❕ Flaky test report

No test was executed to be analysed.

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host #23513 Winlogbeat #23563 Auditbeat auditd #23594 Journalbeat #23737 Packetbeat #23783 Filebeat: auditd #23723 cisco #23819 cef #23832 crowdstrike falcon #23875 fortinet firewall #23902 microsoft #23897 elasticsearch/audit #24000 Gsuite/Workspace #23709 o365 #23896 zoom #23904 okta #23929 aws/cloudtrail #23911 aws/s3access #23920 azure #23927 juniper/srx #23936 panw #23931 sophos/xg #23967 system/auth #23961 mysqlenterprise #23978 zeek #23847 Make all Beats and modules report ECS 1.8.0 #23992 Closes #23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host elastic#23513 Winlogbeat elastic#23563 Auditbeat auditd elastic#23594 Journalbeat elastic#23737 Packetbeat elastic#23783 Filebeat: auditd elastic#23723 cisco elastic#23819 cef elastic#23832 crowdstrike falcon elastic#23875 fortinet firewall elastic#23902 microsoft elastic#23897 elasticsearch/audit elastic#24000 Gsuite/Workspace elastic#23709 o365 elastic#23896 zoom elastic#23904 okta elastic#23929 aws/cloudtrail elastic#23911 aws/s3access elastic#23920 azure elastic#23927 juniper/srx elastic#23936 panw elastic#23931 sophos/xg elastic#23967 system/auth elastic#23961 mysqlenterprise elastic#23978 zeek elastic#23847 Make all Beats and modules report ECS 1.8.0 elastic#23992 Closes elastic#23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> (cherry picked from commit 048c3cc)

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host #23513 Winlogbeat #23563 Auditbeat auditd #23594 Journalbeat #23737 Packetbeat #23783 Filebeat: auditd #23723 cisco #23819 cef #23832 crowdstrike falcon #23875 fortinet firewall #23902 microsoft #23897 elasticsearch/audit #24000 Gsuite/Workspace #23709 o365 #23896 zoom #23904 okta #23929 aws/cloudtrail #23911 aws/s3access #23920 azure #23927 juniper/srx #23936 panw #23931 sophos/xg #23967 system/auth #23961 mysqlenterprise #23978 zeek #23847 Make all Beats and modules report ECS 1.8.0 #23992 Closes #23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> (cherry picked from commit 048c3cc)

Updates zoom pipeline with new ECS 1.8 mappings (multiuser). Fixes a couple of issues with the existing module: - user events: missing mapping for event.category (wrongly mapped to event.type). - chat_channel events: fixed an error in the pipeline that caused some events to be dropped on ingestion.

adriansr added enhancement review ecs Team:Security-External Integrations labels Feb 8, 2021

adriansr requested review from P1llus and marc-gr February 8, 2021 11:11

adriansr requested review from a team as code owners February 8, 2021 11:11

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 8, 2021

adriansr changed the base branch from master to feature-ecs-1.8 February 8, 2021 11:13

adriansr mentioned this pull request Feb 8, 2021

[ECS] Upgrade modules to 1.8 #23118

Closed

89 tasks

adriansr added 13 commits February 10, 2021 16:19

Set user.* fields from operator

5d16d9a

Most events that represent actions include operator_id and operator (email) fields to identify the user performing the action.

Set multiuser fields for account events

9b97bec

Fix chat_channel pipeline not ingesting some events

3b53b6a

Update meeting pipeline to ECS 1.8

965586b

phone pipeline: Enrich source and destination users

a4d9754

Update recording to ECS 1.8

2f66a22

user pipeline: fix event.category

75a5c47

user: populate multiuser fields

d8439c2

Webinar: support multiuser

1f55349

Zoomroom: remove unused processor

65ce254

There's no zoom.user in zoomroom events

Update ECS version to 1.8.0

d5932bf

Update changelog

0e79a7c

Fix ingestion of recording events with empty dates

10e8273

adriansr force-pushed the update-zoom-ecs-1.8 branch from c4455d8 to 10e8273 Compare February 10, 2021 15:58

marc-gr approved these changes Feb 11, 2021

View reviewed changes

Merge branch 'feature-ecs-1.8' into update-zoom-ecs-1.8

5ccd483

Merge branch 'feature-ecs-1.8' into update-zoom-ecs-1.8

62759e4

adriansr merged commit c4b6fd2 into elastic:feature-ecs-1.8 Feb 12, 2021

adriansr mentioned this pull request Feb 12, 2021

Update Beats to ECS 1.8.0 #23465

Merged

28 tasks

adriansr mentioned this pull request Feb 16, 2021

Cherry-pick #23465 to 7.x: Update Beats to ECS 1.8.0 #24072

Merged

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update zoom module to ECS 1.8#23904

Update zoom module to ECS 1.8#23904
adriansr merged 15 commits intoelastic:feature-ecs-1.8from
adriansr:update-zoom-ecs-1.8

adriansr commented Feb 8, 2021

Uh oh!

elasticmachine commented Feb 8, 2021

Uh oh!

elasticmachine commented Feb 8, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

elasticmachine commented Feb 12, 2021

Build stats

Trends 🧪

`Git fetch`

`Archive the artifacts`

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adriansr commented Feb 8, 2021

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Feb 8, 2021

Uh oh!

elasticmachine commented Feb 8, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Feb 12, 2021

💔 Build Failed

Build stats

Trends 🧪

Steps errors

Git fetch

Archive the artifacts

Log output

❕ Flaky test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elasticmachine commented Feb 8, 2021 •

edited by jenkins-beats-ci bot

Loading

`Git fetch`

`Archive the artifacts`