Skip to content

[CI] Enables APM collection#81731

Merged
tylersmalley merged 16 commits intoelastic:masterfrom
tylersmalley:enable-apm-ci
Dec 3, 2020
Merged

[CI] Enables APM collection#81731
tylersmalley merged 16 commits intoelastic:masterfrom
tylersmalley:enable-apm-ci

Conversation

@tylersmalley
Copy link
Copy Markdown
Member

@tylersmalley tylersmalley commented Oct 27, 2020 •
edited
Loading

Enables APM collection for functional tests on tracked branches, with a 1% sample rate on pull requests.

Stats are accessible at: https://ela.st/kibana-ops-ci-stats

@tylersmalley
[CI] Enables APM collection
5c3601e 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Copy Markdown
Member Author

tylersmalley commented Oct 27, 2020

Early checks appear to increase CI functional test times by 10% This doesn't seem enough to justify preventing us from proceeding, but it's worth investigating to understand if there is anything causing large increases in some tests that could be improved.

job master 81731 diff
kibana-ciGroup1 0:05:44 0:06:11 7.28%
kibana-ciGroup2 0:17:00 0:18:00 5.56%
kibana-ciGroup3 0:24:00 0:24:00 0.00%
kibana-ciGroup4 0:22:00 0:22:00 0.00%
kibana-ciGroup5 0:13:00 0:14:00 7.14%
kibana-ciGroup6
kibana-ciGroup7
kibana-ciGroup8 0:15:00 0:15:00 0.00%
kibana-ciGroup9
kibana-ciGroup10 0:15:00 0:16:00 6.25%
kibana-ciGroup11 0:24:00 0:25:00 4.00%
kibana-ciGroup12 0:29:00 0:30:00 3.33%
xpack-kibana-ciGroup1 0:53:00 1:02:00 14.52%
xpack-kibana-ciGroup2 0:40:00 0:44:00 9.09%
xpack-kibana-ciGroup3 0:51:00 0:59:00 13.56%
xpack-kibana-ciGroup4 0:51:00 0:55:00 7.27%
xpack-kibana-ciGroup5 0:48:00 0:50:00 4.00%
xpack-kibana-ciGroup6
xpack-kibana-ciGroup7
xpack-kibana-ciGroup8 1:00:00 1:25:00 29.41%
xpack-kibana-ciGroup9
xpack-kibana-ciGroup10 0:41:00 0:43:00 4.65%
8:28:44 9:28:11 10.46%

@tylersmalley
Copy link
Copy Markdown
Member Author

tylersmalley commented Nov 4, 2020

@IgorGuz2000 any luck with these failures?

@tylersmalley tylersmalley mentioned this pull request Nov 16, 2020
Closed
@tylersmalley
Skip Security Endpoint test
3288738 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@elastic elastic deleted a comment Nov 19, 2020
tylersmalley
tylersmalley commented Nov 19, 2020
View reviewed changes
@tylersmalley
Updates token
c7ee220 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Enable APM for FTR only
d93b0a7 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Merge remote-tracking branch 'upstream/master' into enable-apm-ci
8bc8914
@tylersmalley
Set sample rate
f380562 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Decrease sample rate to 1%
963addf 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@mshustov
Copy link
Copy Markdown
Contributor

mshustov commented Dec 1, 2020 •
edited
Loading

@tylersmalley you can adjust API config to reduce performance overhead. see #78792 (comment)

# optional metrics to adjust performance 
# see https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuration.html
elastic.apm.centralConfig: false
elastic.apm.breakdownMetrics: false
elastic.apm.transactionSampleRate: 0.1
elastic.apm.metricsInterval: '120s'

@tylersmalley
Increase metricsInterval
3ac2e10 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Increase metricsInterval for distributable
1aafb07 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Copy Markdown
Member Author

tylersmalley commented Dec 1, 2020

@restrry, thanks. Since the functional tests run on the distribution, we were just missing the metricsInterval here: https://github.com/elastic/kibana/blob/master/packages/kbn-apm-config-loader/src/config.ts#L30-L49

For the sampleRate - I was taking 100% for the tracked branches (master/7.x/7.10), and 1% on PR's. Do you think we should reduce that rate even on the tracked branches?

@tylersmalley tylersmalley marked this pull request as ready for review December 1, 2020 23:13
@tylersmalley tylersmalley requested review from a team and vigneshshanmugam as code owners December 1, 2020 23:13
@tylersmalley tylersmalley added release_note:skip Skip the PR/issue when compiling release notes Team:Operations Kibana-Operations Team v7.11.0 v8.0.0 labels Dec 1, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 1, 2020

Pinging @elastic/kibana-operations (Team:Operations)

@tylersmalley
Copy link
Copy Markdown
Member Author

tylersmalley commented Dec 1, 2020

@elasticmachine merge upstream

vigneshshanmugam
vigneshshanmugam approved these changes Dec 2, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@mshustov
Copy link
Copy Markdown
Contributor

mshustov commented Dec 2, 2020

For the sampleRate - I was taking 100% for the tracked branches (master/7.x/7.10), and 1% on PR's. Do you think we should reduce that rate even on the tracked branches?

I'd expect it's all or nothing for PRs.
With master/7.x/7.10: it depends on how often we are going to gather those. If we collect metrics on every build (>20 per day), maybe 100% is overhead?

spalger
spalger approved these changes Dec 2, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tylersmalley
Disable for PRs and set to 10% sample rate
9331fcf 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Copy Markdown
Member Author

tylersmalley commented Dec 2, 2020

@restrry, I have disabled entirely for PR's and set the sample rate to 10% for when it does run on the tracked branches per your suggestion. If we find this is insufficient we can increase it.

@tylersmalley
Allow disabling by env and pass sample rate to frontend
dc4bb17 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Adds prId label and removes jobName
626fdd5 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley tylersmalley merged commit 74ef540 into elastic:master Dec 3, 2020
@tylersmalley tylersmalley mentioned this pull request Dec 3, 2020
Merged
tylersmalley added a commit to tylersmalley/kibana that referenced this pull request Dec 3, 2020
@tylersmalley
[CI] Enables APM collection (elastic#81731)
f9b6492 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit that referenced this pull request Dec 3, 2020
@tylersmalley
[CI] Enables APM collection (#81731) (#84929)
9a8351c 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Dec 4, 2020
@gmmorris
Merge branch 'master' into alerts/custom-recovery-action-group
8940592 
* master: (28 commits)
  [Actions] fixes bug where severity is auto selected but not applied to the action in PagerDuty (elastic#84891)
  Only attempt to rollover signals index if version < builtin version (elastic#84982)
  skip flaky suite (elastic#84978)
  skip lens rollup tests
  Add geo containment tracking alert type (elastic#84151)
  Changed rollup tests to use test user rather than elastic super user. (elastic#79567)
  skip 'should allow creation of lens xy chart' elastic#84957
  [APM] Add log_level/sanitize_field_names config options to Python Agent (elastic#84810)
  [Maps] geo line source (elastic#76572)
  [data.search] Move search method inside session service and add tests (elastic#84715)
  skip lens drag and drop test.  elastic#84941
  [Ingest Node Pipelines] Integrate painless autocomplete (elastic#84554)
  [Lens] allow drag and drop reorder on xyChart for y dimension (elastic#84640)
  [Lens] Fix error when selecting the current field again (elastic#84817)
  [Metrics UI] Add metadata tab to node details flyout (elastic#84454)
  [CI] Enables APM collection (elastic#81731)
  [Workplace Search] Migrate Sources Schema tree (elastic#84847)
  Disable checking for conflicts when copying saved objects (elastic#83575)
  [SECURITY_SOLUTION] delete advanced Policy fields when they are empty (elastic#84368)
  y18n 4.0.0 -> 4.0.1 (elastic#84905)
  ...
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Dec 7, 2020

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@peterschretlen peterschretlen mentioned this pull request Dec 22, 2020
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jbudz jbudz jbudz approved these changes

@vigneshshanmugam vigneshshanmugam vigneshshanmugam approved these changes

+1 more reviewer

@spalger spalger spalger approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release_note:skip Skip the PR/issue when compiling release notes Team:Operations Kibana-Operations Team v7.11.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@tylersmalley @mshustov @elasticmachine @kibanamachine @spalger @jbudz @vigneshshanmugam