[Reporting] APM integration for baseline performance measurements by tsullivan · Pull Request #59967 · elastic/kibana

tsullivan · 2020-03-11T23:33:09Z

Summary

Refer to #43548 for information about Kibana's APM integrations.

Adds APM transactions and spans around the slow parts of Reporting code.

This lets us see a trace of where the flow is taking the most time when screenshots are captured. Here is an example of capturing a Dashboard with "print_layout" mode. You can see the multiple calls to get_screenshots:

Checklist

Delete any items that are not applicable to this PR.

Need some findings on this issue: Kibana crash: TypeError: request.abort is not a function #59712
~~[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~[ ] Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~[ ] This was checked for keyboard-only and screenreader accessibility~~
~~[ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser~~
~~[ ] This was checked for cross-browser compatibility, including a check against IE11~~

For maintainers

This was checked for breaking API changes and was labeled appropriately

src/core/server/elasticsearch/cluster_client.ts

tsullivan · 2020-03-12T01:02:01Z

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/get_screenshots.ts

asyncDurationLogger was hiding a type error, because we're resolving a string, not a Buffer

More context about this: since we'll have APM to measure the duration of all these modules, we no longer need this helper function. After having removed it, I noticed the surrounding code treated the helper function return value as a Buffer but it is actually a string of base64 content.

elasticmachine · 2020-03-12T21:47:00Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts

vigneshshanmugam

Amazing job on doing the full custom instrumentation using the APM Node.js agent. Love to see we are dog fooding more.

I just have one small suggestion, It just personal opinion as Span can be null | Span at any point in time. You could have a generic tracer utility

// utility
function startTrace() {
   const span = apm.startSpan(...);
   return () {
        span && span.end()
   }
}


// in actual code
const endTrace = startTrace(name, type)

endTrace()

Helps to avoid the if check everywhere.

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/open_url.ts

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts

...ck/legacy/plugins/reporting/export_types/common/lib/screenshots/get_element_position_data.ts

src/core/server/elasticsearch/cluster_client.ts

…nce-measurement

tsullivan · 2020-04-29T19:21:38Z

x-pack/legacy/plugins/reporting/export_types/png/server/execute_job/index.ts

        );
      }),
-      map(({ buffer, warnings }) => {
+      map(({ base64, warnings }) => {


I had to rename this - it was very unclear that the return payload actually is a string

tsullivan · 2020-04-30T23:58:23Z

@elasticmachine merge upstream

tsullivan · 2020-05-05T23:44:12Z

Will merge this after the 7.8 branch is cut

kibanamachine · 2020-05-07T21:09:05Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e4495c4

History

💚 Build #45714 succeeded dd9c55b
💔 Build #45678 failed 6b5c392
💔 Build #44683 failed 7618e42
💔 Build #44205 failed e79e875
💔 Build #43854 failed 3db397f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…astic#59967) * apm stuff * fix cluster_client * fix snapshot * tracker utility for generate_pdf * call apm.startSpan instead of txn.startSpan * Fix async call to end transaction * fix typescript * remove captuureErrors * restore accidental removal * add startTrace lib * fix import * fix imports * ts fix * fix generate_png to not format base64 to buffer and back to base64 * 💅 * revert change to cluster client * fix unused translation Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…or-part-mvp-2 * 'master' of github.com:elastic/kibana: (259 commits) SavedObjects bulkCreate API should return migrationVersion and strip the type & namespace from the id (elastic#65150) Drilldown count tooltip (elastic#65105) plugins logs start with "plugins." prefix (elastic#65710) [ML] Fix pagination reset on search query update. (elastic#65668) [SIEM] Add types to the mappings objects so extra keys cannot be introduced [apm] Update machine learning flyout and service maps docs (elastic#65517) change api endpoint and throw error (elastic#65790) [Maps] remove SLA percentage metric (elastic#65718) [Reporting] APM integration for baseline performance measurements (elastic#59967) fix(NA): noParse regex for windows on kbn optimizer (elastic#65755) [ML] DFA: ensure at least one field is included in analysis before job can be created (elastic#65320) [Data plugin] cleanup - remove unused getRoutes / routes from indexPattern object (elastic#65683) Removed skip to enable test. (elastic#65575) [Lens] Type safe migrations (elastic#65576) [Canvas] Fix nav link behavior in Canvas (elastic#65590) [Event log] Fix flaky test (elastic#65658) [Alerting] changes preconfigured actions config from array to object (elastic#65397) remove immediate functions from esqueue worker cycles (elastic#65375) [Metrics UI] Fix isAbove to only display when threshold set (elastic#65540) draft search profiler accessibility tests (elastic#62357) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form_fields.tsx

…9967) (#65797) * apm stuff * fix cluster_client * fix snapshot * tracker utility for generate_pdf * call apm.startSpan instead of txn.startSpan * Fix async call to end transaction * fix typescript * remove captuureErrors * restore accidental removal * add startTrace lib * fix import * fix imports * ts fix * fix generate_png to not format base64 to buffer and back to base64 * 💅 * revert change to cluster client * fix unused translation Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

tsullivan changed the title ~~Reporting/performance measurement~~ [Reporting] APM integration for baseline performance measurements Mar 11, 2020

tsullivan force-pushed the reporting/performance-measurement branch from 5fca169 to 1b2a777 Compare March 11, 2020 23:39

tsullivan commented Mar 11, 2020

View reviewed changes

src/core/server/elasticsearch/cluster_client.ts Outdated Show resolved Hide resolved

tsullivan commented Mar 12, 2020

View reviewed changes

tsullivan force-pushed the reporting/performance-measurement branch from 1b2a777 to 4a5af92 Compare March 12, 2020 01:04

tsullivan added 3 commits March 12, 2020 09:29

fix cluster_client

2521d76

apm stuff

912a4ad

fix snapshot

a865bcc

tsullivan force-pushed the reporting/performance-measurement branch from 4a5af92 to a865bcc Compare March 12, 2020 16:48

tsullivan added 3 commits March 12, 2020 13:15

tracker utility for generate_pdf

0fc8813

call apm.startSpan instead of txn.startSpan

cbd094b

Fix async call to end transaction

c037d4d

tsullivan marked this pull request as ready for review March 12, 2020 21:22

tsullivan requested a review from a team as a code owner March 12, 2020 21:22

tsullivan requested review from a team, joelgriffith, poffdeluxe and vigneshshanmugam March 12, 2020 21:22

fix typescript

0aa13e6

tsullivan added zDeprecated Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes Team:Reporting Services v8.0.0 labels Mar 12, 2020

tsullivan commented Mar 12, 2020

View reviewed changes

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts Show resolved Hide resolved

vigneshshanmugam reviewed Mar 12, 2020

View reviewed changes

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/open_url.ts Outdated Show resolved Hide resolved

x-pack/legacy/plugins/reporting/export_types/printable_pdf/server/lib/tracker.ts Show resolved Hide resolved

vigneshshanmugam requested a review from watson March 12, 2020 22:08

pgayvallet reviewed Mar 16, 2020

View reviewed changes

...ck/legacy/plugins/reporting/export_types/common/lib/screenshots/get_element_position_data.ts Outdated Show resolved Hide resolved

src/core/server/elasticsearch/cluster_client.ts Outdated Show resolved Hide resolved

mshustov mentioned this pull request Mar 16, 2020

Kibana crash: TypeError: request.abort is not a function #59712

Closed

tsullivan added the blocked label Mar 30, 2020

tsullivan added 5 commits March 30, 2020 10:40

add startTrace lib

33debce

fix import

cc57a7a

Merge remote-tracking branch 'elastic/master' into reporting/performa…

498a629

…nce-measurement

fix imports

fd63584

Merge branch 'master' into reporting/performance-measurement

2b564ff

joelgriffith approved these changes Apr 27, 2020

View reviewed changes

tsullivan added 4 commits April 28, 2020 10:54

Merge branch 'master' into reporting/performance-measurement

890e5f2

ts fix

3db397f

Merge branch 'master' into reporting/performance-measurement

e193ecf

fix generate_png to not format base64 to buffer and back to base64

e79e875

tsullivan commented Apr 29, 2020

View reviewed changes

elasticmachine and others added 5 commits April 30, 2020 19:58

Merge branch 'master' into reporting/performance-measurement

7618e42

Merge branch 'master' into reporting/performance-measurement

0867800

💅

4ac5ce0

revert change to cluster client

6b5c392

fix unused translation

dd9c55b

tsullivan added the v7.9.0 label May 5, 2020

Merge branch 'master' into reporting/performance-measurement

e4495c4

tsullivan removed the blocked label May 7, 2020

tsullivan merged commit 6bf0890 into elastic:master May 7, 2020

tsullivan mentioned this pull request May 8, 2020

[7.x] [Reporting] APM integration for baseline performance measurements (#59967) #65797

Merged

tsullivan deleted the reporting/performance-measurement branch May 8, 2020 19:27

tsullivan mentioned this pull request May 8, 2020

[Reporting] Improvements needed for screenshot capture flow #59396

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting] APM integration for baseline performance measurements#59967

[Reporting] APM integration for baseline performance measurements#59967
tsullivan merged 27 commits intoelastic:masterfrom
tsullivan:reporting/performance-measurement

tsullivan commented Mar 11, 2020 •

edited

Loading

Uh oh!

Uh oh!

tsullivan Mar 12, 2020 •

edited

Loading

Uh oh!

tsullivan May 5, 2020

Uh oh!

elasticmachine commented Mar 12, 2020

Uh oh!

Uh oh!

vigneshshanmugam left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsullivan Apr 29, 2020

Uh oh!

tsullivan commented Apr 30, 2020

Uh oh!

tsullivan commented May 5, 2020

Uh oh!

kibanamachine commented May 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

tsullivan commented Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

For maintainers

Uh oh!

Uh oh!

tsullivan Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsullivan May 5, 2020

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 12, 2020

Uh oh!

Uh oh!

vigneshshanmugam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsullivan Apr 29, 2020

Choose a reason for hiding this comment

Uh oh!

tsullivan commented Apr 30, 2020

Uh oh!

tsullivan commented May 5, 2020

Uh oh!

kibanamachine commented May 7, 2020

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

tsullivan commented Mar 11, 2020 •

edited

Loading

tsullivan Mar 12, 2020 •

edited

Loading