[APM] Add elasticsearch queries to api response by sorenlouv · Pull Request #95146 · elastic/kibana

sorenlouv · 2021-03-23T11:56:36Z

This PR appends elasticsearch queries to the response to allow the end user to inspect them.

Noteworthy changes:

_debug=true query param has been renamed to _inspect=true. In addition to logging elasticsearch queries in the node process, they will also be send to the client as part of the api response. Only queries authenticated as the end user will be send to the client due to security concerns.
API's must return an object. Arrays and primitive return types (string, number, null, undefined) is no longer allowed.
sessionStorage.set('apm_debug', 'true') has been replaced with a setting in Kibana Advanced Settings

Debug mode is disabled by default but can be enabled in Advanced Settings

When debug mode is enabled a callout will make the user aware of this

When debug mode all API requests will contain _debugQueries

elasticmachine · 2021-03-23T11:56:40Z

Pinging @elastic/apm-ui (Team:apm)

elasticmachine · 2021-03-23T11:56:40Z

Pinging @elastic/uptime (Team:uptime)

x-pack/plugins/apm/server/ui_settings.ts

smith

Works!

I think if you're adding an advanced setting you'll also need to add the field in src/plugins/kibana_usage_collection/server/collectors/management/schema.ts and src/plugins/kibana_usage_collection/server/collectors/management/types.ts and run the telemetry script.

x-pack/plugins/apm/public/components/shared/search_bar.tsx

shahzad31 · 2021-03-23T19:29:13Z

Can we please make this observability wide setting?

We have the same kind of debug mode in uptime. Consolidating UI setting observability wide will make lot of sense. WDYT?

sorenlouv · 2021-03-23T20:26:12Z

Can we please make this observability wide setting?

It's already in the Observability section of Advanced Settings. I can move the settings implementation out of the APM app if that's what you mean? And then each app can choose whether they want to implement the feature. Sounds ok?

dgieselaar

Would it be feasible to add an API test to verify that it doesn't leak data from calls made as the internal user?

shahzad31 · 2021-03-24T09:04:40Z

Can we please make this observability wide setting?

It's already in the Observability section of Advanced Settings. I can move the settings implementation out of the APM app if that's what you mean? And then each app can choose whether they want to implement the feature. Sounds ok?

I also meant in terms of UI setting name, instead of saying APMDebug queries, we can say Observability Debug queries.

That way i don't want to add another checkbox there with Uptime Debug queries or User Experience debug quries.

shahzad31 · 2021-03-24T09:19:32Z

does this still logs to kibana logs?

sorenlouv · 2021-03-24T09:36:02Z

I also meant in terms of UI setting name, instead of saying APMDebug queries, we can say Observability Debug queries.

That way i don't want to add another checkbox there with Uptime Debug queries or User Experience debug quries.

Makes sense. I've updated the setting description to not mention APM.

does this still logs to kibana logs?

Yes, nothing changes there.

sorenlouv · 2021-03-26T09:32:35Z

Would it be feasible to add an API test to verify that it doesn't leak data from calls made as the internal user?
@dgieselaar

Good idea, will add that

shahzad31

Code Changes LGTM for UX and Uptime !!

ts fixes Fix APIs to always return objects minor fixes use beaker icon update copy move setting to observability Rename `_debug` to `_inspect` fix api tests disallow empty responses rename test fix i18n WIP fix param type check fix remaining tests minor improvements Fix jest Add api test for inspect flag Fix tsc Fix jest tests Fix tsc issues

kibanamachine · 2021-03-29T05:53:08Z

💚 Backport successful

✅ 7.x / #95625

This backport PR will be merged automatically after passing CI.

Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>

kibanamachine · 2021-03-29T08:44:57Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	1569	1570	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	4.2MB	4.2MB	+4.7KB
`observability`	327.1KB	327.1KB	+21.0B
`uptime`	879.8KB	879.8KB	+2.0B
total			+4.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`apm`	22.1KB	22.2KB	+5.0B
`observability`	26.0KB	26.6KB	+641.0B
total			+646.0B

History

💚 Build #115680 succeeded 1b9def7
💔 Build #115674 failed be064b60c33593b75388c8c7ec9a2a8ee1fa04e3
💔 Build #115459 failed 4b7b88da7867b08cc0360281b220e6b48b57dcb0
💔 Build #115442 failed 58c1c2cccdbf71e4908e7b64e7603a11db8d4db9
💚 Build #115350 succeeded cace7744e06c901c2e0a35526c9f6e2dfe60b7ed

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match.

* tests/system: adapt to new API response In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match. * tests/system: add user_agent.device.type elastic/elasticsearch#69322 added support for extracting device types to the user_agent ingest processors. Update approvals to match.

* tests/system: adapt to new API response In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match. * tests/system: add user_agent.device.type elastic/elasticsearch#69322 added support for extracting device types to the user_agent ingest processors. Update approvals to match. (cherry picked from commit 0e09aa6)

* tests/system: fix system tests (#5037) * tests/system: adapt to new API response In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match. * tests/system: add user_agent.device.type elastic/elasticsearch#69322 added support for extracting device types to the user_agent ingest processors. Update approvals to match. (cherry picked from commit 0e09aa6) * user_agent.device.type isn't in 7.x yet * make update * systemtest: revert approvals changes Co-authored-by: Andrew Wilkins <axw@elastic.co> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

cauemarcondes · 2021-05-06T15:51:40Z

tests ok

* tests/system: adapt to new API response In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match. * tests/system: add user_agent.device.type elastic/elasticsearch#69322 added support for extracting device types to the user_agent ingest processors. Update approvals to match.

sorenlouv requested a review from a team March 23, 2021 11:56

sorenlouv requested a review from a team as a code owner March 23, 2021 11:56

botelastic bot added Team:APM - DEPRECATED Use Team:obs-ux-infra_services. Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability labels Mar 23, 2021

sorenlouv mentioned this pull request Mar 23, 2021

[APM] WIP: Send inspected queries to client #85052

Closed

sorenlouv added v7.13.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed labels Mar 23, 2021

sorenlouv changed the title ~~[APM] Add debug queries to api response~~ [APM] Add elasticsearch queries to api response Mar 23, 2021

dgieselaar reviewed Mar 23, 2021

View reviewed changes

x-pack/plugins/apm/server/ui_settings.ts Outdated Show resolved Hide resolved

smith approved these changes Mar 23, 2021

View reviewed changes

x-pack/plugins/apm/public/components/shared/search_bar.tsx Outdated Show resolved Hide resolved

sorenlouv requested review from a team as code owners March 23, 2021 23:36

sorenlouv force-pushed the add-apm-debug-mode-setting branch from edba126 to 85c8480 Compare March 23, 2021 23:51

dgieselaar reviewed Mar 24, 2021

View reviewed changes

pgayvallet approved these changes Mar 25, 2021

View reviewed changes

sorenlouv mentioned this pull request Mar 25, 2021

[APM] Update api tests to accomodate changes in API response sorenlouv/kibana#16

Merged

sorenlouv force-pushed the add-apm-debug-mode-setting branch from e98bf85 to 39b6572 Compare March 25, 2021 19:49

shahzad31 approved these changes Mar 26, 2021

View reviewed changes

sorenlouv force-pushed the add-apm-debug-mode-setting branch 2 times, most recently from 4b7b88d to be064b6 Compare March 28, 2021 18:12

sorenlouv force-pushed the add-apm-debug-mode-setting branch from 152c41e to 1b9def7 Compare March 28, 2021 22:40

sorenlouv merged commit 84adfe5 into elastic:master Mar 29, 2021

sorenlouv deleted the add-apm-debug-mode-setting branch March 29, 2021 05:50

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Mar 29, 2021

[APM] Add elasticsearch queries to api response (elastic#95146)

e09f426

kibanamachine mentioned this pull request Mar 29, 2021

[7.x] [APM] Add elasticsearch queries to api response (#95146) #95625

Merged

kibanamachine added a commit that referenced this pull request Mar 29, 2021

[APM] Add elasticsearch queries to api response (#95146) (#95625)

ecedd6b

Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>

axw added a commit to axw/apm-server that referenced this pull request Mar 31, 2021

tests/system: adapt to new API response

d548dff

In elastic/kibana#95146 the response structure for listing APM agent central config changed. Update system tests to match.

sorenlouv added the apm:test-plan-7.13.0 label Apr 20, 2021

formgeist mentioned this pull request May 3, 2021

[APM] Bug: Remove callout that ES queries are enabled (debug flag) #99029

Closed

cauemarcondes self-assigned this May 6, 2021

kertal mentioned this pull request Jan 22, 2024

Introduce/enable enableInspectEsQueries to be implemented for each application #132283

Closed

Conversation

sorenlouv commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 23, 2021

Uh oh!

elasticmachine commented Mar 23, 2021

Uh oh!

Uh oh!

smith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shahzad31 commented Mar 23, 2021

Uh oh!

sorenlouv commented Mar 23, 2021

Uh oh!

dgieselaar left a comment

Choose a reason for hiding this comment

Uh oh!

shahzad31 commented Mar 24, 2021

Uh oh!

shahzad31 commented Mar 24, 2021

Uh oh!

sorenlouv commented Mar 24, 2021

Uh oh!

sorenlouv commented Mar 26, 2021

Uh oh!

shahzad31 left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Mar 29, 2021

💚 Backport successful

Uh oh!

kibanamachine commented Mar 29, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

Uh oh!

cauemarcondes commented May 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

sorenlouv commented Mar 23, 2021 •

edited

Loading