Core usage data by rudolf · Pull Request #79101 · elastic/kibana

rudolf · 2020-10-01T11:28:59Z

Summary

Core Telemetry Service for supplying core metrics to a usage collector (usage collector not implemented yet). I aimed to collect the easy metrics to get this in before FF, but this doesn't collect all the metrics we identified in #56762

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

TinaHeiligers · 2020-10-01T19:38:55Z

@rudolf If it's not too late, can we please rename from "telemetry" to "usage data"? We're trying really hard to reserve using "Telemetry" for only sending the data to the service. We already have a telemetry plugin that it responsible for sending the data and don't want to get the terminology mixed up. It makes it really difficult to debug SDH issues. 🙏

elasticmachine · 2020-10-02T05:56:03Z

Pinging @elastic/kibana-platform (Team:Platform)

rudolf · 2020-10-02T05:58:07Z

~~Still blocked on~~ worked around the issue

node scripts/telemetry_check.js --fix
 ✔ Checking .telemetryrc.json files
 ❯ Extracting Collectors
   ✖ Extracting collectors in /Users/rudolf/dev/kibana/src/plugins/
     → TypeError: Cannot read property 'resolvedFileName' of undefined
     Extracting collectors in /Users/rudolf/dev/kibana/x-pack/plugins/
   Checking Compatible collector.schema with collector.fetch type
   Generating new telemetry mappings
   Updating telemetry mapping files
ERROR Unhandled exception!
ERROR Error: Error extracting collector in src/plugins/kibana_usage_collection/server/collectors/core/core_usage_collector.ts
      TypeError: Cannot read property 'resolvedFileName' of undefined
          at createFailError (/Users/rudolf/dev/kibana/packages/kbn-dev-utils/target/run/fail.js:26:26)
          at parseUsageCollection (/Users/rudolf/dev/kibana/packages/kbn-telemetry-tools/src/tools/ts_parser.ts:205:17)
          at parseUsageCollection.next (<anonymous>)
          at extractCollectors (/Users/rudolf/dev/kibana/packages/kbn-telemetry-tools/src/tools/extract_collectors.ts:73:12)
          at extractCollectors.next (<anonymous>)
          at Task.task (/Users/rudolf/dev/kibana/packages/kbn-telemetry-tools/src/tools/tasks/extract_collectors_task.ts:54:36)

rudolf · 2020-10-02T07:05:06Z

src/core/server/core_usage_data/types.ts

+
+export interface CoreUsageData {
+  config: CoreConfigUsageData;
+  // services: CoreServicesUsageData;


Because the telemetry team asked us not to use ...Telemetry I've switched to ...UsageData but CoreUsageData['usageData'] is a bit awkward, so I deviated from the "spec" in the original issue by using the key "services" with the idea that we would group usage data under the respective Core services they originate from e.g. savedObjects / plugins.

afharo

It looks great! Just added a couple of NITs.

Reading through your PR, I acknowledge we (the telemetry team) need to work on our documentation 😅

src/core/server/core_usage_data/core_usage_data_service.ts

src/plugins/kibana_usage_collection/server/collectors/core/core_usage_collector.ts

joshdover

LGTM. Tested with localhost:5601/api/stats?extended and it seemed to work correctly 😄. A couple small nits is all.

src/core/server/core_usage_data/types.ts

src/plugins/kibana_usage_collection/server/collectors/core/core_usage_collector.ts

kibanamachine · 2020-10-05T10:02:24Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: afe0114

Metrics [docs]

distributable file count

id	before	after	diff
`default`	44119	44125	+6
`oss`	28595	28601	+6

History

💔 Build #79445 failed 062280e
💔 Build #79165 failed 65954a6
💔 Build #79123 failed 4de37e2
💔 Build #79061 failed dbee8f9
💔 Build #79055 failed a508a0a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Core usage data (#79101) * Core Telemetry service * CoreTelemetryService mock * Add missing config values back, cleanup * Core usage collector * HttpConfig path is 'server' * Fix tests * CoreTelemetry -> CoreUsageData * Improve tests / docs * Fix telemetry_check * Don't catch fetch function exceptions, let usage collector handle it * Code review * Collect saved object index usage data * Fix tests and telemetry_check * explicitly import/export usage data types for telemetry_check * Remove OS data for now, test for SO usage data * Fix tests * Polish core docs * This shouldn't be here * Fix test

rudolf added 2 commits October 1, 2020 13:25

Core Telemetry service

057b83f

Merge branch 'master' into core-telemetry

33ba75c

rudolf added Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// v7.10.0 labels Oct 1, 2020

rudolf added 3 commits October 1, 2020 14:23

CoreTelemetryService mock

7227412

Add missing config values back, cleanup

f69a00d

Core usage collector

4678edf

rudolf force-pushed the core-telemetry branch from 6599c42 to 4678edf Compare October 1, 2020 18:12

rudolf added the release_note:skip Skip the PR/issue when compiling release notes label Oct 1, 2020

rudolf added 3 commits October 1, 2020 20:17

Merge branch 'master' into core-telemetry

e1b7af4

HttpConfig path is 'server'

9c5be45

Fix tests

fd290e0

rudolf changed the title ~~Core telemetry~~ Core usage data Oct 1, 2020

CoreTelemetry -> CoreUsageData

0a83399

rudolf marked this pull request as ready for review October 2, 2020 05:56

rudolf requested review from a team as code owners October 2, 2020 05:56

rudolf commented Oct 2, 2020

View reviewed changes

rudolf added 3 commits October 2, 2020 09:23

Improve tests / docs

aa2ef67

Fix telemetry_check

a508a0a

Don't catch fetch function exceptions, let usage collector handle it

dbee8f9

afharo reviewed Oct 2, 2020

View reviewed changes

rudolf added 2 commits October 2, 2020 14:16

Code review

4de37e2

Collect saved object index usage data

65954a6

afharo mentioned this pull request Oct 2, 2020

[@kbn/telemetry-tools] Add support to proxy exports #79265

Closed

joshdover approved these changes Oct 2, 2020

View reviewed changes

src/core/server/core_usage_data/types.ts Show resolved Hide resolved

src/plugins/kibana_usage_collection/server/collectors/core/core_usage_collector.ts Outdated Show resolved Hide resolved

Merge branch 'master' into core-telemetry

c36e3e5

rudolf added 3 commits October 5, 2020 00:03

Fix tests and telemetry_check

a1f476a

explicitly import/export usage data types for telemetry_check

949e6a5

Remove OS data for now, test for SO usage data

062280e

rudolf mentioned this pull request Oct 5, 2020

Add OS usage data (including pretty_name) to Core usage data #42008

Open

rudolf added 3 commits October 5, 2020 09:58

Fix tests

012acd2

Polish core docs

333b3d4

This shouldn't be here

afe0114

rudolf merged commit cd38380 into elastic:master Oct 5, 2020

rudolf mentioned this pull request Oct 5, 2020

[7.x] Core usage data (#79101) #79557

Merged

rudolf deleted the core-telemetry branch October 7, 2020 09:52

rudolf mentioned this pull request Nov 9, 2020

[Advanced Settings] Introducing telemetry #82860

Merged

2 tasks

pgayvallet mentioned this pull request Jul 4, 2024

Collect non-sensitive platform-specific config stats #56762

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core usage data#79101

Core usage data#79101
rudolf merged 21 commits intoelastic:masterfrom
rudolf:core-telemetry

rudolf commented Oct 1, 2020 •

edited

Loading

Uh oh!

TinaHeiligers commented Oct 1, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Oct 2, 2020

Uh oh!

rudolf commented Oct 2, 2020 •

edited

Loading

Uh oh!

rudolf Oct 2, 2020 •

edited

Loading

Uh oh!

afharo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshdover left a comment

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Oct 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rudolf commented Oct 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

For maintainers

Uh oh!

TinaHeiligers commented Oct 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Oct 2, 2020

Uh oh!

rudolf commented Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rudolf Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshdover left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Oct 5, 2020

💚 Build Succeeded

Metrics [docs]

distributable file count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rudolf commented Oct 1, 2020 •

edited

Loading

TinaHeiligers commented Oct 1, 2020 •

edited

Loading

rudolf commented Oct 2, 2020 •

edited

Loading

rudolf Oct 2, 2020 •

edited

Loading