telemetry-export: 5.2.1 default enablement by bobheadxi · Pull Request #57605 · sourcegraph/sourcegraph-public-snapshot

bobheadxi · 2023-10-14T18:48:43Z

This change implements the license-based enforcement of telemetry export as outlined in the plan for the 5.2.1 release.

Most of this functionality has been tested in various environments for some time: dotcom, s2, rctest, cody-dev, demo, and 9 trial Cloud instances (with more to be enabled before the final 5.2.1 release, see https://github.com/sourcegraph/customer/issues/2416), processing about 2 million events in the last 7 days (a small set of events currently instrumented in the new system). The following changes are made to how this feature is enabled:

An export enablement mode is added to package internal/licensing that evaluates what we export based on the configured license, configured based on license plan or tags in the license, but the levels are unlikely to change):
- all events (default going forward)
- cody-only (Feature=~'cody|cody.*'), since Cody events are a special case covered by terms of use
- nothing (air-gapped instance)
When persisting events to the export queue, the enablement mode is checked to decide if anything gets written to the database.
Exporter background workers are now enabled by default, checking the mode on each iteration to decide if we need to run the exporter at all. We also create a new connection to Telemetry Gateway on each run now, to ensure Telemetry Gateway downtime does not break instances.

The feature flag telemetry-export and env var TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR remain configurable, but in a default-enabled state - in 5.2.2, the ability to disable these capabilities via configuration will be removed, and the license-based mechanism will become the only way to disable this feature, as requested in the telemetry export rollout plan. Parts that require follow-up changes are denoted with TODO(5.2.2) comments.

A changelog entry and docs are added in https://github.com/sourcegraph/sourcegraph/pull/57606

Part of https://github.com/sourcegraph/sourcegraph/issues/56816

Test plan

New unit/integration test coverage
Existing tests pass with minimal changes
Manual testing:
- Comment out sg.config.yaml line TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "http://127.0.0.1:10080" and add TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s", exporter starts by default. Run some searches to generate events, and see that worker doesn't export anything, as the default mode for an older license is to disable export (seen in logs and metrics)
- Set dev-private license to a new, real one from sourcegraph.com, remove all mentions of SOURCEGRAPH_LICENSE_GENERATION_KEY from sg.config.yaml, repeat above, events now export (seen in logs and metrics):
  [ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:129 events exported {"maxBatchSize": 10000, "succeeded": 1}

bobheadxi · 2023-10-14T19:29:57Z

Current dependencies on/for this PR:

main
- PR telemetry-export: 5.2.1 default enablement #57605 👈

This comment was auto-generated by Graphite.

sourcegraph-bot · 2023-10-16T20:05:25Z

Codenotify: Notifying subscribers in CODENOTIFY files for diff b621b8b...cce3f4c.

Notify	File(s)
@unknwon	internal/license/BUILD.bazel internal/license/license_test.go internal/license/licensetest/BUILD.bazel internal/license/licensetest/licensetest.go internal/licensing/BUILD.bazel internal/licensing/tags.go internal/licensing/telemetryexport.go internal/licensing/telemetryexport_test.go

unknwon

No blocking comments!

unknwon · 2023-10-17T17:01:04Z

+
+	// SetMockExportMode configures the store's mock export mode for use in testing.
+	//
+	// ⚠️ Use in tests only!


Wondering if we should do similar thing as https://github.com/sourcegraph/sourcegraph/blob/2af7aecbf68cdeab3e2a038577b8dee68b23bd57/internal/lazyregexp/lazyre.go#L104

to programmatically prevent misuse outside tests.

This seems a bit delicate, and is only used on one place in Sourcegraph right now, so I'm a bit hesitant to add this 😅

I think hopefully the need to cast this into a mock interface and call a method with "mock" in the name is a sufficient deterrence

Usually I would have this test helpers accept a testing.T but because it's a bit engrained into the implementatin, and we shouldn't import testing due to side effects as part of the program, I opted not to here

Co-authored-by: Joe Chen <joe@sourcegraph.com>

…telemetry-export_5.2.1_default_enablement

sourcegraph-release-bot · 2023-10-17T20:05:05Z

The backport to 5.2 failed at https://github.com/sourcegraph/sourcegraph/actions/runs/6552227523:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-5.2 5.2
# Navigate to the new working tree
cd .worktrees/backport-5.2
# Create a new branch
git switch --create backport-57605-to-5.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 20c771bc3194152e81f30ea2a4529e4f5fcd7fd4
# Push it to GitHub
git push --set-upstream origin backport-57605-to-5.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.2

If you encouter conflict, first resolve the conflict and stage all files, then run the commands below:

git cherry-pick --continue
# Push it to GitHub
git push --set-upstream origin backport-57605-to-5.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.2

Follow above instructions to backport the commit.
Create a pull request where the base branch is 5.2 and the compare/head branch is backport-57605-to-5.2., click here to create the pull request.
Make sure to tag @sourcegraph/release-guild in the pull request description.
Once the backport pull request is created, kindly remove the release-blocker from this pull request.

bobheadxi · 2023-10-17T22:23:08Z

Opened backport manually in #57681

This change implements the license-based enforcement of telemetry export as outlined in [the plan for the 5.2.1 release](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Most of this functionality has been tested in various environments for some time: dotcom, s2, rctest, cody-dev, demo, and 9 trial Cloud instances (with more to be enabled before the final 5.2.1 release, see sourcegraph/customer#2416), processing about 2 million events in the last 7 days (a small set of events currently instrumented in the new system). The following changes are made to how this feature is enabled: 1. An export enablement mode is added to package `internal/licensing` that evaluates what we export based on the configured license, configured based on license plan or tags in the license, but the levels are unlikely to change): - all events (default going forward) - cody-only (`Feature=~'cody|cody.*'`), since Cody events are a special case covered by terms of use - nothing (air-gapped instance) 3. When persisting events to the export queue, the enablement mode is checked to decide if anything gets written to the database. 4. Exporter background workers are now enabled by default, checking the mode on each iteration to decide if we need to run the exporter at all. We also create a new connection to Telemetry Gateway on each run now, to ensure Telemetry Gateway downtime does not break instances. The feature flag `telemetry-export` and env var `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR` remain configurable, but in a default-enabled state - in 5.2.2, the ability to disable these capabilities via configuration will be removed, and the license-based mechanism will become the only way to disable this feature, as requested in the [telemetry export rollout plan](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Parts that require follow-up changes are denoted with `TODO(5.2.2)` comments. A changelog entry and docs are added in https://github.com/sourcegraph/sourcegraph/pull/57606 Part of https://github.com/sourcegraph/sourcegraph/issues/56816 - [x] New unit/integration test coverage - [x] Existing tests pass with minimal changes - [x] Manual testing: - Comment out `sg.config.yaml` line `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "http://127.0.0.1:10080"` and add `TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s"`, exporter starts by default. Run some searches to generate events, and see that worker doesn't export anything, as the default mode for an older license is to disable export (seen in logs and metrics) - Set `dev-private` license to a new, real one from sourcegraph.com, remove all mentions of `SOURCEGRAPH_LICENSE_GENERATION_KEY` from `sg.config.yaml`, repeat above, events now export (seen in logs and metrics): `[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:129 events exported {"maxBatchSize": 10000, "succeeded": 1}` --------- Co-authored-by: Joe Chen <joe@sourcegraph.com> (cherry picked from commit 20c771b)

…57681) telemetry-export: 5.2.1 default enablement (#57605) This change implements the license-based enforcement of telemetry export as outlined in [the plan for the 5.2.1 release](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Most of this functionality has been tested in various environments for some time: dotcom, s2, rctest, cody-dev, demo, and 9 trial Cloud instances (with more to be enabled before the final 5.2.1 release, see sourcegraph/customer#2416), processing about 2 million events in the last 7 days (a small set of events currently instrumented in the new system). The following changes are made to how this feature is enabled: 1. An export enablement mode is added to package `internal/licensing` that evaluates what we export based on the configured license, configured based on license plan or tags in the license, but the levels are unlikely to change): - all events (default going forward) - cody-only (`Feature=~'cody|cody.*'`), since Cody events are a special case covered by terms of use - nothing (air-gapped instance) 3. When persisting events to the export queue, the enablement mode is checked to decide if anything gets written to the database. 4. Exporter background workers are now enabled by default, checking the mode on each iteration to decide if we need to run the exporter at all. We also create a new connection to Telemetry Gateway on each run now, to ensure Telemetry Gateway downtime does not break instances. The feature flag `telemetry-export` and env var `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR` remain configurable, but in a default-enabled state - in 5.2.2, the ability to disable these capabilities via configuration will be removed, and the license-based mechanism will become the only way to disable this feature, as requested in the [telemetry export rollout plan](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Parts that require follow-up changes are denoted with `TODO(5.2.2)` comments. A changelog entry and docs are added in https://github.com/sourcegraph/sourcegraph/pull/57606 Part of https://github.com/sourcegraph/sourcegraph/issues/56816 - [x] New unit/integration test coverage - [x] Existing tests pass with minimal changes - [x] Manual testing: - Comment out `sg.config.yaml` line `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "http://127.0.0.1:10080"` and add `TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s"`, exporter starts by default. Run some searches to generate events, and see that worker doesn't export anything, as the default mode for an older license is to disable export (seen in logs and metrics) - Set `dev-private` license to a new, real one from sourcegraph.com, remove all mentions of `SOURCEGRAPH_LICENSE_GENERATION_KEY` from `sg.config.yaml`, repeat above, events now export (seen in logs and metrics): `[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:129 events exported {"maxBatchSize": 10000, "succeeded": 1}` --------- Co-authored-by: Joe Chen <joe@sourcegraph.com> (cherry picked from commit 20c771b)

See https://github.com/sourcegraph/sourcegraph/pull/57681 and https://github.com/sourcegraph/sourcegraph/pull/57605

cla-bot Bot added the cla-signed label Oct 14, 2023

bobheadxi added the backport 5.2 label Oct 14, 2023

bobheadxi mentioned this pull request Oct 14, 2023

doc/admin/telemetry: telemetry export reference and changelog #57606

Merged

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch 3 times, most recently from 13471bc to 2d66db1 Compare October 14, 2023 19:29

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch from 2d66db1 to 4cb709f Compare October 16, 2023 16:55

bobheadxi added 2 commits October 16, 2023 10:27

telemetry-export: 5.2.1 default enablement

4eb7a91

licensing: only disable on airgapped plans

b2841f6

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch from 4cb709f to 46b74db Compare October 16, 2023 17:27

fixup

032d2b4

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch from 46b74db to 032d2b4 Compare October 16, 2023 19:54

bobheadxi marked this pull request as ready for review October 16, 2023 20:03

bobheadxi requested review from a team and nathan-downs October 16, 2023 20:04

bobheadxi added the backport/improvement Final touching of existing features label Oct 16, 2023

add compat with the legacy EXPORT_USAGE_DATA_ENABLED

be4bd05

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch from df2250a to be4bd05 Compare October 16, 2023 20:28

unknwon approved these changes Oct 17, 2023

View reviewed changes

bobheadxi and others added 2 commits October 17, 2023 10:43

Update internal/database/telemetry_export_store.go

ed92b5b

Co-authored-by: Joe Chen <joe@sourcegraph.com>

improve security comments

ad60b04

bobheadxi force-pushed the 10-14-telemetry-export_5.2.1_default_enablement branch from e631e17 to ad60b04 Compare October 17, 2023 17:59

Merge branch 'main' of github.com:sourcegraph/sourcegraph into 10-14-…

cce3f4c

…telemetry-export_5.2.1_default_enablement

bobheadxi merged commit 20c771b into main Oct 17, 2023

bobheadxi deleted the 10-14-telemetry-export_5.2.1_default_enablement branch October 17, 2023 20:03

sourcegraph-release-bot added backports failed-backport-to-5.2 labels Oct 17, 2023

sourcegraph-release-bot added the release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases label Oct 17, 2023

bobheadxi mentioned this pull request Oct 17, 2023

database/telemetryexport: fixup for metric and trace attribute #57684

Merged

bobheadxi removed the release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases label Oct 17, 2023

bobheadxi referenced this pull request Oct 18, 2023

doc/admin/telemetry: telemetry export reference and changelog (#57606)

8f48c6c

See https://github.com/sourcegraph/sourcegraph/pull/57681 and https://github.com/sourcegraph/sourcegraph/pull/57605

bobheadxi mentioned this pull request Oct 18, 2023

licensing: export events in dotcom mode #57717

Merged

varungandhi-src mentioned this pull request Jan 16, 2024

vg/matlab fix #59635

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

telemetry-export: 5.2.1 default enablement#57605

telemetry-export: 5.2.1 default enablement#57605
bobheadxi merged 7 commits into
mainfrom
10-14-telemetry-export_5.2.1_default_enablement

bobheadxi commented Oct 14, 2023 •

edited

Loading

Uh oh!

bobheadxi commented Oct 14, 2023

Uh oh!

sourcegraph-bot commented Oct 16, 2023 •

edited

Loading

Uh oh!

unknwon left a comment

Uh oh!

Uh oh!

Uh oh!

unknwon Oct 17, 2023

Uh oh!

bobheadxi Oct 17, 2023 •

edited

Loading

Uh oh!

bobheadxi Oct 17, 2023

Uh oh!

sourcegraph-release-bot commented Oct 17, 2023 •

edited by bobheadxi

Loading

Uh oh!

bobheadxi commented Oct 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bobheadxi commented Oct 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Uh oh!

bobheadxi commented Oct 14, 2023

Uh oh!

sourcegraph-bot commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

unknwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

unknwon Oct 17, 2023

Choose a reason for hiding this comment

Uh oh!

bobheadxi Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobheadxi Oct 17, 2023

Choose a reason for hiding this comment

Uh oh!

sourcegraph-release-bot commented Oct 17, 2023 • edited by bobheadxi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobheadxi commented Oct 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bobheadxi commented Oct 14, 2023 •

edited

Loading

sourcegraph-bot commented Oct 16, 2023 •

edited

Loading

bobheadxi Oct 17, 2023 •

edited

Loading

sourcegraph-release-bot commented Oct 17, 2023 •

edited by bobheadxi

Loading