This repository was archived by the owner on Sep 30, 2024. It is now read-only.
[Backport 5.2] telemetry-export: 5.2.1 default enablement (#57605)#57681
Merged
Conversation
6423a7d to
94f8aca
Compare
This was referenced Oct 17, 2023
Member
Author
|
I've run through the manual test plan again on this branch and verified everything is intact |
Contributor
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff f4b23d6...35f4eca.
|
Contributor
This change implements the license-based enforcement of telemetry export as outlined in [the plan for the 5.2.1 release](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Most of this functionality has been tested in various environments for some time: dotcom, s2, rctest, cody-dev, demo, and 9 trial Cloud instances (with more to be enabled before the final 5.2.1 release, see sourcegraph/customer#2416), processing about 2 million events in the last 7 days (a small set of events currently instrumented in the new system). The following changes are made to how this feature is enabled: 1. An export enablement mode is added to package `internal/licensing` that evaluates what we export based on the configured license, configured based on license plan or tags in the license, but the levels are unlikely to change): - all events (default going forward) - cody-only (`Feature=~'cody|cody.*'`), since Cody events are a special case covered by terms of use - nothing (air-gapped instance) 3. When persisting events to the export queue, the enablement mode is checked to decide if anything gets written to the database. 4. Exporter background workers are now enabled by default, checking the mode on each iteration to decide if we need to run the exporter at all. We also create a new connection to Telemetry Gateway on each run now, to ensure Telemetry Gateway downtime does not break instances. The feature flag `telemetry-export` and env var `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR` remain configurable, but in a default-enabled state - in 5.2.2, the ability to disable these capabilities via configuration will be removed, and the license-based mechanism will become the only way to disable this feature, as requested in the [telemetry export rollout plan](https://docs.google.com/document/d/1Z1Yp7G61WYlQ1B4vO5-mIXVtmvzGmD7PqYHNBQV-2Ik/edit#bookmark=id.w10771x4dx7y). Parts that require follow-up changes are denoted with `TODO(5.2.2)` comments. A changelog entry and docs are added in https://github.com/sourcegraph/sourcegraph/pull/57606 Part of https://github.com/sourcegraph/sourcegraph/issues/56816 - [x] New unit/integration test coverage - [x] Existing tests pass with minimal changes - [x] Manual testing: - Comment out `sg.config.yaml` line `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "http://127.0.0.1:10080"` and add `TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s"`, exporter starts by default. Run some searches to generate events, and see that worker doesn't export anything, as the default mode for an older license is to disable export (seen in logs and metrics) - Set `dev-private` license to a new, real one from sourcegraph.com, remove all mentions of `SOURCEGRAPH_LICENSE_GENERATION_KEY` from `sg.config.yaml`, repeat above, events now export (seen in logs and metrics): `[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:129 events exported {"maxBatchSize": 10000, "succeeded": 1}` --------- Co-authored-by: Joe Chen <joe@sourcegraph.com> (cherry picked from commit 20c771b)
94f8aca to
35f4eca
Compare
keegancsmith
approved these changes
Oct 18, 2023
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change implements the license-based enforcement of telemetry export as outlined in the plan for the 5.2.1 release.
Most of this functionality has been tested in various environments for some time: dotcom, s2, rctest, cody-dev, demo, and 9 trial Cloud instances (with more to be enabled before the final 5.2.1 release, see https://github.com/sourcegraph/customer/issues/2416), processing about 2 million events in the last 7 days (a small set of events currently instrumented in the new system). The following changes are made to how this feature is enabled:
internal/licensingthat evaluates what we export based on the configured license, configured based on license plan or tags in the license, but the levels are unlikely to change):Feature=~'cody|cody.*'), since Cody events are a special case covered by terms of useThe feature flag
telemetry-exportand env varTELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDRremain configurable, but in a default-enabled state - in 5.2.2, the ability to disable these capabilities via configuration will be removed, and the license-based mechanism will become the only way to disable this feature, as requested in the telemetry export rollout plan. Parts that require follow-up changes are denoted withTODO(5.2.2)comments.A changelog entry and docs are added in https://github.com/sourcegraph/sourcegraph/pull/57606
Part of https://github.com/sourcegraph/sourcegraph/issues/56816
sg.config.yamllineTELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "http://127.0.0.1:10080"and addTELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s", exporter starts by default. Run some searches to generate events, and see that worker doesn't export anything, as the default mode for an older license is to disable export (seen in logs and metrics)dev-privatelicense to a new, real one from sourcegraph.com, remove all mentions ofSOURCEGRAPH_LICENSE_GENERATION_KEYfromsg.config.yaml, repeat above, events now export (seen in logs and metrics):[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:129 events exported {"maxBatchSize": 10000, "succeeded": 1}Co-authored-by: Joe Chen joe@sourcegraph.com
(cherry picked from commit 20c771b)
cc @sourcegraph/release-guild