usage-data: push scraped events to pubsub if telemetry job is enabled#39389
Conversation
Instrumentation showed that these endpoints aren't being used and it looks like there weren't even any handlers attached to the routes. Confirmed that they were removed a long time ago: https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/commit/8572e643fa4a0a024a4be7d2441a45649800368f
Finally fixing this after it made me say "ah! oh no" one too many times in the past few weeks. Here's what previously happened on THORSTEN & SG GENERATE: 1. Thorsten has a custom commandset named `horsegraph` in `sg.config.overwrite.yaml`, along with some other custom commands. 2. Thorsten creates a database migration/GraphQL schema addition/... 3. Thorsten runs `sg generate` 4. Thorsten commits and pushes commit 5. Thorsten sees that he pushed commit in which `sg`'s reference in the documentation now contains `"horsegraph"` as an official commandset to be used with `sg start` 6. Thorsten says "ah! oh no" and undoes changes ... multiple times. So what this does here is introduce a `disable-overwrite` flag that causes only the standard config to be read. It's then used in the `go:generate` directive that runs `sg help`.
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff 031f22f...9932595.
|
chwarwick
left a comment
There was a problem hiding this comment.
Some thoughts in comments but nothing blocking.
An easier local testing setup may be to use gcloud and run gcloud auth application-default login This way it runs within your users permissions and don't need to share keys.
I assume this going to be a cross project publish when it runs in production. There will probably need to be some additional considerations that will need to be made in the topic config because I anticipate there are going be issues using the default client credential locations to specify creds to a different project (mainly that they should get picked up by any other running process).
In the MI cloud they will be using workload identity, these credentials are just for testing locally. Thanks for pointing out |
Closes https://github.com/sourcegraph/sourcegraph/issues/39088
Note: If you inspect the pubsub topic below you will see a variety of message formats. There is still some uncertainty in the format, so for now we won't worry so much that the schema is exactly correct in the destination.
Test plan
To run / test locally:
Using credentials: https://start.1password.com/open/i?a=HEDEDSLHPBFGRBTKAKJWE23XX4&h=team-sourcegraph.1password.com&i=fpusirhqcoi744acjtowvzfixq&v=dnrhbauihkhjs5ag6vszsme45a
Copy that file to a location, and then start
sgwithAdd to your site config
A variety of events can be found here: https://console.cloud.google.com/cloudpubsub/topic/detail/usage-data-testing?authuser=0&project=sourcegraph-dogfood&tab=messages from pushing locally.