Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

conf: grpc: support silent retries if frontend isn't ready yet#56217

Merged
ggilmore merged 1 commit into
mainfrom
grpc-retry-conf
Aug 25, 2023
Merged

conf: grpc: support silent retries if frontend isn't ready yet#56217
ggilmore merged 1 commit into
mainfrom
grpc-retry-conf

Conversation

@ggilmore

@ggilmore ggilmore commented Aug 24, 2023

Copy link
Copy Markdown
Contributor

Fixes https://sourcegraph.slack.com/archives/C04HCK4K3DL/p1691782868033789

Our configuration package has logic to automatically retry requests if any of them fail. However, on startup, it's likely the case that any given client will start before the frontend configuration server fully initializes. In the HTTP path, we had logic to "ignore" logging connection errors that occur during this initialization peroid.

We regressed on this behavior when we introduced gRPC powered site configuration in #55648. This PR adds similar logic for gRPC to reduce the logspam on startup.

Test plan

Manual testing.

  1. Before this change, run sg start from scratch with SRC_GRPC_ENABLE_CONF: "true" set in your sg.config.yaml.

See errors like the following pop up in your terminal:

[    gitserver-1] ERROR conf.client conf/client.go:301 received error during background config update {"triggered_by": "waitForSleep", "error": "unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\""}
[       searcher] ERROR conf.client conf/client.go:301 received error during background config update {"triggered_by": "waitForSleep", "error": "unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\""}
[    gitserver-0] ERROR conf.client conf/client.go:301 received error during background config update {"triggered_by": "waitForSleep", "error": "unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\""}
[    zoekt-web-0] INFO server zoekt-webserver/main.go:261 adding reverse proxy {"socket": "/Users/ggilmore/.sourcegraph/zoekt/index-0/indexserver.sock"}
[       searcher] ERROR conf.client conf/client.go:301 received error during background config update {"triggered_by": "waitForSleep", "error": "unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\""}
[    gitserver-0] ERROR conf.client conf/client.go:301 received error during background config update {"triggered_by": "waitForSleep", "error": "unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\""}
  1. After this change, run sg start again with with SRC_GRPC_ENABLE_CONF: "true" set in your sg.config.yaml.

The above errors (unable to fetch new configuration: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:3090: connect: connection refused\"") are no longer present.

@cla-bot cla-bot Bot added the cla-signed label Aug 24, 2023
@ggilmore ggilmore marked this pull request as ready for review August 24, 2023 20:59
@ggilmore ggilmore enabled auto-merge (squash) August 24, 2023 20:59
@ggilmore ggilmore requested a review from a team August 25, 2023 18:23
@github-actions

Copy link
Copy Markdown
Contributor

The backport to 5.1 failed:

Validation Failed: "Could not resolve to a node with the global id of 'T_kwDOADy5QM4AWdx9'."

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-5.1 5.1
# Navigate to the new working tree
cd .worktrees/backport-5.1
# Create a new branch
git switch --create backport-56217-to-5.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 396ef1fd02652f2cd55e2ce949dd24d8ce17bc72
# Push it to GitHub
git push --set-upstream origin backport-56217-to-5.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.1

Then, create a pull request where the base branch is 5.1 and the compare/head branch is backport-56217-to-5.1.
See https://github.com/sourcegraph/sourcegraph/actions/runs/5979283798/jobs/14686 for more information.
Make sure to tag @sourcegraph/release-guild in the pull request description.
Once the backport pull request is created, kindly remove the release-blocker from this pull request.

@github-actions github-actions Bot added backports failed-backport-to-5.1 release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases labels Aug 25, 2023
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

backports cla-signed release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants