gomod: update zoekt to include ngram sort optimization by keegancsmith · Pull Request #54999 · sourcegraph/sourcegraph-public-snapshot

keegancsmith · 2023-07-17T08:46:48Z

Main motivation is to see the effect on performance for attribution search.

Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel.

The changed commits are
sourcegraph/zoekt@7643f3b...45f608f

sourcegraph/zoekt@a176bde1a3 go get -u -t ./...
sourcegraph/zoekt@e2e8aede00 Fix template documentation comments
sourcegraph/zoekt@25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason
sourcegraph/zoekt@9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE
sourcegraph/zoekt@008a775ba8 zoekt-indexserver: use value format directive for bad conf warning
sourcegraph/zoekt@f9d3a0e2e4 zoekt: add fgprof for full profiling
sourcegraph/zoekt@3d0bdd5c9c remove ngram offset code
sourcegraph/zoekt@45f608ff95 sort ngrams before looking them up

Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk.

Main motivation is to see the effect on performance for attribution search. Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel. The changed commits are sourcegraph/zoekt@7643f3b...45f608f - a176bde1a3 go get -u -t ./... - e2e8aede00 Fix template documentation comments - 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason - 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE - 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning - f9d3a0e2e4 zoekt: add fgprof for full profiling - 3d0bdd5c9c remove ngram offset code - 45f608ff95 sort ngrams before looking them up Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk.

sourcegraph-bot · 2023-07-17T09:41:02Z

Codenotify: Notifying subscribers in CODENOTIFY files for diff f3e9e40...467ad1a.

Notify	File(s)
@bobheadxi	docker-images/prometheus/cmd/prom-wrapper/mocks/BUILD.bazel docker-images/prometheus/cmd/prom-wrapper/mocks/mocks_temp.go

sourcegraph-bot · 2023-07-17T09:56:00Z

Codenotify: Notifying subscribers in OWNERS files for diff f3e9e40...467ad1a.

Notify	File(s)
@mrnugget	dev/sg/sg_start_test.go
@sourcegraph/dev-experience	dev/sg/sg_start_test.go

keegancsmith · 2023-07-17T09:59:21Z

-		"Failed to build test-cmd-1: 'bash -c echo 'booting up horsegraph' && exit 1' failed: booting up horsegraph: exit status 1:",
+		"Failed to build test-cmd-1: 'bash -c echo 'booting up horsegraph' && exit 1' failed: booting up horsegraph",
+		": exit status 1:",


I do not know why I needed to update this test, I imagine some sort of tty dependency changed how it detects our test terminal and now newlines happen in a different place? Either way I assume this is a safe update, but cc @mrnugget who likely has context here.

keegancsmith · 2023-07-17T11:08:48Z

FYI I have been blocked since zoekt updated some dependencies... which meant sourcegraph updated some dependencies... which suddenly meant there is some bazel darkarts going on.

bobheadxi · 2023-07-17T17:05:57Z

 	go.etcd.io/bbolt v1.3.6
-	go.opentelemetry.io/collector v0.75.0
-	go.opentelemetry.io/collector/component v0.75.0
+	go.opentelemetry.io/collector v0.81.0 // indirect


Beat me to the upgrade 😁

Note that this requires an upgrade in the service as well: https://github.com/sourcegraph/sourcegraph/pull/54969/files#diff-c4b85f9a40207f45cc97a051e50a08352bbe2302c44b65911deec373410c4ad1

thanks! Is there a test or something we can add to point out that these need to happen in sync?

github-actions · 2023-08-15T18:43:07Z

The backport to 5.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-5.1 5.1
# Navigate to the new working tree
cd .worktrees/backport-5.1
# Create a new branch
git switch --create backport-54999-to-5.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 766a8e13cb499144580f0ec44efbd4fb31399d33
# Push it to GitHub
git push --set-upstream origin backport-54999-to-5.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.1

Then, create a pull request where the base branch is 5.1 and the compare/head branch is backport-54999-to-5.1.

Main motivation is to see the effect on performance for attribution search. Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel. The changed commits are sourcegraph/zoekt@7643f3b...45f608f - a176bde1a3 go get -u -t ./... - e2e8aede00 Fix template documentation comments - 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason - 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE - 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning - f9d3a0e2e4 zoekt: add fgprof for full profiling - 3d0bdd5c9c remove ngram offset code - 45f608ff95 sort ngrams before looking them up Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk. Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>

Main motivation is to see the effect on performance for attribution search. Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel. The changed commits are sourcegraph/zoekt@7643f3b...45f608f - a176bde1a3 go get -u -t ./... - e2e8aede00 Fix template documentation comments - 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason - 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE - 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning - f9d3a0e2e4 zoekt: add fgprof for full profiling - 3d0bdd5c9c remove ngram offset code - 45f608ff95 sort ngrams before looking them up Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk. Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr> (cherry picked from commit 766a8e1)

Main motivation is to see the effect on performance for attribution search. Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel. The changed commits are sourcegraph/zoekt@7643f3b...45f608f - a176bde1a3 go get -u -t ./... - e2e8aede00 Fix template documentation comments - 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason - 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE - 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning - f9d3a0e2e4 zoekt: add fgprof for full profiling - 3d0bdd5c9c remove ngram offset code - 45f608ff95 sort ngrams before looking them up Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk. Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>

…e ngram sort optimization (#56200) * gomod: update zoekt to include ngram sort optimization (#54999) Main motivation is to see the effect on performance for attribution search. Note that this included a bump in the otel version used since zoekt is using a new version. This had a bit of a cascading effect on our third party deps since they removed a package. So this bumps most 3rd party packages that directly interacted with otel. The changed commits are sourcegraph/zoekt@7643f3b...45f608f - a176bde1a3 go get -u -t ./... - e2e8aede00 Fix template documentation comments - 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason - 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE - 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning - f9d3a0e2e4 zoekt: add fgprof for full profiling - 3d0bdd5c9c remove ngram offset code - 45f608ff95 sort ngrams before looking them up Test Plan: tested in the zoekt repo. Our CI will handle the dep updates. I eyeballed them and they all look low risk. Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr> * search: update zoekt with removal of RepositoryRankRequest (#54394) This endpoint was unused which was why it was removed in Zoekt. The changes from Zoekt included in this dep bump: - 1686b50d50 indexserver: remove unused GetRepoRank - 7078a585a9 shards: populate RepoList.Stats.Repos - b9e6d9433e zoekt-indexserver: Check stderr for git fetch - 93f7b0c983 matchtree: capture Stats before pruning - 7643f3b313 matchiter: capture metric NgramLookups Test Plan: CI * fix bazel deps and patches * fix bad merge and update backcompat zoekt * update go.sum for github.com/jackc/pgproto3/v2 * fix go.sum dependency * restore backcompat throttled repo * Revert "restore backcompat throttled repo" This reverts commit e5f66b3. * Fix backcompat by porting #54748 * allow mozilla 2.0 license for retryablehttp --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr> Co-authored-by: ggilmore <geoffrey@sourcegraph.com>

keegancsmith requested a review from stefanhengl July 17, 2023 08:46

cla-bot Bot added the cla-signed label Jul 17, 2023

stefanhengl approved these changes Jul 17, 2023

View reviewed changes

keegancsmith added 3 commits July 17, 2023 11:11

instrumentation: use new noop pkg for meter

8c20d11

somehow lost the zoekt update

92ae642

rm unused prom-wrapper mocks

2780d48

keegancsmith added 2 commits July 17, 2023 11:50

bump zoekt with fixed BUILD.bazel

ff1c8db

TestStartCommandSet_InstallError dep updates

0b92081

keegancsmith commented Jul 17, 2023

View reviewed changes

third_party: update patch for grpc_gateway

f5b4ddd

jhchabran added 2 commits July 17, 2023 13:23

bzl: fix googleapis resolve

7205fe9

patch @back_compat//internal/instrumentation

a994958

jhchabran force-pushed the k/update branch from 9ca95bc to a994958 Compare July 17, 2023 12:58

Patch another otel related package

467ad1a

keegancsmith merged commit 766a8e1 into main Jul 17, 2023

keegancsmith deleted the k/update branch July 17, 2023 13:34

bobheadxi reviewed Jul 17, 2023

View reviewed changes

bobheadxi added a commit that referenced this pull request Jul 17, 2023

add #54999 to changelog

7e9e20c

bobheadxi mentioned this pull request Jul 17, 2023

opentelemetry: upgrade all packages to v1.16/v0.81 #54969

Merged

ggilmore added the backport 5.1 label Aug 15, 2023

github-actions Bot added backports failed-backport-to-5.1 release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases labels Aug 15, 2023

ggilmore mentioned this pull request Aug 15, 2023

gomod: update zoekt to include ngram sort optimization (#54999) #55887

Closed

burmudar mentioned this pull request Aug 24, 2023

backport 5.1: update zoekt - Removal of RepositoryRankRequest, include ngram sort optimization #56200

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gomod: update zoekt to include ngram sort optimization#54999

gomod: update zoekt to include ngram sort optimization#54999
keegancsmith merged 10 commits into
mainfrom
k/update

keegancsmith commented Jul 17, 2023

Uh oh!

sourcegraph-bot commented Jul 17, 2023 •

edited

Loading

Uh oh!

sourcegraph-bot commented Jul 17, 2023 •

edited

Loading

Uh oh!

keegancsmith Jul 17, 2023

Uh oh!

keegancsmith commented Jul 17, 2023

Uh oh!

bobheadxi Jul 17, 2023

Uh oh!

keegancsmith Jul 18, 2023

Uh oh!

github-actions Bot commented Aug 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

keegancsmith commented Jul 17, 2023

Uh oh!

sourcegraph-bot commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sourcegraph-bot commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keegancsmith Jul 17, 2023

Choose a reason for hiding this comment

Uh oh!

keegancsmith commented Jul 17, 2023

Uh oh!

bobheadxi Jul 17, 2023

Choose a reason for hiding this comment

Uh oh!

keegancsmith Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sourcegraph-bot commented Jul 17, 2023 •

edited

Loading

sourcegraph-bot commented Jul 17, 2023 •

edited

Loading