Background is that, I hava a project that depends on 980 packages, each time I run codeql database create ... I see most of the time cost is contributed by running go list to retrieve package and module directory once for each package.
I have counted how much time actually just go list consumes, and here is the log:

[2021-07-25 13:24:58] [build-stderr] 2021/07/25 13:24:58 Running go list.
[2021-07-25 13:51:34] [build-stderr] 2021/07/25 13:51:34 Done running go list, all pkgRoots are:980

The log shows that go list for all packages consumed 26 minutes.

here is the testing code added to extractor/extractor.go:

	log.Printf("Running go list.")
	packages.Visit(pkgs, func(pkg *packages.Package) bool {
		return true
	}, func(pkg *packages.Package) {
		if _, ok := pkgRoots[pkg.PkgPath]; !ok {
			mdir := util.GetModDir(pkg.PkgPath, modFlags...)
			pdir := util.GetPkgDir(pkg.PkgPath, modFlags...)
			// GetModDir returns the empty string if the module directory cannot be determined, e.g. if the package
			// is not using modules. If this is the case, fall back to the package directory
			if mdir == "" {
				mdir = pdir
			}
			pkgRoots[pkg.PkgPath] = mdir
			pkgDirs[pkg.PkgPath] = pdir
		}
	})
	log.Printf("Done running go list, all pkgRoots are:%d", len(pkgRoots))

With go list -deps, the time cost is significantly reduced, here is the log:

[2021-07-25 13:59:40] [build-stderr] 2021/07/25 13:59:40 Running go list deps.
[2021-07-25 13:59:40] [build-stderr] 2021/07/25 13:59:40 Running cmd: /usr/local/go/bin/go list -e -f {{.ImportPath}}{{"\n"}}{{.Dir}}{{"\n"}}{{if .Module}}{{.Module.Dir}}{{else}}<nil>{{end}} -deps ./...
[2021-07-25 13:59:42] [build-stderr] 2021/07/25 13:59:42 Done running go list deps,resolved 980 packags

just 2 seconds.

@smowton @sauyon Thanks for both your commenting!

@smowton @xhd2015 I've pushed the changes I wanted; this still needs a rebase over main, which I won't do for now in case you'd like to do something with the current history.

Test it with a simulated lgtm.com upgrade then merge if good?

@sauyon I shall commit no more changes to it, it's ok to rebase this branch over main. As for the test, I don't know how to do that.

Sounds good, the test is for @sauyon to do

@smowton Made some changes based on the LGTM tests; had about 5 projects fail because of a SEGV that I fix in the last commit, but otherwise I think this is good to go.

Looks like there's currently a build failure. Have you manually retested those projects getting a segfault?

This does now appear to work.