Originally posted by @tiancaiamao in #28555 (comment)

There are two directions to resolve this problem:

1. Factor out benchmarks into another binary.
2. Narrow the range where setups triggered.

The first approach won't work because benchmarks can depend on private methods.

The second approach has two more sub directions:

1. Factor out tests into one parent test and only do setup for the test instead of TestMain.
2. Separately handle logics of tests and benchmarks.

The first approach can work but introduce more boilerplate code, like:

func TestSession(t *testing.T) {
  // setups

  t.Run("hardNaming", func (t *testing. T) { ... })
  // ...
}

The second approach, that I'd prefer, can be achieved by adding a short circuit logic:

func ShortCircuitForBench(m *testing.M) {
	if !flag.Parsed() {
		flag.Parse()
	}

	f := flag.Lookup("test.bench")
	if f != nil && len(f.Value.String()) > 0 {
		os.Exit(m.Run())
	}
}

However, Golang can run benchmarks & tests together, so this method can produce false positive where tests are involved but not trigger setups.

To resolve this problem, we may either assume that benchmarks MUST NOT run along with tests, or introduce a new flag to short circuit explicitly.

What do you think @tiancaiamao ?