tobari

Tobari

Tobari is a scoped coverage measurement tool for Go.

"Tobari" (帷) is a Japanese word meaning "curtain" or "veil," similar to the English word "cover".

Tobari provides coverage measurement capabilities that introduce a new concept called Scoped Coverage in addition to the coverage features provided by runtime/coverage. This feature enables clear mapping between test code and its impact area, allowing for high-precision test code generation using AI and other tools.

Background

To understand what Tobari enables, we first need to understand the current coverage mechanisms provided by Go.

The most common coverage measurement method we use is specifying coverage options with go test, such as go test -cover. We can control coverage target packages with -coverpkg and output coverage results with -coverprofile. However, using coverage through go test means we can only measure coverage when writing test code in Go. This means coverage can only be measured for tests starting from functions like func TestFoo(t *testing.T). For example, it was not possible to measure coverage for a binary created with go build after the fact.

To improve this, Go version 1.20 and later allows the -cover option with go build and supports the runtime/coverage package. With go build -cover, applications can be built with coverage instrumentation points. The runtime/coverage API allows coverage counter initialization and result output at any timing.

This made it possible to measure coverage of servers implemented in Go using E2E testing tools not written in Go when implementing HTTP or gRPC servers. Coverage functionality that was only available during go test execution became available at any timing during application runtime.

However, when actually trying to use runtime/coverage, you'll notice that some operational considerations are needed. Coverage measurement with runtime/coverage increases coverage counters when processing passes through embedded coverage measurement points, but it doesn't care what caused the passage. This is similar to traffic surveys for automobiles, where the number of cars passing a certain location is measured, but the type of cars is not considered. This mechanism becomes problematic in situations like measuring coverage for server applications:

• To measure E2E test effectiveness, you want to measure coverage only when E2E testing tools access the server
  • Accesses other than from E2E tests should not be measured
• For test acceleration, you want to access the server concurrently from E2E testing tools, but manage access contexts separately
  • For example, when E2E test scenarios A and B exist, you want to measure accesses from A and B separately even when accessing the server concurrently
• Asynchronous processing by Goroutines created through methods not originating from E2E test requests should be excluded from coverage measurement

To meet these requirements, the server application must serialize and handle requests, ensuring that accesses from A and B are not processed simultaneously. Additionally, you need to implement mechanisms to determine E2E test accesses by referencing headers and reject other requests. Furthermore, what if you want to use this running server application for purposes other than E2E testing? For example, when performing manual verification in parallel with E2E test verification. In this case, other requests may reach the server while E2E tests are running, and these requests should not be rejected. Also, serializing processing defeats the purpose of concurrent access for test acceleration. Moreover, there's no way to ignore asynchronous processing not originating from E2E tests.

Therefore, I conceived the Scoped Coverage approach and decided to develop Tobari.

What is Scoped Coverage ?

What Scoped Coverage provides over runtime/coverage coverage measurement is measuring "what passed through". Using the E2E testing example, when measuring coverage from scenario A and B accesses, it distinguishes between "access from A" and "access from B". Additionally, it measures only asynchronous processing originating from E2E tests. This enables coverage measurement limited to the scope you want to measure. Concurrent access is also possible.

Coverage also needs to record "places that should be passed" in addition to recording "places that were passed". Coverage is calculated using a formula like this:

Coverage (%) = (Places passed / Places that should be passed) * 100

In normal coverage measurement, all files in packages specified by coverpkg could be considered "places that should be passed", requiring no special processing. However, Scoped Coverage is different. How should we define "places that should be passed" ? For example, when calculating coverage for scenario A, if places that scenario A will never pass are included in "places that should be passed", coverage will never reach 100% no matter how hard you try.

Tobari determines "places that should be passed" by reverse calculation from the results of what was passed. It extracts inter-function dependencies through static analysis in advance and calculates functions that could potentially be passed based on the dependency relationships of actually passed functions. Considering this, the formula becomes:

Coverage (%) = (Places passed / All places in functions that could potentially be called from passed functions) * 100

How to Use

Using tobari is very simple with 3 steps:

1. Installation

First, install the tobari tool with the following command:

go install github.com/goccy/tobari/cmd/tobari@latest

Important: The version of the tobari CLI tool and the tobari library used in your application must match. For example, if you install tobari@v0.2.1, your go.mod should also require github.com/goccy/tobari v0.2.1. Version mismatch will cause fingerprint errors during linking.

2. Use the API (like using the runtime/coverage package)

Using a gRPC server as an example:

tobari.CoverWithName serves as the entry point for coverage measurement. Specify a name in the first argument to distinguish coverage units. The behavior with runtime/coverage is the same as specifying an empty name. For gRPC, the name is obtained from metadata. If there's no metadata, it's treated as a normal request and the function exits without measuring coverage.

tobariInterceptor := func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
  md, ok := metadata.FromIncomingContext(ctx)
  if !ok { return handler(ctx, req) }
  scenarioNames, exists := md["E2E-Tool-Scenario-Name"]
  if !exists { return handler(ctx, req) }
  if len(scenarioNames) == 0 { return handler(ctx, req) }

  scenarioName := scenarioNames[0]

  var (
    res any
    err error
  )
  tobari.CoverWithName(scenarioName, func() {
    res, err = handler(ctx, req)
  })
  return res, err
}

grpcServer := grpc.NewServer(grpc.UnaryInterceptor(tobariInterceptor))

When outputting coverage data, you can use WriteCoverprofileByName or CoverprofileMap:

• WriteCoverprofileByName: Get coverprofile results for a specified name
• CoverprofileMap: Return the relationship between names and coverprofiles in map format

These APIs can be executed by creating a separate gRPC server and calling specific endpoints when E2E tests finish.

3. Build the Application

Then, when building the application you want to measure coverage for, simply specify GOFLAGS as follows:

GOFLAGS="$(tobari flags)" go build .

This example shows distinguishing coverage results by name, but you can also use it the same way as runtime/coverage. For specific APIs, please refer here.

Embedding Source Code

Tobari supports embedding the original source code into instrumented binaries with the --embed-code (-E) option. This is useful for archiving the exact source that was compiled, enabling offline coverage analysis without access to the original source tree.

# Using tobari flags
GOFLAGS="$(tobari flags -E)" go build .

# Using tobari directly as toolexec
go build -cover -toolexec='tobari --embed-code' ./...

To retrieve the embedded sources at runtime, use ReadCoverArchivedFile():

reader := tobari.ReadCoverArchivedFile()
if reader == nil {
    // No sources were embedded
    return
}
// reader provides a tar.gz archive of the original source files

Extracting Embedded Sources from a Binary

You can also extract embedded sources directly from a built binary using the tobari extract command, without writing any code:

tobari extract -o sources.tar.gz ./my-binary

This runs the instrumented binary internally, extracts all embedded source files, and writes them as a tar.gz archive to the specified path. The -o flag is required.

The extraction hook is injected at compile time into the main package, so it works regardless of -coverpkg settings. When the TOBARI_EXTRACT_SOURCES environment variable is set (which tobari extract does internally), the binary writes the archive and exits immediately without running the application's main function.

Generating HTML Coverage Reports

The tobari html command generates an HTML coverage report from a coverprofile or tobari.json file. It wraps go tool cover -html and adds support for tobari's JSON format and source resolution from embedded binaries or tar.gz archives.

# Generate HTML from a coverprofile
tobari html -o coverage.html profile.cover

# Generate HTML from tobari.json (merges all test entries with summed counts)
tobari html -o coverage.html tobari.json

When the source code is not available locally (e.g., on a CI server or a different machine), you can use the -b or -s flags to provide the source files needed for HTML generation:

# Use embedded sources from a tobari-built binary
tobari html -o coverage.html -b ./my-binary profile.cover

# Use sources from a previously extracted tar.gz archive
tobari html -o coverage.html -s sources.tar.gz tobari.json

The -b and -s flags are mutually exclusive. When -b is specified, the command runs the binary with TOBARI_EXTRACT_SOURCES to obtain the source archive, then uses it to resolve file paths in the coverage data.

Example

We will use a more practical example to give you a better idea of how to use tobari. The example code is located in examples/http, so you can run it on your own environment as well.

First, install the tobari CLI using the following command.

go install github.com/goccy/tobari/cmd/tobari@latest

Next, to run the example, clone the repository and move to the repository root.

git clone https://github.com/goccy/tobari.git
cd tobari

The examples/http directory contains code structured as follows.

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"os"

	"github.com/goccy/tobari"
)

func coverageMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if v := r.Header.Get("X-GO-COVERAGE"); v != "" {
			// When measuring coverage, wrap the function with tobari.Cover.
			tobari.Cover(func() { next.ServeHTTP(w, r) })
		} else {
			next.ServeHTTP(w, r)
		}
	})
}

var ch = make(chan struct{})

func run(ctx context.Context) error {
	mux := http.NewServeMux()

	// This is the endpoint used to start measuring coverage.
	mux.HandleFunc("/coverstart", func(w http.ResponseWriter, req *http.Request) {
		// Similar to ClearCounters in runtime/coverage, this resets the currently active counters.
		// It is intended to be called at the start of coverage measurement.
		tobari.ClearCounters()

		// This process is used to ensure that goroutines not subject to measurement are correctly ignored.
		// Normally, when coverage measurement begins, all goroutines are included in the measurement.
		// However, in Tobari, it is possible to count only specific processes, so the coverage of this goroutine will be ignored.
		go func() {
			<-ch
		}()

		fmt.Fprintf(w, "started")
	})

	// This is the endpoint used to stop coverage measurement and retrieve the results.
	mux.HandleFunc("/coverend", func(w http.ResponseWriter, req *http.Request) {
		// Writes data in coverprofile format.
		// The resulting output can be directly used with `go tool cover`.
		tobari.WriteCoverprofile(tobari.SetMode, w)
	})

	mux.HandleFunc("/foo", func(w http.ResponseWriter, req *http.Request) {
		if v := req.Header.Get("xxxx"); v != "" {
			uncoveredFunc()
		}
		fmt.Fprintf(w, "foo")
	})
	mux.HandleFunc("/bar", func(w http.ResponseWriter, req *http.Request) {
		fmt.Fprintf(w, "bar")
	})

	// A middleware is added to switch behavior based on the presence of the coverage flag.
	srv := httptest.NewServer(coverageMiddleware(mux))
	defer srv.Close()

	cli := new(http.Client)

	// start coverage.
	if err := func() error {
		req, err := http.NewRequest("GET", srv.URL+"/coverstart", nil)
		if err != nil {
			return err
		}
		resp, err := cli.Do(req)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return err
		}
		fmt.Println(string(b))
		return nil
	}(); err != nil {
		return err
	}

	// access foo endpoint with X-GO-COVERAGE header.
	if err := func() error {
		req, err := http.NewRequest("GET", srv.URL+"/foo", nil)
		if err != nil {
			return err
		}
		req.Header.Add("X-GO-COVERAGE", "true")
		resp, err := cli.Do(req)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return err
		}
		fmt.Println(string(b))
		return nil
	}(); err != nil {
		return err
	}

	// access bar endpoint without coverage header.
	if err := func() error {
		req, err := http.NewRequest("GET", srv.URL+"/bar", nil)
		if err != nil {
			return err
		}
		resp, err := cli.Do(req)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return err
		}
		fmt.Println(string(b))
		return nil
	}(); err != nil {
		return err
	}

	// end coverage.
	if err := func() error {
		req, err := http.NewRequest("GET", srv.URL+"/coverend", nil)
		if err != nil {
			return err
		}
		resp, err := cli.Do(req)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return err
		}
		return os.WriteFile("test.cover", b, 0o600)
	}(); err != nil {
		return err
	}
	return nil
}

func uncoveredFunc() {
	uncoveredFunc2()
}

func uncoveredFunc2() {
	uncoveredFunc3()
}

func uncoveredFunc3() {
	fmt.Println("uncovered func3")
}

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

Run this code using the following command.

GOFLAGS="$(tobari flags)" go run ./examples/http/main.go

Then, a test.cover file should be created in the current directory. Let’s view it using go tool cover -html.

go tool cover -html test.cover

This will produce an output like the following. Only the coverage related to foo is displayed.

Using with go test

Tobari can also be used with go test to collect coverage data. When running tests with tobari, coverage data is written to tobari/tobari.json and tobari/tobari.toon files.

Basic Usage

GOFLAGS="$(tobari flags)" go test ./...

This will create tobari/tobari.json and tobari/tobari.toon files in the current directory.

Specifying Output Directory

You can use the TOBARI_COVERDIR environment variable to specify where the coverage data should be written:

TOBARI_COVERDIR=/path/to/output GOFLAGS="$(tobari flags)" go test ./...

This will create /path/to/output/tobari/tobari.json and /path/to/output/tobari/tobari.toon.

Using with go test -c

When building a test binary with go test -c, you can specify TOBARI_COVERDIR at runtime:

# Build the test binary
GOFLAGS="$(tobari flags)" go test -c .

# Run with default output (current directory)
./pkg.test

# Run with custom output directory
TOBARI_COVERDIR=/path/to/output ./pkg.test

Output Formats

Tobari generates coverage data in two formats:

JSON Format (tobari.json)

A structured JSON format where each test name maps to an array of coverage entries:

{
  "TestAdd": [
    {
      "FileName": "/path/to/file.go",
      "Start": {"Line": 7, "Column": 24},
      "End": {"Line": 9, "Column": 2},
      "StatementCount": 1,
      "Count": 4
    }
  ]
}

TOON Format (tobari.toon)

A compact Token-Oriented Object Notation format optimized for LLM consumption. TOON uses approximately 40% fewer tokens than JSON while maintaining the same information:

TestAdd[1]{FileName,StartLine,StartCol,EndLine,EndCol,StatementCount,Count}:
	/path/to/file.go,7,24,9,2,1,4
TestMultiply[3]{FileName,StartLine,StartCol,EndLine,EndCol,StatementCount,Count}:
	/path/to/file.go,11,29,12,22,1,5
	/path/to/file.go,15,2,15,14,1,3
	/path/to/file.go,12,22,14,3,1,2

The TOON format is particularly useful when feeding coverage data to AI coding agents, as it reduces token usage while preserving all necessary information for coverage analysis.

Using Coverage Data with AI

The coverage data generated by tobari can be used to improve your test suite with AI assistance. By analyzing which code paths are covered (Count > 0) and which are not (Count = 0), AI coding agents can:

• Identify duplicate test cases that cover nearly identical code paths
• Suggest new test cases to improve coverage
• Find dead code that is never executed

Agent Skills

Tobari provides Agent Skills to help automate test coverage improvement workflows. Agent Skills is an open standard for giving AI coding agents new capabilities and expertise. Skills are supported by many coding agents including Claude Code, Cursor, GitHub Copilot, OpenAI Codex, Gemini CLI, and more.

Available Skills

The following skills are available in the skills/ directory:

How to Use

1. Run tests with tobari to generate coverage data:

   GOFLAGS="$(tobari flags)" go test ./...
   

2. Use the agent skills with your preferred coding agent (Claude Code, Cursor, Codex, etc.) to analyze and improve your test coverage.

The skills will automatically read the tobari.toon file and guide you through:

• Detecting and removing redundant tests
• Identifying uncovered code blocks
• Adding new test cases to improve coverage

How It Works

Tobari records which Goroutine increased the counter by obtaining the Goroutine ID (GID) and Parent Goroutine ID (PGID) when increasing coverage counters. At the same time, when calling the coverage measurement entry function passed to tobari.Cover or tobari.CoverWithName, it creates a Goroutine and records its ID. By tracing Goroutines that have the GID from when coverage measurement started as their parent, it can target only Goroutines related to coverage. To implement this functionality, Tobari passes two options during go build: -cover and -toolexec.

• -cover: Added to have the Go compiler determine coverage instrumentation targets
• -toolexec: Hooks execution of Go build tools to dynamically add APIs to the runtime package for obtaining GID and PGID (which are not public APIs), and to embed measurement points that include GID and PGID

These options are output by the tobari flags command, so they can be added to go build options by simply specifying GOFLAGS=$(tobari flags).

License

MIT