Skip to main content
When working with code coverage, you might occasionally see unexpected changes in coverage metrics. This guide helps you understand why these changes occur and how to investigate them. Unexpected coverage changes often surface when you notice entries in the Indirect Changes section of a pull request’s coverage summary — coverage that moved on files the pull request didn’t touch. There are two high level reasons why your coverage might change unexpectedly:
  1. Execution Path Changes
  2. Incomplete or Missing Coverage Data

​
Execution Path Changes

Qlty is designed to trust code coverage data exactly as it is generated by coverage tools like Cobertura, LCOV, SimpleCov, etc. These coverage tools often reflect reality: coverage changed because tests executed code differently. The question that naturally follows is: what caused these changes? While tests may not fail, coverage can change for all the same reasons that tests can occassionally fail: code changes, dependency changes, non-determinism, and other environment changes can cause your code to execute slightly differently on subsequent runs, even for the same commit.

​
Indirect Code Paths

Although unexpected, this category of change represents a real change in your coverage data caused by a change in your code or tests — not by non-determinism or a problem collecting coverage data. The coverage of one file shifts because of how changed code elsewhere exercises it:
  • Function Call Removal: If File A stops calling a function in File B, File B’s coverage might decrease
  • Conditional Logic Changes: Changing a condition in one file might prevent code in another file from executing
  • Error Handling Changes: Changes to how errors are thrown or caught can affect which code paths are exercised

​
Example:

// Before: UserService.js
async function getUser(id) {
    return await UserRepository.findById(id); // UserRepository.findById is called every time
}

// After: UserService.js
async function getUser(id) {
    if (!id) return null; // Now UserRepository.findById is only called when id exists
    return await UserRepository.findById(id);
}
This change would reduce coverage in UserRepository.js even though it wasn’t modified.
Some code paths are gateway paths that control access to large portions of code:
  • Auth/Permission Checks: Changes to authentication logic might prevent large sections of code from running
  • Feature Flags: Toggling feature flags can enable/disable entire features
  • Dependency Injection: Changing what’s injected can alter large execution paths

​
Dependency Changes

Any sizeable software project relies on dependencies, some explicit and direct, and some less obvious and less direct. Here’s an (incomplete) list of dependencies that can cause execution changes and therefore coverage changes:
  • Explicit dependency changes (e.g. Bumping a package from version 1.2.3 to 1.3.4)
  • Floating dependencies (e.g. your code runs on the latest version of nodejs, and the latest version of nodejs changes)
  • Time: a time change can cause a test to follow a different execution path (even for the same commit)
  • Network connectivity
  • Other environmental changes

​
Non-Determinism and Flaky Coverage

As noted, unexpected coverage changes can be viewed through the same lens as engineers often view unexpected test failures. Just like a test can be flaky and fail occassionally, coverage can change for similar reasons. Some amount of non-determinism is often intentional. A common configuration is to run tests in a random order, which forces an engineering team to surface and address any issues that are the result of unexpected order dependencies between tests. This is healthy for the test suite, but it also means the exact set of lines executed — and therefore coverage — can vary slightly from run to run. Non-determinism tends to come up more often for larger, more complex test suites. As a suite grows, both the chance of some non-determinism and the surface area over which it can occur increase. Statistically, non-determinism tends to affect files that a given pull request didn’t edit: a pull request usually touches only a small fraction of the codebase, so any run-to-run variation is far more likely to land on unedited files. As a result, non-determinism usually appears as indirect coverage changes — coverage shifting in files you didn’t directly modify — which moves your Total Coverage number. These small movements in Total Coverage are usually tolerable: for large repositories, Total Coverage is generally a less helpful signal than Diff Coverage for understanding the coverage impact of a pull request. A small overall percentage change is normal, especially for large repositories. Large swings in coverage, on the other hand, are typically not the result of runtime non-determinism — they usually point to an issue with test coverage data collection, such as incomplete or missing coverage reports. To diagnose whether non-determinism is the cause of a coverage change (rather than specific code or test changes), run the full test suite against the same commit (and/or tag, if applicable) enough times to gain confidence in the stability or instability of your coverage. You can retrieve the raw coverage data for each of these runs to compare them directly. If your coverage changes between these runs, you know the cause is non-deterministic behavior in your test suite. Isolating the test and/or bisecting the test suite are good starting points for investigating what causes the non-determinism.
Because the causes of non-determinism depend on your specific test environment, diagnosing the root cause is not something the Qlty support team can typically help with. Support can, however, help analyze the coverage data sent to us — if you’re unsure how to interpret what you’re seeing, reach out to support.

​
Incomplete or missing coverage reports

​
Test optimization tools

Test optimization tools sometimes intentionally or unintentionally (buggy) run less than 100% of tests. If the test optimization is new, this behavior would consistently result in Qlty reporting coverage drops. Depending on your test sharding and upload strategy, this truncation can sometimes be apparent in the raw coverage data: you may see fewer coverage data files than expected, or individual files that are unusually truncated. See Retrieving Raw Coverage Data for Diagnosis to inspect the files Qlty received.
Coordinating coverage with test optimization tools is not a configuration Qlty currently supports, though it’s something we’ve discussed supporting. If this is a setup you’re interested in, reach out to support.

​
Overwritten or overlooked coverage data files

Test suites commonly generate multiple coverage data files. Qlty must receive all coverage data files in order to accurately report coverage. While this can be straight-forward — a file was not sent that should have — there are a few less straight-forward ways this can happen:
  • A build parallelizes their test suite using multiple threads or processes, but each unit of parallelization overwrites one another’s coverage data file, clobbering or corrupting coverage data
  • A build parallelizes their test suite using multiple machines, where each machine generates unique coverage data files. Then, either some machines’s coverage data files are missed or the files, when collected together from different machines, because similarly named, overwrite one another.
Review our Coverage Merging documentation to ensure you’re sending coverage data correctly. See Retrieving Raw Coverage Data for Diagnosis below for how to verify Qlty has received all coverage reports.

​
Retrieving Raw Coverage Data for Diagnosis

For several of the scenarios above, the most direct way to find the root cause is to examine the raw coverage data Qlty received — the exact files your coverage tools generated, before Qlty processes or merges them. There are various reasons you might want this data: comparing the base and head commits of a pull request, comparing multiple runs of the same commit to diagnose non-determinism, or verifying that every coverage file you expected was actually received.
Qlty is designed to trust code coverage data exactly as it is generated by coverage tools like Cobertura, LCOV, SimpleCov, etc.Therefore, the Qlty technical support team is unable to investigate unexpected code coverage changes, as the causes are always unique to the codebase.If you find a discrepancy between the raw coverage data generated by your coverage tool and the data reported by Qlty, please let us know.

​
Retrieving the raw coverage data

To download and read the raw coverage data files for any coverage upload:
  1. Navigate to coverage uploads: In your Qlty project, go to Project Settings > Code Coverage and search by commit SHA or branch to find the relevant coverage upload(s).
  2. Click “Download Zip”
If using server-side merging, you’ll need to download each “incomplete” coverage zip file by following the upload links marked as incomplete.
  1. Examine the coverage data files: Extract the ZIP file and open the coverage data files in the raw_files/ folder. Search for the file path you’re investigating and read the relevant sections.
If the folder seems empty note that some tools, like SimpleCov (.resultset.json), use a naming convention that starts with a . causing the file to appear hidden in some operating systems.
Each coverage file format is different, but you can often identify discrepancies by reading the data in a text editor.

​
Comparing commits on a pull request

A common, specific investigation is confirming that an unexpected coverage change on a pull request is real by comparing the coverage data for the base and head commits.
  1. Obtain commit SHAs: Obtain the commit SHAs for both the branch head and merge base commits from Git or GitHub.
While the commit sha for a the head of a pull request is often easy to determine, the merge base is not currently listed on Qlty or GitHub. However, you can use git merge-base (documentation) to determine the relevant base commit. Make sure you pull from remote first before using git merge-base.
  1. Compare the coverage metrics in Qlty Cloud: For each commit, go to Project Settings > Code Coverage, search by commit SHA (NB: you can also search by branch), and use the on-page search to filter by the specific file path that’s showing unexpected changes. Confirm whether the Covered Lines and/or Missed Lines values differ between the two uploads.
  2. Compare the raw coverage data: For a deeper look, retrieve the raw coverage data for both commits and compare the relevant sections between the two files.
This process helps confirm whether the indirect coverage change is internally consistent (caused by actual test execution differences) or if there might be an issue with how coverage is being generated or reported in your CI/CD pipeline. After completing your investigation, you’ll encounter one of two outcomes:
  • If the coverage data file shows different coverage values, this confirms the unexpected coverage change is real and legitimate. The coverage difference exists in your actual test execution between commits, meaning Qlty is correctly reporting what your coverage tools generated.
  • If the coverage data shows identical coverage for both the base and head commits, this suggests an inconsistency in Qlty’s processing. If this is the case, feel free to report it to Qlty support with your findings, including the commit SHAs and file paths.
An inconsistency in Qlty’s processing is exceedingly rare. In nearly all cases that customers investigate, the root cause is a difference in the raw coverage data being provided to Qlty.