🎩 What? Why?

UPDATE: I think the root cause is with the merging of the reports and a workaround for that is provided at #9686. This is still relevant as there can be still small fluxuation with the coverage between different runs depending on the execution path (some code may not be run at all test runs due to the randomized run order).

UPDATE 2: The Codecov report can still fail due to the GitHub action failing to upload the report of some of the workflow runs to Codecov. There are some workarounds suggested at #9832 as it seems Codecov seems unwilling to add retry option to their GitHub action.

It has been a consistent issue over the years that the results of the Codecov reports cannot be trusted because they report seemingly random coverage changes for the diffs and whole PRs. This is rather bothering for the developers and can cause extra work for newcomers as they start to investigate why the coverage was changed with their PRs.

I started to investigate this issue and found this very useful piece of documentation:
https://docs.codecov.com/docs/unexpected-coverage-changes

It says the following:

There are a number of reasons that cause line coverage to change unexpectedly. These can be due to:

1. Adding or removing tests.
2. Failing to upload coverage reports, or a different number of reports between head and base
3. Time-sensitive tests (e.g., one case before 2pm UTC, a different case after 2pm UTC).
4. Missing coverage reports or failed builds.
5. Dependencies changed resulting in a different execute plan.
6. Encrypted variables may prevent some execution paths.
7. In the case of changes to a PR, the base or head commits may not have had an uploaded report, or may not be the commit you expect
8. Different error handling paths: sometimes exception handlers are tested and some are not, oftentimes the root of "flaky" tests
9. Due to long running test builds or dynamic test selection, you are intentionally not running all tests on every commit. If this is the case, please check out our feature Carryforward Flags

Especially those bold sections are interesting regarding Decidim as we have:

• Some functionality that can behave differently during different times
• Randomized test running order which causes multiple different execution paths
• Different run sets for different modules which can cause different specs to be run for different changesets

I started to investigate the coverage reports when they are failing and it seems that most of the times the reason is that some methods are not just executed during every test run while they were executed during previous runs.

So I believe we can fix this issue by increasing the unit test coverage which would ensure that all methods are executed on all test runs consistently.

Another thing I am suspecting that can happen when only a subset of the tests are run is e.g. if we execute the accountability tests but they call some methods from the core during their run, but the rest of the methods in the core class were not called during that run, it may also produce weird coverage results.

This will be a long effort but I try to work on this occasionally as we bump into these issues by adding more unit specs to those methods that are reported as not covered by Codecov.

Testing

We just need to consistently add more unit specs for those methods showing up in the coverage reports.