Disable CodeCov expired reports validation#6251
Conversation
|
You can trigger optional UI/connected tests for these changes by visiting CircleCI here. |
Generated by 🚫 dangerJS |
You can test the changes on this Pull Request by downloading an installable build, or scanning this QR code:
|
Codecov Report
@@ Coverage Diff @@
## trunk #6251 +/- ##
============================================
+ Coverage 0 24.55% +24.55%
- Complexity 0 2552 +2552
============================================
Files 0 786 +786
Lines 0 45301 +45301
Branches 0 5692 +5692
============================================
+ Hits 0 11125 +11125
- Misses 0 33154 +33154
- Partials 0 1022 +1022
Continue to review full report at Codecov.
|
|
An update: I've asked the CodeCov team how they determine JaCoCo report creation time. Because if it's time of generating JaCoCo https://community.codecov.com/t/how-codecov-knows-time-of-jacoco-report-generation/3571 For now, let's keep this PR as a draft 🙂. |
|
👋 @wzieba !
Thank you for this update! 👍
👍 PS: I'll be AFK from next week, for 2 weeks, so please don't wait on me to review this change, that is if you end up making it ready for review as some next week. |
|
Gotcha, thanks for the heads up @ParaskP7 ! 🙌 |
|
We got the answer from the CodeCov team:
I also checked how JaCoCo sessions are created and it looks like there's a separate session for each So what happens eventually is that when PR e.g. doesn't change anything in So, in conclusion, as we trust the Gradle Cache system for our tests, the more we should have it for code coverage. Hence, we don't need additional validation from CodeCov. |
AliSoftware
left a comment
There was a problem hiding this comment.
Rationale in above discussion (especially last comments around trusting our cache) make sense to me 👍
To disable PR checks on GitHub. `max_report_age: off` has this context: woocommerce/woocommerce-android#6251
Description
The majority of code coverage reports are considered incorrect by CodeCov (see here).
As valid reports exist (see e.g. #6241 ) I'm leaning toward the theory that the problem is with time-based validation from CodeCov (docs).
How CodeCov knows when a report has been generated?
At the beginning of the XML report, there's a piece of information about the sessions that created the report. Usually, there are two of them (because of two
.execfiles I suppose). There's timestamp for session creation and "dump" there. This is the only time-based information (I could find) that we send to CodeCov so I believe they take one of those values. You can find it by opening any failed report for e.g. commit, on the right underUploadsthere'sDownloadsection to see payload. Then, look forstart="phrase.If you convert those milliseconds to date, you can see that sometimes those point to a date a few days back. Also, I'm not sure if
session dumpmeans exactly the time of generating the report - I have a feeling that it's more like time of generating.execfile.Why are the reports outdated?
Probably Gradle Build cache or cached up-to-date checks for generating
.execfile (artifact from tests task). Just disabling cache forjacocoTestReportdoesn't solve the issue. I'm not 100% sure how it works under the hood though, but I decided to timebox my time for that as it's still an experiment.How does that influence CodeCov?
CodeCov by default doesn't accept reports than were generated 12h+ before sending payload. While this might work for other frameworks, I'm not sure if it's a good fit for JaCoCo (and Gradle).
I'm suggesting removing this validation - if our reports will happen to be invalid, we have a problem in our cache configuration and not with CodeCov.
See similar CodeCov issue:
https://community.codecov.com/t/reports-are-not-displayed-there-was-an-error-processing-coverage-reports/750