Affected version
3.6.0
Bug description
When the artifact:compare check fails, it currently produces error messages like:
[ERROR] size mismatch log4j-api-2.25.1-sources.jar: investigate with diffoscope log4j-api/target/reference/org.apache.logging.log4j/log4j-api-2.25.1-sources.jar log4j-api/target/log4j-api-2.25.1-sources.jar
[ERROR] size mismatch log4j-api-2.25.1.module: investigate with diffoscope log4j-api/target/reference/org.apache.logging.log4j/log4j-api-2.25.1.module log4j-api/target/publications/maven/module.json
[ERROR] Reproducible Build output summary: 3 files ok, 2 different
[ERROR] see diff log4j-api/target/reference/log4j-api-2.25.1.buildinfo log4j-api/target/log4j-api-2.25.1.buildinfo
Although this message includes all the necessary information to investigate a reproducibility failure, it can be confusing—especially for users unfamiliar with reproducible builds.
In our recent release votes, multiple PMC members misinterpreted the output and ran:
diff log4j-api/target/reference/log4j-api-2.25.1.buildinfo log4j-api/target/log4j-api-2.25.1.buildinfo
Running this command is not particularly helpful—it simply confirms that files differ, without revealing why. While the buildinfo file includes some useful metadata (such as the OS name and the versions of the JDK and Maven), it primarily lists file checksums and sizes. As a result, the diff is often long and not very informative. It mostly reiterates what was included in the error messages.
In contrast, the earlier diffoscope commands offer much more useful insights by showing the exact content-level differences between files. Unfortunately, these commands are often overlooked because the final diff line draws users’ attention away from them.
Suggested Improvements
The error output can be made significantly more user-friendly by restructuring and simplifying the message. Specifically:
- Remove the final
diff command comparing the buildinfo files, as it often leads users down an unhelpful path.
- Group all
diffoscope commands together at the end, clearly marked as the recommended next step for debugging.
An improved version of the output might look like:
[ERROR] Size mismatch: log4j-api-2.25.1-sources.jar
[ERROR] Size mismatch: log4j-api-2.25.1.module
[ERROR] Reproducible Build summary: 3 files match, 2 differ
[ERROR] To analyze the differences, run:
diffoscope log4j-api/target/reference/org.apache.logging.log4j/log4j-api-2.25.1-sources.jar log4j-api/target/log4j-api-2.25.1-sources.jar
diffoscope log4j-api/target/reference/org.apache.logging.log4j/log4j-api-2.25.1.module log4j-api/target/publications/maven/module.json
This format:
- Emphasizes the actual mismatched files.
- Promotes
diffoscope as the primary tool for meaningful comparison.
- Avoids misleading or redundant suggestions (like comparing
.buildinfo files).
These small changes can greatly reduce confusion—especially for users encountering reproducibility issues for the first time.
Affected version
3.6.0
Bug description
When the
artifact:comparecheck fails, it currently produces error messages like:Although this message includes all the necessary information to investigate a reproducibility failure, it can be confusing—especially for users unfamiliar with reproducible builds.
In our recent release votes, multiple PMC members misinterpreted the output and ran:
Running this command is not particularly helpful—it simply confirms that files differ, without revealing why. While the
buildinfofile includes some useful metadata (such as the OS name and the versions of the JDK and Maven), it primarily lists file checksums and sizes. As a result, the diff is often long and not very informative. It mostly reiterates what was included in the error messages.In contrast, the earlier
diffoscopecommands offer much more useful insights by showing the exact content-level differences between files. Unfortunately, these commands are often overlooked because the final diff line draws users’ attention away from them.Suggested Improvements
The error output can be made significantly more user-friendly by restructuring and simplifying the message. Specifically:
diffcommand comparing thebuildinfofiles, as it often leads users down an unhelpful path.diffoscopecommands together at the end, clearly marked as the recommended next step for debugging.An improved version of the output might look like:
This format:
diffoscopeas the primary tool for meaningful comparison..buildinfofiles).These small changes can greatly reduce confusion—especially for users encountering reproducibility issues for the first time.