Test/Encoding: Simplify and Parallelize Corpus Checking#52646
Test/Encoding: Simplify and Parallelize Corpus Checking#52646NitzanMordhai wants to merge 1 commit intoceph:mainfrom
Conversation
03f3cb3 to
36c9c7e
Compare
By leveraging parallel processing capabilities during corpus checks, we can speed up the test execution. Currently, the tests are running slowly, posing challenges in identifying mistakes within the encode-decode area and visualizing errors. Implementing the following changes will result in a faster and more efficient process, making it easier to detect and address potential issues. Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
36c9c7e to
54532aa
Compare
|
jenkins test make check |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution! |
rzarzynski
left a comment
There was a problem hiding this comment.
Generally LGTM. The comments are nits.
Taking the complexity into account, I wonder whether we ensure the entire machinery continues to work, similarly to what we do for valgrind testing – there is a test intentionally causing a leak just to verify it's being caught.
| shift | ||
|
|
||
| # import the corpus | ||
| ../src/test/encoding/import.sh \ |
There was a problem hiding this comment.
Do we still need import.sh after these changes. I see only readable.sh gets removed.
| git commit --signoff --message=${version} | ||
| git remote add cc git@github.com:ceph/ceph-object-corpus.git | ||
| # Check if the remote already exists | ||
| if ! git remote | grep -q "cc"; then |
| def process_type(file_path, type): | ||
| print(f"dencoder test for {file_path}") | ||
| cmd1 = [CEPH_DENCODER, "type", type, "import", file_path, "decode", "dump_json"] | ||
| cmd2 = [CEPH_DENCODER, "type", type, "import", file_path, "decode", "encode", "decode", "dump_json"] |
There was a problem hiding this comment.
I think these 2 lines are the crux of the entire idea, and thus deserve more intention-conveying names (or a comment). Something like cmd_decode_recorded_bytestream and cmd_decode_reencoded_bytestream? Still, they are not so good.
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution! |
By leveraging parallel processing capabilities during corpus checks, we can speed up the test execution.
Currently, the tests are running slowly, posing challenges in identifying mistakes within the encode-decode area and visualizing errors. Implementing the following changes will result in a faster and more efficient process, making it easier to detect and address potential issues.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows