Skip to content

qa/workunits/erasure-code: add a new bash script that can be used to#58377

Closed
jamiepryde wants to merge 1 commit intoceph:mainfrom
jamiepryde:ec-plugin-output-comparison
Closed

qa/workunits/erasure-code: add a new bash script that can be used to#58377
jamiepryde wants to merge 1 commit intoceph:mainfrom
jamiepryde:ec-plugin-output-comparison

Conversation

@jamiepryde
Copy link
Contributor

@jamiepryde jamiepryde commented Jul 1, 2024 •
edited
Loading

compare the output of selected ec plugins and techniques

This PR adds a new script that calls ceph_erasure_code_non_regression to generate encoded chunks for a specified list of plugins and techniques (and stripe widths, K and M values).

A hash of each chunk is calculated and added to an output file, which can then be inspected to compare the output of the plugins. This is useful when trying to decide if it would be possible to switch from one plugin to another.

In the following example it can be seen that the ISA reed_sol_van technique produces different parity chunks for M>1 than the Jerasure reed_sol_van technique, so it would not be straightforward to switch the plugin for an existing pool using the reed_sol_van technique.

An example run:

CEPH_ERASURE_CODE_NON_REGRESSION=/ceph/build/bin/ceph_erasure_code_non_regression STRIPE_WIDTHS="256" K=3 M=3 /ceph/qa/workunits/erasure-code/plugin-comparison.sh

[root@9d9c7969fd9a build]# cat plugin_comparison/output
489715dfc67930395b840e6f62f87310  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/0
f2dfcedc0cbd3b8247267941e44ede2b  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/1
3b86c3580ddae1fd8a2bf7b0d94ef931  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/2
3041babbcf88ebd486a4e103e7cdc4ae  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/3
72af377096a80419bc2d607031ad3404  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/4
4d88897a89854d5732fd094b401d3107  plugin_comparison/plugin=isa stripe-width=256 k=3 m=3 technique=reed_sol_van/5

489715dfc67930395b840e6f62f87310  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/0
f2dfcedc0cbd3b8247267941e44ede2b  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/1
3b86c3580ddae1fd8a2bf7b0d94ef931  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/2
3041babbcf88ebd486a4e103e7cdc4ae  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/3
858b48eb0c113ee3f217e91a426fc4fd  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/4
b50e4bdf05f80ba7cab1164505cd6e87  plugin_comparison/plugin=jerasure stripe-width=256 k=3 m=3 technique=reed_sol_van/5

...

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@jamiepryde
qa/workunits/erasure-code: add a new bash script that can be used to
3f53ec1 
compare the output of selected ec plugins and techniques

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
@jamiepryde jamiepryde requested a review from a team as a code owner July 1, 2024 16:51
@jamiepryde
Copy link
Contributor Author

jamiepryde commented Jul 3, 2024 •
edited
Loading

output.txt - here is a bigger set of results.
Some observations:

  1. ISA and Jerasure reed_sol_van techniques produces the same m1 XOR parity chunk (for the same input data), but different parity chunks for m>1.
  2. Jerasure's reed_sol_r6_op technique produces the same m1 and m2 parties as ISA reed_sol_van m1 and m2.
  3. ISA reed_sol_van parity chunks have the same position i.e. m1,m2,m3,m4 etc as m increases (for the same input data). However, when using Jerasure, the parity chunks move into different positions as the m value for the profile changes. So for example m2 where k=4,m=2 becomes m4 when k=4,m=4.
fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/3
5658b6bacd04bb1f794a6354e8175748  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/4
47a5d3e043938273acce80ab350c21c4  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=reed_sol_van/5

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/3
5658b6bacd04bb1f794a6354e8175748  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/4
47a5d3e043938273acce80ab350c21c4  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/5
c78ccd1ce82f1dd96b095c121f96ea06  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/6
45c4c71222daf899807ba631ae6eb76f  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=reed_sol_van/7

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/3
5658b6bacd04bb1f794a6354e8175748  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/4
ddcc23464d2f488e70014935c997db81  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_van/5

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/3
5658b6bacd04bb1f794a6354e8175748  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/4
86ee059a3df8f2669566c8304a5cc8f8  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/5
86b92da1ab9e005aebc0ac4d63f5cfdb  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/6
ddcc23464d2f488e70014935c997db81  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=reed_sol_van/7

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/3
712a4f87254bd9e9ec02baa2ccffd036  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/4
deb070c314393dc4165ce8590fc8c078  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=2 technique=cauchy/5

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/3
712a4f87254bd9e9ec02baa2ccffd036  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/4
deb070c314393dc4165ce8590fc8c078  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/5
593150d30e807bf31042b14b959620bc  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/6
be3502dfae3f2e6837a9ce1713a0339c  plugin_comparison/plugin=isa stripe-width=4096 k=4 m=4 technique=cauchy/7

3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/0
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/1
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/2
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/3
3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/4
3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=cauchy_good/5

3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/0
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/1
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/2
fcd6bcb56c1689fcef28b57c22475bad  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/3
3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/4
3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/5
3df0ce04ad3179750660b31851b85fee  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/6
313ded608d26d0498d797e5a473e75f4  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=4 technique=cauchy_good/7

fe53c731e7b451032f63caf80adb7d24  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/0
e88553b305019e4e341b18f3ef9868e9  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/1
e632bcd9e222023a59c77246a2d13596  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/2
ec78d0d453d82e71e0012024c4fd8f8b  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/3
5658b6bacd04bb1f794a6354e8175748  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/4
47a5d3e043938273acce80ab350c21c4  plugin_comparison/plugin=jerasure stripe-width=4096 k=4 m=2 technique=reed_sol_r6_op/5

@jamiepryde
Copy link
Contributor Author

jamiepryde commented Jul 11, 2024

jenkins test api

@jamiepryde jamiepryde self-assigned this Jul 12, 2024
@jamiepryde jamiepryde requested review from athanatos and markhpc July 19, 2024 12:50
@github-actions
Copy link

github-actions bot commented Sep 17, 2024

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Sep 17, 2024
@jamiepryde jamiepryde mentioned this pull request Oct 10, 2024
Merged
14 tasks
@github-actions
Copy link

github-actions bot commented Oct 17, 2024

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

@github-actions github-actions bot closed this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@athanatos athanatos Awaiting requested review from athanatos

@markhpc markhpc Awaiting requested review from markhpc

Assignees

@jamiepryde jamiepryde

Labels

core stale tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jamiepryde