BTW, do you have an s390x teuthology facility at RH?
Is there any way to setup it at your side?
We have here a small integration test cluster, nevertheless would be great to have a kind of an official test facility :)

@idryomov , thank you!!!

BTW, do you have an s390x teuthology facility at RH? Is there any way to setup it at your side? We have here a small integration test cluster, nevertheless would be great to have a kind of an official test facility :)

We certainly don't have hardware like that in the upstream lab, which is what upstream contributions go through. So we would rely on you to run all the necessary tests and benchmarks and post the results here in the PR ;)

https://github.com/linux-on-ibm-z/crc32-s390x/blob/master/README.md notes a 70x speedup in a synthetic benchmark. Most of that would obviously evaporate in an end-to-end integration test with the rest of Ceph in place, but I'd be curious to see pre and post numbers for e.g. bufferlist::crc32c().

BTW, do you have an s390x teuthology facility at RH? Is there any way to setup it at your side? We have here a small integration test cluster, nevertheless would be great to have a kind of an official test facility :)

We certainly don't have hardware like that in the upstream lab, which is what upstream contributions go through. So we would rely on you to run all the necessary tests and benchmarks and post the results here in the PR ;)

https://github.com/linux-on-ibm-z/crc32-s390x/blob/master/README.md notes a 70x speedup in a synthetic benchmark. Most of that would obviously evaporate in an end-to-end integration test with the rest of Ceph in place, but I'd be curious to see pre and post numbers for e.g. bufferlist::crc32c().

Thank you very much for the clarification!
With the HW crc functions we noticed a huge use of the CPU (fio was employed to load the OSD):

We have a small homebrew teuthology cluster on the s390x nodes. Let me reanimate it and test.

With the HW crc functions we noticed a huge use of the CPU

I'm confused -- did you mean "without" or perhaps "SW" instead of "HW" here?

With the HW crc functions we noticed a huge use of the CPU

I'm confused -- did you mean "without" or perhaps "SW" instead of "HW" here?

Without of course. My typo.

Hi Team,
I manually run teuthology tests @s390x nodes and had no problems.

fio -ioengine=libaio -direct=1 -name=test -group_reporting=1 -runtime=180 -time_based -iodepth=8 -bs=128K -rw=write -numjobs=8 -size=3G

ceph_crc32c_sctp

Samples: 1M of event 'cycles:P', Event count (approx.): 2211775104000
Overhead  Command          Shared Object        Symbol
  19.58%  tp_osd_tp        ceph-osd             [.] ceph_crc32c_sctp
  18.78%  msgr-worker-1    ceph-osd             [.] ceph_crc32c_sctp
   5.88%  msgr-worker-1    [kernel.kallsyms]    [k] raw_copy_to_user_key
   3.48%  tp_osd_tp        libc.so.6            [.] memcpy
   3.04%  tp_osd_tp        libc.so.6            [.] _int_malloc
   2.20%  tp_osd_tp        [kernel.kallsyms]    [k] cpumsf_pmu_enable
   1.19%  tp_osd_tp        libc.so.6            [.] malloc
   1.13%  tp_osd_tp        libc.so.6            [.] malloc_consolidate
   0.88%  tp_osd_tp        ceph-osd             [.] PeeringState::calc_trim_to_aggressive()
   0.87%  bstore_aio       [kernel.kallsyms]    [k] cpumsf_pmu_enable
   0.75%  tp_osd_tp        libc.so.6            [.] cfree@GLIBC_2.2
   0.73%  bstore_kv_sync   [kernel.kallsyms]    [k] cpumsf_pmu_enable
   0.69%  tp_osd_tp        libc.so.6            [.] _int_free
   0.61%  tp_osd_tp        ceph-osd             [.] PGLog::IndexedLog::trim(ceph::common::CephContext*, eversion_t, std::set<eversion_t, std::less<eversion_t>, std::allo

s390x HW crc32

Samples: 874K of event 'cycles:P', Event count (approx.): 1138476612000
Overhead  Command          Shared Object         Symbol
   8.58%  msgr-worker-1    [kernel.kallsyms]     [k] raw_copy_to_user_key
   5.27%  tp_osd_tp        libc.so.6             [.] memcpy
   4.70%  tp_osd_tp        libc.so.6             [.] _int_malloc
   3.03%  tp_osd_tp        [kernel.kallsyms]     [k] cpumsf_pmu_enable
   1.97%  tp_osd_tp        ceph-osd              [.] crc32_le_vgfm_generic
   1.75%  tp_osd_tp        libc.so.6             [.] malloc_consolidate
   1.72%  tp_osd_tp        libc.so.6             [.] malloc
   1.64%  msgr-worker-1    ceph-osd              [.] crc32_le_vgfm_generic
   1.43%  msgr-worker-1    [kernel.kallsyms]     [k] cpumsf_pmu_enable
   1.31%  tp_osd_tp        ceph-osd              [.] PeeringState::calc_trim_to_aggressive()
   1.21%  bstore_aio       [kernel.kallsyms]     [k] cpumsf_pmu_enable
   1.15%  tp_osd_tp        libc.so.6             [.] cfree@GLIBC_2.2
   1.12%  tp_osd_tp        libc.so.6             [.] unlink_chunk.isra.0
   1.02%  tp_osd_tp        libc.so.6             [.] _int_free
   0.86%  tp_osd_tp        ceph-osd              [.] PGLog::IndexedLog::trim(ceph::common::CephContext*, eversion_t, std::set<eversion_t, std::less<eversion_t>, std::allocator<eversion_t> >*, std::set<std
   0.71%  bstore_kv_sync   [kernel.kallsyms]     [k] cpumsf_pmu_enable
   0.70%  bstore_kv_sync   ceph-osd              [.] unsigned int rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Slow_CRC32>(unsigned int, char const*, unsigned long)
   0.66%  bstore_kv_sync   ceph-osd              [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::RecomputeSpliceLevels(rocksdb::Slice const&, rocksdb::InlineSkipList<rocksdb::MemT
   0.64%  msgr-worker-1    libc.so.6             [.] _int_malloc
   0.61%  bstore_kv_sync   libc.so.6             [.] memcpy

@idryomov @cbodley ping :-)

@idryomov @cbodley ping :-)

See the unresolved conversation above and please add Fixes: https://tracker.ceph.com/issues/64952 line like you have in the PR description to the commit message.

@idryomov @cbodley ping :-)

See the unresolved conversation above and please add Fixes: https://tracker.ceph.com/issues/64952 line like you have in the PR description to the commit message.

Done!

jenkins test make check

Hi Casey @cbodley ,
could you please have a look? Thank you very much!!!

my comments were addressed, thanks 👍 this pr already has approvals, so i don't plan to review further

@idryomov @athanatos this is labeled needs-qa but it's not clear who's going to test it. should we add the core label to go through the rados suite?

@cbodley I assumed that the common label was enough for it to be picked up into a rados batch. I went ahead and added the core label, thanks!

Rados approved: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrackercephcomissues67027

PTAL #59809