Project

General

Profile

Actions
Copy link

Bug #70691

open

FAILED ceph_assert(r == 0) in TestLibRBD.ConcurrentOperations

Added by Casey Bodley 12 months ago. Updated 23 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Tags:
make-check-failure

Description

from make check (arm64) https://jenkins.ceph.com/job/ceph-pull-requests-arm64/71049/

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread fdb32740e4e0 time 2025-03-27T16:17:21.453279+0000
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: 13598: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x150) [0xfdb3d263feb4]
 2: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0) [0xfdb3d263fd64]
 3: unittest_librbd(+0x3403670) [0xc2c7e0d03670]
 4: unittest_librbd(+0x340361c) [0xc2c7e0d0361c]
 5: unittest_librbd(+0x34035d0) [0xc2c7e0d035d0]
 6: unittest_librbd(+0x34035a8) [0xc2c7e0d035a8]
 7: unittest_librbd(+0x340357c) [0xc2c7e0d0357c]
 8: unittest_librbd(+0x34034d0) [0xc2c7e0d034d0]
 9: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xd31fc) [0xfdb3d08d31fc]
 10: /lib/aarch64-linux-gnu/libc.so.6(+0x7d5b8) [0xfdb3d06cd5b8]
 11: /lib/aarch64-linux-gnu/libc.so.6(+0xe5edc) [0xfdb3d0735edc]
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/run-rbd-unit-tests.sh: line 20: 735516 Aborted                 (core dumped)

Related issues 1 (0 open — 1 closed)

Related to rbd - Bug #75163: disable known flaky tests in run-rbd-unit-testsResolvedIlya Dryomov

Actions
Actions
Copy link
 #1

Updated by Casey Bodley 12 months ago

  • Subject changed from segfault in TestLibRBD.ConcurrentOperations (arm64?) to FAILED ceph_assert(r == 0) in TestLibRBD.ConcurrentOperations

not arm: https://jenkins.ceph.com/job/ceph-pull-requests/155528/

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 7f2cbdffb640 time 2025-04-04T13:09:09.171952+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: 13593: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f301b6765d8]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x7f301b676b4f]
 3: unittest_librbd(+0x3c34ac0) [0x564b779acac0]
 4: unittest_librbd(+0x3c34aa1) [0x564b779acaa1]
 5: unittest_librbd(+0x3c34a78) [0x564b779aca78]
 6: unittest_librbd(+0x3c34a35) [0x564b779aca35]
 7: unittest_librbd(+0x3c349f5) [0x564b779ac9f5]
 8: unittest_librbd(+0x3c349cd) [0x564b779ac9cd]
 9: unittest_librbd(+0x3c349a5) [0x564b779ac9a5]
 10: unittest_librbd(+0x3c34909) [0x564b779ac909]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f301979a253]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3019421b43]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f30194b3a00]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh: line 20: 3583755 Aborted                 RBD_FEATURES=$i unittest_librbd

Actions
Copy link
 #2

Updated by Casey Bodley 12 months ago

https://jenkins.ceph.com/job/ceph-pull-requests/155532/

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 7ff4ce7fc640 time 2025-04-04T15:20:39.082785+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: 13593: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7ff8b816d5d8]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x7ff8b816db4f]
 3: unittest_librbd(+0x3c34ac0) [0x562d8686bac0]
 4: unittest_librbd(+0x3c34aa1) [0x562d8686baa1]
 5: unittest_librbd(+0x3c34a78) [0x562d8686ba78]
 6: unittest_librbd(+0x3c34a35) [0x562d8686ba35]
 7: unittest_librbd(+0x3c349f5) [0x562d8686b9f5]
 8: unittest_librbd(+0x3c349cd) [0x562d8686b9cd]
 9: unittest_librbd(+0x3c349a5) [0x562d8686b9a5]
 10: unittest_librbd(+0x3c34909) [0x562d8686b909]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe62b3) [0x7ff8b625a2b3]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7ff8b5ed2ac3]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7ff8b5f64850]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh: line 20: 1885619 Aborted                 RBD_FEATURES=$i unittest_librbd

Actions
Copy link
 #3

Updated by Casey Bodley 11 months ago

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72190/

34 - run-rbd-unit-tests-127.sh (Failed)

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread eee00e00e500 time 2025-04-14T05:55:50.102906+0000
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: 13598: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x150) [0xeee0ba878848]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x34) [0xeee0ba878d00]
 3: unittest_librbd(+0x3b9c5f8) [0xbea6f2b1c5f8]
 4: unittest_librbd(+0x3b9c5d4) [0xbea6f2b1c5d4]
 5: unittest_librbd(+0x3b9c59c) [0xbea6f2b1c59c]
 6: unittest_librbd(+0x3b9c550) [0xbea6f2b1c550]
 7: unittest_librbd(+0x3b9c504) [0xbea6f2b1c504]
 8: unittest_librbd(+0x3b9c4dc) [0xbea6f2b1c4dc]
 9: unittest_librbd(+0x3b9c4b0) [0xbea6f2b1c4b0]
 10: unittest_librbd(+0x3b9c404) [0xbea6f2b1c404]
 11: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xd31fc) [0xeee0b88d31fc]
 12: /lib/aarch64-linux-gnu/libc.so.6(+0x7d5b8) [0xeee0b862d5b8]
 13: /lib/aarch64-linux-gnu/libc.so.6(+0xe5edc) [0xeee0b8695edc]
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/run-rbd-unit-tests.sh: line 20: 3455640 Aborted                 (core dumped) RBD_FEATURES=$i unittest_librbd
Actions
Copy link
 #4

Updated by Casey Bodley 9 months ago

https://jenkins.ceph.com/job/ceph-pull-requests/160006/

34 - run-rbd-unit-tests-127.sh (Failed)

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 7f9858ff9640 time 2025-06-07T14:27:12.377244+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: 13598: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f99a996fd08]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x7f99a997027f]
 3: unittest_librbd(+0x3c4c030) [0x561c0f11b030]
 4: unittest_librbd(+0x3c4c011) [0x561c0f11b011]
 5: unittest_librbd(+0x3c4bfe8) [0x561c0f11afe8]
 6: unittest_librbd(+0x3c4bfa5) [0x561c0f11afa5]
 7: unittest_librbd(+0x3c4bf65) [0x561c0f11af65]
 8: unittest_librbd(+0x3c4bf3d) [0x561c0f11af3d]
 9: unittest_librbd(+0x3c4bf15) [0x561c0f11af15]
 10: unittest_librbd(+0x3c4be79) [0x561c0f11ae79]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f99a7a7d253]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f99a7706b43]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f99a7798a00]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh: line 20: 1316394 Aborted                 RBD_FEATURES=$i unittest_librbd
Actions
Copy link
 #5

Updated by Bill Scales 7 months ago

make checks
https://jenkins.ceph.com/job/ceph-pull-requests/164801/console
32 - run-rbd-unit-tests-61.sh (Failed)

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 7f9a20ff9640 time 2025-08-13T10:38:41.391782+0000
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/test_librbd.cc: 13416: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9cf98adf78]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x7f9cf98ae4ef]
 3: unittest_librbd(+0x3b09500) [0x55aa5c0cb500]
 4: unittest_librbd(+0x3b094e1) [0x55aa5c0cb4e1]
 5: unittest_librbd(+0x3b094b8) [0x55aa5c0cb4b8]
 6: unittest_librbd(+0x3b09475) [0x55aa5c0cb475]
 7: unittest_librbd(+0x3b09435) [0x55aa5c0cb435]
 8: unittest_librbd(+0x3b0940d) [0x55aa5c0cb40d]
 9: unittest_librbd(+0x3b093e5) [0x55aa5c0cb3e5]
 10: unittest_librbd(+0x3b09349) [0x55aa5c0cb349]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe62b3) [0x7f9cf793a2b3]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f9cf75b2ac3]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f9cf7644850]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh: line 20: 2630708 Aborted                 RBD_FEATURES=$i unittest_librbd

Actions
Copy link
 #6

Updated by Casey Bodley 6 months ago

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/81060/

The following tests FAILED:
     32 - run-rbd-unit-tests-61.sh (Timeout)
     34 - run-rbd-unit-tests-127.sh (Failed)


[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread f5b6d8f9e500 time 2025-09-16T18:55:13.741300+0000
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/test_librbd.cc: 13414: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x150) [0xf5b952d2e074]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x34) [0xf5b952d2e52c]
 3: unittest_librbd(+0x3a7f0f0) [0xc029b281f0f0]
 4: unittest_librbd(+0x3a7f0cc) [0xc029b281f0cc]
 5: unittest_librbd(+0x3a7f094) [0xc029b281f094]
 6: unittest_librbd(+0x3a7f048) [0xc029b281f048]
 7: unittest_librbd(+0x3a7effc) [0xc029b281effc]
 8: unittest_librbd(+0x3a7efd4) [0xc029b281efd4]
 9: unittest_librbd(+0x3a7efa8) [0xc029b281efa8]
 10: unittest_librbd(+0x3a7eefc) [0xc029b281eefc]
 11: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xd31fc) [0xf5b950ca31fc]
 12: /lib/aarch64-linux-gnu/libc.so.6(+0x7d5b8) [0xf5b9509cd5b8]
 13: /lib/aarch64-linux-gnu/libc.so.6(+0xe5edc) [0xf5b950a35edc]
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/run-rbd-unit-tests.sh: line 20: 3521572 Aborted                 (core dumped) RBD_FEATURES=$i unittest_librbd


319/319 Test  #32: run-rbd-unit-tests-61.sh ..................***Timeout 7200.08 sec
Actions
Copy link
 #7

Updated by Casey Bodley 5 months ago

https://jenkins.ceph.com/job/ceph-pull-requests/168284/

The following tests FAILED:
     32 - run-rbd-unit-tests-61.sh (Timeout)


[       OK ] TestLibRBD.WriteZeroesThickProvision (485 ms)
[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!

99% tests passed, 1 tests failed out of 320

Actions
Copy link
 #8

Updated by Bill Scales 5 months ago

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/82291/

/ceph/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread e3627fffe4e0 time 2025-10-16T18:18:37.262953+0000
/ceph/src/test/librbd/test_librbd.cc: 13420: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x150) [0xe36495f7535c]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x34) [0xe36495f75814]
 3: unittest_librbd(+0x3a7f400) [0xb40d4574f400]
 4: unittest_librbd(+0x3a7f3dc) [0xb40d4574f3dc]
 5: unittest_librbd(+0x3a7f3a4) [0xb40d4574f3a4]
 6: unittest_librbd(+0x3a7f358) [0xb40d4574f358]
 7: unittest_librbd(+0x3a7f30c) [0xb40d4574f30c]
 8: unittest_librbd(+0x3a7f2e4) [0xb40d4574f2e4]
 9: unittest_librbd(+0x3a7f2b8) [0xb40d4574f2b8]
 10: unittest_librbd(+0x3a7f20c) [0xb40d4574f20c]
 11: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xd29cc) [0xe36493ef29cc]
 12: /lib/aarch64-linux-gnu/libc.so.6(+0x80398) [0xe36493c20398]
 13: /lib/aarch64-linux-gnu/libc.so.6(+0xe9e9c) [0xe36493c89e9c]
/ceph/src/test/run-rbd-unit-tests.sh: line 20:   914 Aborted                 RBD_FEATURES=$i unittest_librbd

Actions
Copy link
 #9

Updated by Kefu Chai 3 months ago 

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/ceph/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 7035d57fa640 time 2025-12-23T05:26:15.634822+0000
/ceph/src/test/librbd/test_librbd.cc: 13415: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (stable - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x703d4ec7ec78]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x703d4ec7f1ef]
 3: unittest_librbd(+0x3c5d490) [0x5628901b9490]
 4: unittest_librbd(+0x3c5d471) [0x5628901b9471]
 5: unittest_librbd(+0x3c5d448) [0x5628901b9448]
 6: unittest_librbd(+0x3c5d405) [0x5628901b9405]
 7: unittest_librbd(+0x3c5d3c5) [0x5628901b93c5]
 8: unittest_librbd(+0x3c5d39d) [0x5628901b939d]
 9: unittest_librbd(+0x3c5d375) [0x5628901b9375]
 10: unittest_librbd(+0x3c5d2d9) [0x5628901b92d9]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x703d4ccdc253]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x703d4c894ac3]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x703d4c9268c0]
/ceph/src/test/run-rbd-unit-tests.sh: line 20:   965 Aborted                 (core dumped) RBD_FEATURES=$i unittest_librbd

see https://jenkins.ceph.com/job/ceph-pull-requests/172055/console

Actions
Copy link
 #10

Updated by Kefu Chai 3 months ago

created https://github.com/ceph/ceph/pull/66729 in hope to fix it.

Actions
Copy link
 #11

Updated by Bill Scales 29 days ago

jenkins.ceph.com/job/ceph-pull-requests/174765/console (make checks, main)

[ RUN      ] TestLibRBD.ConcurrentOperations
using new format!
/ceph/src/test/librbd/test_librbd.cc: In function 'auto TestLibRBD_ConcurrentOperations_Test::TestBody()::(anonymous class)::operator()() const' thread 73f5417fa640 time 2026-02-19T14:25:15.743842+0000
/ceph/src/test/librbd/test_librbd.cc: 13685: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x73fc92136638]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x2f) [0x73fc92136baf]
 3: unittest_librbd(+0x3b1eb70) [0x617067007b70]
 4: unittest_librbd(+0x3b1eb51) [0x617067007b51]
 5: unittest_librbd(+0x3b1eb28) [0x617067007b28]
 6: unittest_librbd(+0x3b1eae5) [0x617067007ae5]
 7: unittest_librbd(+0x3b1eaa5) [0x617067007aa5]
 8: unittest_librbd(+0x3b1ea7d) [0x617067007a7d]
 9: unittest_librbd(+0x3b1ea55) [0x617067007a55]
 10: unittest_librbd(+0x3b1e9b9) [0x6170670079b9]
 11: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x73fc900dc253]
 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x73fc8fc94ac3]
 13: /lib/x86_64-linux-gnu/libc.so.6(+0x1268d0) [0x73fc8fd268d0]
/ceph/src/test/run-rbd-unit-tests.sh: line 20:   956 Aborted                 (core dumped) RBD_FEATURES=$i unittest_librbd
Actions
Copy link
 #12

Updated by Ilya Dryomov 24 days ago

  • Tags set to make-check-failure
Actions
Copy link
 #13

Updated by Ilya Dryomov 24 days ago

  • Related to Bug #75163: disable known flaky tests in run-rbd-unit-tests added
Actions
Copy link
 #14

Updated by Ilya Dryomov 23 days ago

Disabled in "make check" context via https://github.com/ceph/ceph/pull/67525.

Actions
Copy link

Also available in: Atom PDF