Actions
Bug #67972
openceph_test_rados - RadosModel.h: 1648: FAILED ceph_assert(omap.size() == old_value.attrs.size())
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Description
/a/nmordech-2024-09-09_04:25:13-crimson-rados-wip-nitzan-crimson-op_gate-alianstore-multi-core-distro-crimson-smithi/7895936
the assertion happened during branch test, and didn't occur on main or on the same branch test after running the failing job for 15 times.
2024-09-09T04:55:49.581 INFO:tasks.rados.rados.0.smithi019.stderr:1640: oid 478 omap.size() is 20 and old is 11 2024-09-09T04:55:49.581 INFO:tasks.rados.rados.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4430-gd14d87b0/rpm/el9/BUILD/ceph-19.3.0-4430-gd14d87b0/src/test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f03af7fe640 time 2024-09-09T04:55:49.582990+0000 2024-09-09T04:55:49.581 INFO:tasks.rados.rados.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4430-gd14d87b0/rpm/el9/BUILD/ceph-19.3.0-4430-gd14d87b0/src/test/osd/RadosModel.h: 1648: FAILED ceph_assert(omap.size() == old_value.attrs.size()) 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: ceph version 19.3.0-4430-gd14d87b0 (d14d87b085ea4e06a54df2d361289915d1a0c5b2) squid (dev) 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x129) [0x7f03c42775ad] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f03c42777f3] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 3: (ReadOp::_finish(TestOp::CallbackInfo*)+0xc9d) [0x452673] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 4: (TestOp::finish(TestOp::CallbackInfo*)+0xe) [0x46138c] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 5: (read_callback(void*, void*)+0x15) [0x4613a6] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 6: /lib64/librados.so.2(+0x98b94) [0x7f03c4ca7b94] 2024-09-09T04:55:49.582 INFO:tasks.rados.rados.0.smithi019.stderr: 7: /lib64/librados.so.2(+0xa12ac) [0x7f03c4cb02ac] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 8: /lib64/librados.so.2(+0x9705d) [0x7f03c4ca605d] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 9: /lib64/librados.so.2(+0xa595a) [0x7f03c4cb495a] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 10: /lib64/librados.so.2(+0xa6267) [0x7f03c4cb5267] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 11: /lib64/librados.so.2(+0xb2583) [0x7f03c4cc1583] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 12: /lib64/librados.so.2(+0xb2d89) [0x7f03c4cc1d89] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 13: /lib64/librados.so.2(+0xb5319) [0x7f03c4cc4319] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 14: /lib64/librados.so.2(+0xb5392) [0x7f03c4cc4392] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 15: /lib64/librados.so.2(+0xb53af) [0x7f03c4cc43af] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 16: /lib64/libstdc++.so.6(+0xdbad4) [0x7f03c4edbad4] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 17: /lib64/libc.so.6(+0x89c52) [0x7f03c3089c52] 2024-09-09T04:55:49.583 INFO:tasks.rados.rados.0.smithi019.stderr: 18: /lib64/libc.so.6(+0x10ec80) [0x7f03c310ec80]
Updated by Radoslaw Zarzynski over 1 year ago
Hi Nitzan! Could you please clarify whether the problem
affects the classical OSD? Or is it just crimson (per the
names in the QA branch name)?
Updated by Nitzan Mordechai over 1 year ago
@Radoslaw Zarzynski it only happened once on a test branch, and on crimson, I couldn't recreate it locally or with a rerun of the test on the same branch, main or other branches. i think we need to leave it for now
Updated by Radoslaw Zarzynski over 1 year ago
- Project changed from RADOS to crimson
Thanks for info, @Nitzan Mordechai!
I'm moving this to crimson.
Updated by MOHIT AGRAWAL over 1 year ago
failed ceph_assert (ceph_assert(omap.size() == old_value.attrs.size())
https://pulpito.ceph.com/moagrawa-2024-11-21_08:55:14-crimson-rados-wip-mohit-crimson-device_class-distro-crimson-smithi/8003283/
Updated by Yingxin Cheng over 1 year ago
crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized}
Updated by Matan Breizman 12 months ago
- Tags set to frequent-failure
- Priority changed from Normal to High
Updated by Nitzan Mordechai 12 months ago
@Matan Breizman i think it will be better to reassign, it will take a while until I'll touch it
Updated by Matan Breizman 10 months ago
- Priority changed from High to Normal
Lowering prio due to frequency
Updated by Matan Breizman 3 months ago
2025-12-07T12:22:39.762 INFO:tasks.rados.rados.0.smithi084.stdout:1780: writing smithi08437057-429 from 3351151 to 3798679 tid 3 2025-12-07T12:22:39.762 INFO:tasks.rados.rados.0.smithi084.stdout:1781: delete oid 566 current snap is 101 2025-12-07T12:22:39.762 INFO:tasks.rados.rados.0.smithi084.stdout:1772: expect (ObjNum 806 snap 0 seq_num 806) 2025-12-07T12:22:39.793 INFO:tasks.rados.rados.0.smithi084.stdout:seeking to 0 2025-12-07T12:22:39.878 INFO:tasks.rados.rados.0.smithi084.stdout:1771: expect (ObjNum 1030 snap 0 seq_num 1030) 2025-12-07T12:22:39.896 INFO:tasks.rados.rados.0.smithi084.stdout:seeking to 2075032 2025-12-07T12:22:39.992 INFO:tasks.rados.rados.0.smithi084.stdout:1774: expect (ObjNum 477 snap 0 seq_num 477) 2025-12-07T12:22:40.048 INFO:tasks.rados.rados.0.smithi084.stderr:1774: oid 478 omap.size() is 20 and old is 11 2025-12-07T12:22:40.048 INFO:tasks.rados.rados.0.smithi084.stderr:/ceph/rpmbuild/BUILD/ceph-20.3.0-4403-g537465ab/src/test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f4bdeffd640 time 2025-12-07T12:22:40.046317+0000 2025-12-07T12:22:40.048 INFO:tasks.rados.rados.0.smithi084.stderr:/ceph/rpmbuild/BUILD/ceph-20.3.0-4403-g537465ab/src/test/osd/RadosModel.h: 1683: FAILED ceph_assert(omap.size() == old_value.attrs.size()) 2025-12-07T12:22:40.048 INFO:tasks.rados.rados.0.smithi084.stderr: ceph version 20.3.0-4403-g537465ab (537465aba0e583c1a3a805746642e7be35504d74) tentacle (dev - Debug) 2025-12-07T12:22:40.048 INFO:tasks.rados.rados.0.smithi084.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f4beb22028a] 2025-12-07T12:22:40.049 INFO:tasks.rados.rados.0.smithi084.stderr: 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x1b) [0x7f4beb220597] 2025-12-07T12:22:40.049 INFO:tasks.rados.rados.0.smithi084.stderr: 3: ceph_test_rados() [0x42bab1] 2025-12-07T12:22:40.049 INFO:tasks.rados.rados.0.smithi084.stderr: 4: ceph_test_rados() [0x41ac10] 2025-12-07T12:22:40.049 INFO:tasks.rados.rados.0.smithi084.stderr: 5: (TestOp::finish(TestOp::CallbackInfo*)+0xe) [0x435478]
Actions