Actions
Bug #69692
opentools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes
Status:
Pending Backport
Priority:
Urgent
Assignee:
Category:
fsck/damage handling
Target version:
% Done:
0%
Source:
Development
Backport:
squid,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, tools
Labels (FS):
crash
Pull request ID:
Tags (freeform):
backport_processed
Merge Commit:
Fixed In:
v19.3.0-7392-gaaee9588b9
Released In:
v20.2.0~1187
Upkeep Timestamp:
2025-11-01T01:00:53+00:00
Description
Credit to Venky for identifying this and creating a test case:
pdonnell@vossi04 ~/mnt/tmp.TPrEUEBexH$ mkdir -p dir_x/dir_xx/dir_xxx/ pdonnell@vossi04 ~/mnt/tmp.TPrEUEBexH$ touch dir_x/dir_xx/dir_xxx/file_y ... 7020 2025-01-28 15:28:43(source vstart_environment.sh && ceph tell mds.a:0 flush journal) 7021 2025-01-28 15:28:48(source vstart_environment.sh && ceph fs fail a) 7023 2025-01-28 15:29:29(source vstart_environment.sh && rados --pool cephfs.a.data ls) 7024 2025-01-28 15:29:44(source vstart_environment.sh && rados --pool cephfs.a.data getxattr 10000000003.00000000 parent | ceph-dencoder type inode_backtrace_t import - decode dump_json) 7025 2025-01-28 15:29:50(source vstart_environment.sh && rados --pool cephfs.a.meta ls) 7028 2025-01-28 15:30:29printf '%x\n' 1099511627777 7029 2025-01-28 15:30:42(source vstart_environment.sh && rados --pool cephfs.a.meta rm 10000000001.00000000) 7030 2025-01-28 15:30:46(source vstart_environment.sh && rados --pool cephfs.a.meta ls) 7031 2025-01-28 15:30:59(source vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 init --force-init) 7032 2025-01-28 15:31:08(source vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 scan_extents) 7033 2025-01-28 15:31:14(source vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 --debug_client=20 --debug_ms=1 --log_file=foo.log scan_inodes)
During scan_inodes step:
2025-01-28T15:31:15.043+0000 7f894fd28940 20 datascan.forall_objects: OSD matched oid 10000000003.00000000 2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.operator(): handling object 10000000003.0 2025-01-28T15:31:15.043+0000 7f894fd28940 1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6834/2564928471,v1:172.21.10.4:6835/2564928471] -- osd_op(unknown.0.0:159 5.b2 5:4def72d4:::10000000003.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380238800 con 0x560380204000 2025-01-28T15:31:15.043+0000 7f894ce4c640 1 -- 172.21.10.4:0/2200173965 <== osd.4 v2:172.21.10.4:6834/2564928471 23 ==== osd_op_reply(159 10000000003.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (158) out=158b,getxattr (30) out=30b,getxattr] v0'0 uv502 ondisk = 0) ==== 416+0+226 (crc 0 0 0) 0x560380200280 con 0x560380204000 2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.inject_with_backtrace: inode: 0x0x10000000003 2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.inject_with_backtrace: backptr: 0x0x10000000002/file_y 2025-01-28T15:31:15.043+0000 7f894fd28940 20 datascan.get_frag_of: dirino=0x10000000002 target_dname=file_y 2025-01-28T15:31:15.043+0000 7f894fd28940 1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:160 4.2 4:4bec3886:::10000000002.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380238c00 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894ce4c640 1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 12 ==== osd_op_reply(160 10000000002.00000000 [getxattr (126) out=126b] v0'0 uv72 ondisk = 0) ==== 164+0+126 (crc 0 0 0) 0x560380200280 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.get_frag_of: got backtrace parent 0x10000000001/dir_xxx 2025-01-28T15:31:15.044+0000 7f894fd28940 1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6810/1523345982,v1:172.21.10.4:6811/1523345982] -- osd_op(unknown.0.0:161 4.7 4:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239000 con 0x560380205800 2025-01-28T15:31:15.044+0000 7f894ce4c640 1 -- 172.21.10.4:0/2200173965 <== osd.1 v2:172.21.10.4:6810/1523345982 17 ==== osd_op_reply(161 10000000001.00000000 [omap-get-vals-by-keys] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x560380200280 con 0x560380205800 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.read_dentry: oid=10000000001.00000000 dname=dir_xxx frag=*, r=-2 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.inject_with_backtrace: don't know fragment for 0x0x10000000002/file_y, will insert to root 2025-01-28T15:31:15.044+0000 7f894fd28940 1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:162 4.2 4:4bec3886:::10000000002.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239400 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894ce4c640 1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 13 ==== osd_op_reply(162 10000000002.00000000 [omap-get-header out=274b] v0'0 uv72 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x560380200280 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894fd28940 10 datascan.find_or_create_dirfrag: read_version = 72 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000002 * 2025-01-28T15:31:15.044+0000 7f894fd28940 1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:163 4.2 4:4bec3886:::10000000002.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239800 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894ce4c640 1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 14 ==== osd_op_reply(163 10000000002.00000000 [omap-get-vals-by-keys out=523b] v0'0 uv72 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x560380200280 con 0x560380e11000 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.read_dentry: oid=10000000002.00000000 dname=file_y frag=*, r=0 2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.inject_with_backtrace: Dentry 0x0x10000000002/file_y already exists and points to me
It does not create the missing dirfrag found in the backtrace.
Updated by Patrick Donnelly about 1 year ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 61562
Updated by Patrick Donnelly about 1 year ago
- Status changed from Fix Under Review to Pending Backport
Updated by Upkeep Bot about 1 year ago
- Copied to Backport #69851: reef: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes added
Updated by Upkeep Bot about 1 year ago
- Copied to Backport #69852: squid: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes added
Updated by Upkeep Bot about 1 year ago
- Tags (freeform) set to backport_processed
Updated by Upkeep Bot 9 months ago
- Merge Commit set to aaee9588b91fd14295579e6185dd6dc895804165
- Fixed In set to v19.3.0-7392-gaaee9588b91
- Upkeep Timestamp set to 2025-07-08T18:12:11+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-7392-gaaee9588b91 to v19.3.0-7392-gaaee9588b91f
- Upkeep Timestamp changed from 2025-07-08T18:12:11+00:00 to 2025-07-14T15:22:24+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-7392-gaaee9588b91f to v19.3.0-7392-gaaee9588b9
- Upkeep Timestamp changed from 2025-07-14T15:22:24+00:00 to 2025-07-14T20:46:51+00:00
Updated by Upkeep Bot 5 months ago
- Released In set to v20.2.0~1187
- Upkeep Timestamp changed from 2025-07-14T20:46:51+00:00 to 2025-11-01T01:00:53+00:00
Actions