Project

General

Profile

Actions

Bug #69692

open

tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes

Added by Patrick Donnelly about 1 year ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
Urgent
Category:
fsck/damage handling
Target version:
% Done:

0%

Source:
Development
Backport:
squid,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, tools
Labels (FS):
crash
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v19.3.0-7392-gaaee9588b9
Released In:
v20.2.0~1187
Upkeep Timestamp:
2025-11-01T01:00:53+00:00

Description

Credit to Venky for identifying this and creating a test case:

pdonnell@vossi04 ~/mnt/tmp.TPrEUEBexH$ mkdir -p dir_x/dir_xx/dir_xxx/
pdonnell@vossi04 ~/mnt/tmp.TPrEUEBexH$ touch dir_x/dir_xx/dir_xxx/file_y
...
 7020  2025-01-28 15:28:43(source  vstart_environment.sh && ceph tell mds.a:0 flush journal)  
 7021  2025-01-28 15:28:48(source  vstart_environment.sh && ceph fs fail a)  
 7023  2025-01-28 15:29:29(source  vstart_environment.sh && rados --pool cephfs.a.data ls) 
 7024  2025-01-28 15:29:44(source vstart_environment.sh && rados --pool cephfs.a.data getxattr 10000000003.00000000 parent | ceph-dencoder type inode_backtrace_t import - decode dump_json) 
 7025  2025-01-28 15:29:50(source  vstart_environment.sh && rados --pool cephfs.a.meta ls) 
 7028  2025-01-28 15:30:29printf '%x\n' 1099511627777
 7029  2025-01-28 15:30:42(source  vstart_environment.sh && rados --pool cephfs.a.meta rm 10000000001.00000000) 
 7030  2025-01-28 15:30:46(source  vstart_environment.sh && rados --pool cephfs.a.meta ls) 
 7031  2025-01-28 15:30:59(source  vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 init --force-init)  
 7032  2025-01-28 15:31:08(source  vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 scan_extents)  
 7033  2025-01-28 15:31:14(source  vstart_environment.sh && bin/cephfs-data-scan --debug_mds=20 --debug_client=20 --debug_ms=1 --log_file=foo.log scan_inodes)  

During scan_inodes step:

2025-01-28T15:31:15.043+0000 7f894fd28940 20 datascan.forall_objects: OSD matched oid 10000000003.00000000
2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.operator(): handling object 10000000003.0
2025-01-28T15:31:15.043+0000 7f894fd28940  1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6834/2564928471,v1:172.21.10.4:6835/2564928471] -- osd_op(unknown.0.0:159 5.b2 5:4def72d4:::10000000003.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380238800 con 0x560380204000
2025-01-28T15:31:15.043+0000 7f894ce4c640  1 -- 172.21.10.4:0/2200173965 <== osd.4 v2:172.21.10.4:6834/2564928471 23 ==== osd_op_reply(159 10000000003.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (158) out=158b,getxattr (30) out=30b,getxattr] v0'0 uv502 ondisk = 0) ==== 416+0+226 (crc 0 0 0) 0x560380200280 con 0x560380204000
2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.inject_with_backtrace:   inode: 0x0x10000000003
2025-01-28T15:31:15.043+0000 7f894fd28940 10 datascan.inject_with_backtrace:   backptr: 0x0x10000000002/file_y
2025-01-28T15:31:15.043+0000 7f894fd28940 20 datascan.get_frag_of: dirino=0x10000000002 target_dname=file_y
2025-01-28T15:31:15.043+0000 7f894fd28940  1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:160 4.2 4:4bec3886:::10000000002.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380238c00 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894ce4c640  1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 12 ==== osd_op_reply(160 10000000002.00000000 [getxattr (126) out=126b] v0'0 uv72 ondisk = 0) ==== 164+0+126 (crc 0 0 0) 0x560380200280 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.get_frag_of: got backtrace parent 0x10000000001/dir_xxx
2025-01-28T15:31:15.044+0000 7f894fd28940  1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6810/1523345982,v1:172.21.10.4:6811/1523345982] -- osd_op(unknown.0.0:161 4.7 4:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239000 con 0x560380205800
2025-01-28T15:31:15.044+0000 7f894ce4c640  1 -- 172.21.10.4:0/2200173965 <== osd.1 v2:172.21.10.4:6810/1523345982 17 ==== osd_op_reply(161 10000000001.00000000 [omap-get-vals-by-keys] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x560380200280 con 0x560380205800
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.read_dentry: oid=10000000001.00000000 dname=dir_xxx frag=*, r=-2
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.inject_with_backtrace: don't know fragment for 0x0x10000000002/file_y, will insert to root
2025-01-28T15:31:15.044+0000 7f894fd28940  1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:162 4.2 4:4bec3886:::10000000002.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239400 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894ce4c640  1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 13 ==== osd_op_reply(162 10000000002.00000000 [omap-get-header out=274b] v0'0 uv72 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x560380200280 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894fd28940 10 datascan.find_or_create_dirfrag: read_version = 72
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000002 *
2025-01-28T15:31:15.044+0000 7f894fd28940  1 -- 172.21.10.4:0/2200173965 --> [v2:172.21.10.4:6850/23978288,v1:172.21.10.4:6851/23978288] -- osd_op(unknown.0.0:163 4.2 4:4bec3886:::10000000002.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131) -- 0x560380239800 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894ce4c640  1 -- 172.21.10.4:0/2200173965 <== osd.6 v2:172.21.10.4:6850/23978288 14 ==== osd_op_reply(163 10000000002.00000000 [omap-get-vals-by-keys out=523b] v0'0 uv72 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x560380200280 con 0x560380e11000
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.read_dentry: oid=10000000002.00000000 dname=file_y frag=*, r=0
2025-01-28T15:31:15.044+0000 7f894fd28940 20 datascan.inject_with_backtrace: Dentry 0x0x10000000002/file_y already exists and points to me

It does not create the missing dirfrag found in the backtrace.


Related issues 2 (2 open0 closed)

Copied to CephFS - Backport #69851: reef: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodesIn ProgressPatrick DonnellyActions
Copied to CephFS - Backport #69852: squid: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodesIn ProgressPatrick DonnellyActions
Actions #1

Updated by Patrick Donnelly about 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 61562
Actions #2

Updated by Patrick Donnelly about 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions #3

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69851: reef: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes added
Actions #4

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69852: squid: tools/cephfs/DataScan: missing dirfrag in backtrace not created during scan_inodes added
Actions #5

Updated by Upkeep Bot about 1 year ago

  • Tags (freeform) set to backport_processed
Actions #6

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to aaee9588b91fd14295579e6185dd6dc895804165
  • Fixed In set to v19.3.0-7392-gaaee9588b91
  • Upkeep Timestamp set to 2025-07-08T18:12:11+00:00
Actions #7

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-7392-gaaee9588b91 to v19.3.0-7392-gaaee9588b91f
  • Upkeep Timestamp changed from 2025-07-08T18:12:11+00:00 to 2025-07-14T15:22:24+00:00
Actions #8

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-7392-gaaee9588b91f to v19.3.0-7392-gaaee9588b9
  • Upkeep Timestamp changed from 2025-07-14T15:22:24+00:00 to 2025-07-14T20:46:51+00:00
Actions #9

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~1187
  • Upkeep Timestamp changed from 2025-07-14T20:46:51+00:00 to 2025-11-01T01:00:53+00:00
Actions

Also available in: Atom PDF