Ceph » CephFS

What’s the folder structure look like on this file system, Dan? Eg depth and number of files per folder?

Unless things have changed, we do a depth-first search and we pin every dentry in the ancestor folders along the way. I don’t remember why exactly but it was necessary for the scrubbing guarantees to hold. :/

Looks like that somehow adds up to 4million inodes, which is possible but definitely surprises me — usually the very large folders are leaf nodes.

Or there’s something completely different happening here. (I also remember flagging the pins as a potential problem so someone may have made a change to work around that which I missed — there were a lot of scrub changes for a couple years.)

Greg Farnum wrote in #note-3:

What’s the folder structure look like on this file system, Dan? Eg depth and number of files per folder?

Unless things have changed, we do a depth-first search and we pin every dentry in the ancestor folders along the way. I don’t remember why exactly but it was necessary for the scrubbing guarantees to hold. :/

That definitely changed to breadth-first search to support multi-mds scrub. See: https://github.com/ceph/ceph/commit/b43af152bab2c9f67fe311ba9450e06fd41e82e4

Looks like that somehow adds up to 4million inodes, which is possible but definitely surprises me — usually the very large folders are leaf nodes.

I don't think its possible to add upto 4M pinned inodes with BFS traversal, unless, the an entire level is kept pinned till scrub moved to the next level. I will have to lookup the scrub code for what's done.

Greg Farnum wrote in #note-3:

What’s the folder structure look like on this file system, Dan? Eg depth and number of files per folder?

sorry, i don't have any insight like that. it was an FS with a bunch of k8s pvc's managed as cephfs subvolumes. I don't know how deep/broad it was.

It is possible to pinning the whole level,

generate a file-system provided below and make mds_max_scrub_ops_in_progress = 1

/ (root)
├── dir00001/
│   ├── subdir00001/
│   ├── subdir00002/
│   ├── subdir00003/
│   ├── ...
│   └── subdir10000/
├── dir00002/
│   ├── subdir00001/
│   ├── subdir00002/
│   ├── ...
│   └── subdir10000/
├── ...
└── dir10000/
    ├── subdir00001/
    ├── subdir00002/
    ├── ...
    └── subdir10000/

I think we can still reduce the memory usage tweaking the BFS. I think it won't be an issue even for multi-mds scrub.

While scrubbing dirfrag, instead of pushing children back of the scrub_stack we can push it front of the scrub_stack and start iterating front of the scrub_stack if any child is pushed. In that case it will be kind of DFS-ish but in DFS memory would have been defined by the tree's height But here if we compare with dfs stack,

Let's say in an intermediate simulation dfs, stack containing (dir1)-->(dir2)-->(dir3)-->(dir4)-->(dir5) path of a tree, where dir_(i + 1) is subdirectory of dir_i

In this approach it will contain (child of dir1), (child of dir2), (child of dir3), (child of dir4), (child of dir5), which will pin much much lesser inode than pinning the whole level also retaining the BFS-ish approach for multi-mds scrub. Also I am not sure whether actually we can do DFS here, for (auto it = dir->begin(); it != dir->end(); /* nop */) having this iterator in the scrub stack.

But for now, I think we can do this minimal change.

@Venky Shankar Aren't we creating backport for squid?