Ceph » CephFS

Case Analysis：

1. When client2 mount server2 through nfsv3, if it create a file and write it, since the nfsv3 protocol has no close opertion, the file inode will always keeps some unwanted caps in the cephfs client cache (issued: pAsxLsXsxFsxcrwb, wanted: pAsxXsxFsxcrwb)

2. Then when client1 read the file through another nfs server1, nfs server1 will perform two operations: getattr and read, at which point server1 will try to get caps (pAsLsXsFscr)

3. At this time, mds will try to recall the Fcwb caps from server2 to meet the caps requirements of server1

4. After server1 completes its getattr and read requests, the mds locker will soon reclaim the Fsc caps from server1 to meet the Fw caps requirements of server2

5. In this way, if a series of getattr and read operations on server1 are executed, mds will repeatedly grant and reclaim Fsc caps to server1

how the read hung case occurs on server1:

1. In the above case, each pair of getattr and reading request on server1 is serial. When server1 receives the pAsLsXsFscr caps granted by mds, it can perform an aysnc reading and a pre-reading to the file. Then in the _read_async function, it will calls get_cap_ref，which result to：in- > cap_refs [CEPH_CAP_FILE_CACHE] ++

2. After the _read_async function completed，the pre-reading may still wait for the osd response, so the in->cap_refs [CEPH_CAP_FILE_CACHE] is still not zero.

3. At which time if the mds locker reclaims the Fsc caps from server1, in the handle_cap_grant function, since the in-> cap_refs[CEPH_CAP_FILE_CACHE] is not 0, which means that check_caps cannot be called to further update the local caps and respond to mds.

4. Then if a newly reading request is executed in server 1, it will wait in Client::get_caps:

int Client::get_caps(Fh *fh, int need, int want, int *phave, loff_t endoff)
{
  Inode *in = fh->inode.get();

  int r = check_pool_perm(in, need);
  if (r < 0)
    return r;

  while (1) {
    ...

    int implemented;
    int have = in->caps_issued(&implemented);

    bool waitfor_caps = false;
    bool waitfor_commit = false;

    ...

    if (!waitfor_caps && !waitfor_commit) {
      /* have: pAsLsXsFr,  need: Fr */
      if ((have & need) == need) {
        int revoking = implemented & ~have;
        ldout(cct, 10) << "get_caps " << *in << " have " << ccap_string(have)
                << " need " << ccap_string(need) << " want " << ccap_string(want)
                << " revoking " << ccap_string(revoking)
                << dendl;
        /* revoking: Fsc, want: Fc */
        if ((revoking & want) == 0) {
          *phave = need | (have & want);
          in->get_cap_ref(need);
          return 0;
        }
      }
      ldout(cct, 10) << "waiting for caps " << *in << " need " << ccap_string(need) << " want " << ccap_string(want) << dendl;
      waitfor_caps = true;
    }

    ...

    if (waitfor_caps)
      wait_on_list(in->waitfor_caps);
    else if (waitfor_commit)
      wait_on_list(in->waitfor_commit);
  }
}

how to fix this bug: when the client drop CEPH_CAP_FILE_CACHE caps，send signal to nodify the waitfor_caps readers

@@ -3240,6 +3240,9 @@ void Client::put_cap_ref(Inode *in, int cap)
     }
     if (drop)
       check_caps(in, 0);
+      if (last & CEPH_CAP_FILE_CACHE) {
+        signal_cond_list(in->waitfor_caps); // wake up blocked readers
+      }
     if (put_nref)
       put_inode(in, put_nref);
   }

@Venky Shankar

hello，

How has this issue progressed? Do you need me to provide more information？

tod chen wrote in #note-3:

@Venky Shankar

hello，

How has this issue progressed? Do you need me to provide more information？

Thanks for your patience. I'm swamped with other tracker - will get to this sometime this week.

Thanks for the detailed reproducer in https://tracker.ceph.com/issues/65971#note-1 - it really helps. I've started look into this.

So, this is a three-way (cap related) deadlock between an asynchronous read which got triggered when the client initially got and Fscr cap, then a cap revoke for Fsc by the MDS (which didn't release the cap, since Fc was in use due to in pending asynchronous read) and finally a read request that wants Fc cap. You suggested fix is to signal the cap waiters in put_cap_ref() seems sensible. Would you mind sending the fix as a pull request, @hit1944 ?

Also, have you checked if the same issue exist with write requests?

Venky Shankar wrote in #note-7:

So, this is a three-way (cap related) deadlock between an asynchronous read which got triggered when the client initially got and Fscr cap, then a cap revoke for Fsc by the MDS (which didn't release the cap, since Fc was in use due to in pending asynchronous read) and finally a read request that wants Fc cap. You suggested fix is to signal the cap waiters in put_cap_ref() seems sensible. Would you mind sending the fix as a pull request, @hit1944 ?

Also, have you checked if the same issue exist with write requests?

ok，i will send the pr.

There is no such problem in the write flow, because in put_cap_ref, it checks whether the subtracted last caps reference contains Fwb, and if so, it will signal the write waiter.

@void Client::put_cap_ref(Inode *in, int cap) {
int last = in->put_cap_ref(cap);
if (last) {
int put_nref = 0;
int drop = last & ~in->caps_issued();
if (in->snapid == CEPH_NOSNAP) {
if ((last & CEPH_CAP_FILE_WR) &&
!in->cap_snaps.empty() &&
in->cap_snaps.rbegin()->second.writing) {
ldout(cct, 10) << func << " finishing pending cap_snap on " << *in << dendl;
in->cap_snaps.rbegin()->second.writing = 0;
finish_cap_snap(in, in->cap_snaps.rbegin()->second, get_caps_used(in));

    signal_cond_list(in->waitfor_caps);  // wake up blocked sync writers

pr: https://github.com/ceph/ceph/pull/59027