Project

General

Profile

Actions
Copy link

Bug #66049

open

qa, tasks/nfs: client.15263 isn't responding to mclientcaps(revoke), ino 0x1 pending pAsLsXs issued pAsLsXsFs

Added by Venky Shankar almost 2 years ago. Updated about 2 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
Mahesh Mohan
Category:
Testing
Target version:
-
% Done:

0%

Source:
Development
Backport:
reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Tags (freeform):
temp-assign
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

/a/vshankar-2024-05-10_15:47:15-fs-wip-vshankar-testing-20240509.114851-debug-testing-default-smithi/7701516

ceph-mgr restarted multiple times

2024-05-11T02:40:49.882+0000 7f11ea1dd200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T02:41:05.676+0000 7f8b2af2b200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T02:41:41.779+0000 7ff73a0c8200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T02:48:52.302+0000 7f34d4817200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T02:50:15.489+0000 7fe83ce53200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T03:08:10.819+0000 7f7237d7d200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7
2024-05-11T03:08:56.297+0000 7f5cbcc7e200  0 ceph version 19.0.0-3590-g7b710596 (7b710596db3f44de0a4b72eff459db857656fe05) squid (dev), process ceph-mgr, pid 7

It's likely the libcephfs handles in ceph-mgr (used by volumes plugin and others) that aren't cleaned up during mgr restart. There is evidence that the MDS has also evicted those clients.

./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:26:31.636+0000 7f469e600700  1 mds.0.154 Evicting (and blocklisting) client session 15975 (172.21.15.154:0/2621613153)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:26:31.636+0000 7f469e600700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 15975 (172.21.15.154:0/2621613153)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:27:29.094+0000 7f469e600700  1 mds.0.154 Evicting (and blocklisting) client session 16000 (172.21.15.154:0/33124601)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:27:29.094+0000 7f469e600700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 16000 (172.21.15.154:0/33124601)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:31:23.368+0000 7f469e600700  1 mds.0.154 Evicting (and blocklisting) client session 16054 (172.21.15.154:0/1715873550)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:31:23.368+0000 7f469e600700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 16054 (172.21.15.154:0/1715873550)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:32:20.147+0000 7f469e600700  1 mds.0.154 Evicting (and blocklisting) client session 16157 (172.21.15.154:0/1333085366)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.hifoie.log.gz:2024-05-11T03:32:20.147+0000 7f469e600700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 16157 (172.21.15.154:0/1333085366)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:02:21.899+0000 7ff589200700  1 mds.0.45 Evicting (and blocklisting) client session 14757 (172.21.15.154:0/1685098280)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:02:21.899+0000 7ff589200700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 14757 (172.21.15.154:0/1685098280)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:03:14.837+0000 7ff589200700  1 mds.0.45 Evicting (and blocklisting) client session 14891 (172.21.15.154:0/3620627595)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:03:14.837+0000 7ff589200700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 14891 (172.21.15.154:0/3620627595)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:04:07.984+0000 7ff589200700  1 mds.0.45 Evicting (and blocklisting) client session 14954 (172.21.15.154:0/755355721)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.nfs-cephfs.smithi154.farbxt.log.gz:2024-05-11T03:04:07.984+0000 7ff589200700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 14954 (172.21.15.154:0/755355721)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz:2024-05-11T02:55:53.022+0000 7fcaa8a00700  1 mds.0.33 Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz:2024-05-11T02:55:53.022+0000 7fcaa8a00700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz: -3077> 2024-05-11T02:55:53.022+0000 7fcaa8a00700  1 mds.0.33 Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz: -3076> 2024-05-11T02:55:53.022+0000 7fcaa8a00700  0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz: -3065> 2024-05-11T02:55:53.198+0000 7fcaa7600700 10 log_client  will send 2024-05-11T02:55:53.027225+0000 mds.user_test_fs.smithi154.kpsfmq (mds.0) 1 : cluster [INF] Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)
./remote/smithi154/log/b5778b1c-0f3f-11ef-bc98-c7b262605968/ceph-mds.user_test_fs.smithi154.kpsfmq.log.gz: -3060> 2024-05-11T02:55:53.914+0000 7fcaa8a00700 10 log_client  logged 2024-05-11T02:55:53.027225+0000 mds.user_test_fs.smithi154.kpsfmq (mds.0) 1 : cluster [INF] Evicting (and blocklisting) client session 14575 (172.21.15.154:0/2551661327)

Related issues 1 (0 open — 1 closed)

Related to CephFS - Bug #66009: qa: `fs volume ls` command times out waiting for fs to come onlineResolvedMilind Changire

Actions
Actions
Copy link
 #1

Updated by Venky Shankar almost 2 years ago

  • Related to Bug #66009: qa: `fs volume ls` command times out waiting for fs to come online added
Actions
Copy link
 #2

Updated by Venky Shankar almost 2 years ago

  • Assignee deleted (Dhairya Parmar)

Dhairya - unassigning for now as per https://tracker.ceph.com/issues/66009#note-7

Actions
Copy link
 #3

Updated by Venky Shankar almost 2 years ago

  • Assignee set to Milind Changire

Milind, assigning this to you since you own tracker #66009

Actions
Copy link
 #4

Updated by Venky Shankar almost 2 years ago

  • Status changed from New to Triaged
Actions
Copy link
 #6

Updated by Konstantin Shalygin about 1 year ago

  • Backport changed from quincy,reef,squid to reef,squid
Actions
Copy link
 #7

Updated by Patrick Donnelly 9 months ago

  • Target version deleted (v20.0.0)
Actions
Copy link
 #8

Updated by Venky Shankar about 2 months ago

  • Assignee changed from Milind Changire to Mahesh Mohan
  • Tags (freeform) set to temp-assign
Actions
Copy link

Also available in: Atom PDF