ceph-monstore-tool: use a large enough paxos/{first,last}_committed#27465
ceph-monstore-tool: use a large enough paxos/{first,last}_committed#27465tchaikov merged 6 commits intoceph:masterfrom
Conversation
|
retest this please |
|
the tests still fail like a hell.. |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
unstale |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
@tchaikov would it be possible to get this PR in nautilus seeing this issue pops up every few months in teuthology: https://tracker.ceph.com/issues/38219 |
|
jenkins retest this please |
|
@ideepika will revisit this pr by the end of this week. |
so the rebuild paxos transaction won't be overwritten by the ones
created before recovery completes.
when the quorum is recovering, the leader will collect the paxos
transactions from peons. if the quorum accept the proposal for setting
the fingerprint, the peon will update the monitor with the paxos
transaction with a newer "last_committed" than the one created using
update_paxos() in ceph_monstore_tool.cc. the latter "last_committed" is
always 0.
so, to avoid this extra paxos proposal obsoleting the "rebuilding" paxos
transaction, we use a large enough number for {first,last}_committed.
Fixes: http://tracker.ceph.com/issues/38219
Signed-off-by: Kefu Chai <kchai@redhat.com>
|
changelog
|
for better reading experience. Signed-off-by: Kefu Chai <kchai@redhat.com>
for better readability Signed-off-by: Kefu Chai <kchai@redhat.com>
more consistent this way. Signed-off-by: Kefu Chai <kchai@redhat.com>
for better readability Signed-off-by: Kefu Chai <kchai@redhat.com>
mon_tick_interval is 5 seconds by default. monitors update their rotating keys every mon_tick_interval. before monitors forms a quorum, the auth requests from clients are put into the wait list. these requests are re-enqueued once the monitors form a quorum. but there is a small window of mon_tick_interval, before they are able to serve the auth requests even after their claim to be able to server requests. if these re-enqueued requests happen to be served in this window, and if authx is enabled, they will be greeted with errors like handle_auth_bad_method server allowed_methods [2] but i only support [2] in the case of ceph cli, the error would look like: [errno 13] RADOS permission denied (error connecting to the cluster) so, to address this issue, the EACCES error is ignored when waiting for a quorum. Signed-off-by: Kefu Chai <kchai@redhat.com>
in the failed tests, monitors were able to form quorum. but there were two categories of failures:
but i still think this changeset is an improvement. as it addresses two issues:
|
|
changelog
@neha-ojha could you take another look? |
@tchaikov sure, will take a look at it tomorrow |
neha-ojha
left a comment
There was a problem hiding this comment.
@tchaikov a couple of questions 1. do you understand why https://tracker.ceph.com/issues/38219 is not seen in releases after nautilus? 2. how far should we backport this patch? overall, this change makes sense to me
@neha-ojha , thank you for reviewing the changes. yes, the frequency of the failures is much lower in recent releases. but it still surfaces occasionally. i checked
as far as possible, i'd say. even back to nautilus. |
set monitor/cluster_fingerprint for the newly created monstore,
otherwise, the leader will create a new paxos proposal and there is
chance that the quorum will accept it.
when the quorum is recovering, the leader will collect the paxos
transactions from peons. if the quorum accept the proposal for setting
the fingerprint, the peon will update the monitor with the paxos
transaction with a newer "last_committed" than the one created using
update_paxos() in ceph_monstore_tool.cc. the latter "last_committed" is
always 0.
so, to avoid this extra paxos proposal obsoleting the "rebuilding" paxos
transaction, we set "monitor/cluster_fingerprint" when rebuilding the
monstore.
Fixes: http://tracker.ceph.com/issues/38219
Signed-off-by: Kefu Chai kchai@redhat.com