retest this please

Hmm, I doesn't seem like you should be hitting the failsafe threshold.

Oh, it's because vstart sets the thresholds too high:

        osd failsafe full ratio = .99
        mon osd nearfull ratio = .99
        mon osd backfillfull ratio = .99
        mon osd full ratio = .99

should should be .99, .96, .97, .98, or similar. Update vstart.sh?

@liewegas i got the test result by seting these options all to 15, so that This osd Could be filled full quickly. I donnot think This issue is related to option values... Could you give me some suggestion if we dont do This change to resolve this issue?

The important thing is that the full_ratio is less than the failsafe ratio, so that the clsuter is marked full and clients stop writing before hitting the failsafe.

The failsafe is a last-ditch safety check to prevent the OSD from filling itself up. You shouldn't be allowed to override it with the force flag.

@liewegas yes, full_ratio is normally less than the failsafe ratio, but in my case, there is possible the failsafe reached first: that is because the "statfs" is called in osd(every one or five seconds?), and set cur_stat of this OSD to full by fail_safe ratio, and then send to monitor, and check osdmap change by full_ratio, and send to client, then pause client io... So there is a time interval...

In addition, I didn't override with full_force, but full_try.

I will not insist on if you think we should tune failsafe and full_ratio to avoid this issue... But I think it maybe a little tricky for this tuning...

@liewegas And in this case, I set all this options to 15%, but I found both full_ratio and full_try are 20% when fio pause... I use 1m bs to write it... For this case, I think 1 or 2 percent difference between full_try and fail_safe could not reolve it...

Do you mind updating the commit description?

@liewegas commit description has been updated, thanks.

Ok, since this is just a FULL_TRY, it's probably harmless... we will only proceed if the transaction is a net reduction in usage. There is still some risk, though: it may be that the operation forces recovery of an object that then fills things up. The failsafe should block that from happening, though!

I searched the code, and found there were no other places set this CEPH_OSD_FLAG_FULL_TRY... I think we could avoid this risk by strictly limit the operations to set FULL_TRY/FULL_FORCE flag, so that failsafe will really safe to block that.

• http://pulpito.ceph.com/kchai-2017-08-27_16:28:46-rados-wip-kefu-testing-distro-basic-mira/
• http://pulpito.ceph.com/kchai-2017-08-28_15:54:18-rados-wip-kefu-testing-distro-basic-smithi/