jenkins test make check

This is fine as far as it goes. What I don't understand is this:

1. "ceph -s" is run in a try-wait loop. I.e. the "ceph -s" command is issued and the process waits some number of seconds for it to produce output
2. in the log excerpt on the tracker ticket, we see that at least 4 attempts go by without getting any output
3. while cephadm is waiting for output from the 5th attempt, suddenly we see "ceph -s" output in the log!
4. alas, this "ceph -s" output we see is not from the 5th attempt, but (presumably) from the 1st, so it is ignored. The try-wait loop continues.
5. we continue to get output from successive (2nd, 3rd, 4th) tries, but while this is happening, the try-wait loop is issuing new commands (6th, 7th, 8th, 9th, finally 10th try)
6. by the time we get to the end of the try-wait loop, the first 4 tries have successfully completed, but each time they are ignored because the loop has cycled in the meantime

So, I'm thinking something more clever is needed, but I don't know what, exactly. In the reported case, the first attempt did not complete until the 5th attempt was in progress, which means it took more than 30 * 4 = 120 seconds to complete. So it's doubtful that this PR would have had any effect in that case.

http://pulpito.ceph.com/swagner-2020-06-10_12:32:13-rados:cephadm-wip-swagner-testing-2020-06-10-1105-distro-basic-smithi/

This is fine as far as it goes. What I don't understand is this:

1. "ceph -s" is run in a try-wait loop. I.e. the "ceph -s" command is issued and the process waits some number of seconds for it to produce output
2. in the log excerpt on the tracker ticket, we see that at least 4 attempts go by without getting any output
3. while cephadm is waiting for output from the 5th attempt, suddenly we see "ceph -s" output in the log!
4. alas, this "ceph -s" output we see is not from the 5th attempt, but (presumably) from the 1st, so it is ignored. The try-wait loop continues.
5. we continue to get output from successive (2nd, 3rd, 4th) tries, but while this is happening, the try-wait loop is issuing new commands (6th, 7th, 8th, 9th, finally 10th try)
6. by the time we get to the end of the try-wait loop, the first 4 tries have successfully completed, but each time they are ignored because the loop has cycled in the meantime

So, I'm thinking something more clever is needed, but I don't know what, exactly. In the reported case, the first attempt did not complete until the 5th attempt was in progress, which means it took more than 30 * 4 = 120 seconds to complete. So it's doubtful that this PR would have had any effect in that case.

It's a sequential loop (not async). The data from stdout is exactly the data read during the attempt in which it occurred.

Despite having some data on stdout, for every single attempt, the pid did not exit before the timeout duration was reached and a SIGKILL was issued causing the pid to return with exitcode -9.

I've added a debug log line to profile this loop, hopefully it will make the behavior more apparent.

DEBUG:cephadm:/usr/bin/podman:profile rt=0.1542048454284668, stop=False, exit=None, reads=[3]
DEBUG:cephadm:/usr/bin/podman:profile rt=0.15837812423706055, stop=False, exit=0, reads=[3, 5]
DEBUG:cephadm:/usr/bin/podman:profile rt=0.1585376262664795, stop=True, exit=0, reads=[3, 5]
...
...
DEBUG:cephadm:/usr/bin/ceph:profile rt=60.063154220581055, stop=False, exit=None, reads=[]
INFO:cephadm:/usr/bin/ceph:timeout after 60 seconds
DEBUG:cephadm:/usr/bin/ceph:profile rt=65.00448799133301, stop=True, exit=-9, reads=[3, 5]
...

jenkins test dashboard backend