Bug #66991
closedqa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
0%
Description
See this again after https://tracker.ceph.com/issues/59413
2024-07-13T01:44:26.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing
2024-07-13T01:44:30.852 INFO:tasks.ceph.mds.d.smithi189.stderr:2024-07-13T01:44:30.855+0000 7f80519a8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request.
2024-07-13T01:44:31.610 INFO:tasks.workunit.client.0.smithi156.stderr:error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
2024-07-13T01:44:31.624 INFO:tasks.workunit.client.0.smithi156.stderr:fetch-pack: unexpected disconnect while reading sideband packet
2024-07-13T01:44:31.642 INFO:tasks.workunit.client.0.smithi156.stderr:fatal: early EOF
2024-07-13T01:44:31.643 INFO:tasks.workunit.client.0.smithi156.stderr:fatal: fetch-pack: invalid index-pack output
2024-07-13T01:44:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 128
2024-07-13T01:44:32.038 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2024-07-13T01:44:32.038 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2024-07-13T01:44:32.338 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 145, in task
_spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 424, in _run_tests
remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi156 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aae77c1a1f23b888a62f052045327bcaad529749 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
Updated by Xiubo Li over 1 year ago
- Related to Bug #59413: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" added
Updated by Venky Shankar over 1 year ago
I'm seeing somewhat similar problem in this run: /a/vshankar-2024-07-31_11:42:32-fs-wip-vshankar-testing-20240730.074544-debug-testing-default-smithi/7828285
2024-07-31T16:35:33.970 INFO:tasks.workunit.client.0.smithi146.stderr:error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. 2024-07-31T16:35:33.976 INFO:tasks.workunit.client.0.smithi146.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-07-31T16:35:33.983 INFO:tasks.workunit.client.0.smithi146.stderr:fatal: early EOF 2024-07-31T16:35:33.984 INFO:tasks.workunit.client.0.smithi146.stderr:fatal: fetch-pack: invalid index-pack output 2024-07-31T16:35:35.162 INFO:tasks.workunit.client.0.smithi146.stderr:+ retry 2024-07-31T16:35:35.163 INFO:tasks.workunit.client.0.smithi146.stderr:+ rm -rf ceph 2024-07-31T16:35:35.164 INFO:tasks.workunit.client.0.smithi146.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-07-31T16:35:35.175 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into 'ceph'...
But one can see from the above, cloning the repo is retried, but then the workunit hits a failure after 100% clone is done:
^MUpdating files: 98% (12567/12823)^MUpdating files: 98% (12686/12823)^MUpdating files: 99% (12695/12823)^MUpdating files: 100% (12823/12823)^MUpdating files: 100% (12823/12823), done. 2024-07-31T16:42:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-07-31T16:42:45.565 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
I'm not sure at this point if this is just a remote process failure or a bug with ceph-fuse.
Updated by Xiubo Li over 1 year ago
Venky Shankar wrote in #note-3:
I'm seeing somewhat similar problem in this run: /a/vshankar-2024-07-31_11:42:32-fs-wip-vshankar-testing-20240730.074544-debug-testing-default-smithi/7828285
[...]
But one can see from the above, cloning the repo is retried, but then the workunit hits a failure after 100% clone is done:
[...]
I'm not sure at this point if this is just a remote process failure or a bug with ceph-fuse.
I will dig it further today.
Updated by Xiubo Li over 1 year ago
As I remembered months ago I also hit the a similar issue, which is the clone succeeded, but later it reported a failure, and this last for days. It seems it was an issue to access the github server without the VPN.
Updated by Venky Shankar over 1 year ago
- Priority changed from Normal to Low
Xiubo Li wrote in #note-5:
As I remembered months ago I also hit the a similar issue, which is the clone succeeded, but later it reported a failure, and this last for days. It seems it was an issue to access the github server without the VPN.
It is possible, yes. A rerun makes the test pass, so it could be network/remote endpoint related somewhat. I'm lowering the prio for now.
Updated by Xiubo Li over 1 year ago
Locally I have two commit to improve and enable the debug log for git, let's see could we get more useful info in future.
Updated by Xiubo Li over 1 year ago
- Status changed from New to Fix Under Review
- Pull request ID set to 59072
Updated by Venky Shankar over 1 year ago
- Category set to Testing
- Status changed from Fix Under Review to Pending Backport
- Assignee changed from Xiubo Li to Venky Shankar
- Target version set to v20.0.0
- Source set to Q/A
Taking this from Xiubo.
Updated by Upkeep Bot over 1 year ago
- Copied to Backport #68763: squid: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Updated by Upkeep Bot over 1 year ago
- Copied to Backport #68764: quincy: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Updated by Upkeep Bot over 1 year ago
- Copied to Backport #68765: reef: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Updated by Upkeep Bot over 1 year ago
- Tags (freeform) set to backport_processed
Updated by Milind Changire over 1 year ago
main:
https://pulpito.ceph.com/mchangir-2024-12-10_11:05:18-fs-wip-mchangir-testing-mon-caps-main-debug-testing-default-smithi/8028286
https://pulpito.ceph.com/mchangir-2024-12-10_11:05:18-fs-wip-mchangir-testing-mon-caps-main-debug-testing-default-smithi/8028430
2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:error: RPC failed; curl 18 transfer closed with outstanding read data remaining 2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:fatal: early EOF 2024-12-10T17:32:08.986 INFO:tasks.workunit.client.0.smithi070.stderr:fatal: fetch-pack: invalid index-pack output 2024-12-10T17:32:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-12-10T17:32:09.010 INFO:tasks.workunit:failed to check out 'e335c9ae9404c849797e5ab5b34f451fc9d1d719' from https://git.ceph.com/ceph-ci.git; will also try in https://git.ceph.com/ceph.git
Updated by Jaya Prakash 10 months ago
teuthology-2025-05-11_20:00:15-rados-main-distro-default-smithi/8280296
Updated by Upkeep Bot 9 months ago
- Merge Commit set to 94ce78a599a29fac526d4608c891347f38b60b8e
- Fixed In set to v19.3.0-5824-g94ce78a599a
- Upkeep Timestamp set to 2025-07-08T22:37:56+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-5824-g94ce78a599a to v19.3.0-5824-g94ce78a599a2
- Upkeep Timestamp changed from 2025-07-08T22:37:56+00:00 to 2025-07-14T15:46:30+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-5824-g94ce78a599a2 to v19.3.0-5824-g94ce78a599
- Upkeep Timestamp changed from 2025-07-14T15:46:30+00:00 to 2025-07-14T21:10:36+00:00
Updated by Upkeep Bot 5 months ago
- Status changed from Pending Backport to Resolved
- Upkeep Timestamp changed from 2025-07-14T21:10:36+00:00 to 2025-10-08T13:09:29+00:00
Updated by Upkeep Bot 5 months ago
- Released In set to v20.2.0~1700
- Upkeep Timestamp changed from 2025-10-08T13:09:29+00:00 to 2025-11-01T01:39:36+00:00