Project

General

Profile

Actions

Bug #66991

closed

qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.

Added by Xiubo Li over 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Backport:
squid,reef,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v19.3.0-5824-g94ce78a599
Released In:
v20.2.0~1700
Upkeep Timestamp:
2025-11-01T01:39:36+00:00

Description

https://pulpito.ceph.com/jcollin-2024-07-12_00:27:44-fs-wip-jcollin-testing-20240711.125835-reef-distro-default-smithi/7798434/

See this again after https://tracker.ceph.com/issues/59413

2024-07-13T01:44:26.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing
2024-07-13T01:44:30.852 INFO:tasks.ceph.mds.d.smithi189.stderr:2024-07-13T01:44:30.855+0000 7f80519a8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request.
2024-07-13T01:44:31.610 INFO:tasks.workunit.client.0.smithi156.stderr:error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
2024-07-13T01:44:31.624 INFO:tasks.workunit.client.0.smithi156.stderr:fetch-pack: unexpected disconnect while reading sideband packet
2024-07-13T01:44:31.642 INFO:tasks.workunit.client.0.smithi156.stderr:fatal: early EOF
2024-07-13T01:44:31.643 INFO:tasks.workunit.client.0.smithi156.stderr:fatal: fetch-pack: invalid index-pack output
2024-07-13T01:44:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 128
2024-07-13T01:44:32.038 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2024-07-13T01:44:32.038 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2024-07-13T01:44:32.338 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 145, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients
    with parallel() as p:
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_aae77c1a1f23b888a62f052045327bcaad529749/qa/tasks/workunit.py", line 424, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/remote.py", line 523, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_454a46106b61e201598ae5e1c392af2328539bfe/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi156 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aae77c1a1f23b888a62f052045327bcaad529749 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'


Related issues 4 (0 open4 closed)

Related to CephFS - Bug #59413: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"DuplicateXiubo Li

Actions
Copied to CephFS - Backport #68763: squid: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.ResolvedVenky ShankarActions
Copied to CephFS - Backport #68764: quincy: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.RejectedVenky ShankarActions
Copied to CephFS - Backport #68765: reef: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.ResolvedVenky ShankarActions
Actions #2

Updated by Xiubo Li over 1 year ago

  • Related to Bug #59413: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" added
Actions #3

Updated by Venky Shankar over 1 year ago

I'm seeing somewhat similar problem in this run: /a/vshankar-2024-07-31_11:42:32-fs-wip-vshankar-testing-20240730.074544-debug-testing-default-smithi/7828285

2024-07-31T16:35:33.970 INFO:tasks.workunit.client.0.smithi146.stderr:error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
2024-07-31T16:35:33.976 INFO:tasks.workunit.client.0.smithi146.stderr:fetch-pack: unexpected disconnect while reading sideband packet
2024-07-31T16:35:33.983 INFO:tasks.workunit.client.0.smithi146.stderr:fatal: early EOF
2024-07-31T16:35:33.984 INFO:tasks.workunit.client.0.smithi146.stderr:fatal: fetch-pack: invalid index-pack output
2024-07-31T16:35:35.162 INFO:tasks.workunit.client.0.smithi146.stderr:+ retry
2024-07-31T16:35:35.163 INFO:tasks.workunit.client.0.smithi146.stderr:+ rm -rf ceph
2024-07-31T16:35:35.164 INFO:tasks.workunit.client.0.smithi146.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git
2024-07-31T16:35:35.175 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into 'ceph'...

But one can see from the above, cloning the repo is retried, but then the workunit hits a failure after 100% clone is done:

^MUpdating files:  98% (12567/12823)^MUpdating files:  98% (12686/12823)^MUpdating files:  99% (12695/12823)^MUpdating files: 100% (12823/12823)^MUpdating files: 100% (12823/12823), done.
2024-07-31T16:42:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 128
2024-07-31T16:42:45.565 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...

I'm not sure at this point if this is just a remote process failure or a bug with ceph-fuse.

Actions #4

Updated by Xiubo Li over 1 year ago

Venky Shankar wrote in #note-3:

I'm seeing somewhat similar problem in this run: /a/vshankar-2024-07-31_11:42:32-fs-wip-vshankar-testing-20240730.074544-debug-testing-default-smithi/7828285

[...]

But one can see from the above, cloning the repo is retried, but then the workunit hits a failure after 100% clone is done:

[...]

I'm not sure at this point if this is just a remote process failure or a bug with ceph-fuse.

I will dig it further today.

Actions #5

Updated by Xiubo Li over 1 year ago

As I remembered months ago I also hit the a similar issue, which is the clone succeeded, but later it reported a failure, and this last for days. It seems it was an issue to access the github server without the VPN.

Actions #6

Updated by Venky Shankar over 1 year ago

  • Priority changed from Normal to Low

Xiubo Li wrote in #note-5:

As I remembered months ago I also hit the a similar issue, which is the clone succeeded, but later it reported a failure, and this last for days. It seems it was an issue to access the github server without the VPN.

It is possible, yes. A rerun makes the test pass, so it could be network/remote endpoint related somewhat. I'm lowering the prio for now.

Actions #7

Updated by Xiubo Li over 1 year ago

Locally I have two commit to improve and enable the debug log for git, let's see could we get more useful info in future.

Actions #8

Updated by Xiubo Li over 1 year ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 59072
Actions #9

Updated by Venky Shankar over 1 year ago

  • Category set to Testing
  • Status changed from Fix Under Review to Pending Backport
  • Assignee changed from Xiubo Li to Venky Shankar
  • Target version set to v20.0.0
  • Source set to Q/A

Taking this from Xiubo.

Actions #10

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #68763: squid: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Actions #11

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #68764: quincy: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Actions #12

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #68765: reef: qa: error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. added
Actions #13

Updated by Upkeep Bot over 1 year ago

  • Tags (freeform) set to backport_processed
Actions #14

Updated by Milind Changire over 1 year ago

main:
https://pulpito.ceph.com/mchangir-2024-12-10_11:05:18-fs-wip-mchangir-testing-mon-caps-main-debug-testing-default-smithi/8028286
https://pulpito.ceph.com/mchangir-2024-12-10_11:05:18-fs-wip-mchangir-testing-mon-caps-main-debug-testing-default-smithi/8028430

2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:error: RPC failed; curl 18 transfer closed with outstanding read data remaining
2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:fetch-pack: unexpected disconnect while reading sideband packet
2024-12-10T17:32:08.983 INFO:tasks.workunit.client.0.smithi070.stderr:fatal: early EOF
2024-12-10T17:32:08.986 INFO:tasks.workunit.client.0.smithi070.stderr:fatal: fetch-pack: invalid index-pack output
2024-12-10T17:32:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 128
2024-12-10T17:32:09.010 INFO:tasks.workunit:failed to check out 'e335c9ae9404c849797e5ab5b34f451fc9d1d719' from https://git.ceph.com/ceph-ci.git; will also try in https://git.ceph.com/ceph.git

Actions #15

Updated by Jaya Prakash 10 months ago

teuthology-2025-05-11_20:00:15-rados-main-distro-default-smithi/8280296

Actions #16

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 94ce78a599a29fac526d4608c891347f38b60b8e
  • Fixed In set to v19.3.0-5824-g94ce78a599a
  • Upkeep Timestamp set to 2025-07-08T22:37:56+00:00
Actions #17

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-5824-g94ce78a599a to v19.3.0-5824-g94ce78a599a2
  • Upkeep Timestamp changed from 2025-07-08T22:37:56+00:00 to 2025-07-14T15:46:30+00:00
Actions #18

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-5824-g94ce78a599a2 to v19.3.0-5824-g94ce78a599
  • Upkeep Timestamp changed from 2025-07-14T15:46:30+00:00 to 2025-07-14T21:10:36+00:00
Actions #20

Updated by Upkeep Bot 5 months ago

  • Status changed from Pending Backport to Resolved
  • Upkeep Timestamp changed from 2025-07-14T21:10:36+00:00 to 2025-10-08T13:09:29+00:00
Actions #21

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~1700
  • Upkeep Timestamp changed from 2025-10-08T13:09:29+00:00 to 2025-11-01T01:39:36+00:00
Actions

Also available in: Atom PDF