Project

General

Profile

Actions
Copy link

Bug #69838

open

LibRadosTwoPoolsPP.HitSetWrite FAILED on rados/test.sh

Added by Lee Sanders about 1 year ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Tags:
low-hanging-fruit main-failures

Description

Command failed (workunit test rados/test.sh) on smithi164 with status 1: 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=093e0de0b870f3cc10663e24dc60c991ac615612 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

Failed on main (19.3.x)

2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (136998 ms)
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (34998 ms)
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (8050 ms)
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (8041 ms)
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (11069 ms)
2025-02-05T02:47:29.421 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (5015 ms)
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (475303 ms total)
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp:
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [----------] Global test environment tear-down
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1912181 ms total)
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ PASSED ] 76 tests.
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ FAILED ] 1 test, listed below:
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp:
2025-02-05T02:47:29.422 INFO:tasks.workunit.client.0.smithi164.stdout: api_tier_pp: 1 FAILED TEST


025-02-05T02:47:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:2025-02-05T02:47:29.586+0000 7f3b01ffb640  1 -- 172.21.15.60:0/1725172903 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) ==== 72+0+203943 (secure 0 0 0) 0x7f3afc05e550 con 0x7f3b100ca5a0
2025-02-05T02:47:29.630 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument
2025-02-05T02:47:29.630 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2025-02-05T02:47:29.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight
2025-02-05T02:47:29.642 INFO:tasks.ceph.osd.9.smithi120.stderr:2025-02-05T02:47:29.639+0000 7fa03c6b0640 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9  (PID: 102664) UID: 0
2025-02-05T02:47:29.743 INFO:tasks.ceph.osd.4.smithi060.stderr:2025-02-05T02:47:29.740+0000 7fe853608640 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4  (PID: 23610) UID: 0
2025-02-05T02:47:29.743 INFO:teuthology.orchestra.run.smithi143.stdout:ERROR: (22) Invalid argument
2025-02-05T02:47:29.743 INFO:teuthology.orchestra.run.smithi143.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2025-02-05T02:47:29.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops
2025-02-05T02:47:29.772 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_093e0de0b870f3cc10663e24dc60c991ac615612/qa/tasks/workunit.py", line 126, in task
    with parallel() as p:
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_093e0de0b870f3cc10663e24dc60c991ac615612/qa/tasks/workunit.py", line 434, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/orchestra/remote.py", line 535, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/orchestra/run.py", line 461, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=093e0de0b870f3cc10663e24dc60c991ac615612 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2025-02-05T02:47:29.950 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2887e0a7c19a421183116e743f4c4083
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5eda30d4a0d087fd373518d7806759bd2f5557b0/teuthology/run_tasks.py", line 83, in run_one_task

/a/ljsanders-2025-02-04_09:31:40-rados-main-distro-default-smithi/8114227
Actions
Copy link
 #1

Updated by Shraddha Agrawal about 1 year ago

  • Backport set to reef

/a/skanta-2025-02-14_03:51:29-rados-wip-bharath5-testing-2025-02-13-2020-reef-distro-default-smithi/8130365

2025-02-14T23:08:49.261 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: verifying foo0 is eventually promoted
2025-02-14T23:08:49.261 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (25138 ms)
2025-02-14T23:08:49.261 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.ProxyRead
2025-02-14T23:08:49.261 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.ProxyRead (34188 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.CachePin
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.CachePin (32549 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.SetRedirectRead
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.SetRedirectRead (7055 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.SetChunkRead
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.SetChunkRead (7103 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.ManifestPromoteRead
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (8024 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsECPP.TrySetDedupTier
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (5033 ms)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (437357 ms total)
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp:
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [----------] Global test environment tear-down
2025-02-14T23:08:49.262 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [==========] 77 tests from 4 test suites ran. (1792463 ms total)
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [  PASSED  ] 76 tests.
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [  FAILED  ] 1 test, listed below:
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp: [  FAILED  ] LibRadosTwoPoolsPP.HitSetWrite
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp:
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stdout:              api_tier_pp:  1 FAILED TEST
2025-02-14T23:08:49.263 INFO:tasks.workunit.client.0.smithi142.stderr:+ echo 'error in api_tier_pp (37950)'
Actions
Copy link
 #2

Updated by Adam Kupczyk 12 months ago

  • Description updated (diff)
Actions
Copy link

Also available in: Atom PDF