Actions
Bug #74524
openTEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote"
Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:
0%
Source:
Backport:
squid,tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Tags (freeform):
backport_processed
Merge Commit:
Fixed In:
v20.3.0-5533-g157336f4d8
Released In:
Upkeep Timestamp:
2026-02-23T18:48:50+00:00
Description
/a/lflores-2026-01-21_20:56:39-rados-main-distro-default-trial/12028
2026-01-22T01:55:37.208 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out
2026-01-22T01:55:37.209 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item'
2026-01-22T01:55:37.209 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: eval ITEM=null
2026-01-22T01:55:37.209 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: ITEM=null
2026-01-22T01:55:37.210 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:445: TEST_backfill_pool_priority: '[' null '!=' 1.0 ']'
2026-01-22T01:55:37.210 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: echo 'The primary PG 1.0 didn'\''t become the in progress item'
2026-01-22T01:55:37.210 INFO:tasks.workunit.client.0.trial038.stdout:The primary PG 1.0 didn't become the in progress item
2026-01-22T01:55:37.210 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: expr 0 + 1
2026-01-22T01:55:37.211 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: ERRORS=1
2026-01-22T01:55:37.211 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out
2026-01-22T01:55:37.211 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item'
2026-01-22T01:55:37.213 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: eval ITEM=null
2026-01-22T01:55:37.213 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: ITEM=null
2026-01-22T01:55:37.213 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:460: TEST_backfill_pool_priority: '[' null '!=' 1.0 ']'
2026-01-22T01:55:37.214 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: echo 'The primary PG 1.0 didn'\''t become the in progress item on remote'
2026-01-22T01:55:37.214 INFO:tasks.workunit.client.0.trial038.stdout:The primary PG 1.0 didn't become the in progress item on remote
2026-01-22T01:55:37.214 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: expr 1 + 1
2026-01-22T01:55:37.214 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: ERRORS=2
2026-01-22T01:55:37.215 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out
2026-01-22T01:55:37.215 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item'
2026-01-22T01:55:37.217 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: eval ITEM=null
2026-01-22T01:55:37.217 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: ITEM=null
2026-01-22T01:55:37.217 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:475: TEST_backfill_pool_priority: '[' null '!=' 2.0 ']'
2026-01-22T01:55:37.218 INFO:tasks.workunit.client.0.trial038.stdout:The primary PG 2.0 didn't become the in progress item
2026-01-22T01:55:37.218 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: echo 'The primary PG 2.0 didn'\''t become the in progress item'
2026-01-22T01:55:37.218 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: expr 2 + 1
2026-01-22T01:55:37.218 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: ERRORS=3
2026-01-22T01:55:37.219 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out
2026-01-22T01:55:37.219 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item'
2026-01-22T01:55:37.221 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: eval ITEM=null
2026-01-22T01:55:37.221 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: ITEM=null
2026-01-22T01:55:37.221 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:490: TEST_backfill_pool_priority: '[' null '!=' 2.0 ']'
2026-01-22T01:55:37.221 INFO:tasks.workunit.client.0.trial038.stdout:The primary PG 2.0 didn't become the in progress item on remote
2026-01-22T01:55:37.222 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: echo 'The primary PG 2.0 didn'\''t become the in progress item on remote'
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: expr 3 + 1
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: ERRORS=4
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:503: TEST_backfill_pool_priority: wait_for_clean
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd=
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1
2026-01-22T01:55:37.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true
2026-01-22T01:55:37.224 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace
2026-01-22T01:55:37.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5')
2026-01-22T01:55:37.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays
2026-01-22T01:55:37.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0
2026-01-22T01:55:37.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats
2026-01-22T01:55:37.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300
2026-01-22T01:55:37.305 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls
2026-01-22T01:55:37.545 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0
2026-01-22T01:55:37.545 INFO:tasks.workunit.client.0.trial038.stderr:1'
2026-01-22T01:55:37.545 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs=
2026-01-22T01:55:37.545 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids
2026-01-22T01:55:37.545 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats
2026-01-22T01:55:37.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493
2026-01-22T01:55:37.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493
2026-01-22T01:55:37.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836493'
2026-01-22T01:55:37.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids
2026-01-22T01:55:37.613 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats
2026-01-22T01:55:37.677 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705676
2026-01-22T01:55:37.678 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705676
2026-01-22T01:55:37.678 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836493 1-38654705676'
2026-01-22T01:55:37.678 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs
2026-01-22T01:55:37.678 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836493
2026-01-22T01:55:37.678 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1
2026-01-22T01:55:37.679 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0
2026-01-22T01:55:37.679 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493
2026-01-22T01:55:37.679 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2
2026-01-22T01:55:37.681 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836493
2026-01-22T01:55:37.681 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836493'
2026-01-22T01:55:37.681 INFO:tasks.workunit.client.0.trial038.stdout:waiting osd.0 seq 21474836493
2026-01-22T01:55:37.681 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0
2026-01-22T01:55:37.910 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836490 -lt 21474836493
2026-01-22T01:55:37.910 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1
2026-01-22T01:55:38.911 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']'
2026-01-22T01:55:38.911 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0
2026-01-22T01:55:39.141 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836493 -lt 21474836493
2026-01-22T01:55:39.141 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs
2026-01-22T01:55:39.141 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705676
2026-01-22T01:55:39.142 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1
2026-01-22T01:55:39.142 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1
2026-01-22T01:55:39.143 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705676
2026-01-22T01:55:39.143 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2
2026-01-22T01:55:39.144 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705676
2026-01-22T01:55:39.145 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705676'
2026-01-22T01:55:39.145 INFO:tasks.workunit.client.0.trial038.stdout:waiting osd.1 seq 38654705676
2026-01-22T01:55:39.145 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1
2026-01-22T01:55:39.375 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705676 -lt 38654705676
2026-01-22T01:55:39.375 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs
2026-01-22T01:55:39.376 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status
2026-01-22T01:55:39.376 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs
2026-01-22T01:55:39.712 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 2 == 0
2026-01-22T01:55:39.712 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true
2026-01-22T01:55:39.712 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean
2026-01-22T01:55:39.713 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression
2026-01-22T01:55:39.713 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | '
2026-01-22T01:55:39.713 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)'
2026-01-22T01:55:39.713 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs
2026-01-22T01:55:39.713 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length'
2026-01-22T01:55:39.931 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=2
2026-01-22T01:55:39.932 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs
2026-01-22T01:55:39.932 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status
2026-01-22T01:55:39.932 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs
2026-01-22T01:55:40.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 2 = 2
2026-01-22T01:55:40.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break
2026-01-22T01:55:40.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0
2026-01-22T01:55:40.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:505: TEST_backfill_pool_priority: '[' 4 '!=' 0 ']'
2026-01-22T01:55:40.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:507: TEST_backfill_pool_priority: echo '4 error(s) found'
2026-01-22T01:55:40.250 INFO:tasks.workunit.client.0.trial038.stdout:4 error(s) found
2026-01-22T01:55:40.250 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: delete_pool test1
2026-01-22T01:55:40.250 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test1
2026-01-22T01:55:40.250 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it
2026-01-22T01:55:40.565 INFO:tasks.workunit.client.0.trial038.stderr:pool 'test1' does not exist
2026-01-22T01:55:40.573 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:513: TEST_backfill_pool_priority: delete_pool test2
2026-01-22T01:55:40.573 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test2
2026-01-22T01:55:40.573 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it
2026-01-22T01:55:40.845 INFO:tasks.workunit.client.0.trial038.stderr:pool 'test2' does not exist
2026-01-22T01:55:40.852 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:514: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio
2026-01-22T01:55:40.853 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace
2026-01-22T01:55:40.853 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true
2026-01-22T01:55:40.853 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true
2026-01-22T01:55:40.853 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true
2026-01-22T01:55:40.853 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace
2026-01-22T01:55:46.165 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0
2026-01-22T01:55:46.165 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:515: TEST_backfill_pool_priority: return 4
2026-01-22T01:55:46.165 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: return 1
Updated by Laura Flores about 2 months ago
- Related to Bug #48842: qa/standalone/osd/osd-recovery-prio.sh: TEST_recovery_pool_priority failed added
Updated by Laura Flores about 2 months ago
Added an old issue where this happened due to a slow test machine. This new issue might be similar in that the trial machines have a different speed than the smithi's. Just a hunch...
Updated by Laura Flores about 2 months ago ยท Edited
Here is a link to the test that failed: https://github.com/ceph/ceph/blob/9c1a6fe13f87fb1516c7e22b4ad13bc46010ddfc/qa/standalone/osd-backfill/osd-backfill-prio.sh#L328
Updated by Radoslaw Zarzynski about 2 months ago
I think that even after https://github.com/ceph/ceph/pull/38857 the test is fragile to timings.
Updated by Laura Flores about 2 months ago
- Backport set to reef,squid,tentacle
/a/yuriw-2026-01-23_21:16:01-rados-wip-yuri13-testing-2026-01-23-1852-reef-distro-default-trial/15614
Found on reef. This really does seem connected to the trial nodes being at a different speed than the smithis.
Updated by Laura Flores about 2 months ago
/a/lflores-2026-01-23_19:07:45-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/15247
Updated by Aishwarya Mathuria about 2 months ago
/a/yuriw-2026-01-27_16:21:32-rados-wip-yuri10-testing-2026-01-22-2036-tentacle-distro-default-trial/21911/
Updated by Laura Flores about 2 months ago
/a/lflores-2026-01-26_23:21:06-rados-wip-yuri12-testing-2026-01-22-2045-distro-default-trial/19114
Updated by Radoslaw Zarzynski about 2 months ago
Looks correlated to the new machines.
Updated by Sridhar Seshasayee about 2 months ago
/a/skanta-2026-01-27_05:35:03-rados-wip-bharath1-testing-2026-01-26-1242-distro-default-trial/19782
Updated by Laura Flores about 2 months ago
/a/skanta-2026-01-28_09:27:17-rados-wip-bharath2-testing-2026-01-27-0639-squid-distro-default-trial/23143
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-01-30_23:46:16-rados-wip-bharath7-testing-2026-01-29-2016-distro-default-trial/28587
Updated by Connor Fawcett about 1 month ago
/a/skanta-2026-01-27_07:02:07-rados-wip-bharath3-testing-2026-01-26-1323-distro-default-trial/19880
Updated by Laura Flores about 1 month ago
/a/yuriw-2026-02-03_16:00:06-rados-wip-yuri4-testing-2026-02-02-2122-distro-default-trial/31983
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
/a/skanta-2026-01-29_13:10:13-rados-wip-bharath6-testing-2026-01-29-0855-squid-distro-default-trial/25768
Updated by Nitzan Mordechai about 1 month ago
/a/yuriw-2026-02-04_23:08:40-rados-wip-yuri3-testing-2026-02-04-1948-tentacle-distro-default-trial/35497
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-02-07_00:02:26-rados-wip-bharath7-testing-2026-02-06-0906-distro-default-trial/39134
Updated by Lee Sanders about 1 month ago
/a/skanta-2026-01-29_13:05:02-rados-wip-bharath5-testing-2026-01-28-2018-distro-default-trial/25733
Updated by Jaya Prakash about 1 month ago
/a/jayaprakash-2026-02-06_12:54:34-rados-jaya-bs-testing-05-02-2026-distro-default-trial/38097
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
- Assignee set to Kamoltat (Junior) Sirivadhna
Taking a look
Updated by Nitzan Mordechai about 1 month ago
/a/nmordech-2026-02-09_08:34:53-rados-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/41033
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
- Backport changed from reef,squid,tentacle to squid,tentacle
lets backport only squid and tentacle since reef is EOL
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
- Status changed from New to Fix Under Review
- Pull request ID set to 67295
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-02-05_03:38:32-rados-wip-bharath2-testing-2026-02-03-0542-distro-default-trial/35669
Updated by Laura Flores about 1 month ago
/a/yuriw-2026-01-27_16:01:09-rados-reef-release-distro-default-trial/20731
Updated by Naveen Naidu about 1 month ago
/a/skanta-2026-01-26_08:54:40-rados-wip-bharath4-testing-2026-01-26-1300-distro-default-trial/17851
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
PR is ready for review
Updated by Nitzan Mordechai about 1 month ago
/a/skanta-2026-02-02_23:43:28-rados-wip-bharath9-testing-2026-02-02-0839-distro-default-trial/30430
Updated by Aishwarya Mathuria about 1 month ago
/a/yuriw-2026-02-17_20:43:43-rados-wip-yuri6-testing-2026-02-17-1732-squid-distro-default-trial/53899
Updated by Lee Sanders 26 days ago
/a/skanta-2026-02-07_14:54:11-rados-wip-bharath5-testing-2026-02-06-2052-distro-default-trial/39492
Updated by Radoslaw Zarzynski 26 days ago
- Status changed from Fix Under Review to Pending Backport
Updated by Upkeep Bot 26 days ago
- Merge Commit set to 157336f4d881ef1d0e92e403552771cc491e0f02
- Fixed In set to v20.3.0-5533-g157336f4d8
- Upkeep Timestamp set to 2026-02-23T18:48:50+00:00
Updated by Upkeep Bot 26 days ago
- Copied to Backport #75115: squid: TEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote" added
Updated by Upkeep Bot 26 days ago
- Copied to Backport #75116: tentacle: TEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote" added
Updated by Jaya Prakash 25 days ago
/a/yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial/62218
Updated by Sridhar Seshasayee 19 days ago
/a/sseshasa-2026-02-26_14:56:45-rados-wip-sseshasa-testing-2026-02-26-1772100687-distro-default-trial/72480
Updated by Nitzan Mordechai 18 days ago
/a/yaarit-2026-03-03_02:44:53-rados-wip-rocky10-branch-of-the-day-2026-03-02-1772457124-tentacle-distro-default-trial/77514
Updated by Nitzan Mordechai 17 days ago
/a/yaarit-2026-03-04_01:35:25-rados-wip-rocky10-branch-of-the-day-2026-03-03-1772543694-tentacle-distro-default-trial/80272
Updated by Aishwarya Mathuria 10 days ago
/a/skanta-2026-03-07_15:39:05-rados-wip-bharath4-testing-2026-03-05-1456-tentacle-distro-default-trial/93064
Updated by Connor Fawcett 9 days ago
/a/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68649
Updated by Jaya Prakash 5 days ago
/a/jayaprakash-2026-03-06_10:20:34-rados-wip-jaya-bs-testing-06-03-2025-distro-default-trial/90321
Updated by Lee Sanders 2 days ago
/a/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71082
/a/yuriw-2026-02-26_16:03:22-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/72557
Actions