Ceph QA

No sure what happened, sorry
rebasing
@Laura Flores

@Matan Breizman - note that the failure in 7707685 (scrub-repair) is unrelated.

@Laura Flores, there are almost 50 red jobs. Should we schedule a re-run?

Ronen Friedman wrote in #note-14:

@Matan Breizman - note that the failure in 7707685 (scrub-repair) is unrelated.

Thanks.

Laura Flores wrote in #note-16:

@Yuri Weinstein can you schedule a rerun?

For some reason I could schedule a rerun but a full run is being scheduled

@Matan Breizman @Laura Flores pls review when ready

• Related Failures :* :
• https://github.com/ceph/ceph/pull/57015 - bluefs: bluefs alloc unit should only be shrink (7707982)
• https://github.com/ceph/ceph/pull/56531 shouldn't be merged yet as the issue persists. Left a comment in the PR. No need for a rerun since it's only a qa change. (7707707, 7707787)

• We still have around 60 failed jobs on each rados run. It looks like most of them are not related. However, it may be tricky to ignore 60 jobs on each batch (@Laura Flores).

• There are new ones failures which look suspicious, may be related to the failures above(?). I'll investigate on rerun.
  "reached maximum tries (51) after waiting for 300 seconds" - 7707938, 7707827.

• https://tracker.ceph.com/issues/65728 - Had 7 instances: 7707712, 7707734, 7707751, 7707848, 7707867, 7707905, 7707972
• https://tracker.ceph.com/issues/65824 - 4 instances: 7707711, 7707754, 7707919, 7707960

New, not related:
https://tracker.ceph.com/issues/66209

@Laura Flores I can't schedule a rerun
Can you pls try so we can see if this is my setup, this run, or the actual problem?

I see in the output looping thru log:

2024-05-23 15:02:41,096.096 DEBUG:teuthology.suite.merge:postmerge script running:

2024-05-23 15:02:41,098.098 DEBUG:teuthology.suite.merge:skipping config rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} due to postmerge filter
2024-05-23 15:02:41,098.098 DEBUG:teuthology.suite.merge:merging config thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps}

Here are my lines:

SHA1=
CEPH_BRANCH=
CEPH_QA_MAIL=
CEPH_REPO=
SUITE_REPO=
LIMIT=
DISTRO=
PRIO=
TEUTH=
MACHINE_NAME=
echo $SHA1
echo $CEPH_BRANCH

CEPH_REPO=https://github.com/ceph/ceph-ci.git

SHA1=6cd0f8013e2dea00c3a29a0d8b10656b132d7c80
CEPH_BRANCH=wip-yuri5-testing-2024-05-15-0804
SUITE_REPO=$CEPH_REPO
SUITE_BRANCH=$CEPH_BRANCH

LIMIT=10000
DISTRO=distro
TEUTH=main
MACHINE_NAME=smithi
CEPH_QA_MAIL=yweinste@redhat.com
PRIO=99

echo $SHA1
echo $CEPH_BRANCH
echo $SUITE_BRANCH

RERUN=yuriw-2024-05-20_19:47:14-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi

teuthology-suite -v -c $CEPH_BRANCH -m $MACHINE_NAME -r $RERUN --suite-repo $SUITE_REPO --suite-branch $SUITE_BRANCH --ceph-repo $CEPH_REPO -p $PRIO -R fail,dead,running,waiting --force-priority -k $DISTRO -t $TEUTH -S $SHA1

Yuri Weinstein wrote in #note-21:

@Laura Flores I can't schedule a rerun
Can you pls try so we can see if this is my setup, this run, or the actual problem?

Hey Yuri,
This run will probably need a rebuild either way - without the 2 blocking PRs. Thanks!

Matan Breizman wrote in #note-22:

Yuri Weinstein wrote in #note-21:

@Laura Flores I can't schedule a rerun
Can you pls try so we can see if this is my setup, this run, or the actual problem?

Hey Yuri,
This run will probably need a rebuild either way - without the 2 blocking PRs. Thanks!

When can I rebase, pls LMK

The scheduling issue is a stand-alone one

rerun bug => https://github.com/ceph/teuthology/pull/1946

When can I rebase, pls LMK

• https://github.com/ceph/ceph/pull/57015 - new changes, fixed previous issue.
• https://github.com/ceph/ceph/pull/56531 - should be excluded from this batch.

added a rerun for failed
@Matan Breizman fyi

The following comment was probably missed:

• https://github.com/ceph/ceph/pull/56531 - should be excluded from this batch.

However, the PR is only a `qa` change so no rerun is needed.

@Matan Breizman merged all but https://github.com/ceph/ceph/pull/56531

I think you forgot to add your comment/link to all PRs

I think you forgot to add your comment/link to all PRs

@Yuri Weinstein, I was about to but you were quicker than me :)