Project

General

Profile

Actions
Copy link

Bug #72947

open

Dedup failures from unclean run

Added by Adam Emerson 7 months ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Gabriel BenHanokh
Target version:
-
% Done:

0%

Source:
Q/A
Backport:
tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
65783
Tags (freeform):
backport_processed
Merge Commit:
f92cc238729af286f039fdf91afa448cdd83e791
Fixed In:
v20.3.0-3547-gf92cc23872
Released In:
Upkeep Timestamp:
2025-10-12T12:28:59+00:00

Description

We have intermittent failures of dedup tests that look to be a matter of not initializing/cleaning properly, from the warnings:

2025-08-21T13:07:12.336 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_md5_collisions
2025-08-21T13:07:12.336 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:12.336 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:12.336 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:12.385 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [  5%]
2025-08-21T13:07:12.495 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_small
2025-08-21T13:07:12.495 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:12.495 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:12.496 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:12.561 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [  8%]
2025-08-21T13:07:12.672 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_small_with_tenants
2025-08-21T13:07:12.672 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:12.672 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:12.672 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:12.722 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 11%]
2025-08-21T13:07:12.831 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_0_with_tenants
2025-08-21T13:07:12.831 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:12.832 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:12.832 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:12.910 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 14%]
2025-08-21T13:07:13.019 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_0
2025-08-21T13:07:13.019 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.019 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.020 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.069 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 17%]
2025-08-21T13:07:13.178 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_1_with_tenants
2025-08-21T13:07:13.178 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.178 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.178 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.227 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 20%]
2025-08-21T13:07:13.336 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_1
2025-08-21T13:07:13.336 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.336 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.336 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.386 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 23%]
2025-08-21T13:07:13.498 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_2_with_tenants
2025-08-21T13:07:13.498 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.499 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.499 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.548 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 26%]
2025-08-21T13:07:13.660 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_2
2025-08-21T13:07:13.660 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.660 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.660 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.734 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 29%]
2025-08-21T13:07:13.842 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_with_remove_multi_tenants
2025-08-21T13:07:13.843 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:13.843 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:13.843 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:13.893 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 32%]
2025-08-21T13:07:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_inc_with_remove
2025-08-21T13:07:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:14.050 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 35%]
2025-08-21T13:07:14.160 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_multipart_with_tenants
2025-08-21T13:07:14.160 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:14.160 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:14.160 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:14.209 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 38%]
2025-08-21T13:07:14.318 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_multipart
2025-08-21T13:07:14.318 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:14.318 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:14.318 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:14.392 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 41%]
2025-08-21T13:07:14.501 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_basic_with_tenants
2025-08-21T13:07:14.501 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------
2025-08-21T13:07:14.501 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1142 The system was left dirty from previous run
2025-08-21T13:07:14.501 INFO:teuthology.orchestra.run.smithi017.stdout:WARNING  dedup.test_dedup:test_dedup.py:1143 Make sure to remove all objects before starting
2025-08-21T13:07:14.551 INFO:teuthology.orchestra.run.smithi017.stdout:FAILED                                                                   [ 44%]
2025-08-21T13:07:14.662 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_dedup_basic
2025-08-21T13:07:14.662 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------- live log call ---------------------------------

http://qa-proxy.ceph.com/teuthology/anuchaithra-2025-08-21_12:44:18-rgw-wip-anrao2-testing-2025-08-20-1151-distro-default-smithi/8456389/teuthology.log

Related issues 1 (1 open — 0 closed)

Copied to rgw - Backport #73506: tentacle: Dedup failures from unclean runNewGabriel BenHanokhActions
Actions
Copy link
 #1

Updated by J. Eric Ivancich 6 months ago

  • Assignee set to Gabriel BenHanokh

@Gabriel BenHanokh -- please take a look.

Actions
Copy link
 #2

Updated by J. Eric Ivancich 6 months ago

I emailed Gabriel on 9/18/2025.

Actions
Copy link
 #3

Updated by Gabriel BenHanokh 6 months ago ยท Edited

test_dedup.py:: test_dedup_etag_corruption

2025-08-21T13:06:29.153 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1313 full dedup is enabled!
2025-08-21T13:06:38.620 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:06:44.073 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:06:49.580 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:06:55.086 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:07:00.591 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:07:06.095 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:07:11.604 INFO:teuthology.orchestra.run.smithi017.stdout:INFO dedup.test_dedup:test_dedup.py:1096 dedup completed in 5 seconds
2025-08-21T13:07:12.225 INFO:teuthology.orchestra.run.smithi017. stdout:FAILED [ 2%]
2025-08-21T13:07:12.336 INFO:teuthology.orchestra.run.smithi017.stdout:test_dedup.py::test_md5_collisions

The failure is on the first dedup-test (test_dedup_etag_corruption) everything after this is not interesting

I need to understand what is the issue causing this failure.
It seems no dedup can run at all which might be caused by lack of permissions (on a system using permissions)
This is something I fixed in https://github.com/ceph/ceph/pull/64730 which is ready to go

Actions
Copy link
 #4

Updated by J. Eric Ivancich 6 months ago

How does the full teuthology rgw suite come out when that PR is included?

Actions
Copy link
 #5

Updated by Gabriel BenHanokh 6 months ago

https://github.com/ceph/ceph/pull/64730 adds a throttling mechanism to dedup allowing the admin to slow down the process.
It also changes the permissions assigns to the dedup process giving it full RGW permissions.
This was done after a test failed because the system was setup with limited permissions blocking dedup from storing intermediate data

Actions
Copy link
 #6

Updated by J. Eric Ivancich 6 months ago

Gabriel BenHanokh wrote in #note-5:

https://github.com/ceph/ceph/pull/64730 adds a throttling mechanism to dedup allowing the admin to slow down the process.
It also changes the permissions assigns to the dedup process giving it full RGW permissions.
This was done after a test failed because the system was setup with limited permissions blocking dedup from storing intermediate data

Do you think the permissions change will fix the teuthology issue?

Actions
Copy link
 #7

Updated by J. Eric Ivancich 6 months ago

I'm wondering if this PR (https://github.com/ceph/ceph/pull/65783) is the intended solution to this tracker, @Gabriel BenHanokh .

Actions
Copy link
 #8

Updated by Yuval Lifshitz 5 months ago

  • Status changed from New to Pending Backport
  • Backport set to tentacle
  • Pull request ID set to 65783
Actions
Copy link
 #9

Updated by Upkeep Bot 5 months ago

  • Copied to Backport #73506: tentacle: Dedup failures from unclean run added
Actions
Copy link
 #10

Updated by Upkeep Bot 5 months ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #11

Updated by Upkeep Bot 5 months ago

  • Merge Commit set to f92cc238729af286f039fdf91afa448cdd83e791
  • Fixed In set to v20.3.0-3547-gf92cc23872
  • Upkeep Timestamp set to 2025-10-12T12:28:59+00:00
Actions
Copy link

Also available in: Atom PDF