Ceph » rgw

We observed a bug in Ceph where we corrupted around 60k objects in S3. The objects are still getting listed by S3 clients (the metadata seems to be fine) but when you try download the object you get a NoSuchKey error or the download starts but never finishes (missing parts).

After some research we were able to reproduce it (in v17.2.6, v17.2.8 and v19.2.2).

Steps to reproduce it:

1. Create 2 buckets with the same credential
2. Upload a file thats greater than 4MB to the first bucket (I used a 10MB file). This makes sure we get a shadow object in Rados. (I have not tested with multipart uploads)
3. Use rclone to server-side copy the file to the second bucket:
   

   rclone copyto -v ceph-remote:test-corruption-source-bucket/myfile_small ceph-remote:test-corruption-destination--bucket/myfile_small
   

4. When you inspect the rados location of the file in the destination bucket you will see that it refers to the same location on rados as the file in the source bucket.
   

   sudo radosgw-admin object stat --bucket test-corruption-source-bucket --object myfile_small
   sudo radosgw-admin object stat --bucket test-corruption-destination-bucket --object myfile_small
   

5. Delete the destination bucket but use flag bypass-gc
   

   sudo radosgw-admin bucket rm --bucket=test-corruption-destination-bucket --purge-objects --bypass-gc
   

6. Source file gets corrupted because bypass-gc does not check if the files are still referenced somewhere else.

I tested a normal delete and this does not corrupt the source. I also tested without the bypass-gc flag, this also does not corrupt source. In both cases I could see the shadow objects being scheduled for GC:

sudo radosgw-admin gc list --include-all

But once GC has run, the list is empty and the source file is still intact. So it looks like GC does check the reference in other indexes and the bug only happens if GC is being bypassed.

If you need more information please let me know.