Ceph » rgw

S3 Server-Side copy only copies the head-object and instead of copying the tail-objects it increments the ref_count on the Copy-SRC.
It then copies the manifest from SRC to Dest enabling access to the full object data.
When the SRC object is removed we decrement the ref-count on the tail-objects (correctly), but when the Dest object is removed we pass the wrong ref_tag to cls_refcount_put() so the tail objects are never deleted.

Recreation steps:
0) start with an empty ceph install and no objects in it
1) create bucket1 -> s3cmd mb s3://bucket1
2) choose a source file larger than 4MB (e.g. ceph/build/bin/osdmaptool)
3) put the file in a bucket -> s3cmd put s3://bucket1/osm ceph/build/bin/osdmaptool
4) server-side copy -> s3cmd cp s3://bucket1/osm s3://bucket1/osm_copy
5) list the object in rados -> bin/rados -p default.rgw.buckets.data
output should look something like this with 2 head objects and 3 shared tail-objects
[gbenhano@o09 build]$ bin/rados -p default.rgw.buckets.data ls
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_2
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_1
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_3
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1_osm_copy
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1_osm
6) remove both objects ->
s3cmd del s3://bucket1/osm
s3cmd del s3://bucket1/osm_copy
7) list again and you will see that the tail objects are not removed
gbenhano@o09 build]$ bin/rados -p default.rgw.buckets.data ls
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_2
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_1
6a468ac1-d575-4ff9-9409-8b9d5bec2abd.4182.1__shadow_.oikVkne8F5shYFl84PAS0fvbViMiee4_3
8)force gc to work -> radosgw-admin gc process --include-all --bucket=bucket1
9) list rados again and you will still see the 3 tail-objects
10) listing from rados-admin -> radosgw-admin bucket stats
You will see that usage is set at zero
11) rados df will still report the tail objects

The problem is that when we inc ref_count we pass a "ref_tag" to cls_refcount_ get (op, ref_tag , true), but when we remove the object and call cls_refcount_ put (op, ref_tag , true) we pass in another ref_tag.
The delete pass in the value in the "tag" attribute, but we used another value for the cls_refcount_ get (op, ref_tag , true)