Skip to content

rgw: track compressed size with bucket index#1

Closed
cbodley wants to merge 5915 commits intoaclamk:alyona/wip_rgw_compressionfrom
cbodley:wip-rgw-compression-bi
Closed

rgw: track compressed size with bucket index#1
cbodley wants to merge 5915 commits intoaclamk:alyona/wip_rgw_compressionfrom
cbodley:wip-rgw-compression-bi

Conversation

@cbodley
Copy link

@cbodley cbodley commented Sep 29, 2016

this patch set changes how buckets track the total size as reported in RGWStatBucket. instead of tracking the uncompressed size as a separate attribute on the bucket instance, handle it the same way as our existing bucket size accounting

this adds an actual_size field to rgw_bucket_category_stats, in addition to the existing total_size. total_size (which is used by RGWStatBucket) represents the original, uncompressed size of the objects, while actual_size represents the actual compressed size of the objects as stored in rados (a generic name was chosen instead of compressed_size because we'll eventually handle encryption as well)

this also repurposes the existing rgw_bucket_dir_entry_meta::size field, which is currently set to the same value as its accounted_size. this size will now represent the actual compressed size of the object, and is used in cls to update rgw_bucket_category_stats::actual_size in all the same ways that accounted_size is applied to rgw_bucket_category_stats::total_size

the RGWPutObjProcessor::complete() interface now takes an accounted_size from the caller (generally s->obj_size), and passes that through Object::Write::write_meta() and Bucket::UpdateIndex::complete(), which sets RGWObjEnt::accounted_size for cls_rgw_bucket_complete_op()

there were a few other callers of Write::write_meta() and UpdateIndex::complete() outside of RGWPutObjProcessor which also needed access to an accounted_size. for these, i added a RGWObjState::accounted_size, which RGWRados::get_obj_state() initializes with the compression attribute, if one exists (this should be the only part that needs to be extended when we add encryption)

the output of radosgw-admin bucket stats --bucket=foo was extended to include fields for size_utilized and size_kb_utilized

Igor Fedotov and others added 30 commits September 14, 2016 12:24
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Igor Fedotov <ifedotovo@miranitis.com>
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
os/bluestore: make assert conditional with macro for allocator

Reviewed-by: Sage Weil <sage@redhat.com>
os/bluestore: add checks and kill unreachable code

Reviewed-by: Sage Weil <sage@redhat.com>
compressor/zlib: zlib wrapper fix

Reviewed-by: Sage Weil <sage@redhat.com>
rocksdb: pull up to master (4.12 + a few patches)
rgw: work around curl_multi_wait bug with non-blocking reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Matches rm().

Signed-off-by: Sage Weil <sage@redhat.com>
common: only call crypto::init once per CephContext
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
…ion statistics and hence compression workability

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
run-*make-check.sh: Make DRY_RUN actually mean a dry run

Reviewed-by: Kefu Chai <kchai@redhat.com>
…vement

test/store_test: extend Bluestore compression test to verify compress…
…ilure.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
We need to remove items from the cache LRUs in their
destructors, and it is very awkward to ensure that all
releasing paths either hold the cache lock or have
already cleared the cache items out.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
os/bluestore: fix a few leaks

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
…lugins to ensure PluginRegistry access for them

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
…ypeswithauto

common/Timer.cc: replace long types with auto
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>
The rocksdb transaction is now been logged into ceph log based
on debug level. If transaction is failed, the error code,
error string and the entire transaction is dumped in the log.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
StupidAllocator is probably a more appropriate choice for bluefs.
Either way, it needn't be tied to what BlueStore is using.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
aclamk pushed a commit that referenced this pull request Apr 20, 2017
Fix for:

CID 1399581 (#1 of 1): Data race condition (MISSING_LOCK)
 missing_lock: Accessing this->state.atime without holding lock
 rgw::RGWFileHandle.mtx. Elsewhere, "_ZN3rgw13RGWFileHandle5StateE.atime"
 is accessed with rgw::RGWFileHandle.mtx held 2 out of 3 times
 (2 of these accesses strongly imply that it is necessary).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
aclamk pushed a commit that referenced this pull request Jun 26, 2017
Fixes The Coverity Scan Report:
CID 1412577 (#1 of 1): Division or modulo by float zero (DIVIDE_BY_ZERO)
35. divide_by_zero: In expression (float)mk / k, division by expression k which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jun 26, 2017
Fixes the Coverity Scan Report:
CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jun 27, 2017
Fixes the coverity scan report:
1412839 Uninitialized pointer field
CID 1412839 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)2. uninit_member: Non-static
class member array is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jun 27, 2017
…t::parse_option()

Fixes the Coverity Scan Report:
CID 1412776 (#1 of 1): Explicit null dereferenced (FORWARD_NULL)19. var_deref_model: Passing null pointer option_name to operator <<, which dereferences it.

Fixed the review comments too in this commit.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jul 7, 2017
Fixes the Coverity Scan Report:
CID 1412617 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
3. uninit_member: Non-static class member field_aliases is not initialized in this constructor nor in any functions that it calls.
5. uninit_member: Non-static class member restricted_fields is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jul 7, 2017
Fixes the Coverity Scan Report:
CID 1412616 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member worker is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Jul 7, 2017
Fixed the Coverity Scan Report:
CID 1412615 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member val is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes the Coverity Scan Report:
CID 1351737 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member worker is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes the Coverity Scan Report:
CID 1352180 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member field fs.rgw is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes the Coverity Scan Report:
CID 1396146 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member max_objs is not initialized in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member obj_names is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk pushed a commit that referenced this pull request Aug 17, 2017
I'm seeing sporadic single thread deadlocks on fio stat_mutex during krbd
thrash runs:

  (gdb) info threads
    Id   Target Id         Frame
  * 1    Thread 0x7f89ee730740 (LWP 15604) 0x00007f89ed9f41bd in __lll_lock_wait () from /lib64/libpthread.so.0
  (gdb) bt
  #0  0x00007f89ed9f41bd in __lll_lock_wait () from /lib64/libpthread.so.0
  #1  0x00007f89ed9f17b2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #2  0x00000000004429b9 in fio_mutex_down (mutex=0x7f89ee72d000) at mutex.c:170
  ceph#3  0x0000000000459704 in thread_main (data=<optimized out>) at backend.c:1639
  ceph#4  0x000000000045b013 in fork_main (offset=0, shmid=<optimized out>, sk_out=0x0) at backend.c:1778
  ceph#5  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2195
  ceph#6  0x000000000045b47f in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2400
  ceph#7  0x000000000040cb0c in main (argc=2, argv=0x7fffad3e3888, envp=<optimized out>) at fio.c:63
  (gdb) up 2
  170                     pthread_cond_wait(&mutex->cond, &mutex->lock);
  (gdb) p mutex.lock.__data.__owner
  $1 = 15604

Upgrading to 2.21 seems to make these go away.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1409697 (#1 of 1): Assign instead of compare (PW.ASSIGN_WHERE_COMPARE_MEANT)
1. assign_where_compare_meant: use of "=" where "==" may have been intended
2. primary_file: During compilation of file '/src/rgw/librgw.cc'

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1414508 (#1 of 1): Assign instead of compare (PW.ASSIGN_WHERE_COMPARE_MEANT)
1. assign_where_compare_meant: use of "=" where "==" may have been intended

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1351525 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS)
7. negative_returns: sd is passed to a parameter that cannot be negative.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1395347 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS)
3. negative_returns: fd is passed to a parameter that cannot be negative

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1403249 (#1 of 1): Uninitialized scalar variable (UNINIT)
6. uninit_use: Using uninitialized value snap_protected.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1402141 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member reopen is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** CID 1402142 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member ops_allowance is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:
** 1405280 Explicit null dereferenced
CID 1405280 (#1 of 1): Explicit null dereferenced (FORWARD_NULL)
20. var_deref_model: Passing null pointer diri to check_access, which dereferences it.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixed:

** 1405291 Explicit null dereferenced.
CID 1405291 (#1 of 1): Explicit null dereferenced (FORWARD_NULL)
10. var_deref_op: Dereferencing null pointer s.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes the Coverity Scan Report:

** 1414521 Uninitialized scalar field
CID 1414521 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
3. uninit_member: Non-static class member m_writeRc is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes:

1414515 Uninitialized pointer field
CID 1414515 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
22. uninit_member: Non-static class member perf_counter is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes:

1244200 Uninitialized pointer field
CID 1244200 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member msg is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes:

** 717285 Uninitialized scalar field
CID 717285 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes coverity Issue:

** 717293 Uninitialized scalar field
2. uninit_member: Non-static class member table is not initialized in this constructor nor in any functions that it calls.
  	4. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls.

CID 717293 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member reqid is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes coverity Issue:

** 717296 Uninitialized scalar field
CID 717296 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member handle is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes coverity Issue:

** 717297 Uninitialized scalar field
2. uninit_member: Non-static class member handle is not initialized in this constructor nor in any functions that it calls.
  	4. uninit_member: Non-static class member version is not initialized in this constructor nor in any functions that it calls.

CID 717297 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member oldest_version is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes coverity Issue:

2. uninit_member: Non-static class member epoch is not initialized in this constructor nor in any functions that it calls.
  	4. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls.
   	6. uninit_member: Non-static class member first_committed is not initialized in this constructor nor in any functions that it calls.
   	8. uninit_member: Non-static class member last_committed is not initialized in this constructor nor in any functions that it calls.
   	10. uninit_member: Non-static class member pn_from is not initialized in this constructor nor in any functions that it calls.
   	12. uninit_member: Non-static class member pn is not initialized in this constructor nor in any functions that it calls.
   	14. uninit_member: Non-static class member uncommitted_pn is not initialized in this constructor nor in any functions that it calls.

CID 717298 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
16. uninit_member: Non-static class member latest_version is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk pushed a commit that referenced this pull request Aug 17, 2017
Fixes coverity Issue:

** 717299 Uninitialized scalar field
2. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls.
  	4. uninit_member: Non-static class member paxos_first_version is not initialized in this constructor nor in any functions that it calls.
   	6. uninit_member: Non-static class member paxos_last_version is not initialized in this constructor nor in any functions that it calls.
   	8. uninit_member: Non-static class member has_ever_joined is not initialized in this constructor nor in any functions that it calls.

CID 717299 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
10. uninit_member: Non-static class member required_features is not initialized in this constructor nor in any functions that it calls

Signed-off-by: Amit Kumar amitkuma@redhat.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.