rgw: track compressed size with bucket index#1
Closed
cbodley wants to merge 5915 commits intoaclamk:alyona/wip_rgw_compressionfrom
Closed
rgw: track compressed size with bucket index#1cbodley wants to merge 5915 commits intoaclamk:alyona/wip_rgw_compressionfrom
cbodley wants to merge 5915 commits intoaclamk:alyona/wip_rgw_compressionfrom
Conversation
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Igor Fedotov <ifedotovo@miranitis.com>
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
os/bluestore: make assert conditional with macro for allocator Reviewed-by: Sage Weil <sage@redhat.com>
os/bluestore: add checks and kill unreachable code Reviewed-by: Sage Weil <sage@redhat.com>
compressor/zlib: zlib wrapper fix Reviewed-by: Sage Weil <sage@redhat.com>
rocksdb: pull up to master (4.12 + a few patches)
rgw: work around curl_multi_wait bug with non-blocking reads Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Matches rm(). Signed-off-by: Sage Weil <sage@redhat.com>
common: only call crypto::init once per CephContext Reviewed-by: Orit Wasserman <owasserm@redhat.com>
…ion statistics and hence compression workability Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
run-*make-check.sh: Make DRY_RUN actually mean a dry run Reviewed-by: Kefu Chai <kchai@redhat.com>
…vement test/store_test: extend Bluestore compression test to verify compress…
…ilure. Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
We need to remove items from the cache LRUs in their destructors, and it is very awkward to ensure that all releasing paths either hold the cache lock or have already cleared the cache items out. Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
os/bluestore: fix a few leaks Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
…lugins to ensure PluginRegistry access for them Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
…ypeswithauto common/Timer.cc: replace long types with auto
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>
osd/OSD.h: make some members private
The rocksdb transaction is now been logged into ceph log based on debug level. If transaction is failed, the error code, error string and the entire transaction is dumped in the log. Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
StupidAllocator is probably a more appropriate choice for bluefs. Either way, it needn't be tied to what BlueStore is using. Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Apr 20, 2017
Fix for: CID 1399581 (#1 of 1): Data race condition (MISSING_LOCK) missing_lock: Accessing this->state.atime without holding lock rgw::RGWFileHandle.mtx. Elsewhere, "_ZN3rgw13RGWFileHandle5StateE.atime" is accessed with rgw::RGWFileHandle.mtx held 2 out of 3 times (2 of these accesses strongly imply that it is necessary). Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
aclamk
pushed a commit
that referenced
this pull request
Jun 26, 2017
Fixes the Coverity Scan Report: CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO) 9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Jun 27, 2017
Fixes the coverity scan report: 1412839 Uninitialized pointer field CID 1412839 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)2. uninit_member: Non-static class member array is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Jun 27, 2017
…t::parse_option() Fixes the Coverity Scan Report: CID 1412776 (#1 of 1): Explicit null dereferenced (FORWARD_NULL)19. var_deref_model: Passing null pointer option_name to operator <<, which dereferences it. Fixed the review comments too in this commit. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Jul 7, 2017
Fixes the Coverity Scan Report: CID 1412617 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 3. uninit_member: Non-static class member field_aliases is not initialized in this constructor nor in any functions that it calls. 5. uninit_member: Non-static class member restricted_fields is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Jul 7, 2017
Fixes the Coverity Scan Report: CID 1412616 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member worker is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Jul 7, 2017
Fixed the Coverity Scan Report: CID 1412615 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member val is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes the Coverity Scan Report: CID 1351737 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member worker is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes the Coverity Scan Report: CID 1352180 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member field fs.rgw is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes the Coverity Scan Report: CID 1396146 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member max_objs is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member obj_names is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Jos Collin <jcollin@redhat.com>
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
I'm seeing sporadic single thread deadlocks on fio stat_mutex during krbd
thrash runs:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f89ee730740 (LWP 15604) 0x00007f89ed9f41bd in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f89ed9f41bd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f89ed9f17b2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x00000000004429b9 in fio_mutex_down (mutex=0x7f89ee72d000) at mutex.c:170
ceph#3 0x0000000000459704 in thread_main (data=<optimized out>) at backend.c:1639
ceph#4 0x000000000045b013 in fork_main (offset=0, shmid=<optimized out>, sk_out=0x0) at backend.c:1778
ceph#5 run_threads (sk_out=sk_out@entry=0x0) at backend.c:2195
ceph#6 0x000000000045b47f in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2400
ceph#7 0x000000000040cb0c in main (argc=2, argv=0x7fffad3e3888, envp=<optimized out>) at fio.c:63
(gdb) up 2
170 pthread_cond_wait(&mutex->cond, &mutex->lock);
(gdb) p mutex.lock.__data.__owner
$1 = 15604
Upgrading to 2.21 seems to make these go away.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1409697 (#1 of 1): Assign instead of compare (PW.ASSIGN_WHERE_COMPARE_MEANT) 1. assign_where_compare_meant: use of "=" where "==" may have been intended 2. primary_file: During compilation of file '/src/rgw/librgw.cc' Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1414508 (#1 of 1): Assign instead of compare (PW.ASSIGN_WHERE_COMPARE_MEANT) 1. assign_where_compare_meant: use of "=" where "==" may have been intended Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1351525 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS) 7. negative_returns: sd is passed to a parameter that cannot be negative. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1395347 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS) 3. negative_returns: fd is passed to a parameter that cannot be negative Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1403249 (#1 of 1): Uninitialized scalar variable (UNINIT) 6. uninit_use: Using uninitialized value snap_protected. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1402141 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member reopen is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** CID 1402142 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member ops_allowance is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** 1405280 Explicit null dereferenced CID 1405280 (#1 of 1): Explicit null dereferenced (FORWARD_NULL) 20. var_deref_model: Passing null pointer diri to check_access, which dereferences it. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixed: ** 1405291 Explicit null dereferenced. CID 1405291 (#1 of 1): Explicit null dereferenced (FORWARD_NULL) 10. var_deref_op: Dereferencing null pointer s. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes the Coverity Scan Report: ** 1414521 Uninitialized scalar field CID 1414521 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member m_writeRc is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes: 1414515 Uninitialized pointer field CID 1414515 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 22. uninit_member: Non-static class member perf_counter is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes: 1244200 Uninitialized pointer field CID 1244200 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member msg is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes: ** 717285 Uninitialized scalar field CID 717285 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes coverity Issue: ** 717293 Uninitialized scalar field 2. uninit_member: Non-static class member table is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls. CID 717293 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member reqid is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes coverity Issue: ** 717296 Uninitialized scalar field CID 717296 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member handle is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes coverity Issue: ** 717297 Uninitialized scalar field 2. uninit_member: Non-static class member handle is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member version is not initialized in this constructor nor in any functions that it calls. CID 717297 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member oldest_version is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes coverity Issue: 2. uninit_member: Non-static class member epoch is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member first_committed is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member last_committed is not initialized in this constructor nor in any functions that it calls. 10. uninit_member: Non-static class member pn_from is not initialized in this constructor nor in any functions that it calls. 12. uninit_member: Non-static class member pn is not initialized in this constructor nor in any functions that it calls. 14. uninit_member: Non-static class member uncommitted_pn is not initialized in this constructor nor in any functions that it calls. CID 717298 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 16. uninit_member: Non-static class member latest_version is not initialized in this constructor nor in any functions that it calls. Signed-off-by: Amit Kumar amitkuma@redhat.com
aclamk
pushed a commit
that referenced
this pull request
Aug 17, 2017
Fixes coverity Issue: ** 717299 Uninitialized scalar field 2. uninit_member: Non-static class member op is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member paxos_first_version is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member paxos_last_version is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member has_ever_joined is not initialized in this constructor nor in any functions that it calls. CID 717299 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 10. uninit_member: Non-static class member required_features is not initialized in this constructor nor in any functions that it calls Signed-off-by: Amit Kumar amitkuma@redhat.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this patch set changes how buckets track the total size as reported in
RGWStatBucket. instead of tracking the uncompressed size as a separate attribute on the bucket instance, handle it the same way as our existing bucket size accountingthis adds an
actual_sizefield torgw_bucket_category_stats, in addition to the existingtotal_size.total_size(which is used byRGWStatBucket) represents the original, uncompressed size of the objects, whileactual_sizerepresents the actual compressed size of the objects as stored in rados (a generic name was chosen instead ofcompressed_sizebecause we'll eventually handle encryption as well)this also repurposes the existing
rgw_bucket_dir_entry_meta::sizefield, which is currently set to the same value as itsaccounted_size. thissizewill now represent the actual compressed size of the object, and is used in cls to updatergw_bucket_category_stats::actual_sizein all the same ways thataccounted_sizeis applied torgw_bucket_category_stats::total_sizethe
RGWPutObjProcessor::complete()interface now takes anaccounted_sizefrom the caller (generallys->obj_size), and passes that throughObject::Write::write_meta()andBucket::UpdateIndex::complete(), which setsRGWObjEnt::accounted_sizeforcls_rgw_bucket_complete_op()there were a few other callers of
Write::write_meta()andUpdateIndex::complete()outside ofRGWPutObjProcessorwhich also needed access to anaccounted_size. for these, i added aRGWObjState::accounted_size, whichRGWRados::get_obj_state()initializes with the compression attribute, if one exists (this should be the only part that needs to be extended when we add encryption)the output of
radosgw-admin bucket stats --bucket=foowas extended to include fields forsize_utilizedandsize_kb_utilized