box: dynamically adjust xlog readahead size#8113
Merged
locker merged 1 commit intotarantool:masterfrom Jan 30, 2023
Gumix:iverbin/gh-8108-increase-xlog_read_ahead
Merged
box: dynamically adjust xlog readahead size#8113locker merged 1 commit intotarantool:masterfrom Gumix:iverbin/gh-8108-increase-xlog_read_ahead
locker merged 1 commit intotarantool:masterfrom
Gumix:iverbin/gh-8108-increase-xlog_read_ahead
Conversation
locker
reviewed
Jan 9, 2023
XLOG_READ_AHEAD from 16 KB to 8 MB
locker
reviewed
Jan 16, 2023
locker
reviewed
Jan 18, 2023
locker
approved these changes
Jan 18, 2023
alyapunov
approved these changes
Jan 19, 2023
XLOG_READ_AHEAD defines the number of bytes added to the `count' argument of the `pread' syscall, currently it equals to 16 KB. However xlog and snap files are written by 128 KB chunks of data, which turn into ~80 KB chunks after compression (in average, depending on the data), so the 16 KB read- ahead doesn't make any sense. According to performance experiments, 8 MB readahead gives the best results for large files. However, 8 MB read buffers would increase memory consumption of the replication relays, which usually read small portions of data and does not need such a big buffers. For this reason, dynamically- sized read buffer is implemented by this patch. The minimal readahead is now 128 KB, and the maximal is 8 MB. As a result, the recovery time of a 900 MB snapshot decreased from 25 sec to 13 sec (when reading from HDD). Performance of a recovery from SSD is not affected. Closes #8108 NO_DOC=performance improvement NO_TEST=performance improvement
Contributor
Author
|
Updated the diff --git a/test/unit/xlog.c b/test/unit/xlog.c
index ee9d52971..ae4d920ef 100644
--- a/test/unit/xlog.c
+++ b/test/unit/xlog.c
@@ -131,14 +131,10 @@ test_dynamic_sized_ibuf(void)
fail_if(row.lsn != prev_lsn + 1);
prev_lsn = row.lsn;
}
- size_t capacity = ibuf_capacity(&cursor.rbuf);
- size_t start_capacity = cursor.rbuf.start_capacity;
is(cursor.read_ahead, XLOG_READ_AHEAD_MIN,
"read_ahead decreased to %d", XLOG_READ_AHEAD_MIN);
- ok(capacity >= start_capacity && capacity < 2 * start_capacity,
- "ibuf capacity decreased to [%zu..%zu)",
- start_capacity, 2 * start_capacity);
+ ok(ibuf_capacity(&cursor.rbuf) == 0, "ibuf capacity decreased to 0");
xlog_cursor_close(&cursor, false);
fail_if(xlog_close(&xlog, false) < 0); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
XLOG_READ_AHEADdefines the number of bytes added to thecountargumentof the
preadsyscall, currently it equals to 16 KB. However xlog and snapfiles are written by 128 KB chunks of data, which turn into ~80 KB chunks
after compression (in average, depending on the data), so the 16 KB read-
ahead doesn't make any sense.
According to performance experiments, 8 MB readahead gives the best results
for large files. However, 8 MB read buffers would increase memory
consumption of the replication relays, which usually read small portions of
data and does not need such a big buffers. For this reason, dynamically-
sized read buffer is implemented by this patch. The minimum readahead is
now 128 KB, and the maximum is 8 MB. As a result, the recovery time of a
900 MB snapshot decreased from 25 sec to 13 sec (when reading from HDD).
Performance of a recovery from SSD is not affected.
Closes #8108
Do not merge until tarantool/small#58 is merged!