osd: compress data in recovery process#3303
Conversation
fbcb1ca to
5294e4f
Compare
|
FAIL: the output of run-make-check.sh on bcb0269 is http://paste.pound-python.org/ |
Sent from |
This is pretty interesting! Mostly because it is so simple. Several comments, but I think we should discuss the best compression strategy before investing too much time to make sure we take the right approach. |
src/common/buffer.cc
Outdated
There was a problem hiding this comment.
this allocates, decompresses, copies, then deallocates. Instead, you can do
bufferptr bp = buffer::create_page_aligned(max_compressed_size);
uint32_t actual = LZ4_compress(pch_src, bp.c_str(), input_size);
bp.set_len(actual);
dest.append(bp);
to avoid the copy. we will allocate more memory than we need, which isn't ideal, but probably good enough for now.
|
I wonder if, instead of doing this specifically for recovery, we should do it as a generic feature of the messenger wire protocol. Then we could compress all messages that pass over the wire. Is there any reason not to go this route? |
|
@liewegas, becasuse in recover process, ceph will write 4M object each time. So the network bandwith is bottleneck insteand of cpu in 1000Mbits environment . We can compress data using a little cpu resource to speed up the recover process. In normal 4k write scene, maybe cpu resource is bottleneck, I think we should consider. |
|
Good commit. Agree with sage, it would better let "compress" feature become a impl of Messenger? So caller can call "send_message(m, COMPRESS_FLAGS)" depends on caller config value or let Messenger make a smart decision according to message? |
|
@yuyuyu101, this also looks good to me. |
5294e4f to
d503111
Compare
|
FAIL: the output of run-make-check.sh on 40987d9 is http://paste.pound-python.org/show/4MvE2sUiC38Yf5w1dqkt/ |
d503111 to
7947e6c
Compare
|
FAIL: the output of run-make-check.sh on 9b0a050 is http://paste.pound-python.org/show/CyEYmkl5wViXlUxqw82E/ |
common/buffer.cc:34:17: fatal error: lz4.h: No such file or directory #include "lz4.h" ^ compilation terminated. You need to add the dependency (lz4.h package) to debian/control and ceph.spec.in also. |
ceph maybe run out of network bandwith in recovery process, compress data if needed. Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
|
Since we have to create new buffers anyway, I'd prefer that compress and decompress returned new buffer::list's. |
|
Should there be a size threshold for compression? Actually, with small objects, we tend to combine a bunch of Pushes, so we might want to compress after that. In fact, I think Sage is right, we should add machinery at the Messenger level to allow us to tag Messages as suitable for compression. Perhaps a bool should_compress() const { return false; } in the message interface which Push messages might return true for. |
|
I am wondering should we set compression preference on OSD bases? For example, if communicating with a remote OSD, we can compress all messages since the bandwidth is limited. |
|
See #3490 |
Ceph may run out of network bandwith in recovery process, compress data if needed.
By default, I will use lz4. Maybe we can shoose any compression algorithm by user as ec pools does.
To run of my patch, First of all, we should install lz4 from https://code.google.com/p/lz4/.
Signed-off-by: Xinze Chi xmdxcxz@gmail.com