[Extension] oak-incremental-index: Low resource (RAM and CPU) incremental-index implementation using off-heap key/value map (OakMap)#10001
Conversation
|
This pull request introduces 3 alerts when merging eed0606a67f89b653e8cc402eb0fe36af511de30 into 45b699f - view on LGTM.com new alerts:
|
4156fee to
7495ac9
Compare
|
Hi @jihoonson, @leventov, @gianm, @ebortnik, @b-slim, and @jon-wei. |
|
Glad you guys are still working on this! I will take a look soon. |
|
@jihoonson Thank you. We are eager to hear your valuable feedback. |
|
@clintropolis We noticed your recent commits modified the same benchmarks as we did in this PR (commit #a607c95). Since you are familiar with this part of Druid, we will appreciate it if you can take the time to review the benchmark part of this PR. This commit alone can contribute to Druid since it resolves issues we had with the benchmarks. You can find a summary of the modifications here. |
|
We noticed a lot of Druid users run their workload on Amazon EC2. We want to point out that this PR will not only improve performance but will also reduce operational costs by allowing the users to choose more affordable EC2 instances without sacrificing performance. The figure below shows the operational cost of different required ingestion throughput on Amazon EC2. |
server/src/main/java/org/apache/druid/segment/realtime/plumber/Sink.java
Show resolved
Hide resolved
2aac78d to
0e313e8
Compare
|
Thanks to @yuanlihan for helping us find bugs on the realtime query side. Our system experiments focused on batch ingestion, so his contribution is highly appreciated. |
95d95db to
663f155
Compare
|
Hi @jihoonson, have you had a chance to check out our issue/PR? We will be happy to answer any questions you might have. |
29e8453 to
9c83a01
Compare
UpdatesWe are working with our production teams at Verizon Media toward testing our incremental-index implementation on actual production data. As part of this effort, we discovered some issues with: (1) sketches, and (2) scenarios where multiple indexes are used during ingestion in a single Peon. |
…rsion # Conflicts: # pom.xml
...al-index/src/main/java/org/apache/druid/segment/incremental/oak/OakIncrementalIndexSpec.java
Outdated
Show resolved
Hide resolved
ebef5f6 to
b6ee2b0
Compare
# Conflicts: # processing/src/main/java/org/apache/druid/segment/incremental/IncrementalIndex.java # processing/src/main/java/org/apache/druid/segment/incremental/OnheapIncrementalIndex.java
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions. |
|
Please don't close this PR. |
|
This issue is no longer marked as stale. |
|
Index performance is very critical for us.Too many tasks means too many segments. |
@exherb For batch ingestion, each task can ingest different time periods. Those can be merged when the task is done, similar to how we merge the results from periodic flushes. So we can benefit from better throughput without sacrificing the number of segments. |
It’s take about 3 hours to merge for us. |
@exherb From my experiment, merging fewer, but larger segments take about the same amount of time as merging many small segments. Even so, this extension enables more data to be ingested into memory before it is flushed to disk, resulting in larger segments.
@exherb Yes. Except for some conflicts that I can resolve (upon demand). |
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
Fixes #9967.
Description
This PR introduces an extension that improves Druid’s ingestion memory and CPU efficiency. It uses 60% less memory and 50% less CPU-time to achieve the same performance. This translated to nearly double the system's ingestion-throughput with the same memory budget, and a 75% increase in throughput with the same CPU-time budget. The experimental setup and the full results are available here.
To understand the motivation and rationale behind some of the proposed changes below, it is necessary to read the related issue: #9967.
Introduce
OakIncrementalIndexWe add a new incremental index implementation:
OakIncrementalIndexas a Druid extension. The implementation is mostly borrowed fromOnheapIncrementalIndexandOffheapIncrementalIndex, but has a few notable differences:OakMapinstead of Java’sConcurrentSkipList(CSL)FactsHolder.persistIterable()), even in plain modeTo achieve the best performance of our implementation, we had to refactor some interfaces of
IncrementalIndexRowandIncrementalIndex. This refactoring is explained in #12122.Key changed/added classes in this commit
druid/extensions-contrib/oak-incremental-index. Most notable additions:OakIncrementalIndex: follows theIncrementalIndexAPIOakIncrementalIndexRow: follows theIncrementalIndexRowAPIOakKey: handles the serialization, deserialization, and comparison of keysThis PR has: