[refactor] IncrementalIndex extension preparations#12122
Closed
liran-funaro wants to merge 10 commits intoapache:masterfrom
Closed
[refactor] IncrementalIndex extension preparations#12122liran-funaro wants to merge 10 commits intoapache:masterfrom
liran-funaro wants to merge 10 commits intoapache:masterfrom
Conversation
8 tasks
…reparations # Conflicts: # processing/src/main/java/org/apache/druid/segment/incremental/IncrementalIndex.java # processing/src/main/java/org/apache/druid/segment/incremental/OnheapIncrementalIndex.java
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR refactors the code in preparation for new extensions for
IncrementalIndex(i.e., #10001).It encapsulates common
IncrementalIndexfunctionality inIncrementalIndex.javaas much as possible.In addition, it avoids exposing the dimensions storage outside of
IncrementalIndexRow. That is, fetch each dimension directly without fetching the entire dimension array.The modifications to
IncrementalIndexincludes:getMetricmethods to acceptIncrementalIndexRowinstead ofint rowOffset, becauseOakIncrementalIndexdoes not keep a mapping from row index to an actual row.This does not affect the other implementations' performance because in all the cases these methods are used, the caller already had an
IncrementalIndexRowobject.CachingColumnSelectorFactorytoIncrementalIndexThe modifications to
IncrementalIndexRowallow lazy evaluation of off-heap keys, without adding an overhead to the on-heap keys case. We add/modify the following methods:public Object[] getDims()topublic Object getDim(int index)public int getDimsLength()public boolean isDimNull(int index)public IndexedInts getIndexedDim(int index, @Nullable IndexedInts cachedIndexedInts)getIndexedDim()purpose is to support a lazy-evaluation version of an indexed ints dimension instead of the array of integers that is returned bygetDim().The modified implementation in
StringDimensionIndexer, utilize this method and query using IndexedInts instead of the array. This allows our implementation (OakIncrementalIndex) to use the lazy-evaluation approach, while the other implementations use the existing integer array, without performance degradation.Key changed/added classes in this commit
IncrementalIndexIncrementalIndexRowOnheapIncrementalIndexStringDimensionIndexerThis PR has: