Add hierarchical routing processors for document co-location#18826
Add hierarchical routing processors for document co-location#18826msfroh merged 2 commits intoopensearch-project:mainfrom
Conversation
|
RFC and Design Co Author @abhishekpsingh |
|
Fixes #18816 |
|
❌ Gradle check result for 72e9306: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
msfroh
left a comment
There was a problem hiding this comment.
Looks good overall. I left a couple of nitpicky comments.
It looks like one of your tests is failing, and both modules have a test that asserts on what processors are present, so the newly-added processors break that.
...s/ingest-common/src/main/java/org/opensearch/ingest/common/HierarchicalRoutingProcessor.java
Outdated
Show resolved
Hide resolved
...s/ingest-common/src/main/java/org/opensearch/ingest/common/HierarchicalRoutingProcessor.java
Show resolved
Hide resolved
...s/ingest-common/src/main/java/org/opensearch/ingest/common/HierarchicalRoutingProcessor.java
Outdated
Show resolved
Hide resolved
Thanks, fixed both. |
|
@msfroh Thanks for reviewing. Updated per comments, please see |
|
❌ Gradle check result for 9dfe9cc: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 85368ab: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
...s/ingest-common/src/main/java/org/opensearch/ingest/common/HierarchicalRoutingProcessor.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/opensearch/search/pipeline/common/HierarchicalRoutingSearchProcessor.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/opensearch/search/pipeline/common/HierarchicalRoutingSearchProcessor.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/opensearch/search/pipeline/common/HierarchicalRoutingSearchProcessor.java
Show resolved
Hide resolved
|
@msfroh Updated, please see |
|
❌ Gradle check result for 8194377: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 80db053: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Implements ingest and search pipeline processors to co-locate related documents based on hierarchical paths (e.g., folder structures). Documents with same path prefix are routed to same shard, enabling efficient search within hierarchies. - HierarchicalRoutingProcessor: Routes documents during indexing - HierarchicalRoutingSearchProcessor: Routes queries to relevant shards - Consistent MurmurHash3-based routing between ingest and search - Configurable anchor depth, path separators, and field mapping Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Fix test asserts Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Fix failing tests and update per comments Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Yet more test fixes Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Fix more comments Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Mark flaky test for now Signed-off-by: Atri Sharma <atri.jiit@gmail.com> Add more javadocs Signed-off-by: Atri Sharma <atri.jiit@gmail.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #18826 +/- ##
============================================
+ Coverage 72.75% 72.76% +0.01%
- Complexity 68512 68637 +125
============================================
Files 5570 5572 +2
Lines 314989 315164 +175
Branches 45696 45742 +46
============================================
+ Hits 229165 229340 +175
+ Misses 67240 67233 -7
- Partials 18584 18591 +7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Atri Sharma <atri.jiit@gmail.com>
|
❌ Gradle check result for 12c4e99: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 12c4e99: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Flaky tests #14509 |
…rch-project#18826) Implements ingest and search pipeline processors to co-locate related documents based on hierarchical paths (e.g., folder structures). Documents with same path prefix are routed to same shard, enabling efficient search within hierarchies. - HierarchicalRoutingProcessor: Routes documents during indexing - HierarchicalRoutingSearchProcessor: Routes queries to relevant shards - Consistent MurmurHash3-based routing between ingest and search - Configurable anchor depth, path separators, and field mapping --------- Signed-off-by: Atri Sharma <atri.jiit@gmail.com>
…rch-project#18826) Implements ingest and search pipeline processors to co-locate related documents based on hierarchical paths (e.g., folder structures). Documents with same path prefix are routed to same shard, enabling efficient search within hierarchies. - HierarchicalRoutingProcessor: Routes documents during indexing - HierarchicalRoutingSearchProcessor: Routes queries to relevant shards - Consistent MurmurHash3-based routing between ingest and search - Configurable anchor depth, path separators, and field mapping --------- Signed-off-by: Atri Sharma <atri.jiit@gmail.com>
Implements ingest and search pipeline processors to co-locate related
documents based on hierarchical paths (e.g., folder structures). Documents
with same path prefix are routed to same shard, enabling efficient
search within hierarchies.