Show only intersecting buckets to the Adjacency matrix aggregation by brusic · Pull Request #10226 · opensearch-project/OpenSearch

brusic · 2023-09-25T23:10:49Z

EDIT: Rebasing the code to add the signoff created an inconsistency with the branch. Never used Signed-off-by, willing to start a new branch since this one appears to have prior commits.

Description

Show only intersecting buckets to the Adjacency matrix aggregation

The Adjacency matrix aggregation will compute and return all buckets related to any of the combinations of the filters specified, including hits on a single filter. However, sometimes a user would want to return only the buckets related to an intersection and not for a single filter. An optional show_only_intersecting parameter would stop buckets resulting from a single filter being hit to be created.

Related Issues

#8832

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…-project#9082) This is the first round refactor of the foundation Task classes from the server module to core and common library. This is split into separate commits due to control the surface area impacted by the change. This commit refactors the following classes: * o.o.action.ActionFuture -> :libs:opensearch-common * o.o.action.ActionListener -> :libs:opensearch-core * o.o.action.ActionResponse -> :libs:opensearch-core * o.o.action.NotifyOnceListener -> :libs:opensearch-core * o.o.common.compress.Compressor -> :libs:opensearch-core * o.o.node.ReportingService -> o.o.core.service in :libs:opensearch-core * o.o.tasks.TaskCancelledExceptoin -> :libs:opensearch-core * o.o.tasks.TaskId -> :libs:opensearch-core * o.o.common.xcontent.ObjectParserHelper -> :libs:opensearch-core The following task resource tracking classes are refactored into a new resourcetracker namespace under the o.o.core.tasks packed in :libs:opensearch-core * o.o.tasks.ResourceStats * o.o.tasks.ResourceStatsType * o.o.tasks.ResourceUsageMetric * o.o.tasks.TaskResourceStats * o.o.tasks.TaskResourceUsage * o.o.tasks.TaskThreadUsage * o.o.tasks.ThreadResourceInfo These refactors are part of the effort to split foundation classes into core libraries in support of modularity for serverless and cloud native implementations. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…nsearch-project#9221) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…/repository-azure (opensearch-project#9231) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

… to 1.43.3 (opensearch-project#10126) Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <ssashish@amazon.com> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Provide service accounts tokens to extensions This change adds a new transport action which passes the extension a string representation of its service account auth token. This token is created by the TokenManager interface implementation. The token is expected to be an encoded basic auth credential string which can be used by the extension to interact with its own system index. Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Peter Nied <petern@amazon.com> Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Co-authored-by: Peter Nied <petern@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…t#10178) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* Add support for encrypted async blob read Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Add async blob read support for encrypted containers Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…t#10036) Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…/fixtures/hdfs-fixture (opensearch-project#10206) * Bump org.xerial.snappy:snappy-java in /test/fixtures/hdfs-fixture Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.3 to 1.1.10.4. - [Release notes](https://github.com/xerial/snappy-java/releases) - [Commits](xerial/snappy-java@v1.1.10.3...v1.1.10.4) --- updated-dependencies: - dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ectly (opensearch-project#10089) * Fix remove ingest processor handing ignore_missing parameter not correctly Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Fix test failure and optimize some code Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Minor change Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify test code Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Split a test method to two Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* Bump codecov/codecov-action from 2 to 3 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2 to 3. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v2...v3) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* [Tracing Framework] Add support for SpanKind Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> * Add CHANGELOG Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> * Fix test case Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> * Address review comment Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> * Address review comment Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> * Address review comment Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> --------- Signed-off-by: Gagan Juneja <gjjuneja@amazon.com> Co-authored-by: Gagan Juneja <gjjuneja@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…25.0 in /plugins/repository-gcs (opensearch-project#10208) * Bump com.google.api.grpc:proto-google-common-protos Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.10.0 to 2.25.0. - [Release notes](https://github.com/googleapis/sdk-platform-java/releases) - [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md) - [Commits](googleapis/sdk-platform-java@v2.10.0...v2.25.0) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…rsecting

github-actions · 2023-09-25T23:27:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26289/
CommitID: 3d66a5f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-25T23:33:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26290/
CommitID: 8fc0cb5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-25T23:46:45Z

Compatibility status:

Checks if related components are compatible with change 188ff84

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-26T00:03:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26288/
CommitID: 0c9a4ca
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T00:18:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26292/
CommitID: 188ff84
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-10-26T15:20:05Z

This PR is stalled because it has been open for 30 days with no activity.

brusic · 2024-01-03T23:08:12Z

transitioned to #11733

gbbafna and others added 6 commits September 25, 2023 16:12

[Remote Store] Fix couple of Remote Store flaky test and use bulk api…

c2c20d7

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Automatically update GitHub Actions dependencies with dependabot (ope…

a995c2b

…nsearch-project#9221) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Poojita-Raj and others added 13 commits September 25, 2023 16:15

Bump com.google.http-client:google-http-client-jackson2 from 1.43.2…

2228e76

… to 1.43.3 (opensearch-project#10126) Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Reenable multipart repository within multipart ITs (opensearch-projec…

cb7142c

…t#10178) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Add async segment file download support from remote store (opensearch…

74aad97

…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Honor max segment size during only_expunge_deletes (opensearch-projec…

5c0e568

…t#10036) Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

Bump BouncyCastle to 1.76 (opensearch-project#10219)

3d66a5f

Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

brusic force-pushed the brusic/8832-show-only-intersecting branch from 0c9a4ca to 3d66a5f Compare September 25, 2023 23:15

Merge remote-tracking branch 'origin' into brusic/8832-show-only-inte…

8fc0cb5

…rsecting

brusic marked this pull request as draft September 25, 2023 23:18

brusic changed the title ~~Brusic/8832 show only intersecting~~ Show only intersecting buckets to the Adjacency matrix aggregation Sep 25, 2023

brusic marked this pull request as ready for review September 25, 2023 23:24

Fix errant keystroke

188ff84

opensearch-trigger-bot bot added the stalled Issues that have stalled label Oct 26, 2023

brusic mentioned this pull request Jan 3, 2024

Show only intersecting buckets to the Adjacency matrix aggregation #11733

Merged

6 tasks

brusic closed this Jan 3, 2024

brusic deleted the brusic/8832-show-only-intersecting branch January 3, 2024 23:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Show only intersecting buckets to the Adjacency matrix aggregation#10226

Show only intersecting buckets to the Adjacency matrix aggregation#10226
brusic wants to merge 359 commits intoopensearch-project:mainfrom
brusic:brusic/8832-show-only-intersecting

brusic commented Sep 25, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Sep 25, 2023

Uh oh!

github-actions bot commented Sep 25, 2023

Uh oh!

github-actions bot commented Sep 25, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Sep 26, 2023

Uh oh!

github-actions bot commented Sep 26, 2023

Uh oh!

opensearch-trigger-bot bot commented Oct 26, 2023

Uh oh!

brusic commented Jan 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

brusic commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Sep 25, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 25, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Oct 26, 2023

Uh oh!

brusic commented Jan 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

brusic commented Sep 25, 2023 •

edited

Loading

github-actions bot commented Sep 25, 2023 •

edited

Loading