Fix Graph Filter Error in Search by mingshl · Pull Request #5665 · opensearch-project/OpenSearch

mingshl · 2022-12-30T19:19:05Z

Description:

Reproducing the issue:

[1] . creating index:

curl -XPUT localhost:9200/test-index --data '{ "settings": { "index": { "analysis": { "analyzer": { "search_analyzer": { "filter": [ "stop", "asciifolding", "elision", "lowercase", "synonym_graph", "snowball", "unique" ], "tokenizer": "standard", "type": "custom" }, "index_analyzer": { "filter": [ "stop", "asciifolding", "elision", "word_delimiter", "flatten_graph", "lowercase", "snowball", "unique" ], "tokenizer": "standard", "type": "custom" } }, "filter": { "synonym_graph": { "lenient": "true", "synonyms": [ "head board, bed head, bedhead, headboard" ], "type": "synonym_graph" }, "synonyms": { "synonyms": [ "head board, bed head, bedhead, headboard" ], "type": "synonym" }, "word_delimiter": { "preserve_original": "true", "type": "word_delimiter" } } } } }, "mappings": { "dynamic": "strict", "properties": { "merchandising_keywords": { "analyzer": "index_analyzer", "index_options": "docs", "search_analyzer": "search_analyzer", "type": "text" } } } } ' -H "Content-Type:Application/json"

[2 ] querying by
curl 'localhost:9200/test-index/_search?pretty' --data '{"query": {"match": {"merchandising_keywords": "Gas Lift Storage Bed Frame with Arched Bed Head in King"}}}' -H "Content-Type:Application/json"
[3 ] outcome
'get 404 error' assertion error

Testing with logging information. the Input of the function is a graph created from lucene, GraphTokenStreamFiniteStringsorg.apache.lucene.util.graph.GraphTokenStreamFiniteStrings. Found out when there is no document added, the graph has no path, only the source node. so while walking through the graph to find side path get assertion error, start =0, end = -1, state =0, next_state = 0

[2022-12-29T15:07:35,251][INFO ][o.o.p.PluginsService ] [runTask-0] PluginService:onIndexModule index:[test-index/jVsetp_JTSuPkUBIeKFuYg][2022-12-29T15:07:35,290][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=3} [2022-12-29T15:07:35,423][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received diff cluster state version [4] with uuid [aSNgAs0GRIK8AkGNiu_TjQ], diff size [620] [2022-12-29T15:07:35,453][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=4} [2022-12-29T15:07:49,047][INFO ][o.o.i.s.MatchQuery ] [runTask-0] check the sourceCachingTokenFilter@52e91aa3 term=,bytes=[],startOffset=55,endOffset=55,positionIncrement=0,positionLength=1,type=word,termFrequency=1,keyword=false [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery ] [runTask-0] check the GraphTokenStreamFiniteStringsorg.apache.lucene.util.graph.GraphTokenStreamFiniteStrings@6389dd20 [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery ] [runTask-0] articulationPoints.length: 0 [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery ] [runTask-0] i: 0 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery ] [runTask-0] lastState: -1 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery ] [runTask-0] end: -1 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery ] [runTask-0] catch assertion error: java.lang.AssertionError: state=0 nextState=0

Applying the fix
to catch the error raising from lucene, and return empty query. Because there is no path found, no need to go further to analyze the graph, return empty query results.

** this is the query result as expected after the fix:

curl 'localhost:9200/test-index/_search?pretty' --data '{"query": {"match": {"merchandising_keywords": "Gas Lift Storage Bed Frame with Arched Bed Head in King"}}}' -H "Content-Type:Application/json"

{
"took" : 68,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}

Issues Resolved

#5379

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

github-actions · 2022-12-30T19:42:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8636/
CommitID: b125146
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-30T19:49:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8637/
CommitID: 409e419
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-30T19:50:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8638/
CommitID: d4ea089
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-30T19:57:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8639/
CommitID: 8b431d8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-30T20:05:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8640/
CommitID: 75e83d3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

reta · 2023-01-06T22:23:51Z

+                 * check if the GraphTokenStreamFiniteStrings graph is empty
+                 * return empty BooleanQuery result
+                 */
+                Iterator<TokenStream> graphIt = graph.getFiniteStrings();


Looks much better, thanks @mingshl !

github-actions · 2023-01-06T22:30:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8978/
CommitID: c61d387
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

tlfeng · 2023-01-06T22:45:15Z

test failure in build 8978:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.action.support.replication.TransportReplicationActionTests.testClosedIndexOnReroute" -Dtests.seed=22C2F47F9CF3054C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-SY -Dtests.timezone=Etc/GMT0 -Druntime.java=19

org.opensearch.action.support.replication.TransportReplicationActionTests > testClosedIndexOnReroute FAILED
    java.lang.IllegalStateException: No local node found. Is the node started?
        at __randomizedtesting.SeedInfo.seed([22C2F47F9CF3054C:4E3C56FCFF618C07]:0)
        at org.opensearch.cluster.service.ClusterService.localNode(ClusterService.java:156)
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.<init>(TransportReplicationAction.java:890)
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.<init>(TransportReplicationAction.java:883)
        at org.opensearch.action.support.replication.TransportReplicationActionTests.testClosedIndexOnReroute(TransportReplicationActionTests.java:640)

Doesn't seem related to the code change.

github-actions · 2023-01-06T23:15:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8982/
CommitID: c61d387
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

tlfeng · 2023-01-06T23:46:08Z

In build 8982:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure" -Dtests.seed=4CCBB588DFDFA1DA -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-BO -Dtests.timezone=Europe/Simferopol -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationRelocationIT > testPrimaryRelocationWithSegRepFailure FAILED
    java.lang.AssertionError: Count is 24 hits but 48 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([4CCBB588DFDFA1DA:C854ED08956797F7]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:303)
        at org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure(SegmentReplicationRelocationIT.java:190)

github-actions · 2023-01-07T00:14:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8984/
CommitID: c61d387
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

tlfeng · 2023-01-07T00:38:45Z

in build 8984:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation" -Dtests.seed=A006921DD2141D23 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-BH -Dtests.timezone=Africa/El_Aaiun -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationRelocationIT > testPrimaryRelocation FAILED
    java.lang.AssertionError: Count is 69 hits but 138 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([A006921DD2141D23:180EA5CFF17D4ECD]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:303)
        at org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation(SegmentReplicationRelocationIT.java:114)

github-actions · 2023-01-07T00:58:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8987/
CommitID: c61d387
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-07T01:33:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8988/
CommitID: 52626cf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-07T02:53:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

URL: https://build.ci.opensearch.org/job/gradle-check/8994/
CommitID: 52626cf
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot · 2023-01-17T17:31:02Z

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1.x 1.x
# Navigate to the new working tree
pushd ../.worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-5665-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6a7a9a1b2472f8d4a496d5b976ae16be87893b0e
# Push it to GitHub
git push --set-upstream origin backport/backport-5665-to-1.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-5665-to-1.x.

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1)

* Fix Graph Filter Error in Search (#5665) * fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) * organize package pass compile test Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* fix graph filter out of bound error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * add changelog Signed-off-by: Mingshi Liu <mingshl@amazon.com> * run gradle spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * reproduce error in unit test Signed-off-by: Mingshi Liu <mingshl@amazon.com> * format to pass spotlessApply Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

mingshl and others added 2 commits December 29, 2022 15:40

fix graph filter out of bound error

da120cf

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

Merge branch 'opensearch-project:main' into main

8b431d8

mingshl requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, adnapibar, anasalkouz, andrross, dblock, dreamer-89, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja, tlfeng and xuezhou25 as code owners December 30, 2022 19:19

add changelog

75e83d3

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

run gradle spotlessApply

1c1959d

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

mingshl requested a review from reta January 6, 2023 22:20

reta reviewed Jan 6, 2023

View reviewed changes

reta approved these changes Jan 6, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main'

52626cf

dblock merged commit 6a7a9a1 into opensearch-project:main Jan 9, 2023

dblock added the backport 2.x Backport to 2.x branch label Jan 9, 2023

opensearch-trigger-bot Bot mentioned this pull request Jan 9, 2023

[Backport 2.x] Fix Graph Filter Error in Search #5758

Merged

dblock mentioned this pull request Jan 9, 2023

[BUG] Failed to create query: Index 0 out of bounds for length 0 #5379

Closed

dblock added the backport 2.5 label Jan 9, 2023

opensearch-trigger-bot Bot mentioned this pull request Jan 9, 2023

[Backport 2.5] Fix Graph Filter Error in Search #5769

Merged

mingshl added the backport 1.x label Jan 17, 2023

mingshl mentioned this pull request Jan 17, 2023

[Backport 1.x]Fix Graph Filter Error in Search (#5665) #5912

Merged

6 tasks

mingshl self-assigned this Jan 31, 2023

Conversation

mingshl commented Dec 30, 2022

Description:

Issues Resolved

Check List

Uh oh!

github-actions Bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Dec 30, 2022

Gradle Check (Jenkins) Run Completed with:

Uh oh!

reta Jan 6, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

tlfeng commented Jan 6, 2023

Uh oh!

github-actions Bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

tlfeng commented Jan 6, 2023

Uh oh!

github-actions Bot commented Jan 7, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

tlfeng commented Jan 7, 2023

Uh oh!

github-actions Bot commented Jan 7, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jan 7, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jan 7, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot Bot commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants