Elasticsearch Version
8.19.6
Installed Plugins
No response
Java Version
bundled
OS Version
Ubuntu 24.04 (using the official elastic docker image)
Problem Description
Description
After upgrading from Elasticsearch 7.17.5 to 8.19.6, percolate queries
intermittently fail with a NoSuchElementException when attempting to pop from
an empty stack in NestedScope.previousLevel(). The error occurs during query
phase execution when processing percolate queries that contain nested queries.
Observed Behavior
- Intermittent failures: Not all percolate queries fail, suggesting specific
query patterns trigger the bug
- Shard-specific: Failures occur on specific shards, likely those containing
problematic percolate query patterns
- Query pattern: Percolate queries with nested queries within bool clauses
appear to be affected
Additional Notes
- This issue did NOT occur in Elasticsearch 7.17.5
- We're using the 7.x client with compatibility mode enabled.
Steps to Reproduce
While I haven't isolated a minimal reproduction case yet, the issue seems to be
triggered by:
- Creating an index with nested field mappings
- Storing percolate queries that include:
- Nested queries referencing the nested paths
- Multiple bool query clauses at different nesting levels
- Mix of must/should/filter clauses containing nested queries
- Executing a percolate query with documents matching these stored queries
Reproducing is proving difficult due to the intermittent nature of this bug. I can follow-up with further steps if I can narrow it down to a particular document shape and query. That being said, in order to surface the error, I am executing the exact same percolation requests (documents are not changing between requests) and stored percolate queries are remaining the same as well, however, the response is occasionally partial. Could this be a bug related to parallel query execution?
Our documents contain multiple nested fields (ie various nested paths), each nested fields may have more than 200 nested documents, index is spread across 10 shards, we are using the 7.x ruby client with compatibility mode enabled (we are in the middle of an upgrade to 8.x)
Logs (if relevant)
Stacktrace
org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:652)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:824)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$6(SearchService.java:688)
at org.elasticsearch.server@8.19.6/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
at org.elasticsearch.server@8.19.6/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
at org.elasticsearch.server@8.19.6/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
at org.elasticsearch.server@8.19.6/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at org.elasticsearch.server@8.19.6/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
at org.elasticsearch.server@8.19.6/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.server@8.19.6/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.util.NoSuchElementException
at java.base/java.util.LinkedList.removeFirst(LinkedList.java:282)
at java.base/java.util.LinkedList.pop(LinkedList.java:813)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.support.NestedScope.previousLevel(NestedScope.java:45)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.NestedQueryBuilder.toQuery(NestedQueryBuilder.java:318)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.NestedQueryBuilder.doToQuery(NestedQueryBuilder.java:271)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:119)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:325)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:308)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:119)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:325)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:307)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:119)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:325)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:308)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:119)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:325)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:305)
at org.elasticsearch.server@8.19.6/org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:119)
at org.elasticsearch.percolator@8.19.6/org.elasticsearch.percolator.PercolateQueryBuilder.lambda$createStore$9(PercolateQueryBuilder.java:613)
at org.elasticsearch.percolator@8.19.6/org.elasticsearch.percolator.PercolateQuery$1$2.matchDocId(PercolateQuery.java:170)
at org.elasticsearch.percolator@8.19.6/org.elasticsearch.percolator.PercolateQuery$BaseScorer$1.matches(PercolateQuery.java:283)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.ConjunctionDISI$ConjunctionTwoPhaseIterator.matches(ConjunctionDISI.java:346)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:295)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:236)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.BulkScorer.score(BulkScorer.java:43)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:736)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:374)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:352)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.TaskExecutor$TaskGroup$1.run(TaskExecutor.java:120)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.TaskExecutor$TaskGroup.invokeAll(TaskExecutor.java:176)
at org.apache.lucene.core@9.12.2/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:84)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:356)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:323)
at org.elasticsearch.server@8.19.6/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
... 15 more
Elasticsearch Version
8.19.6
Installed Plugins
No response
Java Version
bundled
OS Version
Ubuntu 24.04 (using the official elastic docker image)
Problem Description
Description
After upgrading from Elasticsearch 7.17.5 to 8.19.6, percolate queries
intermittently fail with a NoSuchElementException when attempting to pop from
an empty stack in NestedScope.previousLevel(). The error occurs during query
phase execution when processing percolate queries that contain nested queries.
Observed Behavior
query patterns trigger the bug
problematic percolate query patterns
appear to be affected
Additional Notes
Steps to Reproduce
While I haven't isolated a minimal reproduction case yet, the issue seems to be
triggered by:
Reproducing is proving difficult due to the intermittent nature of this bug. I can follow-up with further steps if I can narrow it down to a particular document shape and query. That being said, in order to surface the error, I am executing the exact same percolation requests (documents are not changing between requests) and stored percolate queries are remaining the same as well, however, the response is occasionally partial. Could this be a bug related to parallel query execution?
Our documents contain multiple nested fields (ie various nested paths), each nested fields may have more than 200 nested documents, index is spread across 10 shards, we are using the 7.x ruby client with compatibility mode enabled (we are in the middle of an upgrade to 8.x)
Logs (if relevant)
Stacktrace