PROBLEM
We have use case where we are running ES on 100s of nodes and we get a mix of long and small running concurrent queries, where long and short is defined by number of shards it executes on.
https://discuss.elastic.co/t/long-query-starving-other-small-queries-and-slowing-down-the-cluster/163665?u=dextermorgan
I know we have a feature where we can limit the number of concurrent shards per search request and protect our cluster form one big query but what if there are more concurrent big queries with small queries. It would still starve the small queries
PROPOSAL
I propose to introduce a new search queue, maybe called as search_fast with configurable threads and elasticsearch can schedule smaller queries on this queue in case "search" thread queue size is large.
Let me know your thoughts regarding this and I can come up with a design and later, pull request.
P.S. - Something similar in YARN - https://hortonworks.com/blog/yarn-capacity-scheduler/
PROBLEM
We have use case where we are running ES on 100s of nodes and we get a mix of long and small running concurrent queries, where long and short is defined by number of shards it executes on.
https://discuss.elastic.co/t/long-query-starving-other-small-queries-and-slowing-down-the-cluster/163665?u=dextermorgan
I know we have a feature where we can limit the number of concurrent shards per search request and protect our cluster form one big query but what if there are more concurrent big queries with small queries. It would still starve the small queries
PROPOSAL
I propose to introduce a new search queue, maybe called as search_fast with configurable threads and elasticsearch can schedule smaller queries on this queue in case "search" thread queue size is large.
Let me know your thoughts regarding this and I can come up with a design and later, pull request.
P.S. - Something similar in YARN - https://hortonworks.com/blog/yarn-capacity-scheduler/