The Indices stats API does not scale well for very large index counts.
The response size can reach hundreds of MBs for tens of thousands of indices
and response times eventually go into the tens of seconds due to the
CPU time required for creating and serializing very large responses.
We discussed this during the many-shards sync and decided on introducing
pagination into the indices stats API to enable clients to bound the API's
response size to manageable levels.
This can be done by adding two parameters to indices stats that enable pagination
(sorted by index name):
size to limit the number of indices that stats are fetched for
from to provide the index name to search from (analogous to search_after in search or the from parameter in the get-snapshots API)
relates #77466
Also relates to #88094 (sorting in APIs) and #88095 (filtering in APIs)
The Indices stats API does not scale well for very large index counts.
The response size can reach hundreds of MBs for tens of thousands of indices
and response times eventually go into the tens of seconds due to the
CPU time required for creating and serializing very large responses.
We discussed this during the many-shards sync and decided on introducing
pagination into the indices stats API to enable clients to bound the API's
response size to manageable levels.
This can be done by adding two parameters to indices stats that enable pagination
(sorted by index name):
sizeto limit the number of indices that stats are fetched forfromto provide the index name to search from (analogous to search_after in search or the from parameter in the get-snapshots API)relates #77466
Also relates to #88094 (sorting in APIs) and #88095 (filtering in APIs)