Skip to content

Migrate Index Management away from using the cat APIs #57286

@cjcenizal

Description

@cjcenizal

Per elastic/elasticsearch#51772, cat APIs are not intended for consumption by applications. Index Management currently uses _cat/aliases and _cat/indices to list index aliases and indices.

As a side effect of using the cat APIs, Elasticsearch skips shard refreshes by default as of 7.0. This means that if you index a document and go to Index Management, the doc count for that index will be unchanged until that index is searched.

We can use the index stats API in lieu of cat indices:

GET _all/_stats/docs

However, these results are still affected by the skipped shard refreshes. Per discussion with Jason Tedor, we'd have to make an explicit refresh call to keep indices up-to-date, but refreshing every index can be very expensive and often unnecesssary:

for lots of use cases today (time series), users will have a mix of indices that are actively being indexed into, and those that are no longer being indexed into
in that case, of active indexing, a refresh doesn’t help because while the page is loading, more docs are being indexed so the count isn’t going to be the latest anyway, because it will have already been rendered wrong
in the case of indices that are no longer being indexed into, the counts will be accurate after a little bit of time has elapsed
for these use cases, a refresh/toggle to refresh doesn’t do anything except an unnecessary refresh call on the actively indexing indices (hurting performance)

Based on the above input, we should not bake a refresh into the request for fetching indices. If users need to see the most up-to-date doc count of an index, they should refresh it manually:

image

Metadata

Metadata

Assignees

No one assigned

    Labels

    Feature:Index ManagementIndex and index templates UITeam:Kibana ManagementDev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t//enhancementNew value added to drive a business result

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions