[Star-Tree] Search level stats for nodes, indices and shards by sandeshkr419 · Pull Request #18707 · opensearch-project/OpenSearch

sandeshkr419 · 2025-07-08T20:06:35Z

Description

Adding metric data for star-tree search queries.

search.startree_query_total
search.startree_query_time_in_millis
search.startree_query_current

Documentation: opensearch-project/documentation-website#10667

/_nodes/stats/indices/search

{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "runTask",
    "nodes": {
        "ZVVPzxDISt2bzCcpeam25A": {
            "timestamp": 1752005128826,
            "name": "runTask-0",
            "transport_address": "127.0.0.1:9300",
            "host": "127.0.0.1",
            "ip": "127.0.0.1:9300",
            "roles": [
                "cluster_manager",
                "data",
                "ingest",
                "remote_cluster_client"
            ],
            "attributes": {
                "testattr": "test",
                "shard_indexing_pressure_enabled": "true"
            },
            "indices": {
                "search": {
                    "open_contexts": 0,
                    "query_total": 7,
                    "query_time_in_millis": 74,
                    "query_current": 0,
                    "concurrent_query_total": 7,
                    "concurrent_query_time_in_millis": 74,
                    "concurrent_query_current": 0,
                    "concurrent_avg_slice_count": 1.0,
                    "startree_query_total": 4,
                    "startree_query_time_in_millis": 40,
                    "startree_query_current": 0,
                    "fetch_total": 7,
                    "fetch_time_in_millis": 1,
                    "fetch_current": 0,
                    "scroll_total": 0,
                    "scroll_time_in_millis": 0,
                    "scroll_current": 0,
                    "point_in_time_total": 0,
                    "point_in_time_time_in_millis": 0,
                    "point_in_time_current": 0,
                    "suggest_total": 0,
                    "suggest_time_in_millis": 0,
                    "suggest_current": 0,
                    "search_idle_reactivate_count_total": 0,
                    "request": {
                        "took": {
                            "time_in_millis": 136,
                            "current": 0,
                            "total": 7
                        },
                        "dfs_pre_query": {
                            "time_in_millis": 0,
                            "current": 0,
                            "total": 0
                        },
                        "query": {
                            "time_in_millis": 112,
                            "current": 0,
                            "total": 7
                        },
                        "fetch": {
                            "time_in_millis": 3,
                            "current": 0,
                            "total": 7
                        },
                        "dfs_query": {
                            "time_in_millis": 0,
                            "current": 0,
                            "total": 0
                        },
                        "expand": {
                            "time_in_millis": 0,
                            "current": 0,
                            "total": 7
                        },
                        "can_match": {
                            "time_in_millis": 0,
                            "current": 0,
                            "total": 0
                        }
                    }
                }
            }
        }
    }
}

_stats/search

{
    "_shards": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "_all": {
        "primaries": {
            "search": {
                "open_contexts": 0,
                "query_total": 7,
                "query_time_in_millis": 74,
                "query_current": 0,
                "concurrent_query_total": 7,
                "concurrent_query_time_in_millis": 74,
                "concurrent_query_current": 0,
                "concurrent_avg_slice_count": 1.0,
                "startree_query_total": 4,
                "startree_query_time_in_millis": 40,
                "startree_query_current": 0,
                "fetch_total": 7,
                "fetch_time_in_millis": 1,
                "fetch_current": 0,
                "scroll_total": 0,
                "scroll_time_in_millis": 0,
                "scroll_current": 0,
                "point_in_time_total": 0,
                "point_in_time_time_in_millis": 0,
                "point_in_time_current": 0,
                "suggest_total": 0,
                "suggest_time_in_millis": 0,
                "suggest_current": 0,
                "search_idle_reactivate_count_total": 0
            }
        },
        "total": {
            "search": {
                "open_contexts": 0,
                "query_total": 7,
                "query_time_in_millis": 74,
                "query_current": 0,
                "concurrent_query_total": 7,
                "concurrent_query_time_in_millis": 74,
                "concurrent_query_current": 0,
                "concurrent_avg_slice_count": 1.0,
                "startree_query_total": 4,
                "startree_query_time_in_millis": 40,
                "fetch_total": 7,
                "fetch_time_in_millis": 1,
                "fetch_current": 0,
                "scroll_total": 0,
                "scroll_time_in_millis": 0,
                "scroll_current": 0,
                "point_in_time_total": 0,
                "point_in_time_time_in_millis": 0,
                "point_in_time_current": 0,
                "suggest_total": 0,
                "suggest_time_in_millis": 0,
                "suggest_current": 0,
                "search_idle_reactivate_count_total": 0
            }
        }
    },
    "indices": {
        "logs-241998": {
            "uuid": "vcsLlw6ERyaLXI7mJ295nw",
            "primaries": {
                "search": {
                    "open_contexts": 0,
                    "query_total": 7,
                    "query_time_in_millis": 74,
                    "query_current": 0,
                    "concurrent_query_total": 7,
                    "concurrent_query_time_in_millis": 74,
                    "concurrent_query_current": 0,
                    "concurrent_avg_slice_count": 1.0,
                    "startree_query_total": 4,
                    "startree_query_time_in_millis": 40,
                    "fetch_total": 7,
                    "fetch_time_in_millis": 1,
                    "fetch_current": 0,
                    "scroll_total": 0,
                    "scroll_time_in_millis": 0,
                    "scroll_current": 0,
                    "point_in_time_total": 0,
                    "point_in_time_time_in_millis": 0,
                    "point_in_time_current": 0,
                    "suggest_total": 0,
                    "suggest_time_in_millis": 0,
                    "suggest_current": 0,
                    "search_idle_reactivate_count_total": 0
                }
            },
            "total": {
                "search": {
                    "open_contexts": 0,
                    "query_total": 7,
                    "query_time_in_millis": 74,
                    "query_current": 0,
                    "concurrent_query_total": 7,
                    "concurrent_query_time_in_millis": 74,
                    "concurrent_query_current": 0,
                    "concurrent_avg_slice_count": 1.0,
                    "startree_query_total": 4,
                    "startree_query_time_in_millis": 40,
                    "fetch_total": 7,
                    "fetch_time_in_millis": 1,
                    "fetch_current": 0,
                    "scroll_total": 0,
                    "scroll_time_in_millis": 0,
                    "scroll_current": 0,
                    "point_in_time_total": 0,
                    "point_in_time_time_in_millis": 0,
                    "point_in_time_current": 0,
                    "suggest_total": 0,
                    "suggest_time_in_millis": 0,
                    "suggest_current": 0,
                    "search_idle_reactivate_count_total": 0
                }
            }
        }
    }
}

_stats/search?pretty&level=shards

{
    "_shards": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "_all": {
        "primaries": {
            "search": {
                "open_contexts": 0,
                "query_total": 7,
                "query_time_in_millis": 74,
                "query_current": 0,
                "concurrent_query_total": 7,
                "concurrent_query_time_in_millis": 74,
                "concurrent_query_current": 0,
                "concurrent_avg_slice_count": 1.0,
                "startree_query_total": 4,
                "startree_query_time_in_millis": 40,
                "startree_query_current": 0,
                "fetch_total": 7,
                "fetch_time_in_millis": 1,
                "fetch_current": 0,
                "scroll_total": 0,
                "scroll_time_in_millis": 0,
                "scroll_current": 0,
                "point_in_time_total": 0,
                "point_in_time_time_in_millis": 0,
                "point_in_time_current": 0,
                "suggest_total": 0,
                "suggest_time_in_millis": 0,
                "suggest_current": 0,
                "search_idle_reactivate_count_total": 0
            }
        },
        "total": {
            "search": {
                "open_contexts": 0,
                "query_total": 7,
                "query_time_in_millis": 74,
                "query_current": 0,
                "concurrent_query_total": 7,
                "concurrent_query_time_in_millis": 74,
                "concurrent_query_current": 0,
                "concurrent_avg_slice_count": 1.0,
                "startree_query_total": 4,
                "startree_query_time_in_millis": 40,
                "fetch_total": 7,
                "fetch_time_in_millis": 1,
                "fetch_current": 0,
                "scroll_total": 0,
                "scroll_time_in_millis": 0,
                "scroll_current": 0,
                "point_in_time_total": 0,
                "point_in_time_time_in_millis": 0,
                "point_in_time_current": 0,
                "suggest_total": 0,
                "suggest_time_in_millis": 0,
                "suggest_current": 0,
                "search_idle_reactivate_count_total": 0
            }
        }
    },
    "indices": {
        "logs-241998": {
            "uuid": "vcsLlw6ERyaLXI7mJ295nw",
            "primaries": {
                "search": {
                    "open_contexts": 0,
                    "query_total": 7,
                    "query_time_in_millis": 74,
                    "query_current": 0,
                    "concurrent_query_total": 7,
                    "concurrent_query_time_in_millis": 74,
                    "concurrent_query_current": 0,
                    "concurrent_avg_slice_count": 1.0,
                    "startree_query_total": 4,
                    "startree_query_time_in_millis": 40,
                    "fetch_total": 7,
                    "fetch_time_in_millis": 1,
                    "fetch_current": 0,
                    "scroll_total": 0,
                    "scroll_time_in_millis": 0,
                    "scroll_current": 0,
                    "point_in_time_total": 0,
                    "point_in_time_time_in_millis": 0,
                    "point_in_time_current": 0,
                    "suggest_total": 0,
                    "suggest_time_in_millis": 0,
                    "suggest_current": 0,
                    "search_idle_reactivate_count_total": 0
                }
            },
            "total": {
                "search": {
                    "open_contexts": 0,
                    "query_total": 7,
                    "query_time_in_millis": 74,
                    "query_current": 0,
                    "concurrent_query_total": 7,
                    "concurrent_query_time_in_millis": 74,
                    "concurrent_query_current": 0,
                    "concurrent_avg_slice_count": 1.0,
                    "startree_query_total": 4,
                    "startree_query_time_in_millis": 40,
                    "fetch_total": 7,
                    "fetch_time_in_millis": 1,
                    "fetch_current": 0,
                    "scroll_total": 0,
                    "scroll_time_in_millis": 0,
                    "scroll_current": 0,
                    "point_in_time_total": 0,
                    "point_in_time_time_in_millis": 0,
                    "point_in_time_current": 0,
                    "suggest_total": 0,
                    "suggest_time_in_millis": 0,
                    "suggest_current": 0,
                    "search_idle_reactivate_count_total": 0
                }
            },
            "shards": {
                "0": [
                    {
                        "routing": {
                            "state": "STARTED",
                            "primary": true,
                            "node": "ZVVPzxDISt2bzCcpeam25A",
                            "relocating_node": null
                        },
                        "search": {
                            "open_contexts": 0,
                            "query_total": 7,
                            "query_time_in_millis": 74,
                            "query_current": 0,
                            "concurrent_query_total": 7,
                            "concurrent_query_time_in_millis": 74,
                            "concurrent_query_current": 0,
                            "concurrent_avg_slice_count": 1.0,
                            "startree_query_total": 4,
                            "startree_query_time_in_millis": 40,
                            "fetch_total": 7,
                            "fetch_time_in_millis": 1,
                            "fetch_current": 0,
                            "scroll_total": 0,
                            "scroll_time_in_millis": 0,
                            "scroll_current": 0,
                            "point_in_time_total": 0,
                            "point_in_time_time_in_millis": 0,
                            "point_in_time_current": 0,
                            "suggest_total": 0,
                            "suggest_time_in_millis": 0,
                            "suggest_current": 0,
                            "search_idle_reactivate_count_total": 0,
                            "request": {
                                "took": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "dfs_pre_query": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "query": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "fetch": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "dfs_query": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "expand": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                },
                                "can_match": {
                                    "time_in_millis": 0,
                                    "current": 0,
                                    "total": 0
                                }
                            }
                        },
                        "commit": {
                            "id": "rG1Y/6BUJM/miP6EBZWeDA==",
                            "generation": 3,
                            "user_data": {
                                "local_checkpoint": "1000",
                                "max_unsafe_auto_id_timestamp": "-1",
                                "min_retained_seq_no": "0",
                                "translog_uuid": "YLsJ70EGSxSBK8PYKzg8gw",
                                "force_merge_uuid": "yB9uwBF6TuS03_Mz23NYaw",
                                "history_uuid": "bQ8HgtAKSXWxPIMZZhUDmg",
                                "max_seq_no": "1000"
                            },
                            "num_docs": 1001
                        },
                        "seq_no": {
                            "max_seq_no": 1000,
                            "local_checkpoint": 1000,
                            "global_checkpoint": 1000
                        },
                        "retention_leases": {
                            "primary_term": 1,
                            "version": 2,
                            "leases": [
                                {
                                    "id": "peer_recovery/ZVVPzxDISt2bzCcpeam25A",
                                    "retaining_seq_no": 1001,
                                    "timestamp": 1751997664441,
                                    "source": "peer recovery"
                                }
                            ]
                        },
                        "shard_path": {
                            "state_path": "/Users/<hidden>/workplace/opensearch/OpenSearch/build/testclusters/runTask-0/data/nodes/0",
                            "data_path": "/Users/<hidden>/workplace/opensearch/OpenSearch/build/testclusters/runTask-0/data/nodes/0",
                            "is_custom_data_path": false
                        }
                    }
                ]
            }
        }
    }
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-07-08T22:08:51Z

❌ Gradle check result for ad439e8: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-09T04:42:15Z

❌ Gradle check result for 7431dfd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-09T05:53:31Z

❌ Gradle check result for 26fd3d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

server/src/main/java/org/opensearch/rest/action/cat/RestIndicesAction.java

github-actions · 2025-07-09T22:05:26Z

❌ Gradle check result for cb1a9de: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

getsaurabh02 · 2025-07-09T22:42:56Z

Shouldn't we enable a new metric group for STAR TREE: https://docs.opensearch.org/docs/latest/api-reference/nodes-apis/nodes-stats/

sandeshkr419 · 2025-07-10T19:01:24Z

Shouldn't we enable a new metric group for STAR TREE: https://docs.opensearch.org/docs/latest/api-reference/nodes-apis/nodes-stats/

@getsaurabh02 I'd argue otherwise. The reason to place it inside search metric group was that similar metrics like concurrent search / PIT metrics are also part of this. Also, the metric related to star-tree like startree_query_total will be less than or equal to query_total for example. Placing it here gives a good picture of what is your star-tree usage with total queries and concurrent search queries as well.

@expani @bharath-techie Looping you both in the thread, please review/evaluate the placement of metrics if this needs any change than the current state.

github-actions · 2025-07-13T22:03:34Z

❌ Gradle check result for e674fad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T05:18:44Z

❌ Gradle check result for 78480c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

server/src/internalClusterTest/java/org/opensearch/search/stats/StarTreeSearchStatsIT.java

github-actions · 2025-07-16T02:46:52Z

❌ Gradle check result for f75a836: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

github-actions · 2025-07-16T04:15:12Z

✅ Gradle check result for 7218ea6: SUCCESS

…rch-project#18707) * startree node,indices,shards stats Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * changing to builder pattern Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * add query current, fix version Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * fix version Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * add integ test Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * typo Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> --------- Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

sandeshkr419 changed the title ~~startree node,indices,shards stats~~ [Star-Tree] Search level stats for nodes, indices and shards Jul 9, 2025

sandeshkr419 force-pushed the stats branch from ad439e8 to 7431dfd Compare July 9, 2025 03:40

sandeshkr419 force-pushed the stats branch from 7431dfd to 26fd3d9 Compare July 9, 2025 04:45

sandeshkr419 added Search Search query, autocomplete ...etc v3.2.0 labels Jul 9, 2025

This was referenced Jul 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Open

[AUTOCUT] Gradle Check Flaky Test Report for ClientYamlTestSuiteIT #14319

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

sandeshkr419 force-pushed the stats branch 2 times, most recently from d966f1f to cb1a9de Compare July 9, 2025 20:56

expani reviewed Jul 9, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Outdated Show resolved Hide resolved

expani reviewed Jul 9, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Show resolved Hide resolved

expani reviewed Jul 9, 2025

View reviewed changes

server/src/main/java/org/opensearch/rest/action/cat/RestIndicesAction.java Outdated Show resolved Hide resolved

sandeshkr419 mentioned this pull request Jul 10, 2025

[Refactor] Refactor SearchStats.Stats to use a builder pattern for non-breaking extensibility. #18723

Closed

sandeshkr419 force-pushed the stats branch 2 times, most recently from c139f7d to e674fad Compare July 13, 2025 20:58

sandeshkr419 force-pushed the stats branch from e674fad to 78480c6 Compare July 15, 2025 04:09

sandeshkr419 marked this pull request as ready for review July 15, 2025 05:48

sandeshkr419 requested review from Bukhtawar, anasalkouz, andrross and ashking94 as code owners July 15, 2025 05:48

sandeshkr419 added 4 commits July 15, 2025 19:06

changing to builder pattern

2b6cdba

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

add query current, fix version

078cded

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

fix version

e9201d4

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

add integ test

563e7aa

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

sandeshkr419 force-pushed the stats branch from 789ee5c to 563e7aa Compare July 16, 2025 02:06

expani reviewed Jul 16, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Outdated Show resolved Hide resolved

expani approved these changes Jul 16, 2025

View reviewed changes

expani reviewed Jul 16, 2025

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/stats/StarTreeSearchStatsIT.java Show resolved Hide resolved

sandeshkr419 force-pushed the stats branch from 93d5df2 to f75a836 Compare July 16, 2025 02:35

typo

7218ea6

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

sandeshkr419 force-pushed the stats branch from f75a836 to 7218ea6 Compare July 16, 2025 02:53

bharath-techie approved these changes Jul 16, 2025

View reviewed changes

bharath-techie merged commit 1352627 into opensearch-project:main Jul 16, 2025
31 checks passed

sandeshkr419 deleted the stats branch July 16, 2025 04:19

sandeshkr419 mentioned this pull request Jul 16, 2025

[DOC] [Star-Tree] Add documentation for star-tree search stats opensearch-project/documentation-website#10317

Closed

4 tasks

opensearch-ci-bot mentioned this pull request Jun 27, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

Open

This was referenced Aug 5, 2025

Adding star-tree related stats documentation. opensearch-project/documentation-website#10667

Merged

[META] [Star Tree] [Observability] Star-Tree Indexing & Search Observability #19017

Open

BrewTestBot mentioned this pull request Aug 20, 2025

opensearch 3.2.0 Homebrew/homebrew-core#234146

Merged

This was referenced Sep 2, 2025

[Feature Request] [Search] Query Failure Related Stats #19210

Closed

[Refactor] Use Builder pattern instead of constructors for Stats API classes #19225

Closed

This was referenced Jan 10, 2026

docs: add star-tree-index report for v3.2.0 tkykenmt/opensearch-feature-explorer#1217

Merged

[feature] Star Tree Index tkykenmt/opensearch-feature-explorer#1114

Closed

opensearch-ci-bot mentioned this pull request Feb 23, 2026

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStatePublicationIT #20709

Open

Conversation

sandeshkr419 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

getsaurabh02 commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sandeshkr419 commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 13, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sandeshkr419 commented Jul 8, 2025 •

edited

Loading

getsaurabh02 commented Jul 9, 2025 •

edited

Loading