Skip to content

[7.x] Add telemetry for data tiers (#63031)#63140

Merged
dakrone merged 2 commits intoelastic:7.xfrom
dakrone:backport/7.x/pr-63031
Oct 1, 2020
Merged

[7.x] Add telemetry for data tiers (#63031)#63140
dakrone merged 2 commits intoelastic:7.xfrom
dakrone:backport/7.x/pr-63031

Conversation

@dakrone
Copy link
Copy Markdown
Member

@dakrone dakrone commented Oct 1, 2020

Backports the following commits to 7.x:

This commit adds telemetry for our data tier formalization. This telemetry helps determine the
topology of the cluster with regard to the content, hot, warm, & cold tiers/roles.

An example of the telemetry looks like:

```
GET /_xpack/usage?human
{
  ...
  "data_tiers" : {
    "available" : true,
    "enabled" : true,
    "data_warm" : {
      ...
    },
    "data_cold" : {
      ...
    },
    "data_content" : {
      "node_count" : 1,
      "index_count" : 6,
      "total_shard_count" : 6,
      "primary_shard_count" : 6,
      "doc_count" : 71,
      "total_size" : "59.6kb",
      "total_size_bytes" : 61110,
      "primary_size" : "59.6kb",
      "primary_size_bytes" : 61110,
      "primary_shard_size_avg" : "9.9kb",
      "primary_shard_size_avg_bytes" : 10185,
      "primary_shard_size_median" : "8kb",
      "primary_shard_size_median_bytes" : 8254,
      "primary_shard_size_mad" : "7.2kb",
      "primary_shard_size_mad_bytes" : 7391
    },
    "data_hot" : {
       ...
    }
  }
}
```

The fields are as follows:

- node_count :: number of nodes with this tier/role
- index_count :: number of indices on this tier
- total_shard_count :: total number of shards for all nodes in this tier
- primary_shard_count :: number of primary shards for all nodes in this tier
- doc_count :: number of documents for all nodes in this tier
- total_size_bytes :: total number of bytes for all shards for all nodes in this tier
- primary_size_bytes :: number of bytes for all primary shards on all nodes in this tier
- primary_shard_size_avg_bytes :: average shard size for primary shard in this tier
- primary_shard_size_median_bytes :: median shard size for primary shard in this tier
- primary_shard_size_mad_bytes :: [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) of shard size for primary shard in this tier

Relates to elastic#60848
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Features)

@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Oct 1, 2020
@dakrone
Copy link
Copy Markdown
Member Author

dakrone commented Oct 1, 2020

@elasticmachine run elasticsearch-ci/bwc

@dakrone dakrone merged commit f0f0da2 into elastic:7.x Oct 1, 2020
@dakrone dakrone deleted the backport/7.x/pr-63031 branch October 1, 2020 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v7.10.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants