Skip to content

Commit 0f2b29a

Browse files
ycombinatorycombinator
authored
Adding elasticsearch/ccr metricset (#8335)
This PR adds the `elasticsearch/ccr` metricset which collects cross-cluster replication (CCR) stats from the `GET _ccr/stats` Elasticsearch API and indexes them in `metricbeat-*`.
1 parent 0d28221 commit 0f2b29a

20 files changed

Lines changed: 645 additions & 21 deletions

‎CHANGELOG.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
130130
- Add support to renamed fields planned for redis 5.0. {pull}8167[8167]
131131
- Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. {pull}8278[8278]
132132
- Added 'died' PID state to process_system metricset on system module{pull}8275[8275]
133+
- Added `ccr` metricset to Elasticsearch module. {pull}8335[8335]
133134

134135
*Packetbeat*
135136

‎metricbeat/docs/fields.asciidoc‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3459,6 +3459,85 @@ type: keyword
34593459
Elasticsearch state id.
34603460
34613461
3462+
--
3463+
3464+
[float]
3465+
== ccr fields
3466+
3467+
Cross-cluster replication stats
3468+
3469+
3470+
3471+
3472+
*`elasticsearch.ccr.leader.index`*::
3473+
+
3474+
--
3475+
type: keyword
3476+
3477+
Name of leader index
3478+
3479+
3480+
--
3481+
3482+
*`elasticsearch.ccr.leader.max_seq_no`*::
3483+
+
3484+
--
3485+
type: long
3486+
3487+
Maximum sequence number of operation on the leader shard
3488+
3489+
3490+
--
3491+
3492+
3493+
*`elasticsearch.ccr.follower.index`*::
3494+
+
3495+
--
3496+
type: keyword
3497+
3498+
Name of follower index
3499+
3500+
3501+
--
3502+
3503+
*`elasticsearch.ccr.follower.shard.number`*::
3504+
+
3505+
--
3506+
type: long
3507+
3508+
Number of the shard within the index
3509+
3510+
3511+
--
3512+
3513+
*`elasticsearch.ccr.follower.operations_indexed`*::
3514+
+
3515+
--
3516+
type: long
3517+
3518+
Number of operations indexed (replicated) into the follower shard from the leader shard
3519+
3520+
3521+
--
3522+
3523+
*`elasticsearch.ccr.follower.time_since_last_fetch.ms`*::
3524+
+
3525+
--
3526+
type: long
3527+
3528+
Time, in ms, since the follower last fetched from the leader
3529+
3530+
3531+
--
3532+
3533+
*`elasticsearch.ccr.follower.global_checkpoint`*::
3534+
+
3535+
--
3536+
type: long
3537+
3538+
Global checkpoint value on follower shard
3539+
3540+
34623541
--
34633542
34643543
[float]

‎metricbeat/docs/modules/elasticsearch.asciidoc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ This module supports TLS connection when using `ssl` config field, as described
5050

5151
The following metricsets are available:
5252

53+
* <<metricbeat-metricset-elasticsearch-ccr,ccr>>
54+
5355
* <<metricbeat-metricset-elasticsearch-cluster_stats,cluster_stats>>
5456

5557
* <<metricbeat-metricset-elasticsearch-index,index>>
@@ -68,6 +70,8 @@ The following metricsets are available:
6870

6971
* <<metricbeat-metricset-elasticsearch-shard,shard>>
7072

73+
include::elasticsearch/ccr.asciidoc[]
74+
7175
include::elasticsearch/cluster_stats.asciidoc[]
7276

7377
include::elasticsearch/index.asciidoc[]

‎metricbeat/docs/modules/elasticsearch/ccr.asciidoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
////
2+
This file is generated! See scripts/docs_collector.py
3+
////
4+
5+
[[metricbeat-metricset-elasticsearch-ccr]]
6+
=== Elasticsearch ccr metricset
7+
8+
beta[]
9+
10+
include::../../../module/elasticsearch/ccr/_meta/docs.asciidoc[]
11+
12+
13+
==== Fields
14+
15+
For a description of each field in the metricset, see the
16+
<<exported-fields-elasticsearch,exported fields>> section.
17+
18+
Here is an example document generated by this metricset:
19+
20+
[source,json]
21+
----
22+
include::../../../module/elasticsearch/ccr/_meta/data.json[]
23+
----

‎metricbeat/docs/modules_list.asciidoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@ This file is generated! See scripts/docs_collector.py
3333
|<<metricbeat-module-dropwizard,Dropwizard>> beta[] |image:./images/icon-no.png[No prebuilt dashboards] |
3434
.1+| .1+| |<<metricbeat-metricset-dropwizard-collector,collector>> beta[]
3535
|<<metricbeat-module-elasticsearch,Elasticsearch>> beta[] |image:./images/icon-no.png[No prebuilt dashboards] |
36-
.9+| .9+| |<<metricbeat-metricset-elasticsearch-cluster_stats,cluster_stats>> beta[]
36+
.10+| .10+| |<<metricbeat-metricset-elasticsearch-ccr,ccr>> beta[]
37+
|<<metricbeat-metricset-elasticsearch-cluster_stats,cluster_stats>> beta[]
3738
|<<metricbeat-metricset-elasticsearch-index,index>> beta[]
3839
|<<metricbeat-metricset-elasticsearch-index_recovery,index_recovery>> beta[]
3940
|<<metricbeat-metricset-elasticsearch-index_summary,index_summary>> beta[]

‎metricbeat/helper/elastic/elastic.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121
"fmt"
2222
"strings"
2323

24+
"github.com/elastic/beats/libbeat/common"
2425
"github.com/elastic/beats/metricbeat/mb"
2526
)
2627

@@ -92,3 +93,18 @@ func ReportErrorForMissingField(field string, product Product, r mb.ReporterV2)
9293
func MakeErrorForMissingField(field string, product Product) error {
9394
return fmt.Errorf("Could not find field '%v' in %v stats API response", field, strings.Title(product.String()))
9495
}
96+
97+
// IsFeatureAvailable returns whether a feature is available in the current product version
98+
func IsFeatureAvailable(currentProductVersion, featureAvailableInProductVersion string) (bool, error) {
99+
currentVersion, err := common.NewVersion(currentProductVersion)
100+
if err != nil {
101+
return false, err
102+
}
103+
104+
wantVersion, err := common.NewVersion(featureAvailableInProductVersion)
105+
if err != nil {
106+
return false, err
107+
}
108+
109+
return !currentVersion.LessThan(wantVersion), nil
110+
}

‎metricbeat/include/list.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎metricbeat/module/elasticsearch/ccr/_meta/data.json‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{
2+
"@timestamp": "2017-10-12T08:05:34.853Z",
3+
"beat": {
4+
"hostname": "host.example.com",
5+
"name": "host.example.com"
6+
},
7+
"elasticsearch": {
8+
"ccr": {
9+
"follower": {
10+
"global_checkpoint": 1,
11+
"index": "my_follower",
12+
"operations_indexed": 2,
13+
"shard": {
14+
"number": 0
15+
},
16+
"time_since_last_fetch": {
17+
"ms": 4926
18+
}
19+
},
20+
"leader": {
21+
"index": "my_leader",
22+
"max_seq_no": 1
23+
}
24+
},
25+
"cluster": {
26+
"id": "KSGkOjOuSg6whAgtpPyhQw",
27+
"name": "elasticsearch"
28+
}
29+
},
30+
"metricset": {
31+
"host": "127.0.0.1:9200",
32+
"module": "elasticsearch",
33+
"name": "ccr",
34+
"rtt": 115
35+
},
36+
"service": {
37+
"name": "elasticsearch"
38+
}
39+
}

‎metricbeat/module/elasticsearch/ccr/_meta/docs.asciidoc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
This is the `ccr` metricset of the Elasticsearch module. It interrogates the
2+
Cross Cluster Replication Stats API endpoint to fetch information about shards
3+
in the Elasticsearch cluster that are participating in cross-cluster replication.

‎metricbeat/module/elasticsearch/ccr/_meta/fields.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
- name: ccr
2+
type: group
3+
description: >
4+
Cross-cluster replication stats
5+
release: beta
6+
fields:
7+
- name: leader
8+
type: group
9+
fields:
10+
- name: index
11+
type: keyword
12+
description: >
13+
Name of leader index
14+
- name: max_seq_no
15+
type: long
16+
description: >
17+
Maximum sequence number of operation on the leader shard
18+
- name: follower
19+
type: group
20+
fields:
21+
- name: index
22+
type: keyword
23+
description: >
24+
Name of follower index
25+
- name: shard.number
26+
type: long
27+
description: >
28+
Number of the shard within the index
29+
- name: operations_indexed
30+
type: long
31+
description: >
32+
Number of operations indexed (replicated) into the follower shard from the leader shard
33+
- name: time_since_last_fetch.ms
34+
type: long
35+
description: >
36+
Time, in ms, since the follower last fetched from the leader
37+
- name: global_checkpoint
38+
type: long
39+
description: >
40+
Global checkpoint value on follower shard

0 commit comments

Comments
 (0)