-
Notifications
You must be signed in to change notification settings - Fork 5.8k
Incorrect Endpoint for Bucket Metrics In Couchbase #9701
Description
This is using the Cluster wide aggregate endpoint for gathering bucket metrics. If metrics are reported per host, each host reports the bucket metrics, resulting in the values being multiplied.
| req, err := http.NewRequest("GET", server+"/pools/default/buckets/"+bucket+"/stats?", nil) |
The current endpoint being used is http://localhost:8091/pools/default/buckets/{BUCKET}/stats which is the sum of all metrics across the cluster. To get per node metrics the node specific endpoint must be used.
This is easy enough to very by looking at the vb_active_num stat, as a bucket in Couchbase has 1024 vBuckets regardless of however many nodes are in the cluster, the vBuckets are distributed amongst all of the data nodes in the cluster. When using the cluster wide endpoint this value will always be 1024, but in a 3 node configuration for example 2 nodes would report a vb_active_num of 341 and 1 node would report 342 .
Per Node Endpoint: http://localhost:8091/pools/default/buckets/{BUCKET}/nodes/{HOSTNAME}:8091/stats
Note that the {HOSTNAME} must match what is configured in Couchbase Server. Even when using localhost you can still determine the node name as configured in Couchbase. The /pools/default endpoint that is being referenced and iterated over here:
| for i := 0; i < len(pool.Nodes); i++ { |
Needs to look for an attribute called "thisNode": true that is only defined on the current node returning the response, otherwise it is not present. that is the hostname you would use. for example
http://localhost:8091/pools/default/buckets/{BUCKET}/nodes/node1.example.org:8091/stats
The REST Endpoint Monitoring Guide may be useful https://developer.couchbase.com/seo-monitoring-guide