go-carbon icon indicating copy to clipboard operation
go-carbon copied to clipboard

Metric creation throttled w/ limits=0 and hard-max-creates-per-second = false

Open Quentin-M opened this issue 7 years ago • 2 comments

Hi,

Just updated to master (22a6384bc3f8a539f88b34777060f6ed7b125aa8) and now seeing:

[2018-09-11T00:41:26.421Z] ERROR [persister] metric creation throttled {"name": "carbon.agents.graphite-cache-0.persister.maxCreatesPerSecond", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-0_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-1_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.batchesSent", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-1_graphite-cache_dev_svc_cluster_local:2004:None.attemptedRelays", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.blacklistMatches", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.memUsage", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-3_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.419Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.attemptedRelays", "operation": "create", "dropped": true}
[2018-09-11T00:41:57.420Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-49m9z-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.relayMaxQueueLength", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.attemptedRelays", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-0_graphite-cache_dev_svc_cluster_local:2004:None.attemptedRelays", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-3_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-0_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.blacklistMatches", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.metricsReceived", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-3_graphite-cache_dev_svc_cluster_local:2004:None.relayMaxQueueLength", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-3_graphite-cache_dev_svc_cluster_local:2004:None.attemptedRelays", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-0_graphite-cache_dev_svc_cluster_local:2004:None.batchesSent", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.cpuUsage", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-1_graphite-cache_dev_svc_cluster_local:2004:None.sent", "operation": "create", "dropped": true}
[2018-09-11T00:42:00.228Z] ERROR [persister] metric creation throttled {"name": "carbon.relays.graphite-relay-7c945485c5-nqnpx-a.destinations.graphite-cache-2_graphite-cache_dev_svc_cluster_local:2004:None.relayMaxQueueLength", "operation": "create", "dropped": true}

With the following configuration:

max-updates-per-second = 0
max-creates-per-second = 0
hard-max-creates-per-second = false

Quentin-M avatar Sep 11 '18 00:09 Quentin-M

@gunnihinn , fyi

lomik avatar Sep 12 '18 07:09 lomik

Thanks, I'll have a look in a bit.

gunnihinn avatar Sep 12 '18 09:09 gunnihinn