Azure output plugin - properly recover after context deadline exceeded #10950
Description
Feature Request
When there is a network issue (corporate proxy, corporate zscaler and other network elements that can cause this) that triggers "context deadline exceeded" the connection is not reinitiated, and telegraf keeps throwing the same error indefinitely.
Proposal:
Connection is reinitiated after this happens.
Current behavior:
Mar 22 09:52:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:52:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:53:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:53:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:54:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:54:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:55:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:55:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:56:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:56:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:57:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:57:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:58:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:58:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 22 09:59:14 HOSTNAME_CENSORED.com telegraf[6436]: 2022-03-22T09:59:14Z E! [agent] Error writing to outputs.azure_monitor: Post "https://westus.monitoring.azure.com/subscriptions/CENSORED/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Desired behavior:
When context deadline exceeded is detected multiple times, the connections should try to get reinitiated.
We are currently using a workaround where we set the conntimeout to 10 seconds (IdleConnTimeout: 10 * time.Second in below code added in /plugins/outputs/azure_monitor/azure_monitor.go), which is not ideal solution:
a.client = &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
IdleConnTimeout: 10 * time.Second,
},
Timeout: time.Duration(a.Timeout),
}
Use case:
To make telegraf successfully send metrics