-
Notifications
You must be signed in to change notification settings - Fork 4.1k
changefeedccl: mark retryable errors as warnings in the logs #69677
Description
As pointed out in #69561, retryable errors are logged as errors, but are generally not actionable today. We should consider labeling these as warnings, with a more severe error if we are experiencing a high number of the same retryable error.
Today:
2021-08-30T10:07:18.331Z,syd-crdb-cockroachdb-1,cockroach,"[n17,job=‹688920172251709457›] 1319933 CHANGEFEED job ‹688920172251709457› encountered retryable error: ‹retryable changefeed error›: ‹kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.›"
Possible future state:
2021-08-30T10:07:18.331Z,syd-crdb-cockroachdb-1,cockroach,"[n17,job=‹688920172251709457›] 1319933 WARNING: CHANGEFEED job ‹688920172251709457› encountered retryable error: ‹retryable changefeed error›: ‹kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.›"