Skip to content

changefeedccl: mark retryable errors as warnings in the logs #69677

@amruss

Description

@amruss

As pointed out in #69561, retryable errors are logged as errors, but are generally not actionable today. We should consider labeling these as warnings, with a more severe error if we are experiencing a high number of the same retryable error.

Today:
2021-08-30T10:07:18.331Z,syd-crdb-cockroachdb-1,cockroach,"[n17,job=‹688920172251709457›] 1319933 CHANGEFEED job ‹688920172251709457› encountered retryable error: ‹retryable changefeed error›: ‹kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.›"

Possible future state:
2021-08-30T10:07:18.331Z,syd-crdb-cockroachdb-1,cockroach,"[n17,job=‹688920172251709457›] 1319933 WARNING: CHANGEFEED job ‹688920172251709457› encountered retryable error: ‹retryable changefeed error›: ‹kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.›"

Metadata

Metadata

Assignees

Labels

A-cdcChange Data CaptureC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)E-starterMight be suitable for a starter project for new employees or team members.T-cdc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions