Back port of #10429

Description

As part of an initial design decision, VReplication workflows always retry in case it encounters an error after sleeping for 5 seconds. The reasoning was that, for large reshards/migrations and perpetual materialize workflows, we could often encounter recoverable errors like PRS, restarting of vttablets/mysql servers, network partitions etc. So rather than error out waiting for an operator to manually restart workflows we decided to keep retrying.

Since we only retried every five seconds any resource wastage due to continuously retrying unrecoverable workflows would be small and in most cases we would transparently recover and make forward progress with minimum downtime. This is especially important for Materialize workflows were the user is expecting near realtime performance.

Usually the vreplication workflows would be setup manually and the possibility of errors due to schema issues was minimal and so this approach worked well. However with the introduction of vreplication-based online DDL workflows we see a lot of automated use where user-specified DDLs are directly used to configure vreplication workflows. Incorrect DDLs can thus result in errors that result in prolonged retries that are not recoverable.

Error reporting in VReplication is also not great: we update the message column in the _vt.vreplication table, but that can get overwritten when we retry. We do also log errors in the _vt.vreplication_log table

A change was introduced recently in Online DDL workflows to mitigate this: we look up the error against a set of MySQL errors that we knew were not recoverable and in that case we put the workflow in an error state. Then there are no more automated retries and a manual restart after fixing the error is expected.

However there are still unrecoverable schema-related errors that are not yet mapped or do not map cleanly to MySQL errors. There could also be misconfigured workflows (example: no replicas in a keyspace when the tablet type is set to only replicas, incorrect cell settings etc). Continuously retrying workflows in such cases can delay detecting them.

This PR:

• extends the check for unrecoverable errors to all workflow types, not just Online DDLs
• for all workflows, detects errors that persist for more than the 🚩 new vttablet flag
  --vreplication_max_time_to_retry_errors (default: 15 minutes).

For above cases it directly moves the workflow to Error state, which is then reported in Workflow Show.

Checklist

• "Backport me!" label has been added if this change should be backported
  • We should backport this to 14.0.0-rc, but no further
• Tests were added or are not required
• Documentation was added or is not required