extenstion / feature request - omit canary hosts during retry of a failed request

*Title*: *Retry predicate that can omit canary hosts*

*Description*:

In a canary deployment, failures in the canary subset can be retried on other non canary hosts that are part of the cluster.




