fix(policy): guard against deserialization errors in watches#14844
Merged
fix(policy): guard against deserialization errors in watches#14844
Conversation
Signed-off-by: Alex Leong <alex@buoyant.io>
Signed-off-by: Alex Leong <alex@buoyant.io>
zaharidichev
approved these changes
Jan 12, 2026
Member
zaharidichev
left a comment
There was a problem hiding this comment.
This is great. LGTM, pending comments + CI
|
|
||
| // A watch that uses DeserializeGuard to skip resources which fail to deserialize. | ||
| // Any deserialization errors are logged as warnings and the event is skipped. | ||
| fn guarded_watch<T, R>( |
Member
There was a problem hiding this comment.
TIOLI: This can be simplified to:
- avoid writing the logging logic for each event
- drop the
Tparam of the runtime, seems to always beOption<Bound>
// A watch that uses DeserializeGuard to skip resources which fail to deserialize.
// Any deserialization errors are logged as warnings and the event is skipped.
fn guarded_watch<R>(
runtime: &mut kubert::Runtime<Option<Bound>>,
watcher_config: watcher::Config,
) -> impl Stream<Item = watcher::Event<R>>
where
R: Resource + DeserializeOwned + Clone + Debug + Send + 'static,
R::DynamicType: Default,
{
runtime
.watch_all::<DeserializeGuard<R>>(watcher_config)
.filter_map(async |item| {
let log_warning = |t: &DeserializeGuard<R>| {
if let Err(ref err) = t.0 {
tracing::warn!(
"skipping invalid {} resource {}/{}: {}",
R::kind(&Default::default()),
t.namespace().unwrap_or("<cluster>".to_string()),
t.name_any(),
err
);
}
};
match item {
watcher::Event::Apply(t) => {
log_warning(&t);
t.0.ok().map(watcher::Event::Apply)
}
watcher::Event::Delete(t) => {
log_warning(&t);
t.0.ok().map(watcher::Event::Delete)
}
watcher::Event::InitApply(t) => {
log_warning(&t);
t.0.ok().map(watcher::Event::InitApply)
}
watcher::Event::Init => Some(watcher::Event::Init),
watcher::Event::InitDone => Some(watcher::Event::InitDone),
}
})
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14741
When the policy controller watches a resource and encounters a resource which it cannot deserialize, the entire watch fails and needs to be restarted. When this happens, the problematic resource is encountered again leading to an infinite loop of watch restarts. This can happen when a resource has a enum variant which is not present in Linkerd's client bindings, such as with the CORS filter in HttpRoute as described in #14741.
We add a
DeserializeGuardto all of the resource watches in the policy controller so that when a resource cannot be deserialized, that event is logged and the event is skipped, allowing the watch to continue.Prior to this fix, the policy controller would log an repeating stream of this log message when such a resource was encountered:
After this fix, it now logs this message once: