Summary

Currently, canary tests only swap the kcp image but use the current operator codebase. This means the test doesn't validate a real operator upgrade path — only that the new image works with the existing operator binary.

Problem

As highlighted in PR #188, this approach breaks when there are operator-level incompatibilities (e.g., unknown flags introduced in newer versions, like the cache server flag). The test passes for the wrong reasons and fails for reasons unrelated to the image swap.

Proposed Solution

Canary tests should test a full operator upgrade, which means:

1. Start with the released operator image (e.g., the last stable release from the registry)
2. Upgrade the operator itself (not just the kcp image) to the new version being tested
3. Then validate that kcp runs correctly with the new operator

This ensures we catch operator-level regressions, not just kcp image-level ones.

Context

Raised in PR #188 review: #188 (comment)

/cc @mjudeikis