fix: propagate watch callback errors to KFC for cache reliability by samayer12 · Pull Request #3052 · defenseunicorns/pepr

samayer12 · 2026-04-17T16:28:46Z

Summary

Re-throw watch callback errors so KFC knows the callback failed and does not permanently cache items whose callbacks threw. Previously errors were swallowed, causing KFC to skip the item on subsequent relists — making resources like UDSExemptions invisible for hours.
Add soak test metric assertions for cache miss growth, resync failures, and watch controller failures that were previously collected but never used as pass/fail criteria.

Context

Production bug reported in product-support: Pepr occasionally misses UDSExemption resources after K8s API server 429 errors during EKS cluster bootstrap (UDS Core 1.1.0-unicorn, Pepr 1.1.5). The missed exemption stays invisible until pod restart or manual annotation.

Root cause spans two repos:

KFC: Watcher#process caches items before running callbacks — a failed callback leaves the item permanently stuck in cache (companion PR forthcoming)
Pepr (this PR): watchCallback catches and swallows all errors, so KFC never learns a callback failed

Changes

`src/lib/processors/watch-processor.ts`

Add throw e after Log.error() in the watch callback catch block (line 139)

`src/lib/processors/watch-processor.test.ts`

4 new tests: callback error propagation (direct, queue, finalizer) + success regression guard
Mock ../finalizer to isolate finalizer tests

`scripts/soak-test.sh`

Assert watch_controller_failures_total == 0 every iteration
Assert cache miss growth stays below configurable threshold after stabilization
Assert resync failure count stays below configurable threshold

`scripts/soak-summary.sh`

Add "Cache Miss Trend" section with post-stabilization growth analysis

Test plan

New unit tests written first (TDD), confirmed they fail without the fix
Fix applied, all 1338 unit tests pass
ESLint + Prettier + ShellCheck pass
Soak test validates with new assertions (after merge + nightly run)
Companion KFC PR: reorder #process to cache after callback succeeds (see fix: cache items after callback succeeds to prevent permanent loss kubernetes-fluent-client#1105)

samayer12 · 2026-04-17T16:29:31Z

@greptileai

greptile-apps · 2026-04-17T16:40:58Z

Greptile Summary

This PR fixes a production reliability bug where watchCallback silently swallowed errors, causing KFC to permanently cache items whose callbacks failed (making resources like UDSExemption invisible until pod restart). The fix is a single throw e after the existing Log.error call; four new TDD tests confirm error propagation for the direct, queue, and finalizer paths. The soak-test improvements add enforceable pass/fail assertions for watch controller failures, cache miss growth, and resync failures — and thread CACHE_MISS_GROWTH_THRESHOLD explicitly to soak-summary.sh, closing the previously flagged threshold-mismatch gap.

Confidence Score: 5/5

Safe to merge — the core fix is a minimal, well-tested one-liner and all remaining findings are P2 suggestions on soak-test infrastructure.

The production bug fix is correct and backed by four new unit tests covering all code paths. All previous reviewer concerns (forward-looking comment, threshold mismatch) are addressed. The only open finding is a P2 on the cache-miss growth metric logic in the soak script, which does not affect runtime correctness of Pepr itself.

scripts/soak-test.sh — the cache miss growth assertion compares per-interval deltas rather than cumulative post-stabilization sum, which may miss gradual steady-state accumulation.

Important Files Changed

Filename	Overview
src/lib/processors/watch-processor.ts	Core fix: adds `throw e` after logging in the catch block so KFC receives callback failures; includes a clear NOTE about the companion KFC PR dependency
src/lib/processors/watch-processor.test.ts	Adds four well-structured TDD tests covering direct watch, queue, finalizer error propagation, and success regression; finalizer mock correctly isolates the new test suite
scripts/soak-test.sh	Adds three metric assertions (watch failures, cache miss growth, resync failures); cache miss growth check compares per-interval deltas rather than cumulative post-stabilization sum, which can miss gradual steady-state accumulation
scripts/soak-summary.sh	Adds Cache Miss Trend section and now accepts `CACHE_MISS_GROWTH_THRESHOLD` as a positional argument (resolving the previous env-var inconsistency with soak-test.sh)
.github/workflows/soak.yaml	Promotes threshold env vars to job-level `env`, passes `CACHE_MISS_GROWTH_THRESHOLD` explicitly to soak-summary.sh — no issues found

Sequence Diagram

sequenceDiagram
    participant KFC as KFC (Watcher#process)
    participant WP as watchCallback (Pepr)
    participant CB as watchCallback/finalizeCallback

    Note over KFC,CB: Before this PR
    KFC->>KFC: cache item
    KFC->>WP: invoke callback
    WP->>CB: await callback
    CB-->>WP: throws Error
    WP->>WP: Log.error (swallow)
    WP-->>KFC: resolve (KFC unaware of failure)
    Note over KFC: item stuck cached ✗

    Note over KFC,CB: After this PR
    KFC->>KFC: cache item (companion PR will reorder this)
    KFC->>WP: invoke callback
    WP->>CB: await callback
    CB-->>WP: throws Error
    WP->>WP: Log.error
    WP-->>KFC: re-throw Error
    KFC->>KFC: aware callback failed ✓

_{Reviews (3): Last reviewed commit: "fix: address greptile review feedback" | Re-trigger Greptile}

Re-throw errors from watch callbacks after logging so KFC knows the callback failed and does not mark the item as successfully cached. Previously, errors were caught and swallowed, which meant KFC's cache would permanently consider a failed item as processed — subsequent relists would skip it and the resource would stay invisible. This is the Pepr-side fix for a production issue where UDSExemption resources are occasionally missed after K8s API server 429 errors during cluster bootstrap. A companion KFC PR will reorder cache updates to happen after callbacks succeed. Also hardens the soak test with metric assertions for cache miss growth, resync failures, and watch controller failures that were previously collected but never asserted as pass/fail criteria.

Clarify watch-processor comment to reflect current KFC caching behavior and fix threshold mismatch between soak-test.sh and soak-summary.sh by defining thresholds at the CI job level and passing explicitly.

…1105) ## Summary - **Reorder `#process`** to update the cache only after the callback succeeds. Previously, items were cached before the callback ran — a failed callback left the item permanently stuck in cache, invisible to the application. - **Guard concurrent `#list()` calls** with a `#listInProgress` flag to prevent cache corruption when the relist timer fires during an in-progress list. - **Await `#process` calls in `#list()`** instead of fire-and-forget (`void`), ensuring list reconciliation completes atomically. - **Return success/failure from `#list()`** so both `#watch()` and the relist timer can trigger faster resync when list operations or callbacks fail. - **Emit `DATA` event only after callback succeeds and cache is updated**, eliminating phantom events for items that failed processing. - **Refresh `lastSeenTime` during list processing** to prevent spurious resync triggers during slow callback sequences. - **Fix dead pagination guard** by separating `pageCount` from `retryCount` — the guard could never trigger because `retryCount` was always reset to 0 on pagination. - **Remove dead backoff ternary** and unused `startSleep` import. ## Context Production bug in Pepr/UDS Core: K8s API server returns 429 Too Many Requests during EKS cluster bootstrap. The KFC watcher misses `UDSExemption` CRs because: 1. `#process` caches items *before* running callbacks (line 363: `this.#cache.set(...)`) 2. If the callback throws, the error is caught internally (line 377-378) but the item is already in cache 3. On the next `#list()` relist, the item's UID is found in cache at the same resourceVersion → skipped 4. The resource stays invisible until pod restart or manual annotation Companion Pepr PR: defenseunicorns/pepr#3052 ## Changes ### `src/fluent/watch.ts` - `#process`: Move `cache.set()`/`cache.delete()` after `await this.#callback()`. Move `DATA` emit to after cache update. Remove internal try/catch — callers handle errors. - `#list()`: Replace `void this.#process()` with `await this.#process()` wrapped in try/catch. Return `boolean` (false on HTTP error, callback failure, or max pages). Add `#listInProgress` concurrency guard with `finally` cleanup. Add `pageCount` parameter for pagination depth tracking. Refresh `lastSeenTime` before each process call. Remove dead backoff ternary. - `#watch()`: Check `#list()` return value. On failure, set `lastSeenTime = OVERRIDE` to trigger faster resync. - Relist timer: Await `#list()` return value and trigger faster resync on failure. Skipped if `#listInProgress`. 10 new test cases covering: callback failure (no cache, no DATA, DATA_ERROR emitted), relist retry, cached item skip, 429 exhaustion, sequential processing, delete retry, resync on callback/HTTP failure (table-driven via `it.each`), concurrent list prevention, pagination limit. ## Breaking change analysis **No breaking changes.** `#process`, `#list()`, `#cache`, and `#listInProgress` are all private. `DATA_ERROR` event signature unchanged. Cache has no public accessor. **Behavioral change**: `WatchEvent.DATA` now fires only after successful callback + cache update (previously fired before callback). No internal consumers rely on the old timing. External consumers now get a stronger guarantee: DATA means the item was fully processed. ## Test plan - [x] 10 new test cases covering all fix commits - [x] All 205 tests pass - [x] ESLint: 0 errors (pre-existing warnings only) - [x] TypeScript: no new type errors - [x] E2E tests against k3d cluster - [ ] Soak test with Pepr companion PR (defenseunicorns/pepr#3052) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

codecov · 2026-04-24T19:12:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (ff16fc8) to head (1070a2c).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@     Coverage Diff      @@
##   main   #3052   +/-   ##
============================
============================

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-04-24T19:12:46Z

🎉 This PR is included in version 1.1.7 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

github-project-automation Bot added this to Pepr Project Board Apr 17, 2026

samayer12 mentioned this pull request Apr 17, 2026

fix: cache items after callback succeeds to prevent permanent loss defenseunicorns/kubernetes-fluent-client#1105

Merged

6 tasks

greptile-apps Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread src/lib/processors/watch-processor.ts Outdated

Comment thread scripts/soak-test.sh

samayer12 added 2 commits April 23, 2026 16:01

fix: address greptile review feedback

1070a2c

Clarify watch-processor comment to reflect current KFC caching behavior and fix threshold mismatch between soak-test.sh and soak-summary.sh by defining thresholds at the CI job level and passing explicitly.

samayer12 force-pushed the fix/watch-callback-error-propagation branch from 35fd3bb to 1070a2c Compare April 23, 2026 21:01

samayer12 marked this pull request as ready for review April 23, 2026 22:05

samayer12 requested a review from a team as a code owner April 23, 2026 22:05

AmberFryar approved these changes Apr 24, 2026

View reviewed changes

github-project-automation Bot moved this to 👀 In review in Pepr Project Board Apr 24, 2026

samayer12 added this pull request to the merge queue Apr 24, 2026

samayer12 removed this pull request from the merge queue due to a manual request Apr 24, 2026

samayer12 mentioned this pull request Apr 24, 2026

chore: bump KFC to 3.11.7 #3073

Merged

5 tasks

samayer12 added this pull request to the merge queue Apr 24, 2026

Merged via the queue into main with commit 8f84add Apr 24, 2026
93 checks passed

samayer12 deleted the fix/watch-callback-error-propagation branch April 24, 2026 19:11

github-project-automation Bot moved this from 👀 In review to ✅ Done in Pepr Project Board Apr 24, 2026

github-actions Bot added the released label Apr 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: propagate watch callback errors to KFC for cache reliability#3052

fix: propagate watch callback errors to KFC for cache reliability#3052
samayer12 merged 2 commits intomainfrom
fix/watch-callback-error-propagation

samayer12 commented Apr 17, 2026 •

edited

Loading

Uh oh!

samayer12 commented Apr 17, 2026

Uh oh!

greptile-apps Bot commented Apr 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2026

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

samayer12 commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Changes

src/lib/processors/watch-processor.ts

src/lib/processors/watch-processor.test.ts

scripts/soak-test.sh

scripts/soak-summary.sh

Test plan

Uh oh!

samayer12 commented Apr 17, 2026

Uh oh!

greptile-apps Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2026

Codecov Report

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

samayer12 commented Apr 17, 2026 •

edited

Loading

`src/lib/processors/watch-processor.ts`

`src/lib/processors/watch-processor.test.ts`

`scripts/soak-test.sh`

`scripts/soak-summary.sh`

greptile-apps Bot commented Apr 17, 2026 •

edited

Loading