Summary

Improve the screenshot/OCR visual-context pipeline so Cotabby sends richer, sanitized prompt context instead of dropping context when summarization fails. The change keeps Screen Recording required, expands OCR capture quality and budgets, adds deterministic OCR-noise filtering, and replaces the generic summary prompt with an autocomplete-focused context extraction prompt.

Validation

• plutil -lint Cotabby.xcodeproj/project.pbxproj
  • Cotabby.xcodeproj/project.pbxproj: OK
• xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination platform=macOS build-for-testing -derivedDataPath build/DerivedData
  • ** TEST BUILD SUCCEEDED **
• xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination platform=macOS test -only-testing:CotabbyTests/PromptContextSanitizerTests -only-testing:CotabbyTests/ScreenshotContextGeneratorTests -only-testing:CotabbyTests/VisualContextSummaryPromptRendererTests -derivedDataPath build/DerivedData
  • Compiled, then failed to load the app-hosted CotabbyTests.xctest bundle because the test bundle and host app have different Team IDs. This matches the known local signing failure for app-hosted tests.
• rm -rf build/DerivedData

Linked issues

None.

Risk / rollout notes

• Visual context intentionally trades a bit more capture/OCR/summarization latency for higher-quality context: larger screenshot/OCR budgets, Vision accurate mode, and a longer summary timeout.
• Fast Mode still skips screenshot/OCR capture after permissions are granted, but Screen Recording remains required for autocomplete eligibility.
• No hosted APIs, raw screenshots, or unbounded OCR text are introduced.

Greptile Summary

This PR overhauls the visual-context OCR pipeline to deliver richer, sanitized prompt context for autocomplete instead of silently dropping context when summarization fails. Capture budgets are expanded (larger screenshot dimensions, 5 000-char OCR cap, 6 s timeout, Vision accurate mode), and a new deterministic token-scoring pass in PromptContextSanitizer replaces the previous heuristic with layered rules that keep prose, technical tokens, emails, file paths, and non-Latin scripts while dropping OCR garbage.

• Graceful OCR fallback: ScreenshotContextGenerator.resolvedContextText now falls back to bounded, sanitized OCR text whenever summarization fails or produces an empty result, rather than propagating the error and losing all visual context.
• Testability via protocols: WindowScreenshotCapturing and ScreenTextExtracting protocol seams let ScreenshotContextGenerator be exercised with stub implementations, enabling the new ScreenshotContextGeneratorTests to cover every key path without requiring live capture or a GGUF model.
• Autocomplete-focused prompt: VisualContextSummaryPromptRenderer extracts the summarization prompt into a dedicated, testable type with explicit priority ordering and prompt-injection rejection instructions.

Confidence Score: 5/5

Safe to merge; the pipeline changes are well-contained, all fallback paths return bounded sanitized text rather than empty context, and new protocol seams enable meaningful unit coverage.

The OCR fallback logic and token-scoring rules are the most complex additions, and both are exercised by new unit tests. The summarizer now propagates errors instead of swallowing them, but resolvedContextText catches every error and falls back gracefully, so no regression path was opened. Expanded capture budgets increase latency but are isolated to the visual-context path and explicitly documented.

PromptContextSanitizer.swift — the isLikelyShortMixedCaseNoise heuristic (already flagged in a previous review round) still false-positives on lowercase-initial brand names common on macOS screenshots (iPhone, macOS, iCloud, UIKit). No other files require special attention.

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[FocusedInputSnapshot] --> B[captureScreenshot WindowScreenshotCapturing]
    B --> C{OCR extractText}
    C -- noRecognizedText --> D{windowTitle available?}
    D -- no --> ERR1[throw unavailable]
    D -- yes --> E[normalizeRecognizedText windowTitle]
    E --> F{hasMeaningfulSignal?}
    F -- no --> ERR1
    F -- yes --> RETURN1[return VisualContextExcerpt source ocr_fallback]
    C -- success --> G[normalizeRecognizedText sanitizeOCR cap 5000 chars]
    G --> H{hasMeaningfulSignal normalizedText?}
    H -- no --> ERR2[throw unavailable]
    H -- yes --> I[resolvedContextText]
    I --> J{summarizer configured?}
    J -- no --> K[ocrFallback boundedSummaryText]
    J -- yes --> L[summarizer.summarize VisualContextSummaryPromptRenderer 6s timeout 160 tokens]
    L -- throws --> K
    L -- empty result --> K
    L -- success --> M[boundedSummaryText cap 1500 chars]
    M --> N{hasMeaningfulSignal summary?}
    N -- no --> K
    N -- yes --> RETURN2[return VisualContextExcerpt source summary]
    K --> O{hasMeaningfulSignal fallback?}
    O -- no --> ERR2
    O -- yes --> RETURN3[return VisualContextExcerpt source ocr_fallback]

_{Reviews (2): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}