Skip to content

test: fail on missing screenshot#4158

Merged
edemaine merged 2 commits intomainfrom
screenshotter-new
Mar 7, 2026
Merged

test: fail on missing screenshot#4158
edemaine merged 2 commits intomainfrom
screenshotter-new

Conversation

@edemaine
Copy link
Copy Markdown
Member

@edemaine edemaine commented Mar 7, 2026

As noted in #4147 (review)

What is the previous behavior before this PR?
If a screenshot file is missing, CI would pass. #4147 is currently an example.

What is the new behavior after this PR?
CI now fails. Specifically, --verify causes failure if there's no image file to compare against. --new still causes the new screenshot to be saved (which is useful for artifact uploading).

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 7, 2026

Greptile Summary

This PR fixes a CI gap where a missing screenshot file during a --verify run would silently pass. The fix is small and focused: it adds an early break from the retry loop when opts.verify is set and no expected file exists, which correctly falls through to the existing failure-recording code (exitStatus = 3). A companion guard (opts.diff && expected) prevents convert from being invoked against a non-existent base file.

Key changes:

  • opts.verify && !expected now breaks the retry loop immediately and records the test as failed, causing CI to exit with status 3.
  • --new mode is preserved: the newly taken screenshot is still saved to newDir for artifact uploading even when the baseline was absent.
  • The diff-image generation step is now guarded by expected to avoid passing a missing file to convert.

Confidence Score: 4/5

  • This PR is safe to merge; the logic change is correct and well-scoped.
  • The change is minimal (two small hunks), the failure path was already in place, and all edge cases (--new, --diff, retry loop) are handled correctly. The only notable concern is a harmless write-then-delete cycle in the --diff-only + missing-screenshot edge case.
  • No files require special attention beyond the one changed file.

Important Files Changed

Filename Overview
dockers/screenshotter/screenshotter.js Adds a guard to fail CI when a screenshot file is missing under --verify, and prevents a crash when --diff is used without an expected file. Logic is correct; minor style inconsistency in logging and a write-then-delete side-effect in the --diff-only+missing-screenshot edge case.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[takeScreenshot called] --> B{opts.verify?}
    B -->|yes| C{file exists?}
    C -->|yes| D[expected = readFile]
    C -->|no| E[expected = null]
    B -->|no| F[expected = null]
    D & E & F --> G[while retry loop]
    G --> H[take screenshot, assign buf]
    H --> I{opts.verify AND not expected?}
    I -->|yes| J[log error missing screenshot, break]
    I -->|no| K{expected?}
    K -->|yes| L{buf equals expected?}
    L -->|yes| M[log ok, return success]
    L -->|no| N[log error, retry]
    K -->|no| O[writeFile baseline, return success]
    J --> Q
    N --> P{retries exhausted?}
    P -->|no| G
    P -->|yes| Q[console.error FAIL, exitStatus=3]
    Q --> R{opts.diff or opts.new?}
    R -->|yes| S[writeFile buf to outputDir]
    S --> T{opts.diff AND expected?}
    T -->|yes| U[generate diff image via convert]
    T -->|no| V{opts.new?}
    U --> V
    V -->|no| W[unlink bufFile]
    V -->|yes| X[keep bufFile in newDir]
    R -->|no| Y[end]
Loading

Comments Outside Diff (1)

  1. dockers/screenshotter/screenshotter.js, line 608-635 (link)

    Write-then-immediately-delete when --diff is set but screenshot is missing

    When --verify --diff is used (without --new) and the expected screenshot is absent, buf is written to diffDir (line 616), the diff-image step is correctly skipped (line 618), but then bufFile is immediately deleted again on line 634 because !opts.new is true. The end state is correct (no leftover file), but the unnecessary write/delete pair could be avoided by guarding the whole block:

    if ((opts.diff || opts.new) && (expected || opts.new)) {

    This is a minor inefficiency rather than a bug, but worth noting.

Last reviewed commit: 5a82ab0

@grigoriy-reshetniak
Copy link
Copy Markdown
Collaborator

Looks great!

@edemaine edemaine merged commit 8c2bf2e into main Mar 7, 2026
8 checks passed
@edemaine edemaine deleted the screenshotter-new branch March 7, 2026 15:37
@edemaine
Copy link
Copy Markdown
Member Author

edemaine commented Mar 8, 2026

🎉 This PR is included in version 0.16.38 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants