When a test reports a digest (read: an image with some metadata) to Skia Gold, there is a tri-modal state of how it is considered:

The way our (read: flutter/flutter and flutter/engine) trees work, is that when a test uses Skia Gold, and uploads a digest, the test passes regardless of what match state occurred⁴, and later⁵, asynchronously, our Flutter Gold custom check runs and verifies that no untriaged digests exist at the current commit.

That is, the following workflow causes a post-submit build breakage:

1. User Octocat pushes a PR that, as a result, produces an update to draw_square_test.
2. The test suite/builder, a hypothetical linux: framework_draw_shapes, runs, and reports all tests passing.
3. The Flutter Gold check holds a pending state, and will report back pending untriage digests, similar to below:

4. The square test now draws a circle. Oops! It is marked "negative".
5. Now, all tests are passing (as they were in step 2), and Flutter Gold is also marked as passing.
6. Either manually, or via auto-submit, the PR lands.
7. ... eventually, it's run on post-submit, which cause a tree breakage, because a test matches a negative digest.

Possible improvements (to discuss):

• Update our documentation on Skia Gold
• Update the GitHub bot to print a scary message that "negative doesn't work like you think it does"
• Update the Flutter Gold check to consider matching negative digests as a failure (note that from talking to @kjlubick, this currently is not something available in the Skia Gold API - we'd either need to request the API, help implement it, or find another way to get this functionality)
• Update the Skia Gold Client logic to fail pre-submit on matching negatives (we could keep the current untriaged behavior)

/cc @Piinks @jonahwilliams @gaaclarke @goderbauer