A bug gets reported, a developer pushes a fix, and your CI turns green. Then a customer hits the same issue again—because the fix never actually addressed the failing behavior, the reproduction steps changed, or the test asserted the wrong thing. I’ve seen this happen in teams with strong engineers and modern pipelines, and the root cause is almost always the same: we treated the act of “fixing” as proof, instead of running a focused check that the specific defect is gone.
Confirmation testing is that focused check. It’s the moment you take the exact bug you saw earlier and prove—using the same intent, same inputs, and same observable outcome—that it no longer exists. Done well, it reduces reopens, keeps regression suites from becoming a crutch, and gives you a clean handoff from defect-fixing back to broader risk coverage.
I’ll walk you through how I design confirmation tests, how I keep them tight and reliable, where they sit in a modern defect workflow, and how I recommend blending manual retesting with automation and 2026-era AI-assisted tooling without turning your pipeline into a flaky mess.
Confirmation testing: the smallest test that answers the only question that matters
Confirmation testing (often called retesting) is a change-driven testing technique with a narrow purpose: confirm that a previously reported defect is fixed. That’s it. It’s not a general “quality check,” and it’s not a substitute for regression testing.
I like to phrase the objective as a single sentence:
- Given the same preconditions and the same trigger, the previously observed wrong behavior does not occur anymore.
That objective forces you to be concrete. A bug report usually contains some mix of symptoms, steps, expectations, and environment details. Confirmation testing should be built from the parts that were essential to reproduce the defect in the first place.
A simple analogy I use with teams: regression testing is checking the whole building after a repair; confirmation testing is turning the exact doorknob that used to fall off.
What confirmation testing is not
Here are the traps I see repeatedly:
- “The code changed, so it must be fixed.” That’s a code review conclusion, not a test outcome.
- “The unit tests passed.” Great, but unless a test encoded the exact failing behavior, you still haven’t confirmed the bug.
- “We ran the full regression suite.” That can still miss the defect, and it’s slower than necessary for the immediate question.
- “We can’t reproduce it now, so it’s fixed.” Not reproducible might mean the environment changed, the data changed, or the reproduction steps were incomplete.
I’ll add a fifth one that’s subtle:
- “The error stopped showing up in logs.” That might mean you silenced the symptom, reduced observability, or shifted the failure into a different layer. Confirmation needs an explicit, user-relevant observable outcome.
Why it’s planned work (even when it’s fast)
Confirmation testing is planned because it depends on coordination:
- The dev team provides a fix (and ideally notes what changed and how to validate).
- The test team (or the developer, in a shift-left workflow) reruns the defect scenario.
- The result updates defect status: fixed, not fixed, or needs more info.
In practice, the actual retest might take 30 seconds—but the planning ensures you’re retesting the right thing.
To make that planning frictionless, I treat confirmation testing like a “micro-contract” between the bug report and the fix:
- The bug report describes the failing contract: inputs + context + observed wrong outcome.
- The fix describes the intended contract: what should happen instead.
- The confirmation test proves the contract is restored.
Where confirmation testing fits in a defect workflow (and how I keep it from turning into churn)
Most teams already have a defect lifecycle. The mistake is treating confirmation testing as an informal “yeah I checked it” step instead of a first-class gate.
Here’s the workflow I recommend because it’s both strict and lightweight:
- Bug reported with a stable reproduction
– Minimum: environment, steps, observed behavior, expected behavior.
– Best: a failing test, a request/response trace, or a recorded UI run.
- Triage decision
– Fixed now, deferred, won’t fix, can’t reproduce, duplicate.
- Fix implemented with a validation note
– I ask developers to include one sentence in the PR: “To confirm: do X, expect Y.”
- Confirmation test executed
– Prefer the same inputs and the same observable output.
– Record evidence: log snippet, screenshot, test run link, or CI artifact.
- Regression testing executed
– Scope based on risk: touched modules, nearby features, integration boundaries.
What “Verified” means on my teams
I’m opinionated here: “Verified” is not “I tried something similar and it seemed okay.” I use “Verified” to mean:
- The original defect scenario (or its updated equivalent, if the workflow changed) was executed.
- The original failure signature was checked explicitly.
- Evidence exists that a future teammate can interpret without re-running the test.
If you’re using a bug tracker with statuses, the semantics matter more than the labels. I’d rather have fewer statuses with strict meaning than a dozen statuses everyone uses differently.
“Ignore,” “hold,” and “rejected” aren’t the end of the story
If a bug is rejected or marked “can’t reproduce,” that’s often a signal that your reproduction isn’t precise enough. I treat confirmation testing here as a reproduction exercise:
- Re-run the scenario on the exact build where it was reported.
- Capture concrete evidence (network calls, DB state, timestamps, feature flag values).
- Convert “sometimes” into “when condition Z is true.”
If you can turn an intermittent report into a deterministic reproduction, you’ve already raised quality—even before a fix exists.
One tactic that works surprisingly well: define an “acceptance reproduction.” That’s a narrower reproduction that doesn’t match every user story detail, but reliably triggers the same failure signature. It becomes the seed for your confirmation test.
I always run confirmation testing before regression testing
Order matters. If the original defect is still present, running regression is wasted time and creates noise. Confirm the fix first; then verify nothing else broke.
The only time I intentionally invert the order is when the fix itself is risky to validate directly (for example, a security mitigation that needs a safe environment), and I want a quick sanity sweep to ensure I’m not about to break staging. Even then, I still run confirmation before I consider the defect closed.
How I keep confirmation from turning into churn
Churn happens when the “fix → not fixed → fix → not fixed” loop drags on. The usual causes are ambiguous expected behavior, hidden environment assumptions, or missing observability.
My anti-churn rules:
- If a confirmation test fails, I attach the failure signature to the ticket immediately (screenshot, stack trace, response payload, and the exact build ID).
- If a defect requires a specific flag/data condition, I store it as part of the reproduction, not tribal knowledge.
- If the “expected behavior” is not written down, I stop and force that decision—product, design, and engineering alignment is part of quality.
Designing confirmation tests: making them reproducible, minimal, and hard to misread
A good confirmation test is boring. It should be short, deterministic, and obvious to interpret.
Start from the failure signature, not from the patch
When people retest based on the code change, they drift into “testing the fix” rather than “testing the bug.” I anchor the test around the failure signature:
- What exactly was wrong? A 500 response? A missing row? A mispriced invoice? A double charge?
- What signal proves it’s gone? Status code, rendered text, emitted event, DB constraint, audit log.
A useful way to think about this: the patch is an explanation; the failure signature is evidence. Confirmation testing is evidence-driven.
Keep the scope narrow on purpose
If your confirmation test touches five screens, a payment provider sandbox, and an email inbox, you’re no longer confirming—you’re redoing an end-to-end scenario. That might be valuable, but it’s not the fastest way to answer the immediate question.
I prefer this structure:
- Setup: minimal data needed (and no more)
- Trigger: the exact action that previously caused failure
- Assert: the exact symptom is gone (plus one sanity check)
When I say “plus one sanity check,” I mean one. Confirmation tests fail in two annoying ways:
- False positives: they pass while the bug is still present (asserted the wrong thing).
- False negatives: they fail due to unrelated instability (environment, test data, timing).
The sanity check reduces the chance of false positives without broadening scope too much.
Use the same test cases (and update them only when reality changes)
Retesting typically reuses the original test case. If the user flow changes, don’t rewrite history—update the test case and explicitly note the change:
- Old reproduction no longer applies because UI changed on 2026-02-08.
- New reproduction steps still produce the same defect signature.
Concrete dates like this prevent “we tested it” claims from floating free of context.
Test the externally observable behavior, not an implementation detail
A confirmation test is at its best when it checks what the user (or integrator) actually experiences:
- UI: the correct message appears; the button is disabled/enabled appropriately; the state persists after refresh.
- API: the response status and schema are correct; idempotency works; errors are consistent.
- Data: the correct row exists; the correct aggregate is computed; constraints are respected.
Implementation details (like “function X was called”) can be helpful inside unit tests, but I don’t rely on them for final confirmation unless the defect itself was internal.
Example: confirmation test as a small, runnable unit test (Python + pytest)
Imagine a bug: checkout sometimes created duplicate orders when the client retried a request after a timeout. The fix adds an idempotency key.
# file: testcheckoutconfirmation.py
# Run: python -m pip install pytest
# Then: pytest -q
import uuid
class OrderStore:
def init(self):
self.ordersbyidempotency_key = {}
self.orders = []
def createorder(self, customerid: str, amountcents: int, idempotencykey: str):
if idempotencykey in self.ordersbyidempotencykey:
return self.ordersbyidempotencykey[idempotencykey]
order_id = str(uuid.uuid4())
order = {"orderid": orderid, "customerid": customerid, "amountcents": amountcents}
self.orders.append(order)
self.ordersbyidempotencykey[idempotencykey] = order
return order
def testconfirmationnoduplicateorderonretry():
"""Confirmation test for a previously reported duplicate-order defect."""
store = OrderStore()
idempotency_key = "checkout-2026-02-08T10:15:00Z-customer-742"
first = store.createorder(customerid="742", amountcents=2599, idempotencykey=idempotency_key)
second = store.createorder(customerid="742", amountcents=2599, idempotencykey=idempotency_key)
assert first["orderid"] == second["orderid"], "Retry should return the same order"
assert len(store.orders) == 1, "Store should contain only one order"
Why this works as confirmation testing:
- It targets the exact symptom (duplicate orders on retry).
- It’s deterministic and fast.
- The assertions are hard to misinterpret.
In a real system, this might be an API-level test instead, but the shape stays the same.
Edge cases I include when the defect involves retries, concurrency, or time
If the original bug involved “sometimes,” it often hides in one of these edges:
- Two requests in flight at the same time (true concurrency, not sequential retries).
- A partial failure (first request created the row but timed out before responding).
- Clock sensitivity (midnight boundaries, DST transitions, token expiration).
- Exactly-once vs at-least-once delivery semantics in queues.
For confirmation, I still keep it minimal, but I’ll intentionally recreate the edge that made it fail. Example mental checklist for idempotency:
- Same idempotency key, same payload → same result.
- Same idempotency key, different payload → either rejected or consistent with your spec (this is where many systems get it wrong).
- Two concurrent calls with same key → one order.
If you don’t have a written product decision for those, you don’t have a testable requirement. That’s not a testing failure; it’s a specification gap.
Confirmation testing vs regression testing: I treat them as different tools for different risks
These two are often confused because both happen after a change. I keep the difference crisp:
- Confirmation testing answers: “Is the specific defect fixed?”
- Regression testing answers: “Did the fix break anything else?”
Here’s the comparison I use with teams:
Confirmation testing
—
Verify the previously failing behavior no longer fails
Narrow: one defect scenario
Immediately after a fix is delivered
Bug status can move to Verified
Often manual or a single targeted automated test
Traditional vs modern execution (what I actually recommend in 2026)
People sometimes claim “confirmation is manual, regression is automated.” That’s a pattern, not a rule. In 2026, I push for a hybrid that respects speed and evidence.
Traditional
—
Tester note: “retested OK”
Rerun a big suite “just to be safe”
Test team only
Rerun until green
Manual steps in a document
If you do one thing differently: make confirmation testing produce a stable artifact. A screenshot, a HAR file, a log excerpt, or a CI run URL makes “verified” meaningful.
How I choose the regression scope after confirmation passes
This is where good teams differentiate themselves. I don’t default to “run everything.” I default to “run what could plausibly be impacted.”
My quick risk questions:
- What boundaries did the fix cross? (DB schema, caching, external APIs, auth, serialization)
- What invariants could it have affected? (money totals, authorization rules, uniqueness)
- What failure mode could it introduce? (latency, deadlocks, memory pressure, retries)
Then I pick a regression slice:
- A small smoke suite (always)
- A module suite for the touched area (often)
- A couple of integration tests around the boundary (if the fix touched one)
- One end-to-end happy path (when user flow risk is high)
Confirmation gives you the right to move on; regression gives you the right to ship.
Automation patterns I use for confirmation testing (without slowing teams down)
I’m selective about what I automate. The goal is to confirm the fix quickly and reliably, not to build a second regression suite.
Pattern 1: link a bug to a single targeted test in CI
When a defect is fixed, I like adding one targeted automated test that would have failed before the fix. Then I wire CI to run:
- The targeted test (confirmation gate)
- A small risk-based regression set
This keeps feedback fast. Typically, you can keep confirmation gates in the 30–120s range even on medium services.
How I keep this clean in practice:
- Name the test after the failure signature, not after the implementation.
- Reference the bug ID in a comment (or in test metadata) so it’s searchable.
- Ensure it fails on a pre-fix commit if feasible. That’s the strongest proof you wrote the right test.
Pattern 2: contract-level confirmation for API bugs
For API defects, I prefer tests that operate at the HTTP boundary because they reflect real usage. Here’s a runnable Node.js example using a minimal server and node:test.
# file: confirmationapiidempotency.test.js
# Run: node confirmationapiidempotency.test.js
import test from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
function createServer() {
const ordersByKey = new Map();
return http.createServer((req, res) => {
if (req.method !== "POST" || req.url !== "/orders") {
res.statusCode = 404;
res.end();
return;
}
const idempotencyKey = req.headers["idempotency-key"];
if (!idempotencyKey) {
res.statusCode = 400;
res.setHeader("content-type", "application/json");
res.end(JSON.stringify({ error: "missing idempotency-key" }));
return;
}
if (ordersByKey.has(idempotencyKey)) {
res.statusCode = 200;
res.setHeader("content-type", "application/json");
res.end(JSON.stringify(ordersByKey.get(idempotencyKey)));
return;
}
const order = { orderId: ord_${ordersByKey.size + 1}, status: "created" };
ordersByKey.set(idempotencyKey, order);
res.statusCode = 201;
res.setHeader("content-type", "application/json");
res.end(JSON.stringify(order));
});
}
function request(port, headers = {}) {
return new Promise((resolve, reject) => {
const req = http.request(
{ method: "POST", hostname: "127.0.0.1", port, path: "/orders", headers },
(res) => {
let data = "";
res.on("data", (chunk) => (data += chunk));
res.on("end", () => resolve({ status: res.statusCode, body: JSON.parse(data || "{}") }));
}
);
req.on("error", reject);
req.end();
});
}
test("confirmation: retry does not create a second order", async () => {
const server = createServer();
await new Promise((r) => server.listen(0, r));
const port = server.address().port;
try {
const key = "checkout-2026-02-08T10:15:00Z-customer-742";
const first = await request(port, { "idempotency-key": key });
const second = await request(port, { "idempotency-key": key });
assert.equal(first.status, 201);
assert.equal(second.status, 200);
assert.equal(first.body.orderId, second.body.orderId);
} finally {
server.close();
}
});
This test is small, local, and deterministic. You can then add a separate suite for broader behavior.
A practical improvement I often add in real services: assert response schema and headers too. Bugs often reappear as “fixed behavior, broken contract,” where the happy path works but clients still fail due to an unexpected field or status.
Pattern 3: AI-assisted reproduction, with human verification
In 2026, many teams use AI assistants to summarize bug reports, generate reproduction scripts, or propose assertions. That’s useful if you treat it as drafting, not as proof.
My rule:
- I’ll accept an AI-generated test as a starting point.
- I won’t accept “AI says it’s fixed” as confirmation.
I always validate that the generated test encodes the original failure signature and that it fails on the last known bad build (or a commit before the fix) when possible.
Where AI genuinely helps confirmation work:
- Turning messy narrative bug reports into crisp “setup/trigger/assert” steps.
- Extracting environment assumptions (flags, roles, account types) from logs.
- Generating a first-pass test that I then harden against flakiness.
Where it hurts:
- It guesses expected behavior when the spec is unclear.
- It writes assertions that look plausible but don’t match the failure signature.
- It produces tests that pass trivially because they never reach the failing path.
Pattern 4: “Repro harness” scripts that run outside the test suite
Not every confirmation belongs in your main test runner. For gnarly bugs (race conditions, memory leaks, long-tail timeouts), I often create a small, dedicated repro harness:
- A script that runs the scenario 100–10,000 times.
- Instrumentation enabled.
- Output that clearly indicates success vs failure.
I treat these as disposable tools unless they become valuable long-term. If the bug class repeats, I promote the harness into a permanent stress test job.
Performance considerations: confirmation tests should be cheap by default
Confirmation testing is about fast truth. If your confirmation step takes 20 minutes, people will skip it or parallelize it sloppily.
Targets I aim for (ranges, not absolutes):
- Unit-level confirmation: under a second.
- API-level confirmation in CI: a few seconds to a couple of minutes.
- UI confirmation: a couple of minutes, ideally less.
If a bug can only be confirmed via a long end-to-end scenario, that’s a signal to invest in testability: add hooks, add observability, add stable test data, or expose a safe API to set up state.
Common mistakes (and the specific habits that prevent them)
Mistake: confirmation test asserts a side effect, not the bug
Example: the bug was “invoice total rounds incorrectly,” but the test checks “invoice record exists.” That can pass even when totals are wrong.
Habit I recommend:
- Write one assertion that directly reflects the defect (total equals expected cents).
- Add one sanity assertion (invoice exists).
If you ever feel tempted to only assert a side effect, ask yourself: “If the bug came back tomorrow, would this test fail?” If the answer is “maybe,” it’s not a confirmation test.
Mistake: retesting on the wrong environment
If a defect was found in production-like config (feature flags, caches, timeouts), confirming on a developer machine can lie.
Habit I recommend:
- Confirm on the closest environment that can reproduce safely.
- Record critical config values: flag states, build id, database snapshot id.
When I’m forced to confirm in a different environment, I write that down explicitly: “Confirmed in staging with config A; production uses config B; remaining risk: X.” That’s honest and actionable.
Mistake: flakiness gets treated as “fixed enough”
A flaky confirmation test is worse than no test because it makes bug status a coin flip.
Habit I recommend:
- If confirmation is flaky, treat the bug as not confirmed until the test is stabilized.
- Add instrumentation before adding retries. Retries can hide the failure signature.
When flakiness is unavoidable (for example, an external dependency sandbox), I isolate it:
- Run confirmation against a local fake or contract test.
- Use the external sandbox only for a separate “integration confidence” job.
Mistake: “works for me” confirmation that never hit the failing path
This one is extremely common in UI and role-based systems. People retest with the wrong account, wrong tenant, wrong permissions, or wrong data state.
Habit I recommend:
- Encode preconditions explicitly: user role, account type, feature flags, and data shape.
- Build a one-click setup for test accounts (seed scripts, fixtures, or admin endpoints).
If you can’t set up the failing state reliably, your confirmation test will always be shaky.
Mistake: confirming the happy path when the defect was an error path
Many bugs live in error handling: validation messages, rollback behavior, partial failures.
Habit I recommend:
- Confirm with the same invalid input or failure trigger.
- Assert not only that the error is gone, but that the replacement behavior is correct (proper message, no partial writes, consistent state).
Mistake: closing the bug without capturing evidence
Without evidence, you’re relying on memory—and memory is not a test artifact.
Habit I recommend:
- Attach one durable artifact: CI run link, screenshot, log excerpt, trace ID, or a saved test result.
It doesn’t have to be fancy. It has to be unambiguous.
Manual confirmation testing: how I keep it rigorous without making it slow
Manual retesting gets a bad reputation because it’s easy to do casually. But manual confirmation is still essential for many classes of defects:
- Visual/UI bugs (alignment, truncation, animations, accessibility)
- Cross-browser/mobile issues
- Complex multi-system workflows where automation cost is high
- One-off production incidents where the goal is fast stabilization
My manual confirmation playbook:
- Restate the failure signature in one line
– Example: “Submitting profile form returns 500 and the user sees a blank page.”
- List explicit preconditions
– Account role, feature flag states, locale, device/browser, build ID.
- Run the exact trigger steps
– I don’t “improve” the flow. I mirror what failed.
- Capture before/after evidence
– Before fix (if available): screenshot/log/trace.
– After fix: screenshot/log/trace.
- Write the verification note like a mini-report
– “On build X in environment Y, with flags A/B, did steps 1–4; expected Z; observed Z.”
This takes slightly longer than “retested OK,” but it saves hours later.
Hard cases: intermittent, non-reproducible, and production-only bugs
These are the bugs that make teams cynical. They’re also where confirmation testing, done properly, is most valuable.
Intermittent bugs: confirm the fix by controlling randomness
Intermittent failures often correlate with:
- Concurrency
- Timing and retries
- Cache state
- Resource contention
- External dependency variability
I approach confirmation in layers:
- First, reproduce deterministically by controlling variables (seed randoms, freeze time, isolate dependencies).
- Then, confirm with repetition: run the scenario many times.
A confirmation note for an intermittent issue should include repetition details:
- How many runs
- Over what duration
- Under what load/concurrency
- What would count as failure
Production-only bugs: use “production parity” and safe validation
When a bug only occurs in production, it’s often because production is meaningfully different:
- Data volume and shape
- Real traffic patterns
- Feature flag exposure
- Permissions/tenancy
- Infrastructure (CDN, WAF, caches)
I still try to confirm outside production first. If I must validate in production, I do it safely:
- Use internal accounts and non-customer-impacting flows.
- Use read-only validation when possible.
- Add temporary guardrails (rate limits, additional logging) during validation.
Confirmation testing in production becomes a coordination exercise with operations: it’s not just “click around,” it’s “validate safely with traceability.”
Observability as confirmation: when “the test” is a trace
For distributed systems, sometimes the best confirmation artifact is a trace ID:
- The original trace shows a failure path.
- The post-fix trace shows the corrected path.
I like to capture:
- Request IDs
- Key logs (not everything)
- The exact status/response
- Any state transitions (events emitted, DB writes)
This turns “verified” into a defensible claim.
Confirmation testing across levels: unit, integration, end-to-end
Confirmation testing is a technique, not a test type. You can confirm at different levels depending on the defect.
Unit-level confirmation
Best when:
- The bug is a pure logic error.
- Inputs/outputs can be expressed without external dependencies.
Risks:
- You might miss integration behaviors.
My habit: if I confirm at unit level, I still do at least a light boundary check (API or UI) when the defect was user-visible.
Integration-level confirmation
Best when:
- The bug involves serialization, DB queries, caching, message queues, or auth.
This is often the sweet spot: realistic enough to catch real failures, cheap enough to run fast.
End-to-end confirmation
Best when:
- The bug is in workflow wiring: front-end/back-end mismatch, feature flags, session handling.
Risks:
- Slow and flaky if your environment isn’t controlled.
I try to keep E2E confirmation to “one path, one assertion, one artifact,” and I avoid turning it into a mini-regression.
Practical scenarios: how I design confirmation tests in the real world
Scenario 1: UI regression that only happens on small screens
Defect: “On mobile width, the ‘Confirm’ button is covered by the sticky footer and can’t be tapped.”
Confirmation test (manual or automated):
- Setup: device emulator at a specific viewport, same locale, same account.
- Trigger: navigate to the screen and scroll to the bottom.
- Assert: the button is visible and clickable; optionally assert focus order and that it’s not overlapped.
Evidence: a screenshot or short recording with viewport size shown.
Regression slice after confirmation:
- Nearby screens that share the sticky footer component.
- A couple of common devices/viewport presets.
Scenario 2: Authorization bug (high severity)
Defect: “Users without role X can access endpoint Y by guessing an ID.”
Confirmation test (must be explicit):
- Setup: two accounts: one authorized, one not.
- Trigger: call the endpoint as unauthorized with a valid ID.
- Assert: response is forbidden/unauthorized (whatever your contract states), and no data is leaked.
Important: for auth bugs, I don’t consider confirmation complete without also checking for side-channel leaks (different error messages, timing, or partial data). That doesn’t mean a huge test; it means one careful assertion: “No sensitive fields are present in any response body.”
Scenario 3: Data integrity bug after a migration
Defect: “After migration, some records get duplicated due to missing unique constraint.”
Confirmation test:
- Setup: seed records that would collide.
- Trigger: run the migration or the write path.
- Assert: duplicates are prevented; the system behavior is correct (reject, merge, or update, depending on spec).
Regression slice:
- Other write paths that touch the same table.
- Backfill jobs that may bypass app-level validation.
Scenario 4: Performance bug that looks like a functional bug
Defect: “Search sometimes returns empty results.” Root cause: query times out under load and the API returns an empty list instead of an error.
Confirmation test:
- Setup: representative dataset size; enable the same timeout thresholds.
- Trigger: run the search under a controlled load or with a forced slow path.
- Assert: the API returns a proper error (or succeeds within budget), but does not silently return empty results.
This is a good example where confirmation includes both functional behavior and non-functional constraints.
Alternative approaches: different ways to confirm the same fix
Sometimes there isn’t one “right” confirmation test. I pick based on speed, reliability, and closeness to user impact.
Option A: Confirm through the UI
Pros:
- Most realistic user signal.
Cons:
- Often slower and more brittle.
Option B: Confirm through the API boundary
Pros:
- Stable, fast, close to user integration.
Cons:
- Might miss rendering/UX issues.
Option C: Confirm through data and logs
Pros:
- Great for back-end and job systems.
Cons:
- Risk of confirming an internal state while the user experience is still broken.
When the defect is user-visible, I usually want at least one user-visible confirmation, even if the main confirmation is automated at the API level.
A tight checklist I use for confirmation tests
When I’m reviewing a confirmation test (manual or automated), I ask:
- Does it recreate the original preconditions (or explicitly document changes)?
- Does it execute the same trigger that failed?
- Does it assert the original failure signature directly?
- Is the outcome unambiguous (pass/fail is obvious)?
- Is it deterministic enough to trust (or does it need stabilization)?
- Is there a durable artifact attached to the verification?
If the answer is “no” to any of those, I treat the confirmation as incomplete.
Blending confirmation testing with AI-assisted workflows safely
AI is fantastic at generating plausible tests. Confirmation testing demands correct tests.
My safe workflow:
- Use AI to draft a test from the bug report.
- Manually align the test with the failure signature.
- Prove it fails on a pre-fix version (if possible).
- Run it on the fixed version and capture artifacts.
- Only then promote it into CI or documentation.
I also keep a strict boundary: AI can help with creation and summarization, but the final confirmation decision requires a human-approved artifact.
Closing the loop: what I measure to know confirmation testing is working
You can tell confirmation testing is healthy when:
- Reopen rate goes down (especially “not fixed” reopens).
- Time from fix merged → verified stays short.
- Confirmation artifacts exist for high-severity defects.
- Flaky tests do not control defect status.
If you want one metric that’s both simple and revealing, track: “Percent of fixed defects that have a linked confirmation artifact.” The goal isn’t 100% for every minor UI nit; the goal is discipline where it matters.
Final thought
Confirmation testing is small by design, but it’s the hinge between “we changed code” and “we solved the problem.” When I treat it as a first-class step—with a crisp failure signature, minimal scope, and durable evidence—I ship with less fear, reopen fewer tickets, and spend far less time arguing about whether something is actually fixed.
