Skip to content

Fix backfill completion race during creation#68125

Open
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/64534-backfill-create-race
Open

Fix backfill completion race during creation#68125
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/64534-backfill-create-race

Conversation

@GayathriSrividya

@GayathriSrividya GayathriSrividya commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

closes: #64534

Make backfill creation atomic so the scheduler cannot observe a partially initialized backfill and mark it complete before its DagRuns are created.

This moves the DagRun-info computation ahead of Backfill persistence, replaces the early commit with a flush so the whole create path stays in one transaction, and adds a regression test to verify failed no-run backfill creation does not leave an orphan Backfill row behind.

@GayathriSrividya GayathriSrividya force-pushed the fix/64534-backfill-create-race branch 2 times, most recently from e8be38b to c3d697a Compare June 6, 2026 08:20
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 8, 2026
@potiuk

potiuk commented Jun 25, 2026

Copy link
Copy Markdown
Member

@GayathriSrividya — I've removed the ready for maintainer review label because the next step here is yours: this branch now has merge conflicts with main. It'll go back into the maintainer queue automatically once you rebase onto the latest main and push — no need to re-add the label by hand. There's no rush.

Automated triage note drafted by an AI-assisted tool — may get things wrong; a real Apache Airflow maintainer takes the next look once it's green. (why automated)


Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

@potiuk potiuk removed the ready for maintainer review Set after triaging when all criteria pass. label Jun 25, 2026
@GayathriSrividya GayathriSrividya force-pushed the fix/64534-backfill-create-race branch from c3d697a to 1aa19fc Compare June 27, 2026 09:43
@GayathriSrividya

Copy link
Copy Markdown
Contributor Author

@GayathriSrividya — I've removed the ready for maintainer review label because the next step here is yours: this branch now has merge conflicts with main. It'll go back into the maintainer queue automatically once you rebase onto the latest main and push — no need to re-add the label by hand. There's no rush.

Automated triage note drafted by an AI-assisted tool — may get things wrong; a real Apache Airflow maintainer takes the next look once it's green. (why automated)

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

@potiuk Resolved conflicts and pushed changes

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 2, 2026

@Vamsi-klu Vamsi-klu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix — moving _get_info_list ahead of the Backfill insert and swapping the early commit() for flush() in _create_backfill collapses creation into a single transaction, so _mark_backfills_complete() can no longer catch a committed Backfill with zero DagRuns. That retires the 2-minute heuristic from #62561 instead of just tuning it, and the new test genuinely fails on the old path.

One thing before merge, and it's the exact concern #64534 raised: the AlreadyRunningBackfill guard used to lean on that early commit to block concurrent duplicates. With flush(), the new row stays invisible to other sessions until the whole block commits, so the num_active count in _create_backfill — a plain SELECT with no with_row_locks/FOR UPDATE, and no unique constraint on Backfill.dag_id — now has a wider window where two concurrent creates for the same dag both pass it. Could you add a lock around that check (or a partial unique index on dag_id WHERE completed_at IS NULL), plus a test asserting only one of two same-dag creates wins?

Also a heads-up: it's showing CONFLICTING with main, so it'll need one more rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make backfill creation robust against scheduler race condition

3 participants