Skip to content

Migration 0106 upgrade takes ~6 minutes on large dag_run tables due to full-table UPDATE #63550

Description

@vatsrahul1001

Apache Airflow version

Main

If "Other Airflow 3 version" selected, which one?

No response

What happened?

Migration 0106_3_2_0_add_partition_key_to_backfill_dag_run upgrade is slow on deployments with large dag_run tables. With 10M dag_run rows, the migration took ~6 minutes.

What you think should happen instead?

No response

How to reproduce

  1. Start Airflow on 3.1.8 with PostgreSQL backend
  2. Accumulate dag_run rows (100K+ to see noticeable impact, 1M+ for significant delays)
  3. Upgrade to 3.2.0 (airflow db migrate)
  4. Observe migration 0106 duration scales linearly with dag_run row count

Operating System

MAc

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions