Skip to content

[airflow] Flag runtime-varying values in DAG/task constructor arguments (AIR304)#23631

Merged
ntBre merged 2 commits intoastral-sh:mainfrom
Dev-iL:2602/airflow/runtime
Mar 9, 2026
Merged

[airflow] Flag runtime-varying values in DAG/task constructor arguments (AIR304)#23631
ntBre merged 2 commits intoastral-sh:mainfrom
Dev-iL:2602/airflow/runtime

Conversation

@Dev-iL
Copy link
Copy Markdown
Contributor

@Dev-iL Dev-iL commented Feb 28, 2026

Summary

Add rule AIR304 that flags runtime-varying function calls (e.g., datetime.now(), pendulum.now(), random.randint(), uuid.uuid4()) used as arguments in Airflow DAG/task constructors. These calls cause the serialized DAG hash to change on every parse, creating infinite DAG versions in the dag_version and serialized_dag tables, leading to unbounded database growth and eventual OOM conditions.

Reference: apache/airflow#59430

The rule checks:

  • DAG(...) constructors and @dag(...) decorator calls
  • Operator and sensor constructors (via is_airflow_builtin_or_provider)
  • @task(...) decorator calls

It recursively inspects keyword argument values through binary/unary ops, dicts, lists, sets, tuples, and f-strings to find calls to known runtime-varying functions from datetime, pendulum, time, uuid, and random.

Test Plan

cargo test -p ruff_linter -- airflow::tests — all 43 tests pass, including the new AIR304 test case with 17 violation and 6 non-violation scenarios covering direct calls, binary ops, default_args dicts, f-strings, operators, sensors, decorators, and non-airflow calls.

Related: apache/airflow#43176
CC: @Lee-W @sjyangkevin @wjddn279

@astral-sh-bot
Copy link
Copy Markdown

astral-sh-bot bot commented Feb 28, 2026 •
edited
Loading

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+97 -0 violations, +0 -0 fixes in 1 projects; 55 projects unchanged)

apache/airflow (+97 -0 violations, +0 -0 fixes) 

ruff check --no-cache --exit-zero --no-fix --output-format concise --preview --select ALL

+ airflow-core/src/airflow/example_dags/example_params_ui_tutorial.py:89:16: AIR304 `date.today()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3116:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3161:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3172:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3197:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3215:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3230:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3258:64: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3326:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3335:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3347:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3370:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3394:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3420:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3439:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3450:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3481:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3510:63: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/models/test_dag.py:3524:60: AIR304 `pendulum.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/serialization/test_dag_serialization.py:1859:56: AIR304 `datetime.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/serialization/test_serialized_objects.py:254:52: AIR304 `datetime.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ airflow-core/tests/unit/serialization/test_serialized_objects.py:366:55: AIR304 `datetime.now()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1069:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1093:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1217:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1247:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1277:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1303:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1325:82: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1329:47: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1358:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1403:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:1469:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:183:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:200:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:219:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:236:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:251:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:268:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:289:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:311:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:332:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:352:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:374:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:393:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:415:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:494:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:517:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:550:29: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
+ kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:627:25: AIR304 `uuid.uuid4()` produces a value that changes at runtime; using it in a Dag or task argument causes infinite Dag version creation
... 47 additional changes omitted for project

Changes by rule (1 rules affected)

code total + violation - violation + fix - fix
AIR304 97 97 0 0 0

@ntBre ntBre added rule Implementing or modifying a lint rule preview Related to preview mode features labels Mar 2, 2026
@Dev-iL Dev-iL force-pushed the 2602/airflow/runtime branch from cfc0bce to 0a4d03a Compare March 3, 2026 20:17
Lee-W
Lee-W reviewed Mar 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits, but looks good in general!

@Dev-iL Dev-iL force-pushed the 2602/airflow/runtime branch 2 times, most recently from 18756ae to 2e8e72c Compare March 5, 2026 08:17
Lee-W
Lee-W reviewed Mar 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, looks good to me. @ntBre could you please take a look when you have a moment? thanks a lots!

ntBre
ntBre approved these changes Mar 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ntBre ntBre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This looks good to me. I just had one suggestion to compare to an existing helper method to see if there are any other cases you want to handle in find_runtime_varying_call.

Dev-iL and others added 2 commits March 8, 2026 17:17
@Dev-iL @claude
Add AIR304: flag runtime-varying values in DAG/task constructor args
08cef31 
Using runtime-varying values (like `datetime.now()`) as arguments to
Airflow DAG or task constructors causes the serialized DAG hash to change
on every parse, creating infinite DAG versions in the database. This rule
detects such calls in DAG constructors, @dag decorators, operator/sensor
constructors, and @task decorators, recursively checking through binary
ops, dicts, lists, sets, tuples, and f-strings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Dev-iL
Standardize DAG/dag/Dag capitalization in AIR301 & AIR302 docs
c48870d
@Dev-iL Dev-iL force-pushed the 2602/airflow/runtime branch from 2e8e72c to c48870d Compare March 8, 2026 15:17
@ntBre ntBre changed the title [airflow] Flag runtime-varying values in DAG/task constructor arguments (AIR304) [airflow] Flag runtime-varying values in DAG/task constructor arguments (AIR304) Mar 9, 2026
@ntBre
Copy link
Copy Markdown
Contributor

ntBre commented Mar 9, 2026

I guess one more quick question before landing: is this behavior specific to Airflow 3? I think so far we've used the AIR3 prefix for things that changed in Airflow 3, but it seems like this could affect other versions (to my naive understanding).

@Dev-iL
Copy link
Copy Markdown
Contributor Author

Dev-iL commented Mar 9, 2026 •
edited
Loading

I guess one more quick question before landing: is this behavior specific to Airflow 3? I think so far we've used the AIR3 prefix for things that changed in Airflow 3, but it seems like this could affect other versions (to my naive understanding).

Good question! Yes, this only affects Airflow3. This is because Dag versioning was introduced in Airflow 3, so these patterns weren't problematic before.

@ntBre
Copy link
Copy Markdown
Contributor

ntBre commented Mar 9, 2026

Sounds good, thank you! I'll land this then!

@ntBre ntBre merged commit b82853f into astral-sh:main Mar 9, 2026
43 checks passed
@Dev-iL Dev-iL deleted the 2602/airflow/runtime branch March 9, 2026 18:01
@BrewTestBot BrewTestBot mentioned this pull request Mar 12, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ntBre ntBre ntBre approved these changes

+1 more reviewer

@Lee-W Lee-W Lee-W left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@ntBre ntBre

Labels

preview Related to preview mode features rule Implementing or modifying a lint rule

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Dev-iL @ntBre @Lee-W