[`airflow`] Flag `Variable.get()` calls outside of task execution context (`AIR003`) by Dev-iL · Pull Request #23584 · astral-sh/ruff

Dev-iL · 2026-02-26T14:28:01Z

Summary

Implements a new rule AIR003 (airflow-variable-get-outside-task) that flags Variable.get() calls outside of task execution context.

Per the Airflow best practices documentation, calling Variable.get() at module level or inside operator constructor arguments causes a database query every time the DAG file is parsed by the scheduler. This can degrade parsing performance and even cause DAG file timeouts. The recommended alternative is to pass variables via Jinja templates ({{ var.value.my_var }}), which defer the lookup until task execution.

What the rule flags

from airflow.sdk import Variable
from airflow.operators.bash import BashOperator

# Top-level Variable.get() — runs on every DAG parse
foo = Variable.get("foo")

# Variable.get() in operator constructor args — also runs on every DAG parse
BashOperator(
    task_id="bad",
    bash_command="echo $FOO",
    env={"FOO": Variable.get("foo")},
)

# Variable.get() in a regular helper function — not a task execution context
def helper():
    return Variable.get("foo")

What it allows

Variable.get() is fine inside code that only runs during task execution:

@task-decorated functions (including @task(), @task.branch, @task.short_circuit, etc.)
execute() methods on BaseOperator subclasses

Implementation details

Resolves Variable.get via the semantic model, matching both airflow.models.Variable and airflow.sdk.Variable import paths.
Determines "task execution context" by walking the statement hierarchy (current_statements()) looking for a parent FunctionDef that is either:
- decorated with @task / @task.<variant> (resolved via airflow.decorators.task), or
- named execute inside a class inheriting from BaseOperator.
Handles both @task and @task() forms via map_callable, and attribute-style decorators like @task.branch via Expr::Attribute resolution.

Test Plan

Added snapshot tests in AIR005.py covering:

Violations: Variable.get() at module level, inside operator constructor keyword arguments, inside f-string interpolation in operator args, and in a regular helper function. Tested both airflow.models.Variable and airflow.sdk.Variable import paths.
Non-violations: Variable.get() inside @task, @task(), @task.branch decorated functions, inside a BaseOperator.execute() method, and Jinja template usage (no Variable.get() call).

related: apache/airflow#43176

astral-sh-bot · 2026-02-26T14:39:12Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+8 -0 violations, +0 -0 fixes in 1 projects; 55 projects unchanged)

apache/airflow (+8 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --no-fix --output-format concise --preview --select ALL

+ airflow-core/tests/unit/dag_processing/test_processor.py:156:30: AIR003 `Variable.get()` outside of a task
+ airflow-core/tests/unit/dag_processing/test_processor.py:194:30: AIR003 `Variable.get()` outside of a task
+ airflow-core/tests/unit/dag_processing/test_processor.py:228:30: AIR003 `Variable.get()` outside of a task
+ airflow-core/tests/unit/dag_processing/test_processor.py:268:21: AIR003 `Variable.get()` outside of a task
+ providers/openlineage/tests/system/openlineage/example_openlineage_base_complex_dag.py:56:13: AIR003 `Variable.get()` outside of a task
+ providers/openlineage/tests/system/openlineage/example_openlineage_defer_simple_dag.py:39:14: AIR003 `Variable.get()` outside of a task
+ providers/openlineage/tests/system/openlineage/example_openlineage_mapped_simple_dag.py:46:18: AIR003 `Variable.get()` outside of a task
+ providers/openlineage/tests/system/openlineage/example_openlineage_short_circuit_dag.py:49:22: AIR003 `Variable.get()` outside of a task

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
AIR003	8	8	0	0	0

Dev-iL · 2026-02-26T16:02:10Z

~~Hmm... Looking at the ecosystem results, it appears that this rule needs to be applied more surgically (only in DAG definition scripts). @sjyangkevin @Lee-W any idea how to do that?~~
Looks like I figured it out!

jscheffl · 2026-03-02T22:21:27Z

Cool!

Miretpl · 2026-03-03T18:39:02Z

Hi @Dev-iL, shouldn't we include pre/post_execute method subclasses of BaseOperator + possible callback defined for them 🤔?

Also, I'm not sure about the following violation example:

# Variable.get() in a regular (non-task) function
def helper():
    return Variable.get("foo")  # AIR005

as it depends on where the function will be used. If it will be passed as a callback, e.g. to PythonOperator (directly via python_callable instead of task decorator), it is good. On the other hand, if it is invoked somewhere in the code on the top level, it will be a violation.

sjyangkevin

Thanks for implementing it, some small feedback. I also wonder if we should move those utils to identify taskflow api decorated function to helpers, as it is used across multiple rules.

crates/ruff_linter/src/rules/airflow/rules/variable_get_outside_task.rs

Dev-iL · 2026-03-04T18:34:18Z

Also, I'm not sure about the following violation example:
# Variable.get() in a regular (non-task) function
def helper():
    return Variable.get("foo")  # AIR005
as it depends on where the function will be used. If it will be passed as a callback, e.g. to PythonOperator (directly via python_callable instead of task decorator), it is good. On the other hand, if it is invoked somewhere in the code on the top level, it will be a violation.

Right. This is a limitation of static analysis - we cannot easily determine how a function is used. The current implementation errs on the side of caution, allowing some false positive which the user can suppress with # noqa: AIR005. I can add a note to the rule docs that it is expected to have some false positives.

we should move those utils to identify taskflow api decorated function to helpers, as it is used across multiple rules.

Indeed. This rule was implemented before the shared utilities, and it wasn't refactored after rebasing. I'll import from helpers.

can we make the message more generic and we put specific message when reporting diagnostic. For traditional operator we can suggest to use jinja but if it is a function can be taskflow api decorated we can suggest move in

Agreed. Will provide different suggestions depending on whether the violation is at module level (suggest Jinja) vs in a function (suggest @task decorator).

would be better to check on DAG definition line instead of import

Claude says this:

Important architectural constraint: Ruff uses a single-pass AST visitor. At the point where a Variable.get() call is analyzed, only bindings/statements before it in the file have been processed. This means:

Checking for DAG() calls that appear after the Variable.get() line won't work in a single pass

However, imports are always at the top, so checking if DAG/dag bindings exist (from imports) is reliable during traversal

DAGs can be defined in 3 ways:

dag = DAG(...) — assignment

with DAG(...) as dag: — context manager

@dag — decorator on a function

Proposed approach: Replace is_dag_file() with a function that iterates global-scope bindings and checks their source statements (via binding.source → semantic.statement(node_id)) for actual DAG definitions:

Assignment: source stmt is Assign/AnnAssign whose value is a DAG() call (resolve via semantic.resolve_qualified_name)

Context manager: source stmt is With, check if any with_item.context_expr is a DAG() call

Decorator: source stmt is FunctionDef, check if any decorator resolves to airflow.*.dag

Ordering constraint: Ruff is a single-pass linter. If Variable.get() appears at module level before the DAG definition, the DAG binding won't exist yet. This means we miss that violation. However:

In practice, DAG definitions typically appear before task definitions

The import-based check would catch this, so we keep imports as a fallback: if there's an import of DAG/dag from airflow, also treat it as a DAG file

This hybrid approach avoids false negatives while being more precise than import-only

Lee-W

a few nits, but overall looks good!

crates/ruff_linter/resources/test/fixtures/airflow/AIR005_no_dag.py

crates/ruff_linter/src/rules/airflow/rules/variable_get_outside_task.rs

Lee-W

Probably let's change it to AIR003? This is likely to be approved first.

ntBre

Thanks! This looks good to me overall, just a few minor suggestions. I also agree with moving this to AIR003 if it's ready to land first, which looks likely!

crates/ruff_linter/src/rules/airflow/rules/variable_get_outside_task.rs

Lee-W

Looks good to me!

ntBre

Awesome, thank you!

astral-sh-bot bot assigned ntBre Feb 26, 2026

ntBre added rule Implementing or modifying a lint rule preview Related to preview mode features labels Feb 26, 2026

Dev-iL force-pushed the airflow/variable branch 2 times, most recently from e5b31f8 to 517e77a Compare February 26, 2026 22:19

Dev-iL mentioned this pull request Feb 28, 2026

[airflow] Extract common utilities for use in new rules #23630

Merged

Dev-iL force-pushed the airflow/variable branch 2 times, most recently from c6295c6 to 8adc896 Compare March 4, 2026 06:58

sjyangkevin reviewed Mar 4, 2026

View reviewed changes

crates/ruff_linter/src/rules/airflow/rules/variable_get_outside_task.rs Outdated Show resolved Hide resolved

crates/ruff_linter/src/rules/airflow/rules/variable_get_outside_task.rs Outdated Show resolved Hide resolved

Dev-iL force-pushed the airflow/variable branch from 8adc896 to 0bcf153 Compare March 4, 2026 18:34

Lee-W reviewed Mar 5, 2026

View reviewed changes

Dev-iL force-pushed the airflow/variable branch from 0bcf153 to 93872da Compare March 5, 2026 08:06

Lee-W reviewed Mar 5, 2026

View reviewed changes

ntBre reviewed Mar 9, 2026

View reviewed changes

Add AIR003: flag parse-time Variable uses

383e6f9

Dev-iL force-pushed the airflow/variable branch from 93872da to 383e6f9 Compare March 10, 2026 06:51

Dev-iL changed the title ~~[airflow] Implement airflow-variable-get-outside-task (AIR005)~~ [airflow] Flag Variable.get() calls outside of task execution context (AIR003) Mar 10, 2026

Dev-iL requested a review from ntBre March 10, 2026 06:55

Lee-W reviewed Mar 10, 2026

View reviewed changes

ntBre approved these changes Mar 11, 2026

View reviewed changes

ntBre merged commit 91cc7cd into astral-sh:main Mar 11, 2026
43 checks passed

BrewTestBot mentioned this pull request Mar 12, 2026

ruff 0.15.6 Homebrew/homebrew-core#272048

Merged

Conversation

Dev-iL commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What the rule flags

What it allows

Implementation details

Test Plan

Uh oh!

astral-sh-bot bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Uh oh!

Dev-iL commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl commented Mar 2, 2026

Uh oh!

Miretpl commented Mar 3, 2026

Uh oh!

sjyangkevin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Dev-iL commented Mar 4, 2026

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

ntBre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

ntBre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Dev-iL commented Feb 26, 2026 •

edited

Loading

astral-sh-bot bot commented Feb 26, 2026 •

edited

Loading

`ruff-ecosystem` results

Dev-iL commented Feb 26, 2026 •

edited

Loading