Apache Airflow version
3.0.6
If "Other Airflow 2 version" selected, which one?
2.10.3 and 2.11.0
What happened?
We’re dynamically generating datasets using dataset alias. With a few or even a few hundred datasets, no problem.
But when a task generates thousands, the runtime becomes huge. The thing is: the task itself runs fast — it’s actually spending most of the time outside the task, processing and writing outlets/metadata.
Two main issues here:
- Performance – A task that should take seconds ends up running for minutes because of outlets handling.
- UI confusion – The Airflow UI doesn’t reflect what’s really happening:
- Task durations show as 0s, 0.16s, 50s, 0s.
- But the full DAG takes ~2.47 minutes
- While the task is running, the UI shows 1+ minutes elapsed… then when it finishes, it’s magically reported as ~51s. Users get confused by this mismatch.
There are cases where the discrepancy is more than 4 minutes for a single task.
During task execution:
After DAG finished:
What you think should happen instead?
Optimization how outlets/datasetalias writes are handled at scale, especially for big dataset packs.
Show this “post-task processing time” somewhere in the UI so users understand where the time is going.
How to reproduce
DAG outlet producer:
import os
from datetime import datetime
from airflow.sdk import DAG, Asset, AssetAlias, get_current_context, Param
from airflow.decorators import task
from airflow.providers.standard.operators.empty import EmptyOperator
DAG_ID = os.path.splitext(os.path.basename(__file__))[0]
asset_alias = AssetAlias("performance-test-1")
with DAG(
dag_id=DAG_ID,
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
params={
"dataset_qty": Param(1, type="integer", minimum=1),
}
) as dag:
start = EmptyOperator(task_id="start")
@task
def build_dataset_list():
ctx = get_current_context()
qty = int(ctx["params"].get("dataset_qty", 1))
return [f"example_dataset_{i}" for i in range(1, qty + 1)]
@task(outlets=[asset_alias])
def emit(datasets: list[str], outlet_events):
# Airflow will register an event for the AssetAlias because it is in outlets.
for name in datasets:
print(f"Emitting logical asset event for: {name} via alias {asset_alias.name}")
outlet_events[asset_alias].add(Asset(name))
return datasets
end = EmptyOperator(task_id="end")
ds_list = build_dataset_list()
start >> ds_list >> emit(ds_list) >> end
DAG outlet consumer:
import os
from datetime import datetime
from airflow.sdk import DAG, Asset, AssetAlias, get_current_context, Param
from airflow.decorators import task
from airflow.providers.standard.operators.empty import EmptyOperator
DAG_ID = os.path.splitext(os.path.basename(__file__))[0]
asset_alias = AssetAlias("performance-test-1")
with DAG(
dag_id=DAG_ID,
start_date=datetime(2024, 1, 1),
schedule=[asset_alias],
catchup=False,
) as dag:
@task(inlets=[asset_alias])
def consume_dataset_event_from_dataset_alias(*, inlet_events=None):
for event in inlet_events[asset_alias]:
print(event)
consume_dataset_event_from_dataset_alias()
Operating System
Linux
Versions of Apache Airflow Providers
No response
Deployment
Amazon (AWS) MWAA
Deployment details
Tested on:
- MWAA 2.10.3
- Local environment with Astro CLI:
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
3.0.6
If "Other Airflow 2 version" selected, which one?
2.10.3 and 2.11.0
What happened?
We’re dynamically generating datasets using dataset alias. With a few or even a few hundred datasets, no problem.
But when a task generates thousands, the runtime becomes huge. The thing is: the task itself runs fast — it’s actually spending most of the time outside the task, processing and writing outlets/metadata.
Two main issues here:
There are cases where the discrepancy is more than 4 minutes for a single task.
During task execution:
After DAG finished:
What you think should happen instead?
Optimization how outlets/datasetalias writes are handled at scale, especially for big dataset packs.
Show this “post-task processing time” somewhere in the UI so users understand where the time is going.
How to reproduce
DAG outlet producer:
DAG outlet consumer:
Operating System
Linux
Versions of Apache Airflow Providers
No response
Deployment
Amazon (AWS) MWAA
Deployment details
Tested on:
Anything else?
No response
Are you willing to submit PR?
Code of Conduct