Add wait_for_termination parameter and fix double-deferral in PowerBIDatasetRefreshOperator by shreyas-dev · Pull Request #60369 · apache/airflow

shreyas-dev · 2026-01-10T13:19:59Z

Description

This PR adds an optional wait_for_termination parameter to PowerBIDatasetRefreshOperator and refactors the execution flow to fix the double-deferral anti-pattern present in the current implementation.

Motivation and Context

New Feature:
Users need the ability to trigger Power BI dataset refreshes without waiting for completion. This is useful for:

Long-running refreshes (hours/days) that don't need to block the DAG
Scenarios where downstream tasks don't depend on refresh completion
Reducing worker slot usage for fire-and-forget operations

Bug Fix:
The current implementation uses an inefficient double-deferral pattern:

execute() → defer → triggers refresh and returns dataset_refresh_id
get_refresh_status() → defer again → polls for completion
execute_complete() → processes final result

This is unnecessary because PowerBITrigger.run() already handles both triggering the refresh AND polling for completion in a single async operation when wait_for_termination=True.

Was generative AI tooling used to co-author this PR?

No

boring-cyborg · 2026-01-10T13:20:08Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

dabla

Congratulations @shreyas-dev, looking good to me beside the remark made by @shahar1 I think where are good nice improvement.

shreyas-dev · 2026-01-13T14:20:16Z

Hey @dabla and @shahar1, Added wait_for_termination to docstring, please review.

shreyas-dev · 2026-01-13T18:23:34Z

Hey @dabla, fixed "imperative mood" docstring test case which was failing.

shreyas-dev · 2026-01-14T05:09:38Z

Hey @vincbeck / @dabla, Please check

shreyas-dev · 2026-01-14T18:09:52Z

Hey @vincbeck , Resolved www-hash.txt.

shreyas-dev · 2026-01-16T15:14:23Z

Hey @vincbeck,

_____________________________________________________________________________ TestAwsS3Hook.test_sync_to_local_dir_behaviour _____________________________________________________________________________
[gw1] linux -- Python 3.10.19 /usr/python/bin/python
providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py:1840: in test_sync_to_local_dir_behaviour
    assert "S3 object size" in logs_string
E   AssertionError: assert 'S3 object size' in 'Downloading data from s3://airflow-test-s3-bucket/ to /tmp/pytest-of-root/pytest-0/popen-gw1/test_sync_to_local_dir_b...al_dir_behavio0/s3_sync_dir/subproject1/dag_b.py is up-to-date with S3 object subproject1/dag_b.py. Skipping download.'

This test case failure looks unrelated to this PR. I have updated my branch. Can you please check ?

shreyas-dev · 2026-02-01T14:05:53Z

Ran prek --all-files locally and all results passed

shreyas-dev · 2026-02-02T16:55:06Z

Hey @vincbeck , I'm guessing structlog was introduced in airflow 3+ and airflow 2.11 used standard logging. How to resolve this issue? On the contrary I'm now thinking of removing the warning cause Its a just informational message.

logger.warning(
                "The PowerBIDatasetRefreshOperator now always uses deferrable execution when wait_for_termination=True."
            )

In this case I can remove structlog entirely

vincbeck · 2026-02-02T17:14:56Z

Hey @vincbeck , I'm guessing structlog was introduced in airflow 3+ and airflow 2.11 used standard logging. How to resolve this issue? On the contrary I'm now thinking of removing the warning cause Its a just informational message.
logger.warning(
                "The PowerBIDatasetRefreshOperator now always uses deferrable execution when wait_for_termination=True."
            )
In this case I can remove structlog entirely

Can you check how it is done in the other providers?

shreyas-dev · 2026-02-03T04:17:23Z

Hey @vincbeck, Updated the code.prek --all-files was successful locally. Please check

shreyas-dev · 2026-02-03T16:40:37Z

The failed test doesn't look to be related to this PR https://github.com/apache/airflow/actions/runs/21616745897/job/62362457618?pr=60369

Host python version is 3.10.19 (main, Oct 10 2025, 01:02:36) [GCC 11.4.0]
The image /mnt/ci-image-save-v3-linux_amd64-3.10.tar does not exist.
Error: Process completed with exit code 1.

…o wait_for_completion

shreyas-dev · 2026-02-04T16:39:10Z

Hey @vincbeck, Apologies I have fixed the test case and verified all the test cases by running it locally.

boring-cyborg · 2026-02-04T18:28:42Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…DatasetRefreshOperator (apache#60369)

shreyas-dev requested a review from dabla as a code owner January 10, 2026 13:20

boring-cyborg Bot added area:providers provider:microsoft-azure Azure-related issues labels Jan 10, 2026

shahar1 reviewed Jan 10, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

dabla requested changes Jan 10, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

dabla approved these changes Jan 13, 2026

View reviewed changes

shreyas-dev requested a review from vincbeck as a code owner January 14, 2026 05:08

vincbeck reviewed Jan 14, 2026

View reviewed changes

Comment thread providers/fab/www-hash.txt Outdated

shreyas-dev force-pushed the main branch from eb8a560 to a4cbe8b Compare January 14, 2026 18:07

eladkal reviewed Jan 16, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

shreyas-dev requested review from amoghrajesh, ashb, bolkedebruin, bugraoz93, dstandish, ephraimbuddy, gopidesupavan, hussein-awala, jason810496, jedcunningham, jscheffl, kaxil, mobuchowski and potiuk as code owners January 29, 2026 03:12

shreyas-dev force-pushed the main branch from a040b9a to 3e2af7c Compare February 3, 2026 04:16

vincbeck reviewed Feb 3, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py Outdated

shreyas-dev added 6 commits February 4, 2026 20:05

Add wait_for_termination parameter to PowerBIDatasetRefreshOperator

06f6c6c

add wait_for_termination to docstring

ac8ce27

fix docstring in execute_complete to adhere imperative mood

c50527a

Apply prek formatting fixes

e51e4d2

add deprecated notice in

7fbcac5

Add deferrable warning

e16d2c0

shreyas-dev force-pushed the main branch from cf353bc to 7e393a2 Compare February 4, 2026 14:35

Change PowerBIDatasetRefreshOperator args from wait_for_termination t…

5d24d77

…o wait_for_completion

shreyas-dev force-pushed the main branch from 7e393a2 to 5d24d77 Compare February 4, 2026 16:09

Merge branch 'main' into main

a96f4d3

vincbeck approved these changes Feb 4, 2026

View reviewed changes

vincbeck merged commit 73282bd into apache:main Feb 4, 2026
169 of 170 checks passed

Alok-kumar-priyadarshi pushed a commit to Alok-kumar-priyadarshi/airflow that referenced this pull request Feb 5, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

e56a430

…DatasetRefreshOperator (apache#60369)

jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

2f55edc

…DatasetRefreshOperator (apache#60369)

shahar1 mentioned this pull request Feb 11, 2026

Status of testing Providers that were prepared on February 10, 2026 #61766

Closed

81 tasks

Ratasa143 pushed a commit to Ratasa143/airflow that referenced this pull request Feb 15, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

ca09ddb

…DatasetRefreshOperator (apache#60369)

choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

629eb9c

…DatasetRefreshOperator (apache#60369)

potiuk mentioned this pull request Feb 26, 2026

Status of testing Providers that were prepared on February 26, 2026 #62537

Closed

Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

1db995e

…DatasetRefreshOperator (apache#60369)

Ankurdeewan pushed a commit to Ankurdeewan/airflow that referenced this pull request Mar 15, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

a033378

…DatasetRefreshOperator (apache#60369)

radhwene pushed a commit to radhwene/airflow that referenced this pull request Mar 21, 2026

Add wait_for_termination parameter and fix double-deferral in PowerBI…

2108ef3

…DatasetRefreshOperator (apache#60369)

Conversation

shreyas-dev commented Jan 10, 2026

Description

Motivation and Context

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg Bot commented Jan 10, 2026

Uh oh!

Uh oh!

dabla left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shreyas-dev commented Jan 13, 2026

Uh oh!

shreyas-dev commented Jan 13, 2026

Uh oh!

shreyas-dev commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

shreyas-dev commented Jan 14, 2026

Uh oh!

shreyas-dev commented Jan 16, 2026

Uh oh!

Uh oh!

shreyas-dev commented Feb 1, 2026

Uh oh!

shreyas-dev commented Feb 2, 2026

Uh oh!

vincbeck commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shreyas-dev commented Feb 3, 2026

Uh oh!

Uh oh!

shreyas-dev commented Feb 3, 2026

Uh oh!

shreyas-dev commented Feb 4, 2026

Uh oh!

Uh oh!

boring-cyborg Bot commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shreyas-dev commented Jan 14, 2026 •

edited

Loading

vincbeck commented Feb 2, 2026 •

edited

Loading