Longer running tasks will occasionally fail because of an invalid auth token that has expired.
2025-07-23 01:27:58,629 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0967-7bc4-b053-e6663b1a45ec/state "HTTP/1.1 204 No Content"
2025-07-23 01:27:58 [info ] Task finished [supervisor] duration=1076.4064389939886 exit_code=0 final_state=success
2025-07-23 01:27:58 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['SecretsManagerBackend', 'EnvironmentVariablesBackend'] count=2
2025-07-23 01:27:58,919 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0955-74d4-8db6-2aea4332ab1b/run "HTTP/1.1 403 Forbidden"
2025-07-23 01:27:58 [warning ] Server error [airflow.sdk.api.client] detail=None
2025-07-23 01:27:58 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=1258981 signal_sent=SIGKILL
2025-07-23 01:27:58,929 {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
_execute_work(log, workload)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1683, in supervise
process = ActivitySubprocess.start(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 851, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 862, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 149, in start
resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 707, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
hook(response)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 120, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 110, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
Correlation-id=019835f7-a470-78d0-b268-b388c323926f
Create with multiple tasks, and different connections in the tasks where the first task runs for 30+ minutes. Store connections in AWS Secrets Manager.
When running in production, we use AWS ECS tasks, I have only see the issue happen here, but also haven't run the longer running tasks locally.
Apache Airflow version
3.0.3
If "Other Airflow 2 version" selected, which one?
No response
What happened?
Longer running tasks will occasionally fail because of an invalid auth token that has expired.
What you think should happen instead?
The task instance should get a new token and continue
How to reproduce
Create with multiple tasks, and different connections in the tasks where the first task runs for 30+ minutes. Store connections in AWS Secrets Manager.
Operating System
Debain
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.10.0
apache-airflow-providers-common-compat==1.7.2
apache-airflow-providers-common-io==1.6.1
apache-airflow-providers-common-sql==1.27.3
apache-airflow-providers-fab==2.3.0
apache-airflow-providers-google==16.1.0
apache-airflow-providers-http==5.3.2
apache-airflow-providers-microsoft-mssql==4.3.1
apache-airflow-providers-openai==1.6.1
apache-airflow-providers-oracle==4.1.2
apache-airflow-providers-postgres==6.2.1
apache-airflow-providers-sftp==5.3.2
apache-airflow-providers-smtp==2.1.1
apache-airflow-providers-ssh==4.1.1
apache-airflow-providers-standard==1.4.1
Deployment
Other Docker-based deployment
Deployment details
When running in production, we use AWS ECS tasks, I have only see the issue happen here, but also haven't run the longer running tasks locally.
Anything else?
No response
Are you willing to submit PR?
Code of Conduct