INFO: 172.32.14.123:56178 - "POST /auth/token HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
| await super().__call__(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
| await responder(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
| await super().__call__(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
| await self.app(scope, receive, self.send_with_compression)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in __call__
| raise app_exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
| await self.app(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
| await super().__call__(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 232, in cleanup_session_middleware
| response = await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 159, in call_next
| raise app_exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
| response = await f(request)
| ^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 308, in app
| raw_response = await run_endpoint_function(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 221, in run_endpoint_function
| return await run_in_threadpool(dependant.call, **values)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
| return await anyio.to_thread.run_sync(func)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
| return await future
| ^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
| result = context.run(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/api_fastapi/routes/login.py", line 48, in create_token
| with get_application_builder():
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/usr/python/lib/python3.12/contextlib.py", line 137, in __enter__
| return next(self.gen)
| ^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 69, in get_application_builder
| yield _return_appbuilder(flask_app, db)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 45, in _return_appbuilder
| init_appbuilder(app, enable_plugins=False)
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 606, in init_appbuilder
| return AirflowAppBuilder(
| ^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 159, in __init__
| self.init_app(app, session)
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 215, in init_app
| self._add_admin_views()
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 311, in _add_admin_views
| auth_manager.register_views()
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 599, in register_views
| self.security_manager.register_views()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/usr/python/lib/python3.12/functools.py", line 998, in __get__
| val = self.func(instance)
| ^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 578, in security_manager
| raise AirflowException("AppBuilder is not initialized.")
| airflow.exceptions.AirflowException: AppBuilder is not initialized.
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
await super().__call__(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
await responder(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
await super().__call__(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
await self.app(scope, receive, self.send_with_compression)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in __call__
raise app_exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
await super().__call__(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
with recv_stream, send_stream, collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 232, in cleanup_session_middleware
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 159, in call_next
raise app_exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 308, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 221, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/api_fastapi/routes/login.py", line 48, in create_token
with get_application_builder():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 69, in get_application_builder
yield _return_appbuilder(flask_app, db)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 45, in _return_appbuilder
init_appbuilder(app, enable_plugins=False)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 606, in init_appbuilder
return AirflowAppBuilder(
^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 159, in __init__
self.init_app(app, session)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 215, in init_app
self._add_admin_views()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/extensions/init_appbuilder.py", line 311, in _add_admin_views
auth_manager.register_views()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 599, in register_views
self.security_manager.register_views()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/functools.py", line 998, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 578, in security_manager
raise AirflowException("AppBuilder is not initialized.")
airflow.exceptions.AirflowException: AppBuilder is not initialized.
Apache Airflow Provider(s)
fab
Versions of Apache Airflow Providers
/opt/airflow$ pip freeze | grep providers
apache-airflow-providers-amazon==9.22.0
apache-airflow-providers-celery==3.17.0
apache-airflow-providers-cncf-kubernetes==10.13.0
apache-airflow-providers-common-compat==1.14.0
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-messaging==2.0.2
apache-airflow-providers-common-sql==1.32.0
apache-airflow-providers-docker==4.5.2
apache-airflow-providers-elasticsearch==6.5.0
apache-airflow-providers-fab==3.5.0
apache-airflow-providers-ftp==3.14.1
apache-airflow-providers-git==0.2.4
apache-airflow-providers-google==20.0.0
apache-airflow-providers-grpc==3.9.2
apache-airflow-providers-hashicorp==4.5.0
apache-airflow-providers-http==6.0.0
apache-airflow-providers-microsoft-azure==13.0.0
apache-airflow-providers-mysql==6.5.0
apache-airflow-providers-odbc==4.12.0
apache-airflow-providers-openlineage==2.11.0
apache-airflow-providers-postgres==6.6.0
apache-airflow-providers-redis==4.4.2
apache-airflow-providers-sendgrid==4.2.1
apache-airflow-providers-sftp==5.7.0
apache-airflow-providers-slack==9.7.0
apache-airflow-providers-smtp==2.4.2
apache-airflow-providers-snowflake==6.10.0
apache-airflow-providers-ssh==4.3.1
apache-airflow-providers-standard==1.12.0
Apache Airflow version
3.1.8
Operating System
Debian GNU/Linux 12 (bookworm)
Deployment
Official Apache Airflow Helm Chart
Deployment details
Use apache/airflow:3.1.8-python3.12 image
What happened
When running Airflow 3.1.8 with
apache-airflow-providers-fab==3.5.0(using Python 3.12), the application fails to handle login requests and throws a 500 Internal Server Error.Looking at the traceback, it crashes with
airflow.exceptions.AirflowException: AppBuilder is not initialized.during the FastAPI endpoint execution (/auth/tokenor login routes).I think the error occurs because
auth_manager.register_views()is called, which in turn callsself.security_manager. However, because theAppBuilderinitialization context is not yet fully completed, thesecurity_managerproperty raises the exception.Traceback snippet:
logs in api-server
Workaround:
I found that settingAIRFLOW__FAB__UPDATE_FAB_PERMS=Falsebypasses this issue and allows the server to run successfully.What you think should happen instead
The FastAPI application should initialize successfully and serve auth requests without crashing, even when the default configurationupdate_fab_perms = Trueis used.The internal startup sequence should be fixed so that theAppBuilderis completely instantiated and attached to the context before the Auth Manager attempts to accessself.security_managerto sync roles or register views.How to reproduce
pip install apache-airflow-providers-fab==3.5.0update_fab_perms = False)./auth/tokenendpoint.AppBuilder is not initializedexception in the api-server logs.Anything else
Related? #61108
Are you willing to submit PR?
Code of Conduct