Skip to content

Fix circular import in DB migration job #31204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jedcunningham merged 1 commit into from
May 11, 2023
Merged

Fix circular import in DB migration job #31204

jedcunningham merged 1 commit into from
May 11, 2023

Conversation

@LipuFei
Copy link
Contributor

@LipuFei LipuFei commented May 11, 2023

Related to #30988 and #31033

Version: 2.6.1rc1
Docker image: apache/airflow:2.6.1rc1-python3.10

The root-level import of TaskInstance will cause a circular import problem when both Kubernetes and Sentry are enabled. This will show up in the DB migration job when I use the official Helm chart 1.9.0 to deploy.

I have tested this fix by patching the image apache/airflow:2.6.1rc1-python3.10 with this kubernetes_executor.py

Here is the stack trace:

Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 112, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 84, in upgradedb
db.upgradedb(
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 76, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 1545, in upgradedb
import_all_models()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/__init__.py", line 60, in import_all_models
__getattr__(name)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/__init__.py", line 78, in __getattr__
val = import_string(f"{path}.{name}")
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 36, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dag.py", line 82, in <module>
from airflow.models.dagrun import DagRun
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 57, in <module>
from airflow.models.taskinstance import TaskInstance as TI
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 100, in <module>
from airflow.sentry import Sentry
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py", line 195, in <module>
Sentry = ConfiguredSentry()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py", line 92, in __init__
executor_class, _ = ExecutorLoader.import_default_executor_cls()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 158, in import_default_executor_cls
executor, source = cls.import_executor_cls(executor_name)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 134, in import_executor_cls
return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 129, in _import_and_validate
executor = import_string(path)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 36, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/celery_kubernetes_executor.py", line 26, in <module>
from airflow.executors.kubernetes_executor import KubernetesExecutor
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/kubernetes_executor.py", line 49, in <module>
from airflow.models.taskinstance import TaskInstance
ImportError: cannot import name 'TaskInstance' from partially initialized module 'airflow.models.taskinstance' (most likely due to a circular import) (/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py)

@boring-cyborg boring-cyborg bot added provider:cncf-kubernetes Kubernetes (k8s) provider related issues area:Scheduler including HA (high availability) scheduler labels May 11, 2023
@LipuFei LipuFei mentioned this pull request May 11, 2023
Closed
34 tasks
pankajkoti
pankajkoti reviewed May 11, 2023
View reviewed changes
Fix circular import
8018724 
related: apache#30988, apache#31033

The root-level import of TaskInstance will still cause a circular import
problem when both Kubernetes and Sentry are enabled. This will show up
in the DB migration job if I use the official Helm chart 1.9.0 to
deploy.
@LipuFei LipuFei force-pushed the fix/circular-import branch from 2396806 to 8018724 Compare May 11, 2023 17:18
@jedcunningham jedcunningham merged commit be8f96b into apache:main May 11, 2023
@LipuFei LipuFei deleted the fix/circular-import branch May 11, 2023 22:49
@ephraimbuddy ephraimbuddy added this to the Airflow 2.6.1 milestone May 12, 2023
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label May 12, 2023
ephraimbuddy pushed a commit that referenced this pull request May 12, 2023
@LipuFei @ephraimbuddy
Fix circular import (#31204)
33ff489 
related: #30988, #31033

The root-level import of TaskInstance will still cause a circular import
problem when both Kubernetes and Sentry are enabled. This will show up
in the DB migration job if I use the official Helm chart 1.9.0 to
deploy.

Co-authored-by: Lipu Fei <lipu.fei@kpn.com>
(cherry picked from commit be8f96b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pankajkoti pankajkoti pankajkoti left review comments

@potiuk potiuk potiuk approved these changes

@jedcunningham jedcunningham jedcunningham approved these changes

@dstandish dstandish Awaiting requested review from dstandish

Assignees

No one assigned

Labels

area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes (k8s) provider related issues type:bug-fix Changelog: Bug Fixes

Projects

None yet

Milestone

Airflow 2.6.1

Development

Successfully merging this pull request may close these issues.

5 participants

@LipuFei @potiuk @pankajkoti @jedcunningham @ephraimbuddy