Skip to content

Add more precise description on avoiding generic package/module names #36927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Jan 20, 2024
Merged

Add more precise description on avoiding generic package/module names #36927

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions docs/apache-airflow/administration-and-deployment/modules_management.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ You can do it in one of those ways:
The next chapter has a general description of how Python loads packages and modules, and dives
deeper into the specifics of each of the three possibilities above.


How package/modules loading in Python works
-------------------------------------------

Expand Down Expand Up @@ -159,14 +160,33 @@ Airflow, when running dynamically adds three directories to the ``sys.path``:
as safe because they are part of configuration of the Airflow installation and controlled by the
people managing the installation.

Best practices for module loading
---------------------------------
Best practices for your code naming
-----------------------------------

There are a few gotchas you should be careful about when you import your code.

Sometimes, you might see exceptions that ``module 'X' has no attribute 'Y'`` raised from Airflow or other
library code that you use. This is usually caused by the fact that you have a module or packaged named 'X'
in your ``PYTHONPATH`` at the top level and it is imported instead of the module that the original
code expects.

You should always use unique names for your packages and modules and there are ways how you can make
sure that uniqueness is enforced described below.


Use unique top package name
...........................

Most importantly avoid using generic names for anything that you add directly at the top level of your
``PYTHONPATH``. For example if you add ``airflow`` folder with ``__init__.py`` to your ``DAGS_FOLDER``,
it will clash with the Airflow package and you will not be able to import anything from Airflow
package. Similarly do not add ``airflow.py`` file directly there. Also common names used by standard
library packages such as ``multiprocessing`` or ``logging`` etc. should not be used as top level - either
as packages (i.e. folders with ``__init__.py``) or as modules (i.e. ``.py`` files).

The same applies to ``config`` and ``plugins`` folders which are also at the ``PYTHONPATH`` and anything
you add to your ``PYTHONPATH`` manually (see details in the following chapters).

It is recommended that you always put your DAGs/common files in a subpackage which is unique to your
deployment (``my_company`` in the example below). It is far too easy to use generic names for the
folders that will clash with other packages already present in the system. For example if you
Expand Down