[bug] order of the env variables breaks parsing #869
Description
Describe the bug
If dynamic env variables are in the wrong order, and depend on each other (eg. in the .env file) the config will not parse.
This is problematic in docker environment because the order of env variables might be random.
The problem didn't occur when I deleted config-defaults.yaml from the example or @format was used in place of @jinja.
To Reproduce
Steps to reproduce the behavior:
- Having the following folder structure
Project structure
.
├── Dockerfile
├── config-defaults.yaml
├── config.py
└── requirements.txt
1 directory, 4 files
- Having the following config files:
Config files
.env
APP_PATH_ROOT=@jinja {{this.PATH_ROOT_REL | abspath}}
APP_PATH_DATA=@format {this.PATH_ROOT}/some_data
APP_PATH_ROOT_REL=..
APP_PATH_MODEL=@format {this.PATH_ROOT}/some_modeland
config-defaults.yaml
PATH_ROOT: ../..
PATH_DATA: "@jinja {{this.PATH_ROOT | abspath}}/data"
PATH_MODEL: "@jinja {{this.PATH_ROOT | abspath}}/models"
PATH_CONFIG: "@jinja {{this.PATH_ROOT | abspath}}/config"- Having the following app code:
Code
/config.py
from pathlib import Path
from dynaconf import Dynaconf
from pprint import pprint
src_dir = Path(__file__).parent.resolve()
settings = Dynaconf(
envvar_prefix="APP",
settings_files=["config.yaml", src_dir / "config-defaults.yaml"],
load_dotenv=True,
merge_enabled=True,
)
pprint(settings.as_dict())- Executing under the following environment
Execution
$ python config.py
Traceback (most recent call last):
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/config.py", line 15, in <module>
pprint(settings.as_dict())
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 115, in __getattr__
self._setup()
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 174, in _setup
self._wrapped = Settings(
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 253, in __init__
self.execute_loaders()
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 1035, in execute_loaders
core_loader.load(self, env, silent=silent, key=key)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/loaders/env_loader.py", line 24, in load
load_from_env(obj, global_prefix, key, silent, IDENTIFIER + "_global")
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/loaders/env_loader.py", line 82, in load_from_env
obj.update(data, loader_identifier=identifier)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 952, in update
self.set(
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 869, in set
existing = getattr(self, key, None)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 298, in __getattribute__
return self._store[name]
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/boxing.py", line 26, in evaluate
return recursively_evaluate_lazy_format(value, settings)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/__init__.py", line 413, in recursively_evaluate_lazy_format
value = value(settings)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 188, in __call__
result = self.formatter(self.value, **self.context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 146, in __call__
return self.function(value, **context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 157, in _jinja_formatter
return jinja_env.from_string(value).render(**context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 485, in getattr
return getattr(obj, attribute)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/base.py", line 298, in __getattribute__
return self._store[name]
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/boxing.py", line 26, in evaluate
return recursively_evaluate_lazy_format(value, settings)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/__init__.py", line 413, in recursively_evaluate_lazy_format
value = value(settings)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 188, in __call__
result = self.formatter(self.value, **self.context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 146, in __call__
return self.function(value, **context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/dynaconf/utils/parse_conf.py", line 157, in _jinja_formatter
return jinja_env.from_string(value).render(**context)
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/Users/amadeusz.lisiecki/workspace/dynaconf-test/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "/Users/amadeusz.lisiecki/.pyenv/versions/3.9.14/lib/python3.9/posixpath.py", line 375, in abspath
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not UndefinedExpected behavior
Environment variables can come in random order and freely depend on each other.
Environment (please complete the following information):
- OS: macOS Ventura 13.1
- Dynaconf Version 3.1.11
- Jinja2 Version 3.1.2
Additional context
I faced this problem mostly in a cloud environment where I was running docerized jobs - because environment variables where passed by the job controller (sagemaker in this case). I reproduced this docker container and then locally in virtual environment. Because of this I wasn't able to use dynamic variables in my jobs.