Add PytestPluginSetup.extra_sys_path for pants-plugins#21274
Add PytestPluginSetup.extra_sys_path for pants-plugins#21274cognifloyd merged 4 commits intomainfrom
PytestPluginSetup.extra_sys_path for pants-plugins#21274Conversation
This feature allows plugins to inject additional paths in the PEX_EXTRA_SYS_PATH env var, similar to how a wrapper script could modify PYTHONPATH when running pytest. This will be helpful for plugins that need virtual "source roots" without explicitly registering them. A codegen plugin could create a new "source root" for example, or a python framework could have special PYTHONPATH requirements that do not map well to "source roots", esp if the roots overlap.
|
I'm not sure if this should be category:plugin api change or category:new feature as this new feature is not directly usable without writing a pants-plugin to use it. I'm happy to switch the category either way. |
|
For reference, here is the plugin that needs this feature: |
|
Here are the locations that could also use the additional PYTHONPATH paths. Unlike Tools:
Goals:
I don't know if we'll need support in coverage:
|
This feature allows plugins to inject additional paths in the
PEX_EXTRA_SYS_PATHenv var, similar to how a wrapper script could modifyPYTHONPATHwhen running pytest.This will be helpful for plugins that need virtual "source roots" without explicitly registering them. A codegen plugin could create a new "source root" for example, or a python framework could have special
PYTHONPATHrequirements that do not map well to "source roots", especially if the roots overlap.Background:
I need this for the StackStorm project where I have to deal with "StackStorm Content Packs" with special PYTHONPATH rules. These packs work best as a single source root, as I have a single
pack_metadata(...)target that defaults to a recursive glob inclusion of yaml files (aka "metadata" files) in the pack. However, I also have python files that live in several directories that, by convention, get added to PYTHONPATH. If I try to create a source root for every one of those special directories, I get an explosion of source roots, and I would have to redesignpack_metadata(...)such that I have to add details about these yaml files in many more BUILD files. So, I don't want to expose those directories as source roots.To facilitate this I have local changes to StackStorm's
pack_metadatapants-plugin that record these special modules in the python module mapper (aFirstPartyPythonMappingImplMarker -> FirstPartyPythonMappingImplrule) allowing dependency inference to work correctly for imports that rely on these special directories being onPYTHONPATH. I also extended the plugin to generate the relevantPYTHONPATHbits, so I just need the feature in this PR (or something similar) to allow me to actually include them when running pytest.Alternatives
I could add a generic
extra_envtoPytestPluginSetup, and then setPytestPluginSetup(extra_env={"PEX_EXTRA_SYS_PATH": ":".join(my_extra_sys_path)}), but that would be more complex as it would have to merge thePEX_EXTRA_SYS_PATHthat pants already generates, and possibly protect any other env vars injected from subsystem config or from particular targets. I don't actually need to modify any other env vars, so I opted to only implementextra_sys_path.I also considered something more generic (not pytest-specific) where plugins can say "I have source roots as well", but then I run into the issue with overlapping source roots. Plus, that would mean potentially changing many of the call sites that request source roots. Something more generic, like this, would be nice for injecting the sys.path entries for other tools like
pylintor goals likerunandrepl, but I don't see any obvious interfaces for a generic feature like this.