Skip to content

Use pkgutil.iter_modules to get un-imported module for test_pre_import#1085

Merged
rapids-bot[bot] merged 1 commit intorapidsai:branch-23.02from
charlesbluca:resolve-pre-import-tests
Jan 17, 2023
Merged

Use pkgutil.iter_modules to get un-imported module for test_pre_import#1085
rapids-bot[bot] merged 1 commit intorapidsai:branch-23.02from
charlesbluca:resolve-pre-import-tests

Conversation

@charlesbluca
Copy link
Copy Markdown
Member

Changed this because IIUC pkg_resources.working_set is listing the installed distributions and not necessarily the importable modules; this becomes an issue if the distribution and module names aren't the same (e.g. one would conda install pillow and then import PIL), which was causing some failures in CI that seem unrelated to the changes here.

Originally posted by @charlesbluca in #981 (comment)

@charlesbluca charlesbluca requested a review from a team as a code owner January 17, 2023 18:56
@github-actions github-actions bot added the python python code needed label Jan 17, 2023
@charlesbluca charlesbluca added non-breaking Non-breaking change bug Something isn't working labels Jan 17, 2023
@jakirkham
Copy link
Copy Markdown
Member

Wonder if we can do this with importlib.metadata instead

Note that there is a backport package importlib-metadata for anything that is not yet included in the Python standard library of the Python versions we support

Copy link
Copy Markdown
Member

@pentschev pentschev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are right indeed @charlesbluca , good catch and thanks for the investigation/fixing work! 😄

@pentschev
Copy link
Copy Markdown
Member

/merge

@charlesbluca charlesbluca marked this pull request as draft January 17, 2023 19:15
@charlesbluca
Copy link
Copy Markdown
Member Author

Converted to draft to try out @jakirkham's suggestion of using importlib.metadata/importlib_metadata instead

@charlesbluca
Copy link
Copy Markdown
Member Author

After doing some digging into importlib.metadata's API, I wasn't able to find a suitable method that could grab a list of all working import packages; the closest thing I could find was packages_distributions(), but at least locally this seems to list some non-importable package names:

>>> importlib_metadata.packages_distributions()
{'... 'docker': ['ucx-py'], ...}
>>> import docker
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'

Since this issue is blocking CI at the moment, and my local testing of pkgutil.iter_modules() seems to show that it doesn't have this issue, I'm inclined to go ahead and merge this and revisit if it becomes a problem again. Though if you have a specific API in mind here would be happy to open up / review a follow up 🙂

@charlesbluca charlesbluca marked this pull request as ready for review January 17, 2023 19:57
@charlesbluca
Copy link
Copy Markdown
Member Author

/merge

@rapids-bot rapids-bot bot merged commit 52dd850 into rapidsai:branch-23.02 Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working non-breaking Non-breaking change python python code needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants