Skip to content

Edited Discovery Logic #20631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eleanorjboyd merged 86 commits into from
Apr 17, 2023
Merged

Edited Discovery Logic #20631

eleanorjboyd merged 86 commits into from
Apr 17, 2023

Conversation

@eleanorjboyd
Copy link
Member

@eleanorjboyd eleanorjboyd commented Feb 2, 2023 •
edited
Loading

closes #20078 and closes #20085 (which is about the testing work to support this code)

This logic now successfully works to discover the pytest repo tests. This branch takes into account all the previous comments made to the python discovery logic in the previous PR (located on my personal fork). Therefore this is a second round of edits on this code. It now works for the pytest library (discovers all the tests in the pytest library).

@eleanorjboyd eleanorjboyd self-assigned this Feb 2, 2023
@eleanorjboyd eleanorjboyd added the no-changelog No news entry required label Feb 6, 2023
@eleanorjboyd eleanorjboyd marked this pull request as ready for review February 6, 2023 18:57
@vscodenpa vscodenpa added this to the February 2023 milestone Feb 6, 2023
@brettcannon
Copy link
Member

brettcannon commented Feb 6, 2023

FYI one of your files doesn't pass Black formatting checks.

brettcannon
brettcannon suggested changes Feb 7, 2023
View reviewed changes
@karthiknadig

This comment was marked as resolved.

Sign in to view
karthiknadig

This comment was marked as resolved.

Sign in to view

@brettcannon
Copy link
Member

brettcannon commented Mar 9, 2023

FYI this didn't pass Python type checking.

brettcannon
brettcannon previously requested changes Mar 10, 2023
View reviewed changes
Copy link
Member

@brettcannon brettcannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some inconsistencies in various things (e.g. docstring formatting, application of typing, etc.). It's always good to read through your changes to make sure there's consistency. It helps with readability as people know what to expect, as well as productivity as people's brain can tune to what to expect, having to think less about how to process something.

@eleanorjboyd eleanorjboyd modified the milestones: March 2023, April 2023 Mar 20, 2023
@eleanorjboyd eleanorjboyd force-pushed the python-discovery-code branch from 2686923 to c102dee Compare April 4, 2023 23:18
cwebster-99
cwebster-99 reviewed Apr 6, 2023
View reviewed changes
cwebster-99
cwebster-99 reviewed Apr 6, 2023
View reviewed changes
cwebster-99
cwebster-99 reviewed Apr 6, 2023
View reviewed changes
@eleanorjboyd eleanorjboyd force-pushed the python-discovery-code branch from 2962eb8 to 601affb Compare April 6, 2023 22:52
@karthiknadig
Copy link
Member

karthiknadig commented Apr 14, 2023

Running pytest session to run other pytest session instances on windows seems to have weird problems:
Running it using microsoft store python causes this error:

Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python

Running it using python.org python causes this error:

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pytest\__main__.py", line 5, in <module>

    raise SystemExit(pytest.console_main())

                     ^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 189, in console_main

    code = main()

           ^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 147, in main

    config = _prepareconfig(args, plugins)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 328, in _prepareconfig

    config = pluginmanager.hook.pytest_cmdline_parse(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_hooks.py", line 265, in __call__

    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_manager.py", line 80, in _hookexec

    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_callers.py", line 55, in _multicall

    gen.send(outcome)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\helpconfig.py", line 103, in pytest_cmdline_parse

    config: Config = outcome.get_result()

                     ^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_result.py", line 60, in get_result

    raise ex[1].with_traceback(ex[2])

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_callers.py", line 39, in _multicall

    res = hook_impl.function(*args)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1067, in pytest_cmdline_parse

    self.parse(args)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1354, in parse

    self._preparse(args, addopts=addopts)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1237, in _preparse

    self.pluginmanager.load_setuptools_entrypoints("pytest11")

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_manager.py", line 287, in load_setuptools_entrypoints

    plugin = ep.load()

             ^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\importlib\metadata\__init__.py", line 202, in load

    module = import_module(match.group('module'))

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\importlib\__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\__init__.py", line [101](https://github.com/microsoft/vscode-python/actions/runs/4695498246/jobs/8324731667#step:20:102), in <module>

    from ._core._fileio import AsyncFile, Path, open_file, wrap_file

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\_core\_fileio.py", line 26, in <module>

    from .. import to_thread

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\to_thread.py", line 5, in <module>

    from .abc import CapacityLimiter

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\abc\__init__.py", line 81, in <module>

    from ..from_thread import BlockingPortal

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\from_thread.py", line 2, in <module>

    from asyncio import iscoroutine

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\asyncio\__init__.py", line 42, in <module>

    from .windows_events import *

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\asyncio\windows_events.py", line 8, in <module>

    import _overlapped

OSError: [WinError 10106] The requested service provider could not be loaded or initialized

We may have to figure out a different way to run these tests. May be just run them in a way we run them separately and collect data. Before running pytest session and then validate using pytest.

@karthiknadig karthiknadig force-pushed the python-discovery-code branch from 899676d to 6bf3eac Compare April 14, 2023 03:04
@karthiknadig karthiknadig force-pushed the python-discovery-code branch from 6bf3eac to cd4ddac Compare April 14, 2023 03:08
@karthiknadig
Copy link
Member

karthiknadig commented Apr 14, 2023

These two errors on windows were hardest to debug:

[WinError 10106] The requested service provider could not be loaded or initialized

This is only on windows store python

_Py_HashRandomization_Init: failed to get random numbers to initialize Python

Looks like anyio package gets used if it exists when running pytest. That package seems to fail if you run a pytest sub session under pytest session. Moved the tests around so we run all the python tool tests before we install other packages. This only affects cases where we are testing the test framework itself like in this case testing pytest plugin.

@karthiknadig
Copy link
Member

karthiknadig commented Apr 14, 2023

Thanks to @int19h we finally figured out what was going here. For any one ese that runs into this issue, basically USERPROFILE and SYSTEMROOT are required environment variables for python in windows. If it is missing it can cause errors like above. The pytest session helper script used subprocess.run did not copy the environ variables from the parent process. It set a minimal one that was needed for test, which did not include any of the system variables. This was the cause of the issue.

@karthiknadig karthiknadig mentioned this pull request Apr 17, 2023
Closed
@eleanorjboyd eleanorjboyd dismissed brettcannon’s stale review April 17, 2023 16:54

inconsistencies have been address and verbal OK was given

@eleanorjboyd eleanorjboyd merged commit ed06e55 into main Apr 17, 2023
@eleanorjboyd eleanorjboyd deleted the python-discovery-code branch April 17, 2023 17:03
paulacamargo25
paulacamargo25 approved these changes Apr 17, 2023
View reviewed changes
@eleanorjboyd eleanorjboyd restored the python-discovery-code branch April 17, 2023 17:16
This was referenced Apr 17, 2023
Closed
Closed
@karthiknadig karthiknadig deleted the python-discovery-code branch May 3, 2023 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cwebster-99 cwebster-99 cwebster-99 left review comments

@karthiknadig karthiknadig karthiknadig approved these changes

@brettcannon brettcannon Awaiting requested review from brettcannon

+2 more reviewers

@karrtikr karrtikr karrtikr approved these changes

@paulacamargo25 paulacamargo25 paulacamargo25 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@eleanorjboyd eleanorjboyd

Labels

no-changelog No news entry required

Projects

None yet

Milestone

April 2023

Development

Successfully merging this pull request may close these issues.

Write testing for pytest discovery Finalize Pytest Discovery Logic

9 participants

@eleanorjboyd @brettcannon @karthiknadig @int19h @karrtikr @paulacamargo25 @cwebster-99 @vscodenpa