Pytest discovery using new Python test discovery adapter. #4695

d3r3kk · 2019-03-09T01:47:04Z

Transform results from our new Python test adapter for discovery into the Tests struct we use throughout the extension.

Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
Title summarizes what is changing
Has a news entry file (remember to thank yourself!)
Has sufficient logging.
Has telemetry for enhancements.
Unit tests & system/integration tests are added/updated
~~Test plan is updated as appropriate~~
~~package-lock.json has been regenerated by running npm install (if dependencies have changed)~~
The wiki is updated with any design decisions/details.

DonJayamanne

I think the python API needa to be re-visited.
We are still parsing strings and building a tree, in the TS layer to build and identify parent child relationships.
This needs to be done in the Python layer where the entire structure of known and prescribed, i.e. it should simply return a strongly typed json structure = a tree, making it totally unnecessary to build a tree/parse again in TS.

/cc @ericsnowcurrently

src/client/unittests/common/discoveredTests.ts

src/client/unittests/common/runner.ts

src/client/unittests/common/types.ts

codecov · 2019-03-11T18:49:07Z

Codecov Report

Merging #4695 into master will decrease coverage by 16%.
The diff coverage is 76%.

@@           Coverage Diff            @@
##           master   #4695     +/-   ##
========================================
- Coverage      77%     62%    -15%     
========================================
  Files         447     372     -75     
  Lines       21472   14476   -6996     
  Branches     3527    1145   -2382     
========================================
- Hits        16444    8860   -7584     
- Misses       5024    5413    +389     
- Partials        4     203    +199

Flag	Coverage Δ
#Linux	`?`
#Windows	`?`
#macOS	`?`

- first pass of the new test discovery API exec - trying to not change anything outside of unittest - particularly outside of pytest

- Make the run_adapter placeholder return an array - use JSON.Parse on the returned result in parser

- Add a script that just spits out test data

- Use python.pythonPath for the exe path - Put the run_adapter.py script as the first arg to the process - Add the rest of the necessary args to the script.

- Add `discover` method alongside `run` method - Focusing on running discovery via the new Python adapter - This will be the call for each test framework going forward

- New data type expected from Python `DiscoveredTestData` - New class to transform from DiscoveredTestData to Tests - Updated `pytest/parserService.ts` to use this new method of discovery

- A bit of a logic re-jig

- start testing to ensure the new discovery works the same as the old

- Fix up test data for path case variance.

- Fix bug with parser throwing on empty input - Tests for the discoveredTests - Tests fixed for the discoveryService - simplify tests for the parserService - remove a non-unit test from running during unit test runs

- Use `exec` over `execObservable` in running Python for test discovery. - Create a new interface for test discovery runner.

- Still needs some fixing.

This reverts commit 8ad18c1.

- needs to be refactored for the new parser model - tracked by GH microsoft#4735

src/client/unittests/common/discoveredTests.ts

- Many tests will need refactoring with the new discovery API

DonJayamanne

Please do not merge:

File names, folder names, suite names and function names are not returned by python code (new PR #4695 will solve this)
Extracting names from IDs is not the right approach (that's business logic that needs to sit in python code, as its the python code that knows how ids are formed) (new PR #4695 will solve this)
Id is unique identifier, and we shouldn't assume it contains names of files, suites and functions

As a result of this:

Code lenses will not work (line numbers returned by python code is not used, as user may have modified the code with auto test discover disabled)
Selection of tests for running will not display user friendly names

Summary:

We need to get python code to return the names
Else we leave code in broken state, that should not be done in PRs
We need to wait for PR #4695

DonJayamanne

A lot of the code will change due to the data structure returned by the python code.
Best to wait until that's completed.

DonJayamanne · 2019-03-13T15:32:49Z

pythonFiles/testing_tools/adapter/pytest.py

-    if parts.pop(0) != filename:
-        # TODO: What to do?
+    if os.path.normcase(parts.pop(0)) != filename:
+        # The filename of this node doesn't match the filename within the node id?


We need to file an issue about this and fix this accordingly.
/cc @ericsnowcurrently

DonJayamanne · 2019-03-13T15:33:04Z

pythonFiles/testing_tools/adapter/pytest.py

-    pytestargs.insert(0, '--collect-only')
    pytestargs.insert(0, '-pno:terminal')
+    pytestargs.insert(0, '--collect-only')
+    pytestargs.insert(0, '--cache-clear')


/cc @ericsnowcurrently

DonJayamanne · 2019-03-13T15:33:46Z

pythonFiles/tests/scripts/testing_tools_dummy.py

@@ -0,0 +1,94 @@
+import json


Where is this file being used?

DonJayamanne · 2019-03-14T00:10:27Z

@d3r3kk
Unfortunately I've got to create a branch from yours and start separately.
I just rebased master onto your branch, but I cannot force push upstream
Your commits won't be lost, so nothing lost, just that it'll end up as a new PR..hope thats ok.

DonJayamanne · 2019-03-14T00:15:30Z

@d3r3kk
Better yet, I'm going to create a new branch based off Erics branch and merge your changes on top of that.,, that way i have best of both worlds..

For #4035. This is a refactor of #4695.

d3r3kk requested review from DonJayamanne and ericsnowcurrently March 9, 2019 01:47

DonJayamanne requested changes Mar 9, 2019

View reviewed changes

d3r3kk added 24 commits March 12, 2019 13:43

Update mocks for VS Code used in unit testing

5681acf

Run a python script to obtain pytest results

0368aff

- first pass of the new test discovery API exec - trying to not change anything outside of unittest - particularly outside of pytest

Update for running local discovery of tests

f5a2341

- Make the run_adapter placeholder return an array - use JSON.Parse on the returned result in parser

Fixups to the test discovery python script/module

08c8989

- Add a script that just spits out test data

Change the runner for Pytest test discovery

317f805

- Use python.pythonPath for the exe path - Put the run_adapter.py script as the first arg to the process - Add the rest of the necessary args to the script.

Add default parameters to the pytest adapter

a9118a6

Alter how we execute the runner when doing discovery

35a889c

- Add `discover` method alongside `run` method - Focusing on running discovery via the new Python adapter - This will be the call for each test framework going forward

Get data from the Python adapter instead of stdout

be73a6a

- New data type expected from Python `DiscoveredTestData` - New class to transform from DiscoveredTestData to Tests - Updated `pytest/parserService.ts` to use this new method of discovery

Documentation

26d30fb

Documentation

177e526

- A bit of a logic re-jig

Update pytest parser test data

9160048

- start testing to ensure the new discovery works the same as the old

Fixups

e028823

Fixes to sync TestFile results of new and old parsers.

4bc1384

Further fixup to make new parser match old

c7fd311

Update test data/add test functions for the Tests results

b72bf45

- Fix up test data for path case variance.

Remove unnecessary tests

87741bf

Adding/fixing further tests

a96660d

- Fix bug with parser throwing on empty input - Tests for the discoveredTests - Tests fixed for the discoveryService - simplify tests for the parserService - remove a non-unit test from running during unit test runs

PR suggestions

afbdb3d

- Use `exec` over `execObservable` in running Python for test discovery. - Create a new interface for test discovery runner.

Make addTo_ and toTest_ methods static

0c45383

Bug fixes, unit test fixups.

60f50bc

Fix hygiene problems

140107d

news

d26fb44

Hygiene

a91ad1a

Update parsed data for test data builder for pytest

8ad18c1

- Still needs some fixing.

d3r3kk force-pushed the 4035_pytest_discovery branch from 7f7cf5f to 8ad18c1 Compare March 12, 2019 20:46

Add missing resource member of the Test_ structs.

7ebd483

d3r3kk added 2 commits March 12, 2019 17:07

Revert "Update parsed data for test data builder for pytest"

be2c925

This reverts commit 8ad18c1.

Remove tests for TestMessageService

c6cb6da

- needs to be refactored for the new parser model - tracked by GH microsoft#4735

DonJayamanne approved these changes Mar 13, 2019

View reviewed changes

src/client/unittests/common/discoveredTests.ts Outdated Show resolved Hide resolved

Fix and skip tests

797f4a3

- Many tests will need refactoring with the new discovery API

DonJayamanne requested changes Mar 13, 2019

View reviewed changes

Just comment out the whole thing

e340032

DonJayamanne requested changes Mar 13, 2019

View reviewed changes

Revert changes to Python code

7644160

DonJayamanne mentioned this pull request Mar 14, 2019

WIP - Using new Python code for discovery of Python Tests #4752

Closed

8 tasks

This was referenced Mar 18, 2019

Test discovery using Python pytest #4795

Merged

(Another) Test discovery using Python pytest #4851

Closed

DonJayamanne closed this Mar 21, 2019

DonJayamanne added a commit that referenced this pull request Apr 11, 2019

Test discovery using Python pytest (#4795)

6a12a25

For #4035. This is a refactor of #4695.

lock bot locked as resolved and limited conversation to collaborators Jul 30, 2019

Pytest discovery using new Python test discovery adapter. #4695

Pytest discovery using new Python test discovery adapter. #4695

Uh oh!

Conversation

d3r3kk commented Mar 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonJayamanne left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

DonJayamanne left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DonJayamanne left a comment

Choose a reason for hiding this comment

Uh oh!

DonJayamanne Mar 13, 2019

Choose a reason for hiding this comment

Uh oh!

DonJayamanne Mar 13, 2019

Choose a reason for hiding this comment

Uh oh!

DonJayamanne Mar 13, 2019

Choose a reason for hiding this comment

Uh oh!

DonJayamanne commented Mar 14, 2019

Uh oh!

DonJayamanne commented Mar 14, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

d3r3kk commented Mar 9, 2019 •

edited

Loading

DonJayamanne left a comment •

edited

Loading

codecov bot commented Mar 11, 2019 •

edited

Loading

DonJayamanne left a comment •

edited

Loading