Introduce PyScriptTest, an helper class to write integration tests#663
Conversation
|
Fix #617 |
|
@antocuni @marimeireles I think we are overlapping efforts here... I've started a PR to add JS unit tests and, also another one to add Python tests. Given the skeleton of this PR I'm not sure I understand the goal (what it's testing) and if we are duplicating efforts, but just wanted to give you a heads up. |
I think they are not-overlapping but complementary. If I understand correctly:
yes, the PR is still very early so I agree it's hard to see where it's headed to without reading into our minds :). The idea is to have enough scaffolding and machinery to make writing small tests very easy and natural. For example, eventually I would like to be able to write something along these lines: def test_py_button(self):
src = """
<py-button id="mybutton">
def onClick():
print('button clicked')
</py-button>
"""
self.write_page("mytest.html", src) # the rest of the HTML is automatically added around src
page = self.open("mytest.html")
page.locator('id=mybutton').click()
assert self.console[-1] = "button clicked" # or something like that |
7486d5c to
19fba27
Compare
… test Co-authored-by: Mariana Meireles <marian.meireles@gmail.com>
The test is incomplete because we need to check the DOM, but the basic idea works Co-authored-by: Mariana Meireles <marian.meireles@gmail.com>
for more information, see https://pre-commit.ci
19fba27 to
f747a95
Compare
…ich will test only the machinery
…finishes. Unfortunately, I could not find any good way to test this behavior :(
0d3d7bd to
2bf53ce
Compare
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…reover, use a more informative message
|
Thanks for all the work Antonio! Is really great ✨ :) |
|
@antocuni, I saw you changed from the two logs, one for "pyscript loading" and one for "pyodide loading" to only one "pyscript fully initialized". |
Right sorry, I should have talked to you before doing that.
|
…_run(), which makes writing pyscript tests very easy
d0a833b to
a18c940
Compare
…for some examples it seems that PyodideRuntime.initialize is not called and thus the magic message is never prited to the console. We probably need to investigate more in another PR
a18c940 to
da1475e
Compare
|
A note for those who are following the conversation in this PR: I modified the first message, which now contains a full description of this PR: #663 (comment) |
| py-button. During the page loading, the following JS exception occur, | ||
| in base.ts:BaseEvalElement.evaluate | ||
|
|
||
| [JS exception ] TypeError: Cannot use 'in' operator to search for 'runPythonAsync' in undefined |
There was a problem hiding this comment.
hmm, this happens when runtime is not loaded yet, but it gets eventually loaded after.
So, should we make this test pass? by using self.clear_errors() perhaps?
Ideally, we should not let this occur and change so that this check doesn't happen when runtime isn't ready yet.
There was a problem hiding this comment.
It's debatable. From some point of view, I agree with you that the test "works as intended" so it should pass.
But from another point of view, the fact that we get an unexpected exception indicates that something is wrong and it should be fixed. And also, without investigating I don't really know whether this is a bug in pybutton.ts or somewhere else.
This is also a consequence of not having had proper tests until now. If we followed proper TDD since the beginning, we would have caught the bug as soon as it appeared. Now we are transitioning from "no tests" to (hopefully) "full test coverage", so we will uncover more and more bugs. As a team we need to decide how strict we want to be, and my personal opinion is that we should be very strict and not let any JS exception to happen (unless there is a good reason).
by using self.clear_errors() perhaps?
this is a bad idea. Currently, it is clear that the test is supposed to fail, and if it does not fail pytest reports it as "xpass" (i.e., "unexpectedly passing").
Imagine that you add a clear_errors() at the end; then we fix the bug, but we forget to remove the clear_errors(). Weeks or months later, we introduce a bug which causes a JS exception in this test, but the test will not notice because we call clear_errors(). Better to avoid going down that route.
Ideally, we should not let this occur and change so that this check doesn't happen when runtime isn't ready yet.
yes, but this should be fixed in #673
There was a problem hiding this comment.
@fpliger what do you think of this here? I am also divided but have some inclination to let it pass with a comment to remove self.clear_errors() in future. But of course, I can be wrong.
fpliger
left a comment
There was a problem hiding this comment.
I reviewed "very fast" but overall it's a GREAT PR. Ton of value added and well documented. A couple of things that I think should be added/addressed later:
- would be good to add some timing to also have timestamps of how long things are taking (maybe it's not part of these tests but I wanted to use this say it out loud :) )
- we'll probably need to restructure the
pyscriptjs/testsfolder a little and maybe put this inpyscriptjs/integrationor something like this
| console.all.* same as above, but considering all messages, no filters | ||
| """ | ||
|
|
||
| class View: |
There was a problem hiding this comment.
Any reason for not defining it at the global scope and then passing it explicitly as class attribute?
There was a problem hiding this comment.
no big reason. It's just that this class is used/usable only inside ConsoleMessageCollection, so I didn't want to pollute the global scope unnecessarily.
…ross the test and measure the time passed since the call of self.goto; use Logger to log also the HTTP requests made to the server, which result in a nicer output
this is actually a very good idea, and was simple enough that I did in commit 18c8ec9.
As you might guess, I have opinions on that for fortunately not as strong as for other topics :). Since it seems there are no big objects, I'm going to merge it. |

EDIT: This message was initially left empty, writing it now that the PR is ready to review.
This PR is about integration tests: they use
playwrightto load HTML pages in the browser and check that PyScript works as intended, as opposed to unit tests like the ones being introduced by #665 and #661.The main goal of this PR is to introduce some machinery to make such tests easier to write, read and maintain, with some attention to capture enough information to produce useful error messages in case they fail in the CI.
In order to use the machinery, you need to subclass
tests.support.PyScriptTest, which provides several useful API calls in the formself.xxx().The most useful method is
self.pyscript_run(): it takes an HTML snippet, put it in a full HTML page with all the necessary tags to enable pyscript, open it in the browser and waits until pyscript has finished to load and execute all the tags. Once it returns, you can proceed to check that they did what you expected, like in this basic test:pyscript/pyscriptjs/tests/test_01_basic.py
Lines 7 to 20 in 3d372f7
The other helpers are described in the docstrings:
pyscript/pyscriptjs/tests/support.py
Lines 9 to 35 in 3d372f7
test_00_support.pycontains all the tests needed to check that thePyScriptTestclass itself works as intended:pyscript/pyscriptjs/tests/test_00_support.py
Lines 9 to 15 in 3d372f7
Another useful feature is automatic logging of console messages: they are collected and made available using the
self.consoleobject, which exposes a nice and pythonic API:pyscript/pyscriptjs/tests/support.py
Lines 223 to 239 in 3d372f7
Example of usage:
pyscript/pyscriptjs/tests/test_01_basic.py
Lines 22 to 38 in 3d372f7
Moreover, console messages are also printed to
stdout. This is useful for two reasons:pytest -s, you can see the messages in real timepytestwill show all the logs in the test report, making it easier to understand what's going on.This is a screenshot showing the output of
test_00_support.py:test_console:Another very important feature is the automatic detection of javascript exceptions. By default,
playwrightjust ignores them, which is not ideal: even if a test seems to work, the presence of an uncaught JS exception is likely an indicator that something is wrong and should be fixed.PyScriptTestlogs all the uncaught exceptions, and similarly to console messages they are also printed tostdoutto make debugging easier.Each test can manually call
self.check_errors(), which raises a Python exception in case JS exceptions were logged -- and for example we can put it inside awith pytest.raises():block in case we expect a JS exception to be raised.Moreover,
self.check_errors()is automatically called at the end of each method, to ensure that JS errors don't pass silently.This approach has already proved useful to find pyscript bugs. For example, in the following test the
on_click()event is correctly handled, but an unrelated JS exception is thrown, indicating what likely is a pyscript bug which should be fixed:pyscript/pyscriptjs/tests/test_py_button.py
Lines 22 to 34 in 3d372f7
Screenshot:
A final note about test filenames; currently we have:
I named them
tes_00...andtest_01...to ensure that they are run at the very beginning, and in that order. The idea is that subsequent tests rely on the features which are tested earlier, so that if something fundamental breaks, we discover it very early in the test run.This is the same approach used by e.g. hpy tests.
Once the PR is merged, we should start to write tests for all the various pyscript features. My original plan was to write more of them already in this PR, but it because big enough that it's probably a good idea to merge it first, and the add new tests in subsequent PRs.
Finally, a note for reviewers: the PR is relatively big, so it might be easier to see it commit-by-commit, because we tried to follow proper TDD and each new feature comes with its own test.