Skip to content

Introduce PyScriptTest, an helper class to write integration tests#663

Merged
antocuni merged 39 commits into
mainfrom
antocuni/write-tests
Aug 10, 2022
Merged

Introduce PyScriptTest, an helper class to write integration tests#663
antocuni merged 39 commits into
mainfrom
antocuni/write-tests

Conversation

@antocuni

@antocuni antocuni commented Aug 4, 2022

Copy link
Copy Markdown
Contributor

EDIT: This message was initially left empty, writing it now that the PR is ready to review.

This PR is about integration tests: they use playwright to load HTML pages in the browser and check that PyScript works as intended, as opposed to unit tests like the ones being introduced by #665 and #661.

The main goal of this PR is to introduce some machinery to make such tests easier to write, read and maintain, with some attention to capture enough information to produce useful error messages in case they fail in the CI.

In order to use the machinery, you need to subclass tests.support.PyScriptTest, which provides several useful API calls in the form self.xxx().

The most useful method is self.pyscript_run(): it takes an HTML snippet, put it in a full HTML page with all the necessary tags to enable pyscript, open it in the browser and waits until pyscript has finished to load and execute all the tags. Once it returns, you can proceed to check that they did what you expected, like in this basic test:

def test_pyscript_hello(self):
self.pyscript_run(
"""
<py-script>
print('hello pyscript')
</py-script>
"""
)
# this is a very ugly way of checking the content of the DOM. If we
# find ourselves to write a lot of code in this style, we will
# probably want to write a nicer API for it.
inner_html = self.page.locator("py-script").inner_html()
pattern = r'<div id="py-.*">hello pyscript</div>'
assert re.search(pattern, inner_html)

The other helpers are described in the docstrings:

class PyScriptTest:
"""
Base class to write PyScript integration tests, based on playwright.
It provides a simple API to generate HTML files and load them in
playwright.
It also provides a Pythonic API on top of playwright for the most
common tasks; in particular:
- self.console collects all the JS console.* messages. Look at the doc
of ConsoleMessageCollection for more details.
- self.check_errors() checks that no JS errors have been thrown
- after each test, self.check_errors() is automatically run to ensure
that no JS error passes uncaught.
- self.wait_for_console waits until the specified message appears in the
console
- self.wait_for_pyscript waits until all the PyScript tags have been
evaluated
- self.pyscript_run is the main entry point for pyscript tests: it
creates an HTML page to run the specified snippet.
"""

test_00_support.py contains all the tests needed to check that the PyScriptTest class itself works as intended:

class TestSupport(PyScriptTest):
"""
These are NOT tests about PyScript.
They test the PyScriptTest class, i.e. we want to ensure that all the
testing machinery that we have works correctly.
"""

Another useful feature is automatic logging of console messages: they are collected and made available using the self.console object, which exposes a nice and pythonic API:

class ConsoleMessageCollection:
"""
Helper class to collect and expose ConsoleMessage in a Pythonic way.
Usage:
console.log.messages: list of ConsoleMessage with type=='log'
console.log.lines: list of strings
console.log.text: the whole text as single string
console.debug.* same as above, but with different types
console.info.*
console.error.*
console.warning.*
console.all.* same as above, but considering all messages, no filters
"""

Example of usage:
def test_execution_in_order(self):
"""
Check that they py-script tags are executed in the same order they are
defined
"""
# NOTE: this test relies on the fact that pyscript does not write
# anything to console.info. If we start writing to info in the future,
# we will probably need to tweak this test.
self.pyscript_run(
"""
<py-script>import js; js.console.info('one')</py-script>
<py-script>js.console.info('two')</py-script>
<py-script>js.console.info('three')</py-script>
<py-script>js.console.info('four')</py-script>
"""
)
assert self.console.info.lines == ["one", "two", "three", "four"]

Moreover, console messages are also printed to stdout. This is useful for two reasons:

  1. if you run the tests using pytest -s, you can see the messages in real time
  2. if a test fail (either locally or in CI), pytest will show all the logs in the test report, making it easier to understand what's going on.
    This is a screenshot showing the output of test_00_support.py:test_console:
    image

Another very important feature is the automatic detection of javascript exceptions. By default, playwright just ignores them, which is not ideal: even if a test seems to work, the presence of an uncaught JS exception is likely an indicator that something is wrong and should be fixed.
PyScriptTest logs all the uncaught exceptions, and similarly to console messages they are also printed to stdout to make debugging easier.
Each test can manually call self.check_errors(), which raises a Python exception in case JS exceptions were logged -- and for example we can put it inside a with pytest.raises(): block in case we expect a JS exception to be raised.
Moreover, self.check_errors() is automatically called at the end of each method, to ensure that JS errors don't pass silently.
This approach has already proved useful to find pyscript bugs. For example, in the following test the on_click() event is correctly handled, but an unrelated JS exception is thrown, indicating what likely is a pyscript bug which should be fixed:

self.pyscript_run(
"""
<py-button label="my button">
import js
def on_click(evt):
js.console.info('clicked!')
</py-button>
"""
)
assert self.console.info.lines == []
self.page.locator("text=my button").click()
self.page.locator("text=my button").click()
assert self.console.info.lines == ["clicked!", "clicked!"]

Screenshot:
image

A final note about test filenames; currently we have:

$ ls -1 test_*.py
test_00_support.py
test_01_basic.py
test_examples.py
test_py_button.py

I named them tes_00... and test_01... to ensure that they are run at the very beginning, and in that order. The idea is that subsequent tests rely on the features which are tested earlier, so that if something fundamental breaks, we discover it very early in the test run.
This is the same approach used by e.g. hpy tests.

Once the PR is merged, we should start to write tests for all the various pyscript features. My original plan was to write more of them already in this PR, but it because big enough that it's probably a good idea to merge it first, and the add new tests in subsequent PRs.

Finally, a note for reviewers: the PR is relatively big, so it might be easier to see it commit-by-commit, because we tried to follow proper TDD and each new feature comes with its own test.

@marimeireles marimeireles added the sprint issue has been pulled into current sprint and is actively being worked on label Aug 4, 2022
@marimeireles

Copy link
Copy Markdown
Contributor

Fix #617

@marimeireles marimeireles linked an issue Aug 4, 2022 that may be closed by this pull request
@fpliger

fpliger commented Aug 4, 2022

Copy link
Copy Markdown
Contributor

@antocuni @marimeireles I think we are overlapping efforts here... I've started a PR to add JS unit tests and, also another one to add Python tests.

Given the skeleton of this PR I'm not sure I understand the goal (what it's testing) and if we are duplicating efforts, but just wanted to give you a heads up.

@antocuni

antocuni commented Aug 4, 2022

Copy link
Copy Markdown
Contributor Author

@antocuni @marimeireles I think we are overlapping efforts here... I've started a PR to add JS unit tests and, also another one to add Python tests.

I think they are not-overlapping but complementary. If I understand correctly:

Given the skeleton of this PR I'm not sure I understand the goal (what it's testing) and if we are duplicating efforts, but just wanted to give you a heads up.

yes, the PR is still very early so I agree it's hard to see where it's headed to without reading into our minds :). The idea is to have enough scaffolding and machinery to make writing small tests very easy and natural. For example, eventually I would like to be able to write something along these lines:

def test_py_button(self):
    src = """
    <py-button id="mybutton">
        def onClick():
            print('button clicked')
    </py-button>
    """
    self.write_page("mytest.html", src) # the rest of the HTML is automatically added around src
    page = self.open("mytest.html")
    page.locator('id=mybutton').click()
    assert self.console[-1] = "button clicked" # or something like that

@antocuni antocuni force-pushed the antocuni/write-tests branch from 7486d5c to 19fba27 Compare August 5, 2022 07:52
antocuni and others added 3 commits August 5, 2022 09:55
… test

Co-authored-by: Mariana Meireles <marian.meireles@gmail.com>
The test is incomplete because we need to check the DOM, but the basic idea works

Co-authored-by: Mariana Meireles <marian.meireles@gmail.com>
@antocuni antocuni force-pushed the antocuni/write-tests branch from 19fba27 to f747a95 Compare August 5, 2022 07:56
@antocuni antocuni force-pushed the antocuni/write-tests branch from 0d3d7bd to 2bf53ce Compare August 5, 2022 12:47
@marimeireles

Copy link
Copy Markdown
Contributor

Thanks for all the work Antonio! Is really great ✨ :)

@marimeireles

marimeireles commented Aug 6, 2022

Copy link
Copy Markdown
Contributor

@antocuni, I saw you changed from the two logs, one for "pyscript loading" and one for "pyodide loading" to only one "pyscript fully initialized".
The reason why I had these two logs is to make it easier to the debug. That problem I described to you and that we arrived to the conclusion that was happening because of the delay on the tests on checking the HTML, it'd have been a bit easier for me to debug if I knew exactly which part of pyscript wasn't loaded.
Maybe we can change the '===PYSCRIPT LOADED=== to '===PYSCRIPT FULLY LOADED===, if you think it's better.
What do you think about it? Do you have a different opinion on the number of logs here?

@antocuni

antocuni commented Aug 6, 2022

Copy link
Copy Markdown
Contributor Author

@antocuni, I saw you changed from the two logs, one for "pyscript loading" and one for "pyodide loading" to only one "pyscript fully initialized". The reason why I had these two logs is to make it easier to the debug. That problem I described to you and that we arrived to the conclusion that was happening because of the delay on the tests on checking the HTML, it'd have been a bit easier for me to debug if I knew exactly which part of pyscript wasn't loaded. Maybe we can change the '===PYSCRIPT LOADED=== to '===PYSCRIPT FULLY LOADED===, if you think it's better. What do you think about it? Do you have a different opinion on the number of logs here?

Right sorry, I should have talked to you before doing that.
My reasoning was the following:

  1. in general, we should at some point rationalize our logging, because currently it's a mess; but that's not the job of this PR

  2. ===PYSCRIPT LOADED=== was here:

    console.log('loading pyscript...');
    const output = await pyodide.runPythonAsync(pyscript);
    if (output !== undefined) {
    console.log(output);
    }
    console.log('done setting up environment');
    console.log('===PYSCRIPT LOADED===');

    The new log doesn't add any new information, because it comes directly after done setting up environment. But more importantly, it's a lie :), because at that point in time pyscript has not finished loading yet, at least not in the expected sense IMHO. If you look further in the logs you see that there are still a lot of things happening in the page. I guess the confusion arises from loading pyscript... at line 25, but this should really be importing the pyscript module..., which is just a small fraction of the whole "loading pyscript".

  3. Similarly, ===PYODIDE LOADED=== was here:

    // now we call all post initializers AFTER we actually executed all page scripts
    loader?.log('Running post initializers...');
    if (appConfig_ && appConfig_.autoclose_loader) {
    loader?.close();
    }
    for (const initializer of postInitializers_) {
    await initializer();
    }
    console.log('===PYODIDE LOADED===');

    But again, it's poorly worded: pyodide is fully loaded much earlier, at line 81. That's why I reworded to something more precise, and also more informative to the user which looks at the console.

@antocuni antocuni force-pushed the antocuni/write-tests branch from d0a833b to a18c940 Compare August 8, 2022 21:57
…for some examples it seems that PyodideRuntime.initialize is not called and thus the magic message is never prited to the console. We probably need to investigate more in another PR
@antocuni antocuni force-pushed the antocuni/write-tests branch from a18c940 to da1475e Compare August 9, 2022 10:21
@antocuni antocuni changed the title WIP: start to write tests Introduce PyScriptTest, an helper class to write integration tests Aug 9, 2022
@antocuni

antocuni commented Aug 9, 2022

Copy link
Copy Markdown
Contributor Author

A note for those who are following the conversation in this PR: I modified the first message, which now contains a full description of this PR: #663 (comment)

@antocuni antocuni marked this pull request as ready for review August 9, 2022 13:06
Comment thread pyscriptjs/tests/support.py Outdated
Comment thread pyscriptjs/tests/support.py
py-button. During the page loading, the following JS exception occur,
in base.ts:BaseEvalElement.evaluate

[JS exception ] TypeError: Cannot use 'in' operator to search for 'runPythonAsync' in undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, this happens when runtime is not loaded yet, but it gets eventually loaded after.
So, should we make this test pass? by using self.clear_errors() perhaps?
Ideally, we should not let this occur and change so that this check doesn't happen when runtime isn't ready yet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's debatable. From some point of view, I agree with you that the test "works as intended" so it should pass.
But from another point of view, the fact that we get an unexpected exception indicates that something is wrong and it should be fixed. And also, without investigating I don't really know whether this is a bug in pybutton.ts or somewhere else.

This is also a consequence of not having had proper tests until now. If we followed proper TDD since the beginning, we would have caught the bug as soon as it appeared. Now we are transitioning from "no tests" to (hopefully) "full test coverage", so we will uncover more and more bugs. As a team we need to decide how strict we want to be, and my personal opinion is that we should be very strict and not let any JS exception to happen (unless there is a good reason).

by using self.clear_errors() perhaps?

this is a bad idea. Currently, it is clear that the test is supposed to fail, and if it does not fail pytest reports it as "xpass" (i.e., "unexpectedly passing").
Imagine that you add a clear_errors() at the end; then we fix the bug, but we forget to remove the clear_errors(). Weeks or months later, we introduce a bug which causes a JS exception in this test, but the test will not notice because we call clear_errors(). Better to avoid going down that route.

Ideally, we should not let this occur and change so that this check doesn't happen when runtime isn't ready yet.

yes, but this should be fixed in #673

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fpliger what do you think of this here? I am also divided but have some inclination to let it pass with a comment to remove self.clear_errors() in future. But of course, I can be wrong.

@fpliger fpliger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed "very fast" but overall it's a GREAT PR. Ton of value added and well documented. A couple of things that I think should be added/addressed later:

  • would be good to add some timing to also have timestamps of how long things are taking (maybe it's not part of these tests but I wanted to use this say it out loud :) )
  • we'll probably need to restructure the pyscriptjs/tests folder a little and maybe put this in pyscriptjs/integration or something like this

console.all.* same as above, but considering all messages, no filters
"""

class View:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for not defining it at the global scope and then passing it explicitly as class attribute?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no big reason. It's just that this class is used/usable only inside ConsoleMessageCollection, so I didn't want to pollute the global scope unnecessarily.

…ross the test and measure the time passed since the call of self.goto; use Logger to log also the HTTP requests made to the server, which result in a nicer output
@antocuni

Copy link
Copy Markdown
Contributor Author

I reviewed "very fast" but overall it's a GREAT PR. Ton of value added and well documented. A couple of things that I think should be added/addressed later:

  • would be good to add some timing to also have timestamps of how long things are taking (maybe it's not part of these tests but I wanted to use this say it out loud :) )

this is actually a very good idea, and was simple enough that I did in commit 18c8ec9.
Screenshot of running test_pyscript_hello:
image

  • we'll probably need to restructure the pyscriptjs/tests folder a little and maybe put this in pyscriptjs/integration or something like this

As you might guess, I have opinions on that for fortunately not as strong as for other topics :).
But yes, I agree we can discuss it separately.

Since it seems there are no big objects, I'm going to merge it.

@antocuni antocuni merged commit 513dfe0 into main Aug 10, 2022
@antocuni antocuni deleted the antocuni/write-tests branch August 10, 2022 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sprint issue has been pulled into current sprint and is actively being worked on

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Create infrastructure and write the first Python unit test

4 participants