Skip to content

[labs/testing] testing package with SSR testing utilities#2957

Merged
augustjk merged 36 commits intomainfrom
testing-util
Jun 29, 2022
Merged

[labs/testing] testing package with SSR testing utilities#2957
augustjk merged 36 commits intomainfrom
testing-util

Conversation

@augustjk
Copy link
Copy Markdown
Member

Resolves #2867

This PR adds a @lit-labs/testing package.

The litSsrPlugin plugin for Web Test Runner registers the lit-ssr-render server command that takes the template and module paths, and renders them within a worker thread in the same manner as eleventy-plugin-lit.

The server command is utilized by ssrFixture to be called in the test file supplying lit-html template to render and component definition module path(s).

The README has example usage.

Still to do are to add tests, and a fixture function with client side rendering with the same signature for ease of writing tests across the different rendering methods.

This is still an early draft and I'm looking for feedback on

  • The structure of the package and exports.
  • Possibly a better way to handle the relative paths of the modules.
  • How to test -- I'm currently thinking doing an integration test with a whole sample component, running WTR with Chrome, FF, Safari, with the plugin registered and running tests utilizing all the fixture function. I'm unsure how to unit test this, if necessary.

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented May 26, 2022

🦋 Changeset detected

Latest commit: cd6144e

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 26, 2022

📊 Tachometer Benchmark Results

Summary

⏳ Benchmarks are currently running. Results below are out of date.

nop-update

  • lit-html-kitchen-sink: unsure 🔍 -1% - +2% (-0.28ms - +0.55ms)
    this-change vs tip-of-tree

render

  • lit-element-list: unsure 🔍 -2% - +1% (-1.67ms - +0.80ms)
    this-change vs tip-of-tree
  • lit-html-kitchen-sink: unsure 🔍 -8% - +4% (-2.63ms - +1.18ms)
    this-change vs tip-of-tree
  • lit-html-repeat: unsure 🔍 -3% - +2% (-0.33ms - +0.22ms)
    this-change vs tip-of-tree
  • lit-html-template-heavy: unsure 🔍 -2% - +0% (-0.91ms - +0.10ms)
    this-change vs tip-of-tree
  • reactive-element-list: unsure 🔍 -11% - +12% (-9.85ms - +10.53ms)
    this-change vs tip-of-tree

update

  • lit-element-list: unsure 🔍 -7% - +1% (-55.97ms - +7.85ms)
    this-change vs tip-of-tree
  • lit-html-kitchen-sink: faster ✔ 0% - 5% (0.12ms - 3.81ms)
    this-change vs tip-of-tree
  • lit-html-repeat: unsure 🔍 -4% - +5% (-14.65ms - +20.21ms)
    this-change vs tip-of-tree
  • lit-html-template-heavy: unsure 🔍 -1% - +1% (-1.44ms - +1.74ms)
    this-change vs tip-of-tree
  • reactive-element-list: unsure 🔍 -6% - +16% (-48.75ms - +141.17ms)
    this-change vs tip-of-tree

update-reflect

  • lit-element-list: unsure 🔍 -1% - +0% (-8.17ms - +1.84ms)
    this-change vs tip-of-tree
  • reactive-element-list: unsure 🔍 -2% - +1% (-19.13ms - +12.91ms)
    this-change vs tip-of-tree

Results

⏳ Benchmarks are currently running. Results below are out of date.
lit-element-list

render

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
76.54ms - 78.10ms-unsure 🔍
-2% - +1%
-1.67ms - +0.80ms
faster ✔
18% - 20%
16.86ms - 18.84ms
tip-of-tree
tip-of-tree
76.80ms - 78.72msunsure 🔍
-1% - +2%
-0.80ms - +1.67ms
-faster ✔
17% - 19%
16.28ms - 18.56ms
previous-release
previous-release
94.56ms - 95.79msslower ❌
22% - 25%
16.86ms - 18.84ms
slower ❌
21% - 24%
16.28ms - 18.56ms
-

update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
797.87ms - 825.95ms-unsure 🔍
-7% - +1%
-55.97ms - +7.85ms
faster ✔
34% - 37%
426.23ms - 461.43ms
tip-of-tree
tip-of-tree
807.31ms - 864.63msunsure 🔍
-1% - +7%
-7.85ms - +55.97ms
-faster ✔
31% - 36%
389.21ms - 450.33ms
previous-release
previous-release
1245.13ms - 1266.35msslower ❌
52% - 58%
426.23ms - 461.43ms
slower ❌
45% - 56%
389.21ms - 450.33ms
-

update-reflect

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
808.88ms - 814.40ms-unsure 🔍
-1% - +0%
-8.17ms - +1.84ms
faster ✔
8% - 9%
71.31ms - 82.97ms
tip-of-tree
tip-of-tree
810.62ms - 818.98msunsure 🔍
-0% - +1%
-1.84ms - +8.17ms
-faster ✔
8% - 9%
67.36ms - 80.60ms
previous-release
previous-release
883.64ms - 893.91msslower ❌
9% - 10%
71.31ms - 82.97ms
slower ❌
8% - 10%
67.36ms - 80.60ms
-
lit-html-kitchen-sink

render

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
30.16ms - 30.56ms-unsure 🔍
-8% - +4%
-2.63ms - +1.18ms
faster ✔
12% - 15%
4.03ms - 5.32ms
tip-of-tree
tip-of-tree
29.19ms - 32.98msunsure 🔍
-4% - +9%
-1.18ms - +2.63ms
-faster ✔
6% - 17%
1.96ms - 5.94ms
previous-release
previous-release
34.42ms - 35.65msslower ❌
13% - 18%
4.03ms - 5.32ms
slower ❌
6% - 20%
1.96ms - 5.94ms
-

update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
80.03ms - 82.22ms-faster ✔
0% - 5%
0.12ms - 3.81ms
faster ✔
3% - 8%
2.51ms - 7.21ms
tip-of-tree
tip-of-tree
81.61ms - 84.58msslower ❌
0% - 5%
0.12ms - 3.81ms
-faster ✔
0% - 6%
0.34ms - 5.45ms
previous-release
previous-release
83.90ms - 88.07msslower ❌
3% - 9%
2.51ms - 7.21ms
slower ❌
0% - 7%
0.34ms - 5.45ms
-

nop-update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
25.74ms - 26.32ms-unsure 🔍
-1% - +2%
-0.28ms - +0.55ms
faster ✔
10% - 17%
3.00ms - 5.30ms
tip-of-tree
tip-of-tree
25.60ms - 26.19msunsure 🔍
-2% - +1%
-0.55ms - +0.28ms
-faster ✔
11% - 18%
3.13ms - 5.43ms
previous-release
previous-release
29.06ms - 31.29msslower ❌
11% - 20%
3.00ms - 5.30ms
slower ❌
12% - 21%
3.13ms - 5.43ms
-
lit-html-repeat

render

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
10.83ms - 11.25ms-unsure 🔍
-3% - +2%
-0.33ms - +0.22ms
faster ✔
7% - 11%
0.88ms - 1.34ms
tip-of-tree
tip-of-tree
10.92ms - 11.27msunsure 🔍
-2% - +3%
-0.22ms - +0.33ms
-faster ✔
7% - 10%
0.86ms - 1.26ms
previous-release
previous-release
12.06ms - 12.24msslower ❌
8% - 12%
0.88ms - 1.34ms
slower ❌
8% - 11%
0.86ms - 1.26ms
-

update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
383.74ms - 416.14ms-unsure 🔍
-4% - +5%
-14.65ms - +20.21ms
faster ✔
19% - 28%
98.87ms - 148.60ms
tip-of-tree
tip-of-tree
390.72ms - 403.60msunsure 🔍
-5% - +4%
-20.21ms - +14.65ms
-faster ✔
21% - 27%
106.58ms - 146.45ms
previous-release
previous-release
504.81ms - 542.54msslower ❌
24% - 38%
98.87ms - 148.60ms
slower ❌
27% - 37%
106.58ms - 146.45ms
-
lit-html-template-heavy

render

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
54.44ms - 55.15ms-unsure 🔍
-2% - +0%
-0.91ms - +0.10ms
faster ✔
16% - 18%
10.33ms - 12.05ms
tip-of-tree
tip-of-tree
54.85ms - 55.56msunsure 🔍
-0% - +2%
-0.10ms - +0.91ms
-faster ✔
15% - 17%
9.93ms - 11.64ms
previous-release
previous-release
65.21ms - 66.77msslower ❌
19% - 22%
10.33ms - 12.05ms
slower ❌
18% - 21%
9.93ms - 11.64ms
-

update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
125.10ms - 127.44ms-unsure 🔍
-1% - +1%
-1.44ms - +1.74ms
faster ✔
13% - 15%
18.26ms - 21.57ms
tip-of-tree
tip-of-tree
125.05ms - 127.20msunsure 🔍
-1% - +1%
-1.74ms - +1.44ms
-faster ✔
13% - 15%
18.47ms - 21.65ms
previous-release
previous-release
145.02ms - 147.36msslower ❌
14% - 17%
18.26ms - 21.57ms
slower ❌
15% - 17%
18.47ms - 21.65ms
-
reactive-element-list

render

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
79.61ms - 93.79ms-unsure 🔍
-11% - +12%
-9.85ms - +10.53ms
unsure 🔍
-12% - +11%
-10.47ms - +9.81ms
tip-of-tree
tip-of-tree
79.05ms - 93.68msunsure 🔍
-12% - +11%
-10.53ms - +9.85ms
-unsure 🔍
-13% - +11%
-10.97ms - +9.63ms
previous-release
previous-release
79.78ms - 94.28msunsure 🔍
-11% - +12%
-9.81ms - +10.47ms
unsure 🔍
-11% - +13%
-9.63ms - +10.97ms
-

update

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
884.36ms - 1031.81ms-unsure 🔍
-6% - +16%
-48.75ms - +141.17ms
unsure 🔍
-2% - +19%
-12.78ms - +162.16ms
tip-of-tree
tip-of-tree
852.03ms - 971.72msunsure 🔍
-14% - +5%
-141.17ms - +48.75ms
-unsure 🔍
-6% - +12%
-47.65ms - +104.61ms
previous-release
previous-release
836.33ms - 930.46msunsure 🔍
-16% - +1%
-162.16ms - +12.78ms
unsure 🔍
-11% - +5%
-104.61ms - +47.65ms
-

update-reflect

VersionAvg timevs this-change
vs tip-of-tree
tip-of-tree
vs previous-release
previous-release
this-change
879.35ms - 901.54ms-unsure 🔍
-2% - +1%
-19.13ms - +12.91ms
unsure 🔍
-2% - +2%
-15.76ms - +16.75ms
tip-of-tree
tip-of-tree
881.99ms - 905.11msunsure 🔍
-1% - +2%
-12.91ms - +19.13ms
-unsure 🔍
-1% - +2%
-12.97ms - +20.18ms
previous-release
previous-release
878.07ms - 901.83msunsure 🔍
-2% - +2%
-16.75ms - +15.76ms
unsure 🔍
-2% - +1%
-20.18ms - +12.97ms
-

tachometer-reporter-action v2 for Benchmarks

@kevinpschaaf
Copy link
Copy Markdown
Member

Possibly a better way to handle the relative paths of the modules.

It seems like we'll need to require the fixture caller to pass their import.meta.url in order to allow them to use relative paths, something like:

    const el = await ssrFixture(html`<my-element></my-element>`, {
      base: import.meta.url, // path to resolve relative modules against
      modules: ['./my-element.js'], // path to component definition, relative to base (or cwd if not specified)
      hydrate: false, // whether to hydrate the component after loading to document (default: true)
    });

On the client, it would just import(new URL(module[i], base).href), and on the server it should be able to work out where to import it from relative to where the server is serving from?

@kevinpschaaf
Copy link
Copy Markdown
Member

How to test -- I'm currently thinking doing an integration test with a whole sample component, running WTR with Chrome, FF, Safari, with the plugin registered and running tests utilizing all the fixture function. I'm unsure how to unit test this, if necessary

Yeah, doing this as an integration test seems like the most bang-for-buck. Seems like you'll want to test a variety of component scenarios, obv a happy case where it all passes, but also some cases where we expect the tests to fail e.g. to ensure we catch when users do SSR-incompatible things. I'd expect main focus to be on the latter, since the labs/ssr integration tests themselves focus on all the happy cases.

^^ points out that we'll want to nicely pipe errors raised from render-lit-html:render() on the server back through to the client via a rejection on the async fixture; that should allow writing try/catch blocks in the integration test to ensure they fail when expected.

(hmmm, now that I think about it)

I'd expect main focus to be on the latter, since the SSR tests themselves focus on all the happy cases.

Seems like we might just be able to switch all the labs/ssr integration tests to use this and use a normal WTR setup; it would just require all the element definitions in the LitElement tests to be moved out to their own modules. But that can happen separately after we get some basic tests for this landed.

@augustjk augustjk force-pushed the testing-util branch 2 times, most recently from c3cecdf to 5466d84 Compare May 31, 2022 17:12
@augustjk augustjk force-pushed the testing-util branch 3 times, most recently from 38c3813 to cc44c30 Compare June 8, 2022 17:43
@augustjk augustjk marked this pull request as ready for review June 8, 2022 18:23
Copy link
Copy Markdown
Member

@aomarks aomarks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really great! Pretty minor comments.

@augustjk augustjk requested a review from aomarks June 17, 2022 00:31
Copy link
Copy Markdown
Member

@aomarks aomarks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just some minor comments, LGTM otherwise.

}
);

// TODO(augustinekim) Clean up the container from the document
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will we know when to do the cleanup? I wonder if we should have a cleanupFixtures or something, which tracks all fixtures in a global, that users can put in their after callbacks?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

open-wc's implementation keeps the containers in a module scoped array https://github.com/open-wc/open-wc/blob/master/packages/testing-helpers/src/fixtureWrapper.js and the fixture import has a side effect of registering the cleanup on afterEach and teardown if mocha is registered globally. https://github.com/open-wc/open-wc/blob/master/packages/testing-helpers/src/fixture.js

We could also provide the cleanup function for users to call themselves if they care about cleaning up.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to a cleanupFixtures function, that makes sense.

We could also just have fixture keep a module-scoped lastContainer and remove it before creating another so we're not accumulating loads of containers during a long suite, if it's safe to assume people don't generally create multiple fixtures in tests (?, not sure about that).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a cleanupFixtures for now.

I haven't seen cases of users creating multiple fixtures in a single test but don't want to discount that it might happen. I don't feel like trying to automatically clean up is important enough to potentially break that particular use case.

augustjk and others added 2 commits June 22, 2022 15:29
@augustjk augustjk requested a review from kevinpschaaf June 23, 2022 21:53
Copy link
Copy Markdown
Member

@kevinpschaaf kevinpschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor suggestions and clarifications; feel free to consider/resolve, but I'm good merging this based on your judgement on those.

}
);

// TODO(augustinekim) Clean up the container from the document
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to a cleanupFixtures function, that makes sense.

We could also just have fixture keep a module-scoped lastContainer and remove it before creating another so we're not accumulating loads of containers during a long suite, if it's safe to assume people don't generally create multiple fixtures in tests (?, not sure about that).

/http:\/\/localhost.+(?=\?wtr-session-id)/
);
if (!match) {
throw new Error('Could not find call site for csrFixture');
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what cases do we expect this to fail? It's unclear in the readme if users are expected to pass import.meta.url or if they can reasonably expect this to work without having to pass that. Maybe clarify?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't anticipate us to ever reach this unless how the stack is generated gets changed (or the regex somehow fails for some browser besides chrome, firefox, webkit) or if web test runner changes the query to something other than wtr-session-id.

I've updated the readme to expand more on the base option. Leaving it empty should have the same behavior as passing in import.meta.url.

render(template, container);

// Awaiting for the next microtask to ensure contents are rendered.
await undefined;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what the open-wc fixture does, but it might be good to wait a requestAnimationFrame instead, to ensure the entire tree of components (if there are multiple levels) are done rendering...?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/open-wc/open-wc/blob/master/packages/testing-helpers/src/elementUpdated.js

open-wc looks for specific promises to await on and falls back to requestAnimationFrame if they don't exist. Perhaps we could do something similar and look for updateComplete and await that.

(strings as {raw: TemplateStringsArray['raw']}).raw = strings;

let rendered = '';
for (const str of render({...template, strings}, {deferHydration: true})) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not familiar with how workers work... since this is all executed in module scope, if it throws, then we'll get a worker.on('error') callback?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's correct. Currently the wtr server command simply fails for those cases so the fixture function call will then throw. We see that happen in the bad-element test.

export function litSsrPlugin(): TestRunnerPlugin<Payload> {
return {
name: 'lit-ssr-plugin',
async executeCommand({command, payload}) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking for this PR, but would love to think about how this could be factored a little differently so that it could be called from e.g. a koa/express middleware as well, so we're not tied directly to web-test-runner. Should be straightforward, I think?

Copy link
Copy Markdown
Member

@aomarks aomarks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending 3 comments!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[labs/ssr] Initial setup of SSR fixture package

3 participants