Skip to content

Warn users about perf numbers when report was created using headless chrome#2920

Closed
ebidel wants to merge 1 commit intomasterfrom
2675
Closed

Warn users about perf numbers when report was created using headless chrome#2920
ebidel wants to merge 1 commit intomasterfrom
2675

Conversation

@ebidel
Copy link
Copy Markdown
Contributor

@ebidel ebidel commented Aug 9, 2017

Fixes #2675.

screen shot 2017-08-09 at 2 21 45 pm

Chose to put this at the top of the report instead of just the perf section for a couple of reasons:

  1. makes it more visible
  2. there are perf-related audits in other categories
  3. we'd need to pipe reportJSON through a bunch of functions in report-renderer.

Should running the module also throw a warning?

@patrickhulce
Copy link
Copy Markdown
Collaborator

nice this is a great step, should we expand this for all latencies that weren't realistic?

@brendankenny
Copy link
Copy Markdown
Contributor

I can't find the issue now, but we also talked about something like this for when the run was a total disaster...e.g. navStart well after FMP in the trace

@ebidel
Copy link
Copy Markdown
Contributor Author

ebidel commented Aug 13, 2017

@patrickhulce something different than

if (!areLatenciesAll3G) {
return {
rawValue: true,
// eslint-disable-next-line max-len
debugString: `First Interactive was found at ${Util.formatMilliseconds(timeToFirstInteractive)}, however, the network request latencies were not sufficiently realistic, so the performance measurements cannot be trusted.`,
extendedInfo,
details
};
}
?

@brendankenny let's think warnings for overall disasters in another PR.

Copy link
Copy Markdown
Contributor

@brendankenny brendankenny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need something in the JSON report as well (a top-level warning or whatever)?


it('renders a warning when using headless chrome', () => {
const container = renderer._dom._document.body;
const reportJson = Object.assign(sampleResults, {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably want const reportJson = Object.assign({}, sampleResults, {...}); so sampleResults isn't modified

const reportJson = Object.assign(sampleResults, {
userAgent: 'HeadlessChrome/62.0.3180.0 Safari/537.36'
});
const output = renderer.renderReport(reportJson, container);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing the contents of the container, which is going to affect other tests in this file since setup/teardown is only done once. Should before/after above actually be beforeEach/afterEach?

// Render a warning that perf numbers may be off when using headless chrome.
if (/HeadlessChrome/.test(report.userAgent)) {
const warning = this._dom.cloneTemplate('#tmpl-lh-headless-warning', this._templateContext);
reportSection.insertBefore(warning, categories);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this move above categories (and use appendChild) so that it's in the code in the same order it will appear in the report?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add a warning-container to the report DOM permanently, and then this can easily appendChild reliably?

@patrickhulce
Copy link
Copy Markdown
Collaborator

@ebidel yeah I meant including that warning with yours as well. Having one place to call it out more prominently in addition to the current spot might prevent some of the WPT discrepancy frustration cases.

@paulirish
Copy link
Copy Markdown
Member

Do we need something in the JSON report as well (a top-level warning or whatever)?

nahhh.. no need.

@paulirish
Copy link
Copy Markdown
Member

Blocked by #1512

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants