core: gather source maps by connorjclark · Pull Request #9101 · GoogleChrome/lighthouse

connorjclark · 2019-06-01T01:13:58Z

Gather source maps.

Nothing done with them yet.

connorjclark · 2019-06-01T01:14:04Z

WIP b/c no tests, and:

I want to put source maps in a new artifact, because gathering them is a lot of network requests and parsing that should only be done if an audit explicitly needs it - which, right now, does not exist.

It could be in a new gatherer, but I'd like to use the Debugger.onScriptParsed to collect the sourceMapURL (instead of parsing it ourselves). So we can add sourceMapURL to ScriptElements, and make a computed artifact, cool. But we don't have the driver in computed artifacts (afaict), so can't make fetch requests in the page to collect the source maps.

Is there a nice way to pass the driver to a computed artifact? Or is it better to forgo Debugger.onScriptParsed and just parse for sourceMapURL?

patrickhulce · 2019-06-01T02:27:01Z

Is there a nice way to pass the driver to a computed artifact?

Sadly, no. Computed artifacts are firmly in audit phase territory and there's no way to have dependencies between artifacts unless one is a BaseArtifact

Or is it better to forgo Debugger.onScriptParsed and just parse for sourceMapURL?

I'm not sure I understand the decision here, since it sounds like if you needed a separate gatherer either way you could have the separate one listen on Debugger.onScriptParsed for now? At some point in the future it might make sense to merge these two if the set of audits that need them overlap significantly enough.

connorjclark · 2019-06-01T03:55:35Z

I'm not sure I understand the decision here, since it sounds like if you needed a separate gatherer either way you could have the separate one listen on Debugger.onScriptParsed for now?

Failed to realize it's not required to do that in the script-elements. yeah separate gatherer makes 100% sense now.

connorjclark · 2019-06-01T05:38:31Z

+const a = 1;
+// Use this obnoxious URL because using a non-existent url from localhost will return a
+// bunch of HTML, which will be parsed, and no fetch error occurs.
+//# sourceMappingURL=http://www.this-will-not-exist-blah-go-rockets.com/some-map.js.map


i guess i could do http://localhost:1234 ...

go-rockets

😱

haha 😉 to your actual point though, :1234 is the default for parcel which I tend to have running so that might cause something unexpected too ;) haha

how about an entirely invalid URL? or is that then testing something different you're not concerned about?

i suppose anything that results in a fetch error will work, including a malformed URL.

go-rockets

Did that just to get a rise out of you ;)

codecov · 2019-06-01T06:50:39Z

Codecov Report

Merging #9101 into master will decrease coverage by 0.01%.
The diff coverage is 94.87%.

@@            Coverage Diff             @@
##           master    #9101      +/-   ##
==========================================
- Coverage   91.44%   91.43%   -0.02%     
==========================================
  Files         291      292       +1     
  Lines        9929     9981      +52     
==========================================
+ Hits         9080     9126      +46     
- Misses        849      855       +6

Flag	Coverage Δ
#smoke	`84.43% <94.87%> (-0.33%)`	⬇️
#unit	`89.24% <15.38%> (-0.29%)`	⬇️

Impacted Files	Coverage Δ
lighthouse-core/config/default-config.js	`87.5% <ø> (ø)`	⬆️
lighthouse-core/gather/gatherers/source-maps.js	`94.87% <94.87%> (ø)`
lighthouse-core/audits/user-timings.js	`96% <0%> (-4%)`	⬇️
...house-core/computed/metrics/lantern-speed-index.js	`97.14% <0%> (-2.86%)`	⬇️
lighthouse-core/gather/gather-runner.js	`99.05% <0%> (-0.95%)`	⬇️
lighthouse-core/lib/manifest-parser.js	`90.55% <0%> (+1.07%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0eacf06...5da78cf. Read the comment docs.

connorjclark · 2019-07-23T05:28:51Z

+      sourceMapUrl: isSourceMapADataUri ? undefined : sourceMapUrl,
+      // map is undefined, unless there wasn't an error.
+      map: undefined,
+      ...sourceMapOrError,


idk is this a sin?

it's definitely approaching "hard to follow" :)

Something like this might be a little easier to follow, possibly:

/** * @param {Driver} driver * @param {LH.Crdp.Debugger.ScriptParsedEvent} event * @return {Promise<LH.Artifacts.SourceMap>} */ async _retrieveMapFromScriptParsedEvent(driver, event) { if (!event.sourceMapURL) { throw new Error('precondition failed: event.sourceMapURL should exist'); } // `sourceMapURL` is simply the URL found in either a magic comment or an x-sourcemap header. // It has not been resolved to a base url. const isSourceMapADataUri = event.sourceMapURL.startsWith('data:'); const scriptUrl = event.url; const rawSourceMapUrl = isSourceMapADataUri ? event.sourceMapURL : this._resolveUrl(event.sourceMapURL, scriptUrl); if (!rawSourceMapUrl) { return { scriptUrl, errorMessage: `Could not resolve map url: ${event.sourceMapURL}`, }; } // sourceMapUrl isn't included in the the artifact if it was a data URL. const sourceMapUrl = isSourceMapADataUri ? undefined : rawSourceMapUrl; try { const map = isSourceMapADataUri ? this.parseSourceMapFromDataUrl(rawSourceMapUrl) : await this.fetchSourceMapInPage(driver, rawSourceMapUrl); return { scriptUrl, sourceMapUrl, map, }; } catch (err) { return { scriptUrl, sourceMapUrl, errorMessage: err.toString(), }; } }

googlebot · 2019-07-24T01:19:41Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

connorjclark · 2019-07-24T17:07:48Z

Considerable work needs to be done to support fetching arbitrary resources outside the page context for each of our environments. We have:

node
devtools frontend
lightrider

.

is easy - the http module.
is also easy - DevTools has a host API for this.
is not so easy.

Options moving forward:

a) land as-is, some maps can't be fetched due to CORs. For the audit that will come after, we can ignore fetch errors.
b) hold off landing completely until this is well supported.

I vote for a) to keep the PR smaller.

googlebot · 2019-07-24T17:53:08Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

connorjclark · 2019-07-24T17:56:42Z

alternatively, could do a hybrid: support node fully using http module and fallback to in-page fetch in other environments (or just disable).

brendankenny · 2019-07-24T18:36:42Z

a) land as-is, some maps can't be fetched due to CORs. For the audit that will come after, we can ignore fetch errors.
b) hold off landing completely until this is well supported.

I vote for a) to keep the PR smaller.

I definitely don't think we should block on that, at worst it's only as bad as if they hadn't published their source maps

brendankenny

looking good, I think just these last things (+ axing the logging)

brendankenny · 2019-07-25T00:45:57Z

+      sourceMapUrl: isSourceMapADataUri ? undefined : sourceMapUrl,
+      // map is undefined, unless there wasn't an error.
+      map: undefined,
+      ...sourceMapOrError,


it's definitely approaching "hard to follow" :)

Something like this might be a little easier to follow, possibly:

/** * @param {Driver} driver * @param {LH.Crdp.Debugger.ScriptParsedEvent} event * @return {Promise<LH.Artifacts.SourceMap>} */ async _retrieveMapFromScriptParsedEvent(driver, event) { if (!event.sourceMapURL) { throw new Error('precondition failed: event.sourceMapURL should exist'); } // `sourceMapURL` is simply the URL found in either a magic comment or an x-sourcemap header. // It has not been resolved to a base url. const isSourceMapADataUri = event.sourceMapURL.startsWith('data:'); const scriptUrl = event.url; const rawSourceMapUrl = isSourceMapADataUri ? event.sourceMapURL : this._resolveUrl(event.sourceMapURL, scriptUrl); if (!rawSourceMapUrl) { return { scriptUrl, errorMessage: `Could not resolve map url: ${event.sourceMapURL}`, }; } // sourceMapUrl isn't included in the the artifact if it was a data URL. const sourceMapUrl = isSourceMapADataUri ? undefined : rawSourceMapUrl; try { const map = isSourceMapADataUri ? this.parseSourceMapFromDataUrl(rawSourceMapUrl) : await this.fetchSourceMapInPage(driver, rawSourceMapUrl); return { scriptUrl, sourceMapUrl, map, }; } catch (err) { return { scriptUrl, sourceMapUrl, errorMessage: err.toString(), }; } }

brendankenny · 2019-07-25T01:08:12Z

+        }
+
+        const value = fetchError ?
+          Object.assign(new Error(), {message: fetchError, __failedInBrowser: true}) :


ah, it's because of

lighthouse/lighthouse-core/gather/driver.js

Line 541 in 21d00ac

return Promise.reject(Object.assign(new Error(), value));

(since fetchError.message isn't enumerable when message is set via the Error constructor, so it isn't copied over on that line)

fetchError could be a plain object like is done in the real pageFunctions.wrapRuntimeEvalErrorInBrowserString, but this seems good enough.

brendankenny · 2019-07-25T01:10:12Z

+        /** Error that occurred during fetching or parsing of source map. */
+        errorMessage: string
+        /** No map on account of error. */
+        map: undefined;


this can be map?: undefined so it won't have to be set to undefined manually (the code snippet I posted above relies on that, I believe)

…se into gather-source-maps

googlebot · 2019-07-25T04:45:39Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

googlebot · 2019-07-25T04:47:58Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot · 2019-07-25T05:08:15Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

googlebot · 2019-07-25T05:11:39Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

connorjclark · 2019-07-25T05:17:03Z

it seems half of these comments could be @googlebot 👎

googlebot · 2019-07-25T05:17:13Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

googlebot · 2019-07-25T05:33:09Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

connorjclark · 2019-07-25T05:39:24Z

@brendankenny @exterkamp ya'll wanna add your personal emails to the CLA thing? click the Googler link above

brendankenny

🗺 🚜
LGTM!

googlebot · 2019-07-25T19:25:12Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

source maps gather first pass

3ce3602

googlebot added the cla: yes label Jun 1, 2019

own gatherer

7e9b440

vercel Bot deployed to staging June 1, 2019 03:54 View deployment

smoke test

66bc6c6

connorjclark marked this pull request as ready for review June 1, 2019 05:02

connorjclark requested review from brendankenny, patrickhulce and paulirish as code owners June 1, 2019 05:02

vercel Bot deployed to staging June 1, 2019 05:02 View deployment

no jest test

fceb88c

vercel Bot deployed to staging June 1, 2019 05:03 View deployment

connorjclark mentioned this pull request Jun 1, 2019

JS analysis #9097

Closed

11 tasks

more smoke

c721cbd

vercel Bot deployed to staging June 1, 2019 05:35 View deployment

connorjclark commented Jun 1, 2019

View reviewed changes

snap

e217b37

vercel Bot deployed to staging June 1, 2019 06:01 View deployment

connorjclark changed the title ~~Gather source maps~~ core: gather source maps Jun 1, 2019

coverage fix

baa97c2

vercel Bot deployed to staging June 1, 2019 06:18 View deployment

get with the program

5da78cf

vercel Bot deployed to staging June 1, 2019 06:50 View deployment

remove source-maps from default config

8719ccb

vercel Bot deployed to staging June 2, 2019 03:08 View deployment

revert snapshot

5886359

connorjclark added 3 commits July 22, 2019 22:11

scriptParsedEvent

2a46c4e

resolvedSourceMapUrl

416fe3a

doc

bf0293c

connorjclark commented Jul 23, 2019

View reviewed changes

connorjclark mentioned this pull request Jul 24, 2019

wip: core: universal fetch #9447

Closed

Merge remote-tracking branch 'origin/master' into gather-source-maps

ac943f6

brendankenny reviewed Jul 25, 2019

View reviewed changes

connorjclark added 2 commits July 24, 2019 21:44

brendan changes

755ac81

Merge branch 'gather-source-maps' of github.com:GoogleChrome/lighthou…

ba09eff

…se into gather-source-maps

Merge remote-tracking branch 'origin/master' into gather-source-maps

052e86b

json, do not use gatherer for css lol

fe24ee2

proto

a4aef48

update

bbaafbd

brendankenny approved these changes Jul 25, 2019

View reviewed changes

connorjclark mentioned this pull request Jul 25, 2019

core(source-maps): workaround CORS for fetching maps #9459

Merged

snyk-bot mentioned this pull request Mar 21, 2020

[Snyk] Upgrade lighthouse from 5.1.0 to 5.6.0 godaddy/lighthouse4u#13

Merged

Conversation

connorjclark commented Jun 1, 2019

Uh oh!

connorjclark commented Jun 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickhulce commented Jun 1, 2019

Uh oh!

connorjclark commented Jun 1, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jun 1, 2019

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

googlebot commented Jul 24, 2019

Uh oh!

connorjclark commented Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

googlebot commented Jul 24, 2019

Uh oh!

connorjclark commented Jul 24, 2019

Uh oh!

brendankenny commented Jul 24, 2019

Uh oh!

brendankenny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

connorjclark commented Jul 25, 2019

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

connorjclark commented Jul 25, 2019

Uh oh!

brendankenny left a comment

Choose a reason for hiding this comment

Uh oh!

googlebot commented Jul 25, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

connorjclark commented Jun 1, 2019 •

edited

Loading

connorjclark commented Jul 24, 2019 •

edited

Loading