Skip to content

core: use the main frame url for finalUrl#13819

Closed
adamraine wants to merge 7 commits into
masterfrom
final-frame-url
Closed

core: use the main frame url for finalUrl#13819
adamraine wants to merge 7 commits into
masterfrom
final-frame-url

Conversation

@adamraine

Copy link
Copy Markdown
Contributor

Step 5 of #13706

I did a pass on all usages of finalUrl in the report renderer and treemap. Every usage in the report renderer was either cosmetic or just looking at the origin, so we should be fine there.

In the treemap, we use the finalUrl to determine if a first-level node is for inline html scripts. With the changes in #13802 (comment) we can determine this by looking for "(inline)" prefixes in the children, however we should keep the URL check around as a backport.

@adamraine adamraine requested a review from a team as a code owner April 5, 2022 18:51
@adamraine adamraine requested review from connorjclark and removed request for a team April 5, 2022 18:51
driver = new EmulationDriver(connectionStub);
resetDefaultMockResponses();

fakeDriver.url = jest.fn().mockResolvedValue('about:blank');

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tests use fakeDriver, some use driver, some make their own driver 🙃

// Note that the final URL is the URL of the network requested resource and not that page we end up on.
// http://localhost:10200/push-state
finalUrl: 'http://localhost:10200/redirects-final.html?pushState',
finalUrl: 'http://localhost:10200/push-state',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@brendankenny

brendankenny commented Jun 13, 2022

Copy link
Copy Markdown
Contributor

😬 awkward to bring this up long after you opened #13706: what do you think of a rename to something else for the final frame url?

HTTP Archive analysis out there in particular is really reliant on finalUrl for de-duping and joining Lighthouse to WPT or CrUX data. For example, Annie Sullivan's recently published TBT and INP colab relied on finalUrl to compare Lighthouse and CrUX values.

Since things like that aren't using Lighthouse directly (and are unlikely to read the changelog), will probably be copy/pasted into future analysis scripts, and everything will still appear to work since often mainDocumentUrl would be the same as finalUrl, we could either:

  1. Keep finalUrl as is (chalk up the imprecise name to historical reasons (should be finalNavigatedUrl or whatever)) and find a new name for the final frame url
  2. Drop finalUrl and find a new name for the final frame url, so the name change will result in 0 results from a script like that

@connorjclark

Copy link
Copy Markdown
Collaborator

can you add a smoke test / scenario test that asserts that the URL hash is present in final url?

@paulirish

Copy link
Copy Markdown
Member

I'm +1 on keeping finalUrl and introducing a new finalFrameUrl which is after any pushState or navigation API.

@paulirish

paulirish commented Jun 13, 2022

Copy link
Copy Markdown
Member

me connor adam discussed and are gravitating towards making this a very BREAKING change:

  • delete finalUrl prop on LHR
  • add finalNetworkUrl (bikeshedding welcome) which is the same definition of current finalURL
  • add finalFrameUrl (bikeshedding welcome) which is the definition proposed here (post hash-navigation, pushState or navigation API)

all clients have to choose what they want for each instance

@connorjclark

Copy link
Copy Markdown
Collaborator

I prefer option 2 because I want to avoid introducing permanent confusion (keeping finalUrl, which we all wish didn't exist anymore and readers will assume means something else; while also having moreFinalUrl/realizedUrl/finalFrameUrl/etc...) just to prevent a small number of analysis script writers from needing to make a very small change to their scripts. They'd see things break loudly because of a missing finalUrl, would hopefully think to read the 10.0 changelog where we detail all the changes to the LHR we made re: urls, then be able to fix it by using whatever new name we think of.

My preference:

  • rename finalUrl to finalNetworkUrl
  • finalFrameUrl or finalDocumentUrl or finalMainDocumentUrl for the actual, address bar URL

This forces users of finalUrl to reckon with the change, and avoids us forever needing to remember why finalUrl is really the final url.

@adamraine

Copy link
Copy Markdown
Contributor Author

I think I'm going to merge this pending connor's request for a smoke test and do a rename of finalUrl in a follow-up

Comment thread .github/workflows/devtools.yml Outdated
# errors-expired-ssl errors-infinite-loop
# Disabled because Chrome will follow the redirect first, and Lighthouse will only ever see/run the final URL.
# redirects-client-paint-server redirects-multiple-server redirects-single-server redirects-single-client
# redirects-client-paint-server redirects-multiple-server redirects-single-server redirects-single-client redirects-hash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps our test definitions should begin to declare themselves with excludeRunners.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise local usage of the runner will fail unless you remember to disable these tests :)

@brendankenny

Copy link
Copy Markdown
Contributor

delete finalUrl prop on LHR

I'll stress again that this is really annoying for people consuming LHRs :) Another use case: comparing HTTP Archive results over time. The standard query is going to have use conditionals on the Lighthouse version to get a joinable URL.

Dropping it from the URL artifact makes total sense, it can be renamed to be completely clear what people should be using for their particular purpose. On the LHR it seems less clear to me that there's been any confusion over the name

add finalNetworkUrl (bikeshedding welcome) which is the same definition of current finalURL

finalNavigationUrl? That seems like what we mean. finalDocumentUrl maybe (in the sense of the url of the final documented downloaded), but the html spec ties "document url" to the displayed url (including e.g. pushState changes), not the url of where the document came from.

add finalFrameUrl (bikeshedding welcome) which is the definition proposed here (post hash-navigation, pushState or navigation API)

I still think some kind of finalDisplayUrl might be better. AFAICT that's the sense we're looking to collect (the URL that's displayed to the user) and in what we're going to use it for (is it for anything but displaying in the report?).

@@ -0,0 +1,34 @@
/**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smoke tests are slow, is it possible to combine this with an existing smoke test if it's just the hash we're after? It doesn't need to use pushState, the requestedUrl could even include a hash and that wouldn't currently end up in finalUrl

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did the client and server redirects test because the hash is not preserved over a client redirect (without adding it yourself) but it is preserved over a server redirect.

@adamraine

Copy link
Copy Markdown
Contributor Author

I'm going to try and organize our bikeshedding issues in the parent issue for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants