Skip to content

Please add an option to skip Client App Initialization for certain cases such as Google/Bing Crawlers/Bots #52360

@naveedahmed1

Description

@naveedahmed1

Which @angular/* package(s) are relevant/related to the feature request?

No response

Description

Here's a request to please add an option to skip initialization of Client App for certain User Agents for example Google Bot, when using SSR. This will help developers working with Angular-based websites and their interaction with Google's search crawlers.

Background:
We have been facing challenges related to the crawling and indexing of our Angular-based website by Google's search engine. Despite Google's advanced capabilities to run JavaScript with an evergreen version of Chromium, we have observed issues during the client-side application initialization process that affect the accuracy of indexing and user experience. Specifically, we have encountered problems with Signed Exchanges (SXG) and Google for Jobs Markup, which result in users being redirected to incorrect pages on our website or encountering empty pages with links on Google Jobs box.

These issues seem to arise during the client app initialization phase, and we believe that having an option to bypass this step for specific user agents, such as Googlebot, could mitigate these problems. By serving static HTML content to Google's crawlers, we can ensure that Google indexes our content accurately and enhances the user experience for visitors from search results.

This will also make it easy to troubleshoot issues.

For example in our case, when we check the cached version of the url through Search Console, it had the exact same markup as returned by our server as SSR response, but on clicking on SXG of that page in SERP lead to a completely different page.

In most of the cases the client side app initlization for a bot such as Google is not even necessary. And if an issue arrises while initlizing the Client App, when crawled by the a search bot, apparently there doesnt seem to be a way to report and troubleshoot the issue. The Inspect URL feature in Search console shows the markup which Google Bot received from the server.

Ref: https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering

Proposed solution

Please add an option that allows developers to define specific user agents (e.g., Googlebot) for which the client app initialization process can be completely skipped during rendering. Or maybe add an option to conditionally skip client app initilization based on certain conditions.

Alternatives considered

I tried replacing the type of the server bundles in server rendered markup

Replace(".js\" type=\"module\"", ".js\" type=\"text/skip-hydration\"");

And then conditionally enabling them by changing the type attribute using below code but it didnt work,

If(!isUserAgentBot()){
document.querySelectorAll('script[type="text/skip-hydration"]')
    .forEach(script => script.replaceWith(Object.assign(script.cloneNode(), 
       {type: 'module'})));
}

I see the scripts are loaded in Network Tab of Dev Tools but Angular app is not initialized.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: serverIssues related to server-side rendering

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions