We should expand the current functionality of AMP experiment to work with third party A/B Testing providers.

Design Doc is a draft, and only acts as an idea of how this can be implemented. Does not reflect a final idea or stance from AMP as a whole.

Last Updated: January 22nd, 2018

Project Goal(s)

To expand the current functionality of AMP experiment to work with third party A/B Testing providers.

Background

• Planned Design Review
• amp-experiment reference documentation
• amp-experiment AMP By Example
• amp-mustache reference documentation

Proposed Solution

The proposed solution is to update the variants key on the <amp-experiment>. The variants key would now take in an object, instead of just a float value. This object would have child objects that represent what mutations should be applied to the document.

Proposed / Example Format

Note: Our goal for mutation objects is to match the naming of the properties on an Element Objects.

<amp-experiment>

 <template id=”template_id”>
    <!-- AMP Mustache Template →
    I am the innerHTML of an element.
  </template>

  <script type="application/json">
      {
          "my_experiment": { // Name of the experiment

              "sticky": true, // Does the experiment “stick” with a user

              "consentNotificationId": "consent-notif" // Id of the consent instance

              "schedule": { // Time period for the duration of the experiment.

                  "start_time": "Sat, 02 Feb 2019 03:04:05 GMT", // UTC Date when the experiment stars.

                  "end_time": "Sat, 02 Feb 2020 03:04:05 GMT", // UTC Date when the experiment ends.
              }
              "variants": { // The different experiment variations to run.

                  "treatment1": { // Name of the variation

                      "enabled": true, // Should the variation be applied

                      "weight": 0.5, // 0.0 => 1.0 float percentate on how often variation should be applied

                      "mutations": [{ // Array of mutations that should occur to the document

                              "querySelector": “#my-link-element”, // CSS Selector for the element we are mutating

                              "innerHTML": “template_id”, // replace the innerHTML with a mustache template with the specified id

                              “style”: { // Properties pertaining to element.style

                                  "backgroundColor": “#FFF”, // Set the element.style.backgroundColor

                                  "color": “#000” // Set the element.style.color
                              },

                              "href": “https://www.google.com/” // Set the element.href

                              “remove”: false // Remove the element entirely with element.remove()

                              "className": “.my-new-class-string”, // sets the element.class. Preserves AMP Classes.

                              “insertBefore”: “#some-div” // insertBefore/insertAfetr, matches the element.insertBefore, however it takes in a CSS Selector for an element to be inserted before/after.
                      }]
                  }
              }
          }
      }
  </script>

</amp-experiment>

Common Client A/B Testing Architecture

Currently, publisher who wish to A/B test their pages opt into it by including a javascript tag provided by the vendor. This javascript blocks render during runtime, downloads a list of mutations from vendor site, applies them on the page and shows the page to the user.

Proposed AMP Client A/B Testing Architecture

The common client side A/B testing architecture is unfavorable for AMP for several reasons:

• AMP restricts third-party javascript on the page. Therefore, loading an external script to handle this would require having the script be Iframed, and would greatly complicate the API, and make this model extremely slow.

• AMP’s Design Principles conflict with the idea of being “render-blocking”. Where render-blocking represents the idea of increasing the time of the “First Meaningful Paint” to execute a task. Because of this, AMP tries to minimize the amount of render blocking tasks. Especially ones that are making network requests, as this could be extremely slow depending on network conditions.

• AMP’s Design Principles also conflict with the idea of the idea of hurting the end-user experience in order to improve the developer experience. This current model slows down the time in order for users to receive the content, in order to make A/B Testing much easier for developers and publishers.

In order to make the current A/B model align more with AMP, we can place JSON describing the mutations that need to take place within the document before serving to the AMP cache or canonical users. Therefore, the only remaining “render-blocking” work to be done once the document is downloaded by a user, is to apply mutations. These mutations can be restricted to further ensure the end-user’s performance. See the updated flow below:

Mutation Limitations

In order to preserve fast pages, and abuse on the AMP platform, mutations must be put in place on this new amp-experiment. Please feel free to comment on this doc and suggest improvements and concerns with the limitations. These limitations are as follows:

• The document can have a maximum of 10 variations.

• A single variation can have a maximum of 10 mutations.

• A maximum of 30 total mutations across all variations.

• Can not have a variation that is applied 1.0 (100%) of the time.

• Can not have the following tags that will be in an unallowed list:

  • <body>

  • <script>

  • Any additional tags we feel that would make sense to block later on.

• JSON KB limit

  • Going to shoot for 3KB.

  • Should be determined by the parse time of a Nokia 2 or Moto G4.

• Reach out to vendors to get their opinion.

• Can not change i-amphtml classes, or use CSS with !important.

Developer/User Intended Workflow

1. Developer signs up for a Third-party A/B Testing provider, such as Optimizely, or Google Optimize.

2. Developer defines AMP-friendly mutations they would like to make to their page in the provider dashboard.

3. Developer creates an AMP page with a single amp-experiment tag, with a JSON tag matching the proposed format.

High-Level AMP Implementation Walkthrough

Please note: amp-experiment is already defined as render-blocking by its service, variant. This can be found in render-delaying-services.js.

1. Update amp-experiment’s variant service, to parse and understand the new JSON format. As well as, support the old format.

2. Abstract the current process of determining when to show a treatment based on weight, for this new path. Use this abstracted path for both the old and new process.

3. Implement functionality of finding all elements from selectors before trying to mutate them.

4. Add support for completely enabling / disabling a treatment

5. Add support for selecting an element using the new JSON format.

6. Add support for applying element mutations using the new JSON format.

7. Add support for templates in the new JSON format.

Validation Rules

Validation rules do not change. See the current amp-experiment validator rules. However, a validation will be applied server side to match the size constraints outlined in the “Mutation Limitations” section, and ensure the JSON format.

Implementation / Validation Edge Cases

What if the config is Invalid?

All configs are validated in the current amp-experiment, and will continue to be so. Will follow the same path on invalid configurations.

What if the document is not fully parsed by the time we start mutating?

Amp-experiment will have to wait until the entire document has been parsed before applying mutations. This will be a part of the 3 second budget that amp-experiment must fit within.
What if an element can not be found or mutated for reason X?
Before mutations are applied, amp-experiment will find all elements on the page with the CSS selectors specified.If one or more elements could not be found, nothing is applied and an error is thrown. If a mutation could not be applied due to a timeout, an error is thrown. In both cases, all analytics pertaining to the experiment should not be sent as the experiment is incomplete.

How should unsanitized HTML / CSS be handled?

We should sanitize all HTML and CSS with our current sanitization services that are already in AMP.

What if a mutation creates invalid AMP?

This can be solved by making that available HTML mutations an allowed list, and not an unallowed list. This will allow us to carefully pick and choose what styles can be applied, and ensure the validity of the AMP being displayed. Also, AMP’s validator only does a static analysis of AMP pages. Meaning other extensions like amp-bind and amp-mustache could also create invalid AMP if done correctly. Even though this is the case, we still plan to improve/maintain the current surface for bad AMP pages.

How will Images that are applied as mutations be cached?

The https://img-curls-subdomain.cdn.ampproject.org/ii/ url is simply a proxy for downloading images of of the canonical site, and caching them to be served to amp documents. Thus, when a mutation is applied, we should be able to simply rewrite the images to use this same endpoint to have the images cached as well.

What if an extension is loaded as part of a mutation?

This should be allowed, as all extensions are loaded asynchronously, thus it would just follow the flow the standard extension bootstrapping. However, this would have to be done in the current bounds of the amp mustache template.

Alternative / Upgraded Implementations

The most optimal solution to this problem that satisfies AMP’s principles, specifically to put the end user first, is to create a completely server side solution. To satisfy a need for a completely server side solution, we would have to do the implementation in the AMP Cache. One solution proposed was to have publishers upload a service worker, that could intercept requests before they are served from the cache. For example, see this diagram below:

The user would request a document, which would come from the AMP Cache. The AMP Cache would then be modified to do the following:

• (Currently Implemented) Grab the document from the in-memory cache

• Pass this document to a service worker, if one exists

  • This service worker is uploaded by a publisher, that can perform operations on an AMP document for a certain time period. Here, the publisher can make network requests, and other actions.

  • If the Service Worker times out, the original document is served.

  • NOTE: This layer could be developed by the AMP team for a specific amp-experiment case. And migrated in the future if needed.

• Validate the AMP Document once more before serving to a user.

  • On Pass, the document is served, and can be cached for a certain time period for the new variation of the document

  • On Failure, the original document is served.

This implementation could also serve as an upgrade path for the current client side solution. Since, the same schema can be used, or expanded, to define the modifications that need to be done before serving to users. These modifications could then be handled by the publisher’s own Javascript, or an NPM module provided by third-party A/B testing solutions for AMP Cache Service workers. Lastly, another positive about this solution is that it could also be used for much more than just A/B testing. Often times, publishers will want/need to make modifications to their documents stored in the cache due to errors or legal reasons. Currently, there is an API to do this, but it has had mixed reviews by our partners. This service worker could fulfill these last minute modifications for things like consent changes, or document errors.

Other Concerns

How will we ensure performance on AMP pages?

Currently, in render-delaying-services.js, there is a timeout of 3 seconds for all services that choose to delay render, and be “render blocking”. However, 3 seconds is still a large amount of time. We are considering adding a well-defined limit of how many mutations / and the types of mutations than can be done on a document.

What if the document author just makes the entire page a large mutation?

This also relates to the fact that we are adding a well-defined limit of how many mutations / and the types of mutations than can be done on a document. See the “Mutation Limitations” section.

Why not make a completely server side solution?

Currently, the AMP Cache can only serve one version of a document. Implementing this within the cache would be a large effort, and require a lot of work on both the server and client side, for both AMP developers and publishers. This is definitely the ideal solution, and which is why we outlined an upgrade path after this solution is built.

If AMP is all about performance, why allow for A/B testing on the client?

A/B testing is something that a lot of publishers value greatly to improve their site, and do research on how to improve their metrics. In order to help these publishers adopt AMP and have a well lit path on how to continue their process while making as little sacrifices as possible. AMP should offer a clean and fast solution to A/B testing for publishers with the A/B testing platforms they already use today.

Is this the first time we are allowing Mutations on page load?

Nope, currently amp-bind and amp-list have this ability as well. Even though we assume they are used on user interactions, it is possible to load amp-list from a JSON endpoint, and render HTML accordingly. Which in turn, could be used for all sorts of bad / malicious use cases, similar to this extension. However, just because another extension as the power to do this, does not mean we should disregard the ability users will have with this extension.

Shouldn’t the schema match mutation observer?

No, is meant for more past tense, not very descriptive for our case: https://developer.mozilla.org/en-US/docs/Web/API/MutationRecord