Plan

After the design review (#15347) we have the following plan to resolve this issue:

• Undelete the origin experiments code, adjusting it to use @jridgewell's suggestion to make it not async. (Issue Undelete OriginExperiments #16183)
• Until that is done, use document-level experiments sparingly. We should not make breaking changes to document-level experiments (including the existing set).
• After the origin experiments code is launched, no longer add any document-level experiments (with a comment in prod-config.json).
• After all existing document-level experiments are launched we'll remove the code supporting document-level experiments.

Proposal

The original proposal was to do one of the following (in order of preference):

• removing support for document-level experiments in favor of origin trial experiments
• not allowing breaking changes for features with document-level experiments (with the same policy we use for launched policies), and adding this to our experiment documentation
• providing very clear documentation that document-level experiments may break a page at any time, and should not be used on non-experimental production traffic

Background

AMP currently supports document-level experiments, i.e. by adding a meta "amp-experiments-opt-in" on a page the page author can enable experiments that are whitelisted for document level experiments (specified in allow-doc-opt-in in prod-config.json).

There are three main types of experiments we support that a developer can opt into:

• Experiments enabled in the browser (via cookies or AMP.toggleExperiment), which allows developers to test experimental features but not to push these features to production.
• Document-level experiments, which allows developers to use experimental features in a production setting by adding the meta tag to their document (ideally behind the developers own experiment).
• Origin trial experiments, which allows us to whitelist domains who can enable the experiment features in a production setting (ideally behind the publisher's own experiment).

Document-level vs. origin trial

The biggest issue with document-level experiments is that we have no good way to know everyone who is using the document-level experiment or to notify them of upcoming breaking changes. This means that a breaking change in the experiment carries the risk that a page in production breaks. In contrast, with origin trials the whitelisting process allows us to (a) require developers using the experiment to acknowledge the risks involved in using an experimental feature and (b) collect information that we could use to notify developers using the experimental feature in the case a breaking change will be made.

The main use case we've had for document-level experiments thus far has been cases where a few developers are working closely with the AMP community to refine an upcoming feature. This case is served well by the origin-trial case, and the number of these cases is not very large.

Other use cases for document-level experiments

The document-level experiment I2I (#6869) has an additional use case for the amp-inabox. I don't have complete context for that use case so I'd like some input from @lannka whether origin trials is sufficient for that case. If not we may still be able to move to the "document-level experiments don't allow breaking changes" policy.

/cc @aghassemi @cathyxz @choumx @cramforce @jridgewell @kristoferbaxter