< README

Fresnel is an automated tool for creating and comparing performance reports about web pages.

cli

The command-line interface for Fresnel.

This module is invoked by bin/fresnel and is in charge of reading user input such as CLI parameters, environment variables, and the .fresnel.yml config file.

Interacts with: conductor.

compute

Functions to help with numerical computations.

conductor

The program interface for Fresnel commands.

Interacts with: puppeteer/Browser, Writer, Probe, and Report.

printer

The text printer for the Fresnel command-line interface.

This module is invoked by cli to turn information from conductor into lines of text.

probes/navtiming

Get data from the Navigation Timing API in the browser.

probes/paint

Get data from the Paint Timing API in the browser.

probes/screenshot

Capture a screenshot from the browser viewport after the page has finished loading.

probes/trace

Capture a trace file that can later be opened in Chrome DevTools or timeline-viewer.

probes/transfer

Get data from the Resource Timing API in the browser.

reports/navtiming

Report on Navigation Timing API metrics.

reports/paint

Report on Paint Timing API metrics.

reports/transfer

Report with transfer sizes from the Resource Timing API.

Writer

Represents a directory and an (optional) prefix for files and subdirectories created within it.

Probe : Object

A probe is a set of callbacks to run client-side when recording scenarios.

probe.before

The before callback runs before the web page starts loading. Use this to make changes to the browser tab (via puppeteer/Page, or higher-level objects accessed via that, such as puppeteer/Browser). Examples of things one might do here: set the viewport, start timeline tracing, disable JavaScript, grant permission for Geo location, simulate a certain GPS position, etc.

Parameters:

• puppeteer/Page page: The browser tab.
• Writer writer: Use this to obtain file paths to write artefacts to.

probe.after

The after callback runs once the web page has finished loading. Use this to capture your data. Typically by using page.evaluate() to send JavaScript code to the browser which will be executed client-side in the context of the page. From there, you can access the DOM, other web platform APIs, as well as any custom JavaScript interfaces exposed by code from the web page itself.

Parameters:

• puppeteer/Page page: The browser tab.
• Writer writer: Use this to obtain file paths to write artefacts to.
• Function addData: Use this to store key/value pairs that should be saved as part of the Fresnel record. These must be serialisable as JSON.

Report : Object

A report analyses data from a Probe when recording and comparing scenario data.

const compute = require( 'fresnel/src/compute' );
module.exports = {
    probes: [ 'x' ],
    metrics: {
        example: {
            caption: 'An example metric.'
            unit: 'ms',
            analyse: ( series ) => compute.stats( series.x.mykey ),
            compare: ( a, b ) => compute.diffStdev( a, b )
        }
    }
};

report.probes

Specify one or probes that provide the data needed for this report. The recording phase uses this to determine which probes to run.

report.metrics

An object with one or more metric specifications. The key is the internal name for the metric, and the value is an object with the following properties:

• string caption - A short description of this metric.
• string unit - The unit for this metric. Must be one of: ms, B, P.
• Function analyse - A callback to aggregate and analyse data from thes probes, as gathered from multiple runs.
• Function compare - A callback to compare the two sets of analysed data, from two recordings.
• number threshold (optional) - If the compared difference is more than this value, a warning will be shown.
• string compareUnit (optional) - The unit of the comparison/threshold value. Defaults to the unit property.

metric.analyse callback

During each run of the same scenario, a probe can capture data into an object. Here, the values from those objects have combined from each run into an array.

The analyser for a single metric, has access to all data from probes, and must return an object with a mean and stdev property. This simplest way to do that to pass a series to compute.stats and return its result.

For example, if probe x collects { mykey: 10 } and { mykey: 12 } from two runs of the same scenario, it will be available here as series.x.mykey containing [ 10, 12 ].

Parameters:

• Object series - An object with for each probe, an array of data from multiple runs.

Returns: Object - An stats object with a mean and stdev property.

metric.compare callback

Parameters:

• Object a: A stats object from the analyse callback.
• Object b: A stats object from the analyse callback.

Returns: number - Difference between A and B, in the metric's unit, or 0 if no significant change was found.

The command-line interface for Fresnel.

This module is invoked by bin/fresnel and is in charge of reading user input such as CLI parameters, environment variables, and the .fresnel.yml config file.

Interacts with: conductor.

Functions to help with numerical computations.

• compute
  • ~subtract(seqA, seqB) ⇒ Array.<number>
  • ~stats(values) ⇒ Object
  • ~diffStdev(before, after) ⇒ number
  • ~ranks(values) ⇒ Array
  • ~mannWhitney(before, after) ⇒ number
  • ~diffMannWhitney(before, after) ⇒ number

Perform subtraction on each pair from two sequences.

Example:

const seqA = [ 3.0, 3.0, 4.5 ];
const seqB = [ 2.5, 2.6, 3.3 ];
subtract( seqA, seqB );
// [ 0.5, 0.4, 1.2 ]

Kind: inner method of compute

Compute statistics about a sequence of numbers.

Example:

stats( [ 3, 4, 5 ] );
// mean: 4.0, stdev: 0.82

Kind: inner method of compute
Returns: Object - An object holding the mean average (mean), standard deviation (stdev) and values (values).

Compare two objects from stats().

Example:

const a = stats( [ 3, 4, 5 ] );       // mean: 4.0, stdev: 0.82
const b = stats( [ 1.0, 1.5, 2.0 ] ); // mean: 1.5, stdev: 0.41
diffStdev( a, b );
// -1.27

This is computed by creating a range of 1 stdev aroud each mean, and if they don't overlap, the distance between them is returned.

In the above example, the range for sequence A is 3.18 ... 4.82, and the range for B is 1.09 ... 1.91. The ranges don't overlap and the distance between 3.18 and 1.91 is -1.27.

Kind: inner method of compute
Returns: number - The difference between the before and after mean averages, after having compensated for 1 standard deviation. If lower numbers are better for your metric, then a negative difference represents an improvement.

Find the rank for each value, giving any tied values the mean of the ranks that they cover. The ranks are used to calculate the U score. Also find the adjustment constant, used for calculating the standard deviation of U.

Example:

values: [ 4, 9, 8, 7, 3, 6, 6 ]
sorted: [ 3, 4, 6, 6, 7, 8, 9 ]
place:  [ 1, 6, 5, 4, 0, 2, 3 ]
ranks:  [ 2, 7, 6, 5, 1, 3.5, 3.5 ]

Kind: inner method of compute
Returns: Array - ranks of the values, adjustment constant

Perform an approximate Mann-Whitney U test on two sets of values to test whether the values in the second set are significantly higher. The test is a non-parametric test that compares the ranks of the values without assuming they are distributed in a particular way.

For details of the test and calculations see: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test

This implementation is paraphrased from: https://github.com/JuliaStats/HypothesisTests.jl/blob/b28a4587fe/src/mann_whitney.jl

Assumptions made by this implementation:

• each set contains the same number of values
• we are interested in whether values in the second set are higher
• the sample is large enough to use the approximate test

Kind: inner method of compute
Returns: number - The p-value representing the likelihood of getting the observed U score (or more extreme) under the null hypothesis, that a randomly-chosen value from either set is equally likely to be higher or lower than a randomly-chosen value from the other set.

Compare two sets of values using the Mann-Witney U test.

Kind: inner method of compute
Returns: number - Number between 0.0 and 1.0. A higher number may suggest the values have increased, and a lower number may suggest the values remained the same or got lower. It is computed as 1 minus the Mann-Whitney p-value.
See: #mannWhitney

The program interface for Fresnel commands.

Interacts with: puppeteer/Browser, Writer, Probe, and Report.

• conductor
  • ~record(config, outputDir, label, progress) ⇒ Object
  • ~compare(outputDir, labelA, labelB) ⇒ Object

Create a Fresnel record.

This runs the scenarios from the given configuration object, and saves the record and probe artefacts to an out subdirectory named after the label.

The Scenario URL may have placeholders for variables. These allow scenarios to adapt to the current environment. For example, when testing an app like MediaWiki, the hostname and port of the web server may vary in each CI or development environment.

Kind: inner method of conductor
Returns: Object - Fresnel record.
Throws:

• Error If configuration is invalid.
• Error If Writer can't create the output directory.

Compare two Fresnel records.

Kind: inner method of conductor
Returns: Object - Comparison
Throws:

• Error If records could not be read

The text printer for the Fresnel command-line interface.

This module is invoked by cli to turn information from conductor into lines of text.

• printer
  • ~format(num, unit, [options]) ⇒ string
  • ~progress(writeln, event, message)
  • ~printTable(writeln, rows)
  • ~comparison(writeln, compared)

Kind: inner method of printer

Kind: inner method of printer

Kind: inner method of printer

Kind: inner method of printer

Get data from the Navigation Timing API in the browser.

See

• Probe
• https://www.w3.org/TR/navigation-timing-2/

Get data from the Paint Timing API in the browser.

See

• Probe
• https://www.w3.org/TR/paint-timing/

Capture a screenshot from the browser viewport after the page has finished loading.

See: Probe

Capture a trace file that can later be opened in Chrome DevTools or timeline-viewer.

See: Probe

Get data from the Resource Timing API in the browser.

See

• Probe
• https://www.w3.org/TR/resource-timing-2/

Report on Navigation Timing API metrics.

See

• Report
• https://www.w3.org/TR/navigation-timing-2/

Report on Paint Timing API metrics.

See

• Report
• https://www.w3.org/TR/paint-timing/

Report with transfer sizes from the Resource Timing API.

See

• Report
• https://www.w3.org/TR/resource-timing-2/

Represents a directory and an (optional) prefix for files and subdirectories created within it.

Kind: global class

• Writer
  • new Writer(dir, prefix)
  • .getPath(name) ⇒ string
  • .prefix(prefix) ⇒ Writer
  • .child(name) ⇒ Writer

The specified directory will be created if needed. Any parent directories must exist beforehand.

Throws:

• Error If directory can't be created.

Get the file path for a resource in this writer's directory.

Kind: instance method of Writer
Returns: string - File path

Create a Writer object for the same directory, with an added prefix for any files and subdirectories.

Kind: instance method of Writer

Create a Writer object for a subdirectory of the current one.

Kind: instance method of Writer

A probe is a set of callbacks to run client-side when recording scenarios.

The before callback runs before the web page starts loading. Use this to make changes to the browser tab (via puppeteer/Page, or higher-level objects accessed via that, such as puppeteer/Browser). Examples of things one might do here: set the viewport, start timeline tracing, disable JavaScript, grant permission for Geo location, simulate a certain GPS position, etc.

Parameters:

• puppeteer/Page page: The browser tab.
• Writer writer: Use this to obtain file paths to write artefacts to.

The after callback runs once the web page has finished loading. Use this to capture your data. Typically by using page.evaluate() to send JavaScript code to the browser which will be executed client-side in the context of the page. From there, you can access the DOM, other web platform APIs, as well as any custom JavaScript interfaces exposed by code from the web page itself.

Parameters:

• puppeteer/Page page: The browser tab.
• Writer writer: Use this to obtain file paths to write artefacts to.
• Function addData: Use this to store key/value pairs that should be saved as part of the Fresnel record. These must be serialisable as JSON.

Kind: global typedef
Properties

A report analyses data from a Probe when recording and comparing scenario data.

const compute = require( 'fresnel/src/compute' );
module.exports = {
    probes: [ 'x' ],
    metrics: {
        example: {
            caption: 'An example metric.'
            unit: 'ms',
            analyse: ( series ) => compute.stats( series.x.mykey ),
            compare: ( a, b ) => compute.diffStdev( a, b )
        }
    }
};

Specify one or probes that provide the data needed for this report. The recording phase uses this to determine which probes to run.

An object with one or more metric specifications. The key is the internal name for the metric, and the value is an object with the following properties:

• string caption - A short description of this metric.
• string unit - The unit for this metric. Must be one of: ms, B, P.
• Function analyse - A callback to aggregate and analyse data from thes probes, as gathered from multiple runs.
• Function compare - A callback to compare the two sets of analysed data, from two recordings.
• number threshold (optional) - If the compared difference is more than this value, a warning will be shown.
• string compareUnit (optional) - The unit of the comparison/threshold value. Defaults to the unit property.

During each run of the same scenario, a probe can capture data into an object. Here, the values from those objects have combined from each run into an array.

The analyser for a single metric, has access to all data from probes, and must return an object with a mean and stdev property. This simplest way to do that to pass a series to compute.stats and return its result.

For example, if probe x collects { mykey: 10 } and { mykey: 12 } from two runs of the same scenario, it will be available here as series.x.mykey containing [ 10, 12 ].

Parameters:

• Object series - An object with for each probe, an array of data from multiple runs.

Returns: Object - An stats object with a mean and stdev property.

Parameters:

• Object a: A stats object from the analyse callback.
• Object b: A stats object from the analyse callback.

Returns: number - Difference between A and B, in the metric's unit, or 0 if no significant change was found.

Kind: global typedef
Properties