Automation API
The Automation API is available from the browser.webfuseSession.automation namespace.
- Perception methods (‘see’ 👁️) are grouped under
automation.see. - Actuation methods (‘act’ 👆️) are grouped under
automation.act. - Auxiliary tool methods (‘tool’ 🛠️) are grouped under
automation.tool. - Other methods and properties are available right from
automation.
Targeting
Section titled “Targeting”The Automation API can be utilized with both DOM- and vision-based agents. For this, targeting an element in the page is overloaded with different types:
type Target = HTMLElement | CSSSelector | Point | WebfuseID | MetaTarget;
type CSSSelector = string;type Point = [number, number]; // [x, y];type WebfuseID = string; // Unique ID of an element per Tab in a Webfuse Sessionenum MetaTarget { POINTER, // Current virtual pointer position FOCUS // Currently focused element}// By element referencebrowser.webfuseSession .automation .act .click(document.getElementById('cta'))
// By CSS selectorbrowser.webfuseSession .automation .act .click('main > button.cta')
// By point coordinatebrowser.webfuseSession .automation .act .click([420, 890])
// By meta targetbrowser.webfuseSession .automation .act .click( browser.webfuseSession.automation.Target.POINTER )Cross-Shadow and -Frame Targeting Webfuse Exclusive
Section titled “Cross-Shadow and -Frame Targeting ”Relevant elements may, in some cases, be hidden inside shadow DOM, or even iframes. For example, if the agent-enhanced web page embeds a checkout component from a third-party provider. Targeting with the Automation API is able to pierce shadow root and even iframe boundaries. Point coordinates on iframes simply lead to descending into the iframe and targeting with normalized coordinates until a non-frame element is found. By design, cross-shadow or -frame CSS selectors do not exist. To enable cross-shadow and -frame targeting via CSS selectors, Webfuse considers both types of DOM nodes as ordinary container elements. For shadow DOM, the shadow-root pseudo container tag name is therefore introduced. As a result, shadow roots and frames represent implicit container tags, like, i.a., div or section.
Cross-Shadow Targeting
Section titled “Cross-Shadow Targeting”Shadow DOM subtrees in the browser are usually invisible upon parent DOM serialization (.outerHTML/.innerHTML). The position of a shadow root is visualized with # Shadow in the following example:
<div> <custom-element> # Shadow <b>Slotted</b> </custom-element></div>Webfuse, however, implies a shadow root element, which is reflected with Webfuse-native perception in DOM snapshots. For above given example, this would look as follows:
<div> <custom-element> <shadow-root> <strong>Shadow</strong> <p> <slot></slot> </p> </shadow-root> <b>Slotted</b> </custom-element></div>Now, the shadow root and elements within the shadow DOM can be targeted with valid CSS selector syntax:
// Cross-shadowbrowser.webfuseSession .automation .act .click('body my-component shadow-root button#submit')Cross-Frame Targeting
Section titled “Cross-Frame Targeting”To isolate embedded DOMs from each other, subtrees beneath iframes are usually hidden upon parent DOM serialization (.outerHTML/.innerHTML):
<html> <head></head> <body> <h1>Parent</h1> <iframe src="/child"></iframe> </body></html>