nice 👍

below is the plan I have created and working against

Selenium BiDi Test Coverage Plan - WPT Comparison

Prepared: 2026-03-07
Goal: Identify missing test coverage in Selenium Python BiDi tests compared to WPT

Executive Summary

Selenium currently has 10 BiDi test modules while WPT has 13 main test modules with an extensive sub-module structure. Key gaps include:

• 3 completely missing modules: errors, integration, and log
• Partial coverage gaps: Network, Script, and Session modules have untested edge cases
• New functionality not in Selenium: permissions tests exist but no corresponding WPT module found

Detailed Module-by-Module Comparison

✅ 1. Browser Module

File: bidi_browser_tests.py (335 lines)

WPT Sub-modules:

• create_user_context/
• get_client_windows/
• get_user_contexts/
• remove_user_context/
• set_download_behavior/
• set_download_behavior/ (comprehensive subdirectory for different scenarios)

Selenium Test Coverage:

• test_browser_initialized()
• test_create_user_context()
• test_get_user_contexts()
• test_remove_user_context()
• test_set_download_behavior() - basic test
• test_get_client_windows()

Missing Tests:

• Multiple user context scenarios: Create multiple contexts, test getting contexts after creation
• Download behavior edge cases: Different values, invalid values, clearing behavior
• Client window state transitions: Various window states and properties
• Error handling: Invalid context IDs, operations on non-existent contexts

Estimated Gap: 5-10 tests

✅ 2. Browsing Context Module

File: bidi_browsing_context_tests.py (1177 lines)

WPT Sub-modules (32 total):

• activate/ - Test activating browsing contexts
• capture_screenshot/ - Screenshot functionality
• close/ - Closing contexts
• context_created/ - Context creation events
• context_destroyed/ - Context destruction events
• create/ - Creating contexts with various parameters
• dom_content_loaded/ - DOM content loaded event
• download_end/ - Download end event
• download_will_begin/ - Download start event
• fragment_navigated/ - Fragment navigation event
• get_tree/ - Getting context tree
• handle_user_prompt/ - User prompt handling
• history_updated/ - History update event
• load/ - Page load event
• locate_nodes/ - Node location functionality
• navigate/ - Navigation functionality
• navigation_committed/ - Navigation committed event
• navigation_failed/ - Navigation failed event
• navigation_started/ - Navigation started event
• print/ - Print to PDF
• reload/ - Page reload
• set_viewport/ - Viewport setting
• traverse_history/ - History traversal
• user_prompt_closed/ - User prompt closed event
• user_prompt_opened/ - User prompt opened event

Selenium Test Coverage:

Covered (✓):

• Context creation, navigation, tree rendering
• Event handlers for major events:
  • context_created, context_destroyed
  • navigation_committed, navigation_failed, navigation_started
  • dom_content_loaded, load
  • fragment_navigated, history_updated
  • user_prompt_opened, user_prompt_closed
  • download_will_begin, download_end
• Screenshot capture with parameters
• Viewport management
• User prompt acceptance/dismissal
• Node location with CSS/XPath/text/max_count
• History traversal (back/forward)
• Print to PDF

Missing/Gaps (✗):

• activate/ edge cases - Various context activation scenarios
• capture_screenshot/ edge cases - Clipping regions, formatting options
• handle_user_prompt/ comprehensive handling - Different prompt types
• set_viewport/ edge cases - Device pixel ratio edge cases, unsetting viewport
• locate_nodes/ advanced scenarios - Complex selectors, performance tests
• print/ edge cases - Different format options, margin handling

Estimated Gap: 10-15 tests

✅ 3. Emulation Module

File: bidi_emulation_tests.py (varies)

WPT Sub-modules (10 total):

• combined/ - Combined emulation settings
• set_geolocation_override/ - Geolocation override
• set_locale_override/ - Locale override
• set_network_conditions/ - Network conditions (latency, throughput, offline)
• set_screen_orientation_override/ - Screen orientation
• set_screen_settings_override/ - Screen settings (resolution, pixel ratio)
• set_scripting_enabled/ - Enable/disable scripting
• set_timezone_override/ - Timezone override
• set_touch_override/ - Touch screen emulation
• set_user_agent_override/ - User agent override

Selenium Test Coverage:

Likely basic coverage of:

• User agent override
• Timezone override
• Geolocation override
• Network conditions
• Screen orientation
• Touch emulation

Missing/Gaps:

• combined/ - Multiple emulation settings applied together
• set_screen_settings_override/ edge cases - Different resolution/pixel ratio combinations
• set_network_conditions/ comprehensive testing - Various latency/throughput values
• set_scripting_enabled/ comprehensive - Verify scripts are actually disabled
• set_locale_override/ comprehensive - Various locale formats
• Error cases - Invalid values, reset operations

Estimated Gap: 8-12 tests

✅ 4. Input Module

File: bidi_input_tests.py (varies)

WPT Sub-modules (4 total):

• file_dialog_opened/ - File dialog events
• perform_actions/ - Keyboard/mouse/touch actions
• release_actions/ - Release queued actions
• set_files/ - Setting file input values

Selenium Test Coverage:

Likely basic coverage of:

• Perform actions
• Release actions
• File dialog opened event
• Set files

Missing/Gaps:

• file_dialog_opened/ comprehensive - Various dialog types, multiple files
• perform_actions/ edge cases - Complex action sequences, various input types
• release_actions/ comprehensive - Queue management, state verification
• set_files/ edge cases - Multiple files, invalid paths, permissions

Estimated Gap: 5-10 tests

✅ 5. Network Module

File: bidi_network_tests.py (varies)

WPT Sub-modules (18 total):

• add_data_collector/ - Add network data collector
• add_intercept/ - Add request/response interceptor
• auth_required/ - Authentication required events
• before_request_sent/ - Before request sent event
• combined/ - Multiple network operations
• continue_request/ - Continue intercepted request
• continue_response/ - Continue intercepted response
• continue_with_auth/ - Continue with authentication
• disown_data/ - Disown network data
• fail_request/ - Fail request
• fetch_error/ - Fetch/network error events
• get_data/ - Get network data details
• provide_response/ - Provide custom response
• remove_data_collector/ - Remove network collector
• remove_intercept/ - Remove interceptor
• response_completed/ - Response completed event
• response_started/ - Response started event
• set_cache_behavior/ - Set cache behavior
• set_extra_headers/ - Set extra headers

Selenium Test Coverage:

Basic coverage likely includes:

• Add/remove data collectors
• Before request sent
• Response started/completed
• Fetch errors
• Basic interception

Missing/Gaps:

• add_intercept/ comprehensive - Different intercept patterns
• continue_request/continue_response/ edge cases - Various modification scenarios
• continue_with_auth/ - Authentication handling
• auth_required/ - Various authentication scenarios
• fail_request/ - Error cases and validation
• provide_response/ - Custom response injection
• combined/ - Multiple network operations together
• set_cache_behavior/ - Cache behavior settings
• set_extra_headers/ - Header manipulation

Estimated Gap: 15-20 tests

✅ 6. Script Module

File: bidi_script_tests.py (varies)

WPT Sub-modules (8 total):

• add_preload_script/ - Add preload scripts
• call_function/ - Call functions in realm/browsing context
• disown/ - Disown script results
• evaluate/ - Evaluate scripts
• get_realms/ - Get JavaScript realms
• message/ - Script messages/logs
• realm_created/ - Realm created event
• realm_destroyed/ - Realm destroyed event
• remove_preload_script/ - Remove preload scripts

Selenium Test Coverage:

Likely basic coverage:

• Evaluate scripts
• Call functions
• Add/remove preload scripts
• Get realms
• Events

Missing/Gaps:

• call_function/ edge cases - Different parameter types, error handling
• evaluate/ comprehensive - Various script types, serialization
• add_preload_script/remove_preload_script/ - Complex scenarios
• realm_created/realm_destroyed/ - Realm lifecycle events
• message/ event - Script logging and message handling
• disown/ comprehensive - Serialization edge cases
• get_realms/ comprehensive - Multiple realms, filtering

Estimated Gap: 10-15 tests

✅ 7. Session Module

File: bidi_session_tests.py (varies)

WPT Sub-modules (5 total):

• capabilities/ - Session capabilities
• new/ - Create new session
• status/ - Get session status
• subscribe/ - Subscribe to events
• unsubscribe/ - Unsubscribe from events

Selenium Test Coverage:

Basic coverage likely includes:

• Session creation
• Subscribe/unsubscribe
• Session capabilities
• Status

Missing/Gaps:

• capabilities/ comprehensive - Various capability combinations
• subscribe/unsubscribe/ edge cases - Multiple subscriptions, event filtering
• status/ - Server status checks
• session lifecycle - Session initialization phases

Estimated Gap: 5-8 tests

✅ 8. Storage Module

File: bidi_storage_tests.py (varies)

WPT Sub-modules (3 total):

• delete_cookies/ - Delete cookies
• get_cookies/ - Get cookies
• set_cookie/ - Set cookies

Selenium Test Coverage:

Basic coverage likely:

• Set cookie
• Get cookies
• Delete cookies

Missing/Gaps:

• delete_cookies/ comprehensive - Various deletion scenarios
• get_cookies/ comprehensive - Filtering, domain/path combinations
• set_cookie/ edge cases - Different cookie attributes, expiration
• Cookie domain/path handling - Cross-domain scenarios
• Cookie attribute edge cases - SameSite, Secure, HttpOnly

Estimated Gap: 8-12 tests

✅ 9. Web Extension Module

File: bidi_webextension_tests.py (varies)

WPT Sub-modules (2 total):

• install/ - Install web extension
• uninstall/ - Uninstall web extension

Selenium Test Coverage:

Basic coverage:

• Install extension
• Uninstall extension

Missing/Gaps:

• install/ comprehensive - Different extension types, manifest versions
• uninstall/ edge cases - Non-existent extensions, multiple installations
• Extension lifecycle - State verification

Estimated Gap: 4-6 tests

❌ Completely Missing Modules

10. Errors Module

File: MISSING - bidi_errors_tests.py should be created

WPT Content:

• errors/errors.py - Error handling and validation tests

What It Should Test:

• BiDi protocol error responses
• Invalid command handling
• Invalid parameter handling
• Error message validation
• Status codes and error types
• Error serialization

Estimated Tests Needed: 15-20 tests

Action Items:

• Create py/test/selenium/webdriver/common/bidi_errors_tests.py
• Test various error conditions and exception handling
• Validate error message formats
• Test error recovery mechanisms

11. Integration Module

File: MISSING - bidi_integration_tests.py should be created

WPT Content:

• cookies_with_network_events.py - Cookies + network integration
• navigation.py - Navigation scenarios
• parallel_execution/ - Parallel test execution
• set_viewport_with_preload_script.py - Viewport + preload script integration

What It Should Test:

• Cross-module functionality combinations
• Cookies interaction with network events
• Complex navigation scenarios
• Parallel BiDi operation execution
• Multiple features working together
• State consistency across modules
• Performance under complex scenarios

Estimated Tests Needed: 10-15 tests

Action Items:

• Create py/test/selenium/webdriver/common/bidi_integration_tests.py
• Test scenarios combining cookies + network events
• Test complex navigation with multiple events
• Test parallel execution scenarios
• Test viewport changes with preload scripts
• Test cross-module state consistency

12. Log Module

File: MISSING - bidi_log_tests.py should be created

WPT Content:

• entry_added/ - Log entry added events

What It Should Test:

• Log entry events
• Different log levels (debug, info, warning, error)
• Console messages (log, warn, error, etc.)
• Script errors and exceptions
• Log filtering
• Log serialization

Estimated Tests Needed: 12-18 tests

Action Items:

• Create py/test/selenium/webdriver/common/bidi_log_tests.py
• Test various console methods (log, warn, error, debug)
• Test exception logging
• Test log level filtering
• Test log event subscription/unsubscription
• Test log message serialization

13. External Module

Status: Unknown - appears in WPT but unclear if it contains tests

What Needs Investigation:

• Verify if external/ contains actual tests or is just a placeholder
• Confirm if Selenium needs external tests coverage

Additional Module: Permissions

File: bidi_permissions_tests.py (EXISTS in Selenium)

WPT Coverage:

Not found in WPT BiDi tests - appears to be Selenium-specific

Current Status:

• Exists in Selenium but no corresponding WPT module
• Likely tests permissions like geolocation, camera, microphone, etc.

Recommendation:

• Keep as-is but ensure it's comprehensive
• Consider if this needs alignment with WPT specifications

Summary Table

Overall Gap Assessment

Total Estimated Missing Tests

• Existing modules (edge cases): 60-100 tests
• Missing modules: 37-53 tests
• Total: ~100-150 tests

Priorities for Implementation

🔴 High Priority (Blocks spec compliance)

1. errors module (15-20 tests) - Core error handling validation
2. log module (12-18 tests) - Essential logging functionality
3. integration tests (10-15 tests) - Cross-module compatibility
4. network module gaps (15-20 tests) - Complex request/response handling

🟡 Medium Priority (Improves coverage)

1. browsing_context module gaps (10-15 tests)
2. script module gaps (10-15 tests)
3. emulation module gaps (8-12 tests)
4. storage module gaps (8-12 tests)
5. browser module gaps (5-10 tests)

🟢 Low Priority (Nice to have)

1. input module gaps (5-10 tests)
2. session module gaps (5-8 tests)
3. webextension module gaps (4-6 tests)

Recommended Implementation Plan

Phase 1: Critical Gaps (2-3 weeks)

1. Create bidi_errors_tests.py - Error handling and protocol validation
2. Create bidi_log_tests.py - Logging and console event tests
3. Add network module tests - Request/response interception edge cases

Phase 2: Core Coverage Expansion (3-4 weeks)

1. Expand script module tests - Realm management and function calls
2. Expand browsing_context tests - Navigation and event handling
3. Create bidi_integration_tests.py - Cross-module scenarios

Phase 3: Edge Cases & Polish (2-3 weeks)

1. Add emulation, input, storage edge cases
2. Add browser, session, webextension edge cases
3. Performance and stress testing
4. Error recovery scenarios

Files to Create/Modify

New Files to Create

py/test/selenium/webdriver/common/bidi_errors_tests.py
py/test/selenium/webdriver/common/bidi_integration_tests.py
py/test/selenium/webdriver/common/bidi_log_tests.py

Files to Expand

py/test/selenium/webdriver/common/bidi_browser_tests.py (+5-10 tests)
py/test/selenium/webdriver/common/bidi_browsing_context_tests.py (+10-15 tests)
py/test/selenium/webdriver/common/bidi_emulation_tests.py (+8-12 tests)
py/test/selenium/webdriver/common/bidi_input_tests.py (+5-10 tests)
py/test/selenium/webdriver/common/bidi_network_tests.py (+15-20 tests)
py/test/selenium/webdriver/common/bidi_script_tests.py (+10-15 tests)
py/test/selenium/webdriver/common/bidi_session_tests.py (+5-8 tests)
py/test/selenium/webdriver/common/bidi_storage_tests.py (+8-12 tests)
py/test/selenium/webdriver/common/bidi_webextension_tests.py (+4-6 tests)

Bazel Integration - Automatic Test Discovery

How New BiDi Tests Are Automatically Picked Up

The Selenium build system automatically includes all BiDi test files without requiring changes to the Bazel configuration.

Key Mechanism (py/BUILD.bazel:633):

BIDI_TESTS = glob(["test/selenium/webdriver/common/**/*bidi*_tests.py"])

This glob pattern automatically matches any file in test/selenium/webdriver/common/ that:

1. Contains bidi in the filename
2. Ends with _tests.py

Test Target Organization

Non-BiDi Tests (test-{browser}):

• BiDi tests are explicitly EXCLUDED
• Prevents BiDi-only tests from running without BiDi mode

BiDi Tests (test-{browser}-bidi):

• All BiDi files are INCLUDED
• --bidi flag is automatically passed
• Only excludes print_pdf_tests.py
• Generated for: Chrome, Edge, Firefox (browsers with "bidi": True)

Naming Convention for New Tests

To ensure automatic inclusion, follow this pattern:

bidi_<module>_tests.py

Examples:

• ✅ bidi_errors_tests.py - Automatically included
• ✅ bidi_integration_tests.py - Automatically included
• ✅ bidi_log_tests.py - Automatically included
• ❌ errors_tests.py - NOT included (missing "bidi")
• ❌ bidi_errors.py - NOT included (missing "_tests")

Test Execution

# Run all BiDi tests for a browser
bazel test //py:test-chrome-bidi
bazel test //py:test-firefox-bidi
bazel test //py:test-edge-bidi

# Run with filter
bazel test //py:test-chrome-bidi --test_filter=test_specific_test_name

# Verify cache after creating new files
bazel clean && bazel test //py:test-chrome-bidi

No Bazel Configuration Changes Required ✅

When creating the new test files (bidi_errors_tests.py, bidi_integration_tests.py, bidi_log_tests.py):

• ✅ No changes needed to py/BUILD.bazel
• ✅ No new targets to define
• ✅ Automatic inclusion via glob pattern
• ✅ Automatically excluded from non-BiDi tests
• ✅ Automatically included in test-{browser}-bidi targets

Reference Links

• WPT BiDi Tests: https://github.com/web-platform-tests/wpt/tree/master/webdriver/tests/bidi
• WPT BiDi Spec: https://w3c.github.io/webdriver-bidi/
• Selenium WebDriver: https://github.com/SeleniumHQ/selenium
• Bazel Integration Guide: BIDI_BAZEL_SETUP.md

Style and Standards

Code Style Compliance

All new BiDi tests must follow the established style guide:

• Style Guide: BIDI_TEST_STYLE_GUIDE.md
  • License headers
  • Import organization
  • Test naming conventions
  • Assertion patterns
  • Event handler patterns
  • Resource cleanup patterns
  • Helper function organization
  • Docstring requirements

Key Style Points

• ✅ Apache 2.0 license header on all files
• ✅ One-liner docstrings for test functions
• ✅ Test function names: test_<description> in snake_case
• ✅ Explicit resource cleanup (remove handlers, close contexts)
• ✅ Use @pytest.mark.xfail_* for browser-specific issues
• ✅ Group related tests in classes with @pytest.fixture(autouse=True) setup
• ✅ Use pages.load() and pages.url() for test page access
• ✅ Use WebDriverWait with lambda for async operations

Notes

• This plan assumes WPT is the source of truth for BiDi protocol testing
• Some Selenium-specific features (permissions) may require additional tests beyond WPT
• Test organization should mirror WPT structure for clarity and maintainability
• Each test module should follow Selenium's existing test patterns and fixtures
• All code must match the style documented in BIDI_TEST_STYLE_GUIDE.md