Summary

Several Playwright E2E test stability issues identified during PR #2963 (SRI implementation) review. These are pre-existing patterns that cause intermittent test flakiness, not regressions from #2963.

Flaky Tests

The following 13 tests failed intermittently during CI but pass on re-run. None are caused by the SRI PR — confirmed with A/B testing, network tracing, and JS framework verification.

Export/Import (transient 401s)

• tests/playwright/operations/test_export_import.py::TestExport::test_export_with_type_filter
• tests/playwright/operations/test_export_import.py::TestExport::test_export_includes_inactive
• tests/playwright/operations/test_export_import.py::TestExport::test_selective_export (KeyError: 'id')
• tests/playwright/operations/test_export_import.py::TestImport::test_import_status_list

MCP Registry (transient timeouts)

• tests/playwright/test_mcp_registry_page.py::TestMCPRegistryPage::test_registry_panel_loads
• tests/playwright/test_mcp_registry_page.py::TestMCPRegistryPage::test_overview_card_displays_counts
• tests/playwright/test_mcp_registry_page.py::TestMCPRegistryPage::test_refresh_catalog_button_visible
• tests/playwright/test_mcp_registry_page.py::TestMCPRegistryPage::test_filter_controls_visible
• tests/playwright/test_mcp_registry_page.py::TestMCPRegistryPage::test_statistics_cards_visible

Entity CRUD (transient timeouts / 400s)

• tests/playwright/entities/test_servers_extended.py::TestServersExtended::test_edit_server_button
• tests/playwright/entities/test_users.py::TestUsersCRUD::test_edit_user (400 on creation)
• tests/playwright/entities/test_users.py::TestUsersCRUD::test_force_password_change

Gateway Creation (cleanup race condition — fails ~2/3 runs)

• tests/playwright/test_gateways.py::TestGatewayCreation::test_create_simple_gateway
  • Core test logic passes (gateway created, verified, cleaned up)
  • Failure is in clear_search() after deleting the last gateway — HTMX swap race leaves #gateways-table-body detached
  • clear_search() lacks the recovery path that search_gateways() already has (reload + navigate on timeout)

Root Cause Patterns

1. Fail fast on JS bootstrap failure

Files: tests/playwright/conftest.py:199-222, tests/playwright/test_rbac_permissions.py:110-113

The _ensure_admin_logged_in helper waits 60s for [data-testid="servers-tab"] but catches PlaywrightTimeoutError silently in some paths. If Alpine.js or HTMX fail to load (e.g., CDN blocked, CORS error), tests proceed with a broken page and fail with confusing timeout errors on unrelated selectors instead of a clear "JS framework failed to load" message.

Proposal: Add an explicit JS readiness assertion after _ensure_admin_logged_in returns:

assert page.evaluate("typeof Alpine !== 'undefined' && typeof htmx !== 'undefined'"), \
    "Alpine.js or HTMX failed to load — check CDN availability and CORS configuration"

2. Stop hardcoding servers-tab as shell-ready signal

Files: tests/playwright/conftest.py:200

The admin shell readiness check uses [data-testid="servers-tab"] as the indicator that the page has loaded. This is fragile — if the sidebar HTML structure changes or the servers tab is renamed, all tests break. The intent is to confirm that the SPA shell has initialized, not that a specific tab exists.

Proposal: Use a dedicated data-testid="admin-shell" attribute on the main layout container, or check for showTab function availability (already partially done at line 217).

3. Replace sleep-based synchronization with explicit waits

Files: Multiple test files use page.wait_for_timeout() for synchronization

Users CRUD tests (test_users.py) and other entity tests use fixed wait_for_timeout(1000) calls to wait for HTMX outerHTML swaps. These are inherently racy — too short on slow CI, wasteful on fast machines.

Proposal: Replace wait_for_timeout calls with explicit wait_for_selector or expect(...).to_be_visible() assertions that wait for the actual DOM mutation to complete. For HTMX swaps, wait for the absence of .htmx-request class on the loading indicator.

4. Security token fixture hardcoded admin identity

Files: tests/playwright/conftest.py:113-136, tests/playwright/test_rbac_permissions.py:70-91

Both _set_admin_jwt_cookie and _inject_jwt_cookie hardcode admin@example.com and use _create_jwt_token directly. If the JWT secret key, claims structure, or admin email changes, these helpers silently produce invalid tokens. The InsecureKeyLengthWarning on every test run is a symptom of this.

Proposal: Source the admin email from Settings().platform_admin_email consistently, and consider a shared create_test_jwt() helper that centralizes token creation with proper key length validation.

5. clear_search() missing recovery path

Files: tests/playwright/pages/gateways_page.py:441-479

search_gateways() has a recovery path (reload + navigate) when #gateways-table-body isn't found after an HTMX swap, but clear_search() does not. This causes test_create_simple_gateway to fail ~2/3 of the time during cleanup.

Proposal: Add the same recovery logic from search_gateways lines 424-429 to clear_search.

Context

Discovered during code review of PR #2963 (SRI implementation). The SRI PR's addition of crossorigin="anonymous" to CDN script tags exposed one of these issues (the CORS + set_extra_http_headers interaction), which has been fixed in that PR. The remaining items are independent improvements.