Skip to content

chore: test loading multi accounts#16732

Merged
tommasini merged 72 commits into
mainfrom
test-loading-multi-accounts
Jul 23, 2025
Merged

chore: test loading multi accounts#16732
tommasini merged 72 commits into
mainfrom
test-loading-multi-accounts

Conversation

@tommasini

@tommasini tommasini commented Jun 26, 2025

Copy link
Copy Markdown
Contributor

Description

🎯 Performance Quality Gates

This document outlines the performance thresholds (quality gates) for MetaMask Mobile's critical user flows. These thresholds ensure optimal user experience across different platforms and scenarios.

📱 Platform Overview

  • Android: Generally has higher thresholds due to platform constraints
  • iOS: Lower thresholds leveraging platform optimizations
  • Quality Gates: Hard limits that cause test failures if exceeded

📊 Test Reporting

Performance tests automatically generate detailed reports using the PerformanceTestReporter utility:

  • JSON Reports: Structured performance data for analysis
  • Test Results: Include timing metrics, thresholds, and pass/fail status
  • User Profile Testing: Tests run across different user states (CORE_USER, POWER_USER)

🏠 Account List Performance Tests

Test: render account list efficiently with multiple accounts and networks

Configuration: Multiple accounts, popular networks, profile syncing enabled

Platform Total Max Time Notes
Android 5000ms 5 seconds maximum
iOS 7,500ms 7.5 seconds maximum

Test: handle account list performance with heavy token load

Configuration: Multiple accounts, popular networks, 10 tokens for stress testing

Platform Total Max Time Notes
Android 5000ms 5 seconds maximum
iOS 7,500ms 7.5 seconds maximum

Test: benchmark account list with minimal load

Configuration: Minimal accounts, default network, 2 tokens (baseline measurement)

Platform Total Max Time Notes
Android 45,000ms 45 seconds maximum
iOS 15,000ms 15 seconds maximum

Test: benchmark switching accounts from the account list

Configuration: Account switching/dismissal performance

Platform Dismissal Max Time Notes
Android 5,000ms 5 seconds maximum
iOS 4,000ms 4 seconds maximum

🌐 Network List Performance Tests

Test: render network list efficiently with multiple accounts and all popular networks

Configuration: Multiple accounts, all popular networks

Platform Total Max Time Notes
Android 17,500ms 17.5 seconds maximum
iOS 6,500ms 6.5 seconds maximum

Test: handle network list performance with heavy token load on all popular networks

Configuration: Multiple accounts, popular networks, 10 tokens for stress testing

Platform Total Max Time Notes
Android 17,500ms 17.5 seconds maximum
iOS 6,500ms 6.5 seconds maximum

Test: benchmark network list with minimal load

Configuration: Minimal tokens, popular networks (baseline measurement)

Platform Render Max Time Notes
Android 2,500ms 2.5 seconds maximum
iOS 1,500ms 1.5 seconds maximum

Test: benchmark switching networks from the network list

Configuration: Network switching/dismissal performance

Platform Dismissal Max Time Notes
Android 2,500ms 2.5 seconds maximum
iOS 1,500ms 1.5 seconds maximum

📊 Test Summary

Account List Tests (4 tests)

  • Standard Load: Multiple accounts, popular networks
  • Heavy Load: Multiple accounts, 10 tokens, popular networks
  • Baseline Test: Minimal accounts, 2 tokens, default network
  • Dismissal Test: Account switching performance

Network List Tests (4 tests)

  • Standard Load: Multiple accounts, popular networks
  • Heavy Load: Multiple accounts, 10 tokens, popular networks
  • Baseline Test: Minimal tokens, popular networks
  • Dismissal Test: Network switching performance

Total: 8 performance tests across critical user flows


🚨 Quality Gate Rules

Failure Criteria

Tests fail immediately when total time exceeds the maximum acceptable time for any scenario.

Performance Patterns

  • Heavy Token Load: Increases render times but maintains same thresholds as standard load
  • Platform Differences: iOS consistently performs better than Android
  • Baseline Tests: Should complete quickly but have generous thresholds for stability
  • Dismissal Tests: Focus on UI responsiveness during state transitions

User Profile Testing

Tests run across different user states with varying account complexity:

User Profile Definitions

  • CASUAL_USER: 2 EVM accounts from 1 SRP (currently not used in tests)
  • CORE_USER: 5 EVM accounts and 5 Solana accounts from 1 SRP
  • POWER_USER: 15 EVM accounts from 2 SRPs + 5 Solana accounts

Current Test Coverage

  • CORE_USER: Standard user configuration for baseline performance
  • POWER_USER: Enhanced user configuration with maximum account complexity

📈 Reporting Features

Automated Reports

  • JSON Output: Structured performance data for CI/CD integration
  • Test Metrics: Total time measurements and performance thresholds
  • Threshold Tracking: Pass/fail status with actual vs. expected performance
  • User Profile Results: Separate results for different user configurations

Report Usage

Performance reports can be used for:

  • Continuous integration quality gates
  • Performance regression detection
  • Platform-specific optimization insights
  • User experience benchmarking

What can we do to improve it

  • Instead of measuring the action once, we perform the action multiple times and we get an average and the top duration, both need to have base lines.
  • To get more granular function reports, we can use this sample work done to measure multiple functions of one flow, for example if add account if it's split in 2 functions, instead of one big measure, we can measure both functions, and get more deep knowledge of where we can improve Feat: local dev perf tests using e2e #16806

Bitrise Runs

Table if 8 bitrise runs results

Test Name Platform Total Time (Runs) Min Max Delta Average
render account list efficiently with multiple accounts and networks IOS 2.54, 2.61, 3.65, 2.46, 2.88, 3.1, 2.94, 2.87 2.46 3.65 1.19 2.88
render account list efficiently with multiple accounts and networks ANDROID 2.83, 2.64, 2.92, 2.28, 2.62, 2.76, 1.95, 2.37 1.95 2.92 0.97 2.55
handle account list performance with heavy token load IOS 4.01, 5.72, 4.02, 5.3, 2.49, 2.48, 2.67, 2.78 2.48 5.72 3.24 3.93
handle account list performance with heavy token load ANDROID 2.15, 2.51, 1.97, 2.78, 2.13, 2.29, 2.23, 2.41 1.97 2.78 0.81 2.29
benchmark account list with minimal load IOS 2.86, 3.48, 2.87, 3.32, 2.81, 3.35, 2.85, 3.13 2.81 3.48 0.67 3.08
benchmark account list with minimal load ANDROID 2.08, 2.17, 2.59, 2.28, 2.07, 2.19, 2.13, 2.49 2.07 2.59 0.52 2.25
benchmark switching networks from the network list IOS 6.61, 8.11, 7.65, 8.08, 7.64, 6.53, 6.49, 6.55 6.49 8.11 1.62 7.34
benchmark switching networks from the network list ANDROID 7.68, 4.26, 4.11, 5.6, 4.04, 4.32, 4.11, 4.28 4.04 7.68 3.64 4.92
benchmark switching accounts from the account list IOS 5.99, 4.37, 3.14, 4.55, 3.36, 4.33 3.14 5.99 2.85 4.29
benchmark switching accounts from the account list ANDROID 2.01, 2.28, 1.71, 2.05, 1.77, 2.28, 1.75, 2.39 1.71 2.39 0.68 2.06

Related issues

Fixes:

Manual testing steps

  1. Go to this page...

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

@tommasini tommasini requested a review from a team as a code owner June 26, 2025 23:38
@github-actions

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@tommasini tommasini marked this pull request as draft June 26, 2025 23:38
Comment thread e2e/fixtures/constants.ts
Comment thread e2e/fixtures/constants.ts Outdated
Comment thread e2e/fixtures/constants.ts Outdated
Comment thread e2e/fixtures/constants.ts Outdated
Comment thread e2e/fixtures/fixture-builder.js Outdated
Comment thread e2e/fixtures/fixture-builder.js Outdated
@github-actions

github-actions Bot commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: b3ed039
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/ce2fa8a0-a437-443a-bb5f-92b49a2a9e3c

Note

  • You can rerun any failed steps by opening the Bitrise build, tapping Rebuild on the upper right then Rebuild unsuccessful Workflows
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

Tip

  • Check the documentation if you have any doubts on how to understand the failure on bitrise

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Keyring Account Mismatch and ID Collision

The withImportedHdKeyringAndTwoDefaultAccountsOneImportedHdAccountOneQrAccountOneSimpleKeyPairAccount method has two issues:

  1. The first HD Key Tree keyring now contains only one account, contradicting the "TwoDefaultAccounts" implied by the method's name.
  2. Two distinct keyrings (QR Hardware Wallet Device and an HD Key Tree) within this method are assigned the same non-unique metadata.id.

e2e/fixtures/fixture-builder.js#L1497-L1516

{
accounts: ['0xbacec2e26c5c794de6e82a1a7e21b9c329fa8cf6'],
metadata: { id: '01JXA9M05DGJ92SMSEPFG8VN17', name: '' },
type: 'HD Key Tree',
},
{
accounts: ['0x428f04e9ea21b31090d377f24501065cbb48512f'],
metadata: { id: '01JXA9MYRXNW16JZSZJXD6F9SD', name: '' },
type: 'QR Hardware Wallet Device',
},
{
accounts: ['0x43e1c289177ecfbe6ef34b5fb2b66ebce5a8e05b'],
metadata: { id: '01JXA9MYRXNW16JZSZJXD6F9SD', name: '' },
type: 'HD Key Tree',
},
{
accounts: ['0x84b4ebb7492e6deb8892b16b0ee425e39d3116a4'],
metadata: { id: '01JXA9YPC99YPE39C063RYGVX1', name: '' },
type: 'Simple Key Pair',
},

Fix in CursorFix in Web


Bug: iOS Test Thresholds Set Incorrectly

All iOS performance test thresholds are incorrectly set to 5000ms, instead of the intended 7500ms. This discrepancy causes iOS tests to fail prematurely.

e2e/specs/performance/account-list/render-account-list.spec.ts#L31-L38

const isAndroid = device.getPlatform() === 'android';
const PERFORMANCE_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 5000, // 5 seconds max for Android
}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS
};

e2e/specs/performance/account-list/render-account-list.spec.ts#L124-L131

const isAndroid = device.getPlatform() === 'android';
const HEAVY_LOAD_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 5000, // 5 seconds max for Android
}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS
};

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@github-actions

github-actions Bot commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: eb0ba47
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/d4e59734-5bf8-4958-8970-d3c57981d520

Note

  • You can rerun any failed steps by opening the Bitrise build, tapping Rebuild on the upper right then Rebuild unsuccessful Workflows
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

Tip

  • Check the documentation if you have any doubts on how to understand the failure on bitrise

@github-actions

github-actions Bot commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: c9e69b6
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/1c8f165c-a101-476a-aa02-82ccb78b0df0

Note

  • You can rerun any failed steps by opening the Bitrise build, tapping Rebuild on the upper right then Rebuild unsuccessful Workflows
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

Tip

  • Check the documentation if you have any doubts on how to understand the failure on bitrise

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Network List Test Uses Incorrect Thresholds

The "heavy load" network list performance test uses incorrect thresholds. Both Android and iOS are currently set to 8000ms, but should be 17500ms for Android and 6500ms for iOS, aligning with the pattern established in the first performance test in the same file.

e2e/specs/performance/network-list/render-network-list.spec.ts#L130-L137

const HEAVY_LOAD_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 8000, // 8 seconds max for Android
}
: {
TOTAL_TIME: 8000, // 8 seconds max for iOS
};

Fix in CursorFix in Web


Bug: Test Passes Despite Performance Issues

The performance test logs "Performance test passed!" unconditionally, even when the measured time exceeds the defined threshold and a warning is logged. This creates misleading output where failed tests appear to pass.

e2e/specs/performance/network-list/render-network-list.spec.ts#L88-L95

if (totalTime > PERFORMANCE_THRESHOLDS.TOTAL_TIME) {
console.warn(
`Performance test failed: Total time (${totalTime}ms) exceeded maximum acceptable time (${PERFORMANCE_THRESHOLDS.TOTAL_TIME}ms)`,
);
}
console.log('Performance test passed!');

Fix in CursorFix in Web


Bug: Network List Test Timing Inconsistency

The "handle network list performance with heavy token load" test has an inaccurate and inconsistent timing measurement. The timer starts after tapping the networks button, excluding navigation time. Crucially, the endTime is recorded before the network list is checked for visibility, meaning the time taken for the list to actually render is not included. This makes the reported performance metrics incomplete and inconsistent with other tests.

e2e/specs/performance/network-list/render-network-list.spec.ts#L239-L242

const startTime = Date.now();
await Assertions.checkIfVisible(NetworkListModal.networkScroll);
const endTime = Date.now();

e2e/specs/performance/network-list/render-network-list.spec.ts#L159-L166

const startTime = Date.now();
await WalletView.tapNetworksButtonOnNavBar();
const endTime = Date.now();
await Assertions.checkIfVisible(NetworkListModal.networkScroll);
const totalTime = endTime - startTime;

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@github-actions

github-actions Bot commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: 4917188
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/1e360e54-5f8c-4387-95d8-c4b542abebf9

Note

  • You can rerun any failed steps by opening the Bitrise build, tapping Rebuild on the upper right then Rebuild unsuccessful Workflows
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

Tip

  • Check the documentation if you have any doubts on how to understand the failure on bitrise

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Account List Dismissal Performance Threshold Mismatch

The performance threshold for dismissing the account list in the 'benchmark switching accounts from the account list' test is incorrectly set to 4000ms for Android. According to quality gates documentation, Android's threshold should be 5000ms, while iOS's 4000ms is correct. This inconsistency may cause incorrect test failures on Android.

e2e/specs/performance/account-list/dismiss-account-list.spec.ts#L54-L60

const PERFORMANCE_THRESHOLDS = isAndroid
? {
DISMISS_ACCOUNT_LIST: 4000, // 4 seconds max for Android
}
: {
DISMISS_ACCOUNT_LIST: 4000, // 4 seconds max for iOS
};

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@github-actions

github-actions Bot commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

🔄🔄🔄 pr_smoke_e2e_pipeline started on Bitrise...🔄🔄🔄

Commit hash: 627096c
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/b5ab9a93-14fb-4a0b-be18-a049e44aa8a7

Note

  • This comment will auto-update when build completes
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Performance Test Thresholds Mismatch PR Specs

Performance thresholds are inconsistent with the PR description across all account list performance tests. The code currently sets both Android and iOS thresholds to 5000ms for general rendering, heavy token load, and minimal load baseline tests. However, the PR description specifies different thresholds:

  • General Rendering: iOS should be 7500ms (Android 5000ms).
  • Heavy Load: Android and iOS should have different thresholds (not both 5000ms).
  • Minimal Load Baseline: Android should be 45000ms, iOS should be 15000ms.

This discrepancy leads to incorrect performance expectations and affects test reliability across platforms.

e2e/specs/performance/account-list/render-account-list.spec.ts#L31-L38

const isAndroid = device.getPlatform() === 'android';
const PERFORMANCE_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 5000, // 5 seconds max for Android
}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS
};

e2e/specs/performance/account-list/render-account-list.spec.ts#L124-L131

const isAndroid = device.getPlatform() === 'android';
const HEAVY_LOAD_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 5000, // 5 seconds max for Android
}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS
};

e2e/specs/performance/account-list/render-account-list.spec.ts#L193-L200

const isAndroid = device.getPlatform() === 'android';
const BASELINE_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 5000, // 5 seconds max for Android
}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS
};

Fix in CursorFix in Web


Bug: iOS Tests Misconfigured on Linux Stack

The ios_e2e_test_performance workflow is incorrectly configured to use the linux-docker-android-22.04 stack. iOS tests require a macOS stack to run simulators, making the current configuration invalid and causing the workflow to fail.

bitrise.yml#L1840-L1843

metamask-mobile/bitrise.yml

Lines 1840 to 1843 in 627096c

meta:
bitrise.io:
machine_type_id: elite-xl
stack: linux-docker-android-22.04

bitrise.yml#L1988-L1991

metamask-mobile/bitrise.yml

Lines 1988 to 1991 in 627096c

meta:
bitrise.io:
machine_type_id: elite-xl
stack: linux-docker-android-22.04

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@github-actions

github-actions Bot commented Jul 23, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: c65c0eb
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/10dc9b6d-2c5d-4d76-b347-091deb358606

Note

  • You can rerun any failed steps by opening the Bitrise build, tapping Rebuild on the upper right then Rebuild unsuccessful Workflows
  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

Tip

  • Check the documentation if you have any doubts on how to understand the failure on bitrise

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Token Load Test Has Incorrect Thresholds

The heavy token load performance test uses incorrect thresholds. Both Android and iOS are currently set to 8000ms, but should be 17500ms for Android and 6500ms for iOS, consistent with the render network list efficiently test. This discrepancy makes the Android test too strict and the iOS test too lenient, leading to inaccurate performance results.

e2e/specs/performance/network-list/render-network-list.spec.ts#L130-L137

const HEAVY_LOAD_THRESHOLDS = isAndroid
? {
TOTAL_TIME: 8000, // 8 seconds max for Android
}
: {
TOTAL_TIME: 8000, // 8 seconds max for iOS
};

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@github-actions

github-actions Bot commented Jul 23, 2025

Copy link
Copy Markdown
Contributor

https://bitrise.io/ Bitrise

✅✅✅ pr_smoke_e2e_pipeline passed on Bitrise! ✅✅✅

Commit hash: 2f7a994
Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7bee5831-f2dc-44ef-869f-e0115513bf5e

Note

  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: iOS Account List Load Times Incorrectly Set

The iOS performance thresholds for account list loading tests (general, heavy load, and baseline) are incorrectly set to 5000ms. They should be 7500ms, and distinct from the Android thresholds.

e2e/specs/performance/account-list/render-account-list.spec.ts#L35-L37

}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS

e2e/specs/performance/account-list/render-account-list.spec.ts#L128-L130

}
: {
TOTAL_TIME: 5000, // 5 seconds max for iOS

Fix in CursorFix in Web


Was this report helpful? Give feedback by reacting with 👍 or 👎

@MarioAslau MarioAslau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

INVALID-PR-TEMPLATE PR's body doesn't match template No QA Needed Apply this label when your PR does not need any QA effort. release-7.53.0 Issue or pull request that will be included in release 7.53.0 team-mobile-platform Mobile Platform team team-qa QA team

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

9 participants