Skip to content

feat: integrate token list controller storage service#24019

Merged
sahar-fehri merged 23 commits intomainfrom
feat/integrate-tokenListController-storageService
Jan 28, 2026
Merged

feat: integrate token list controller storage service#24019
sahar-fehri merged 23 commits intomainfrom
feat/integrate-tokenListController-storageService

Conversation

@sahar-fehri
Copy link
Copy Markdown
Contributor

@sahar-fehri sahar-fehri commented Dec 15, 2025

Description

Do not merge until this is released MetaMask/core#7413

Performance Comparison: Per-Chain Token Cache Storage

This PR implements per-chain file storage for tokensChainsCache in TokenListController, replacing the single-file approach. Each chain's token list is now stored in a separate file, reducing write amplification during incremental updates.


📊 Complete Performance Comparison

Cold Restart

Metric This PR Main Branch
getAllPersistedState 235ms 288ms
TokenListController read 0.04KB (shell only) 4,102KB
Cache load 97ms (parallel reads) 135ms (single file)
Total overhead ~332ms ~288ms

Main is ~44ms faster on cold restart (single file read vs parallel reads + getAllKeys overhead)


Onboarding

Metric This PR Main Branch
Total data written 4,070KB 9,472KB
Number of writes 7 (one per chain) 5 (cumulative rewrites)
Total write time ~38ms ~118ms

This PR writes 57% less data and is 3x faster


Add New Chain (Monad)

Metric This PR Main Branch
Data written 33.79KB 4,103KB
Time 0.23ms 45.34ms

This PR is 121x smaller and 197x faster!


Summary

Category This PR Main Branch Winner
Cold restart ~332ms ~288ms Main (+44ms)
Onboarding writes 4,070KB 9,472KB This PR (-57%)
Onboarding time ~38ms ~118ms This PR (3x faster)
Add chain writes 33.79KB 4,103KB This PR (-99%)
Add chain time 0.23ms 45.34ms This PR (197x faster)
Write amplification None Severe This PR

📋 Captured Logs

This PR - Cold Restart

[ControllerStorage PERF] getAllPersistedState started
[ControllerStorage PERF] TokenListController - 0.04KB - read: 89.00ms, parse: 0.00ms, total: 89.00ms
[ControllerStorage PERF] getAllPersistedState complete - 235.37ms

[StorageService PERF] getAllKeys TokenListController - 7 keys found - 277.37ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xa - 96.19KB - read: 3.12ms, parse: 0.47ms, total: 3.59ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x1 - 1608.95KB - read: 30.86ms, parse: 10.57ms, total: 41.43ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x38 - 1288.32KB - read: 48.95ms, parse: 21.65ms, total: 70.60ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x89 - 324.12KB - read: 72.62ms, parse: 5.21ms, total: 77.83ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xa4b1 - 222.52KB - read: 77.90ms, parse: 7.06ms, total: 84.96ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xe708 - 46.92KB - read: 85.16ms, parse: 0.82ms, total: 85.97ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x2105 - 481.64KB - read: 88.85ms, parse: 8.74ms, total: 97.58ms

This PR - Onboarding

[ControllerStorage PERF] getAllPersistedState complete - 731.91ms

[StorageService PERF] getAllKeys TokenListController - 0 keys found - 309.51ms
[StorageService PERF] getItem TokenListController:tokensChainsCache - NOT FOUND - 33.51ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x1 - 1610.14KB - stringify: 8.64ms, write: 8.54ms, total: 17.17ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xe708 - 46.92KB - stringify: 0.19ms, write: 0.08ms, total: 0.26ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x2105 - 481.34KB - stringify: 1.50ms, write: 2.45ms, total: 3.96ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xa4b1 - 222.53KB - stringify: 1.03ms, write: 0.52ms, total: 1.56ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x38 - 1288.32KB - stringify: 4.74ms, write: 6.49ms, total: 11.23ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xa - 96.19KB - stringify: 0.31ms, write: 0.52ms, total: 0.83ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x89 - 324.46KB - stringify: 1.10ms, write: 1.72ms, total: 2.82ms

This PR - Add New Chain (Monad)

[StorageService PERF] setItem TokenListController:tokensChainsCache:0x8f - 33.79KB - stringify: 0.17ms, write: 0.07ms, total: 0.23ms

Main Branch - Cold Restart

[ControllerStorage PERF] getAllPersistedState started
[ControllerStorage PERF] TokenListController - 4102.55KB - read: 112.51ms, parse: 22.77ms, total: 135.27ms
[ControllerStorage PERF] getAllPersistedState complete - 288.21ms

Main Branch - Onboarding

[ControllerStorage PERF] getAllPersistedState complete - 785.03ms

[ControllerStorage PERF] setItem TokenListController - 0.06KB - stringify: 0.00ms, write: 0.02ms, total: 0.02ms
[ControllerStorage PERF] setItem TokenListController - 1609.28KB - stringify: 13.41ms, write: 11.58ms, total: 24.99ms
[ControllerStorage PERF] setItem TokenListController - 1656.21KB - stringify: 12.85ms, write: 12.20ms, total: 25.04ms
[ControllerStorage PERF] setItem TokenListController - 2137.56KB - stringify: 12.47ms, write: 11.40ms, total: 23.87ms
[ControllerStorage PERF] setItem TokenListController - 4068.75KB - stringify: 22.00ms, write: 22.62ms, total: 44.62ms

Main Branch - Add New Chain (Monad)

[ControllerStorage PERF] setItem TokenListController - 4102.55KB - stringify: 23.52ms, write: 21.82ms, total: 45.34ms

🔧 Performance Logging Code (Main Branch)

The following code was added to app/store/persistConfig/index.ts to capture performance metrics:

Read Performance Logging (getAllPersistedState)

async getAllPersistedState(): Promise<Record<string, unknown>> {
  // eslint-disable-next-line no-console
  console.warn('[ControllerStorage PERF] getAllPersistedState started');
  const totalStart = performance.now();
  try {
    const backgroundState: Record<string, unknown> = {};

    await Promise.all(
      Array.from(
        new Set(
          Array.from(BACKGROUND_STATE_CHANGE_EVENT_NAMES).map(
            (eventName) => eventName.split(':')[0],
          ),
        ),
      ).map(async (controllerName) => {
        const key = `persist:${controllerName}`;
        const startTime = performance.now();
        try {
          const data = await FilesystemStorage.getItem(key);
          if (data) {
            const parseStart = performance.now();
            const parsedData = JSON.parse(data);
            const parseDuration = performance.now() - parseStart;
            const totalDuration = performance.now() - startTime;

            // Log performance for TokenListController specifically
            if (controllerName === 'TokenListController') {
              const sizeKB = (data.length / 1024).toFixed(2);
              // eslint-disable-next-line no-console
              console.warn(
                `[ControllerStorage PERF] ${controllerName} - ${sizeKB}KB - ` +
                  `read: ${(totalDuration - parseDuration).toFixed(2)}ms, ` +
                  `parse: ${parseDuration.toFixed(2)}ms, ` +
                  `total: ${totalDuration.toFixed(2)}ms`,
              );
            }
            // ... rest of the function
          }
        } catch (error) {
          // error handling
        }
      }),
    );

    const totalDuration = performance.now() - totalStart;
    // eslint-disable-next-line no-console
    console.warn(
      `[ControllerStorage PERF] getAllPersistedState complete - ${totalDuration.toFixed(2)}ms`,
    );

    return { backgroundState };
  } catch (error) {
    // error handling
  }
}

Write Performance Logging (createPersistController)

export const createPersistController = (debounceMs: number = 200) =>
  debounce(async (filteredState: unknown, controllerName: string) => {
    const startTime = performance.now();
    try {
      const stringifyStart = performance.now();
      const serialized = JSON.stringify(filteredState);
      const stringifyDuration = performance.now() - stringifyStart;

      await ControllerStorage.setItem(`persist:${controllerName}`, serialized);

      const totalDuration = performance.now() - startTime;
      if (controllerName === 'TokenListController') {
        const sizeKB = (serialized.length / 1024).toFixed(2);
        // eslint-disable-next-line no-console
        console.warn(
          `[ControllerStorage PERF] setItem ${controllerName} - ${sizeKB}KB - ` +
            `stringify: ${stringifyDuration.toFixed(2)}ms, ` +
            `write: ${(totalDuration - stringifyDuration).toFixed(2)}ms, ` +
            `total: ${totalDuration.toFixed(2)}ms`,
        );
      }
      Logger.log(`${controllerName} state persisted successfully`);
    } catch (error) {
      // error handling
    }
  }, debounceMs);

🔧 Performance Logging Code (This PR)

The following code was added to app/core/Engine/controllers/storage-service-init.ts to capture performance metrics for the per-chain storage:

getItem - Read Performance Logging

async getItem(namespace: string, key: string): Promise<StorageGetResult> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] getItem called: ${namespace}:${key}`);
  const startTime = performance.now();
  try {
    const fullKey = `${STORAGE_KEY_PREFIX}${namespace}:${key}`;
    const serialized = await FilesystemStorage.getItem(fullKey);

    // Key not found - return empty object
    if (serialized === undefined || serialized === null) {
      const duration = performance.now() - startTime;
      if (
        key.includes('token') ||
        key.includes('Token') ||
        namespace.includes('Token')
      ) {
        // eslint-disable-next-line no-console
        console.warn(
          `[StorageService PERF] getItem ${namespace}:${key} - NOT FOUND - ${duration.toFixed(2)}ms`,
        );
      }
      return {};
    }

    const parseStart = performance.now();
    const result = JSON.parse(serialized) as Json;
    const parseDuration = performance.now() - parseStart;
    const totalDuration = performance.now() - startTime;

    if (
      key.includes('token') ||
      key.includes('Token') ||
      namespace.includes('Token')
    ) {
      const sizeKB = (serialized.length / 1024).toFixed(2);
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] getItem ${namespace}:${key} - ${sizeKB}KB - ` +
          `read: ${(totalDuration - parseDuration).toFixed(2)}ms, ` +
          `parse: ${parseDuration.toFixed(2)}ms, ` +
          `total: ${totalDuration.toFixed(2)}ms`,
      );
    }

    return { result };
  } catch (error) {
    // error handling
  }
}

setItem - Write Performance Logging

async setItem(namespace: string, key: string, value: Json): Promise<void> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] setItem called: ${namespace}:${key}`);
  const startTime = performance.now();
  try {
    const fullKey = `${STORAGE_KEY_PREFIX}${namespace}:${key}`;

    const stringifyStart = performance.now();
    const serialized = JSON.stringify(value);
    const stringifyDuration = performance.now() - stringifyStart;

    await FilesystemStorage.setItem(fullKey, serialized, Device.isIos());

    const totalDuration = performance.now() - startTime;

    if (
      key.includes('token') ||
      key.includes('Token') ||
      namespace.includes('Token')
    ) {
      const sizeKB = (serialized.length / 1024).toFixed(2);
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] setItem ${namespace}:${key} - ${sizeKB}KB - ` +
          `stringify: ${stringifyDuration.toFixed(2)}ms, ` +
          `write: ${(totalDuration - stringifyDuration).toFixed(2)}ms, ` +
          `total: ${totalDuration.toFixed(2)}ms`,
      );
    }
  } catch (error) {
    // error handling
  }
}

getAllKeys - Key Enumeration Logging

async getAllKeys(namespace: string): Promise<string[]> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] getAllKeys called: ${namespace}`);
  const startTime = performance.now();
  try {
    const allKeys = await FilesystemStorage.getAllKeys();

    if (!allKeys) {
      const duration = performance.now() - startTime;
      if (namespace.includes('Token')) {
        // eslint-disable-next-line no-console
        console.warn(
          `[StorageService PERF] getAllKeys ${namespace} - 0 keys - ${duration.toFixed(2)}ms`,
        );
      }
      return [];
    }

    const prefix = `${STORAGE_KEY_PREFIX}${namespace}:`;
    const filteredKeys = allKeys
      .filter((key) => key.startsWith(prefix))
      .map((key) => key.slice(prefix.length));

    const duration = performance.now() - startTime;
    if (namespace.includes('Token')) {
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] getAllKeys ${namespace} - ${filteredKeys.length} keys found - ${duration.toFixed(2)}ms`,
      );
    }

    return filteredKeys;
  } catch (error) {
    // error handling
  }
}

Changelog

CHANGELOG entry: integrates per chain file save for tokenListController.

Related issues

Related: MetaMask/core#7413

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Introduces per-chain StorageService-backed persistence for TokenListController and migrates existing cache data.

  • Adds migration 114 to move TokenListController.tokensChainsCache from Redux state to per-chain filesystem keys (storageService:TokenListController:tokensChainsCache:{chainId}), avoids overwrites, handles errors, and clears in-state cache; includes comprehensive tests
  • Expands TokenListController messenger to allow StorageService:getAllKeys|getItem|setItem|removeItem
  • Updates token-list-controller-init to pass persisted state, subscribe to network changes, and call controller.initialize(); adds tests mocking controller and verifying initialize
  • Bumps @metamask/assets-controllers to ^98.0.0 and registers migration in migrations/index.ts

Written by Cursor Bugbot for commit 385b6a3. This will update automatically on new commits. Configure here.

@github-actions
Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

+ if (typeof state === "function" ? receiver !== state || !f : !state.has(receiver)) throw new TypeError("Cannot write private member to an object whose class did not declare it");
+ return (kind === "a" ? f.call(receiver, value) : f ? f.value = value : state.set(receiver, value)), value;
+};
+var _TokenListController_instances, _a, _TokenListController_mutex, _TokenListController_storageKeyPrefix, _TokenListController_getChainStorageKey, _TokenListController_intervalId, _TokenListController_intervalDelay, _TokenListController_cacheRefreshThreshold, _TokenListController_chainId, _TokenListController_abortController, _TokenListController_loadCacheFromStorage, _TokenListController_saveChainCacheToStorage, _TokenListController_migrateStateToStorage, _TokenListController_onNetworkControllerStateChange, _TokenListController_stopPolling, _TokenListController_startDeprecatedPolling;
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patch file to be removed after release, this is just for testing

async getAllKeys(namespace: string): Promise<string[]> {
// eslint-disable-next-line no-console
console.warn(`[StorageService DEBUG] getAllKeys called: ${namespace}`);
const startTime = performance.now();
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perf logs to be cleaned up

const totalDuration = performance.now() - startTime;

// Log performance for TokenListController specifically
if (controllerName === 'TokenListController') {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to be reverted; only for testing

@sahar-fehri sahar-fehri changed the title Feat/integrate token list controller storage service feat: integrate token list controller storage service Dec 15, 2025
@sahar-fehri sahar-fehri changed the base branch from main to feature/storage-service December 15, 2025 21:22
return allKeys
const filteredKeys = allKeys
.filter((key) => key.startsWith(prefix))
.map((key) => key.slice(prefix.length));
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filter to only keys that belong to the requested namespace

@sahar-fehri sahar-fehri marked this pull request as ready for review December 15, 2025 21:29
@sahar-fehri sahar-fehri requested a review from a team as a code owner December 15, 2025 21:29
@sahar-fehri sahar-fehri added the DO-NOT-MERGE Pull requests that should not be merged label Dec 15, 2025
Base automatically changed from feature/storage-service to main December 16, 2025 01:26
@sahar-fehri sahar-fehri requested a review from a team as a code owner December 16, 2025 08:23
@github-actions github-actions bot added size-L and removed size-XL labels Dec 16, 2025
@sahar-fehri sahar-fehri force-pushed the feat/integrate-tokenListController-storageService branch from dfa8049 to 070448e Compare December 16, 2025 08:59
@georgewrmarshall georgewrmarshall removed the request for review from a team December 18, 2025 02:01
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.06%. Comparing base (0f5817f) to head (69256ae).
⚠️ Report is 150 commits behind head on main.

Files with missing lines Patch % Lines
app/store/migrations/114.ts 81.63% 8 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24019      +/-   ##
==========================================
+ Coverage   79.96%   80.06%   +0.09%     
==========================================
  Files        4257     4279      +22     
  Lines      109331   110175     +844     
  Branches    22889    23103     +214     
==========================================
+ Hits        87429    88214     +785     
+ Misses      15835    15834       -1     
- Partials     6067     6127      +60     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

juanmigdr
juanmigdr previously approved these changes Jan 28, 2026
Copy link
Copy Markdown
Member

@juanmigdr juanmigdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sahar-fehri sahar-fehri enabled auto-merge January 28, 2026 15:03
salimtb
salimtb previously approved these changes Jan 28, 2026
tommasini
tommasini previously approved these changes Jan 28, 2026
@sahar-fehri sahar-fehri dismissed stale reviews from tommasini, juanmigdr, and salimtb via 758ee8f January 28, 2026 16:55
@github-actions
Copy link
Copy Markdown
Contributor

🔍 Smart E2E Test Selection

  • Selected E2E tags: SmokeWalletPlatform, SmokeTrade, SmokeNetworkAbstractions, SmokeConfirmationsRedesigned, SmokePerps
  • Selected Performance tags: @PerformanceAssetLoading
  • Risk Level: high
  • AI Confidence: 85%
click to see 🤖 AI reasoning details

E2E Test Selection:
This PR upgrades @metamask/assets-controllers from v97 to v98 and introduces significant changes to how TokenListController manages its token cache:

  1. Controller Initialization Change: Added controller.initialize() call in token-list-controller-init.ts - this is a new async initialization pattern required by the updated controller.

  2. Storage Mechanism Change: The TokenListController now uses StorageService for persisting per-chain token cache instead of Redux state. The messenger was updated to allow StorageService actions (getAllKeys, setItem, getItem, removeItem).

  3. Migration 114: A new migration moves existing tokensChainsCache from persisted state to FilesystemStorage. This affects all existing users upgrading to this version.

  4. Wide Impact on Token Display: The tokensChainsCache is used by multiple selectors (selectTokenList, selectERC20TokensByChain) which are consumed by:

    • Bridge hooks (useTopTokens)
    • Earn hooks (useMusdConversionStatus)
    • Perps hooks (usePerpsPaymentTokens, useWithdrawTokens)
    • Display name hooks (useERC20Tokens)
    • Multichain EVM selectors

Selected tags rationale:

  • SmokeWalletPlatform: Core wallet functionality, token display, transaction history
  • SmokeTrade: Swap/bridge functionality heavily depends on token lists for token selection
  • SmokeNetworkAbstractions: Token filtering by network uses tokensChainsCache
  • SmokeConfirmationsRedesigned: Token approvals and transfers need token metadata
  • SmokePerps: Directly uses selectTokenList for payment tokens and withdrawals

This is marked as HIGH risk because:

  • Core controller initialization pattern changed
  • Storage mechanism fundamentally changed (state → filesystem)
  • Data migration required for all existing users
  • Multiple critical features depend on token list data

Performance Test Selection:
The changes affect how token lists are loaded and cached. Moving from Redux state to FilesystemStorage could impact token list loading performance. The @PerformanceAssetLoading tag covers token list rendering and balance fetching, which are directly affected by the TokenListController changes. The new async initialization pattern and storage mechanism change could affect how quickly tokens appear in the UI.

View GitHub Actions results

@sonarqubecloud
Copy link
Copy Markdown

@sahar-fehri sahar-fehri added this pull request to the merge queue Jan 28, 2026
Merged via the queue into main with commit 390ecfd Jan 28, 2026
92 checks passed
@sahar-fehri sahar-fehri deleted the feat/integrate-tokenListController-storageService branch January 28, 2026 18:03
@github-actions github-actions bot locked and limited conversation to collaborators Jan 28, 2026
@metamaskbot metamaskbot added release-7.65.0 Issue or pull request that will be included in release 7.65.0 release-7.64.0 Issue or pull request that will be included in release 7.64.0 and removed release-7.65.0 Issue or pull request that will be included in release 7.65.0 labels Jan 28, 2026
@metamaskbot
Copy link
Copy Markdown
Collaborator

Missing release label release-7.64.0 on PR. Adding release label release-7.64.0 on PR and removing other release labels(release-7.65.0), as PR was added to branch 7.64.0 when release was cut.

@metamaskbot
Copy link
Copy Markdown
Collaborator

No release label on PR. Adding release label release-7.64.0 on PR, as PR was added to branch 7.64.0 when release was cut.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

release-7.64.0 Issue or pull request that will be included in release 7.64.0 size-L team-assets

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants