fix(nodedb): fix race between updating fast node cache and db commit by mattac21 · Pull Request #1142 · cosmos/iavl

mattac21 · 2026-04-03T16:30:24Z

Fixes a race condition where concurrent GetFastNode calls (from RPC queries) can repopulate the fast node cache with stale data that is about to be overridden from the tree during SaveVersion. After Commit within SaveVersion the cache is not repopulated with the correct data from the tree, causing the cache to serve that stale data to future readers. At the application level, this leads to an app hash mismatches.

To fix this, we introduce a pendingFastNodeAdditions and pendingFastNodeRemovals that store changes to the fast node cache when adding nodes via saveFastNodeUnlocked or DeleteFastNode. We then defer the addition or removal of nodes from the fast node cache until after the tree changes commit, meaning there is no period of time where we could have removed a node from the cache, then brought back up an incorrect value from the tree.

This also fixes a separate race between SaveVersion and accessing getLatestVersion that was exercised by the regression tests.

mattac21 · 2026-04-03T18:09:35Z

 			// If the tree is of the latest version and fast node is not in the tree
 			// then the regular node is not in the tree either because fast node
 			// represents live state.
-			if t.version == t.ndb.latestVersion {


this is fixing a separate race condition that was exercised during the test

mattac21 · 2026-04-03T18:09:55Z

+	pendingFastNodeAdditions []*fastnode.Node           // Fast nodes to add to cache after batch commit.
+	pendingFastNodeRemovals  [][]byte                   // Fast node keys to remove from cache after batch commit.


new, this is the actual fix for the main race causing the app hash

mattac21 · 2026-04-03T18:11:10Z

 	// return -1, nil
 }

+func (ndb *nodeDB) getCachedLatestVersion() int64 {


already an accessor for getLatestVersion, but it has the potential to do a heavy db scan on latest version being 0. The previous logic did not do this, so this lighter weight accessor was added. Again this is for the separate race condition being fixed to make the test not race, not the app hash.

mattac21 · 2026-04-06T15:22:36Z

@Mergifyio backport release/v1.2.x

mergify · 2026-04-06T15:22:45Z

backport release/v1.2.x

✅ Backports have been created

Details

#1144 fix(nodedb): fix race between updating fast node cache and db commit (backport #1142) has been created for branch release/v1.2.x but encountered conflicts

Cherry-pick of 146f723 has failed:

On branch mergify/bp/release/v1.2.x/pr-1142
Your branch is up to date with 'origin/release/v1.2.x'.

You are currently cherry-picking commit 146f723.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   immutable_tree.go
	modified:   mutable_tree.go
	modified:   nodedb_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   nodedb.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

…1142) (cherry picked from commit 146f723) # Conflicts: # nodedb.go

mattac21 · 2026-04-06T15:23:40Z

@Mergifyio backport release/v1.3.x

mergify · 2026-04-06T15:24:10Z

backport release/v1.3.x

✅ Backports have been created

Details

#1147 fix(nodedb): fix race between updating fast node cache and db commit (backport #1142) has been created for branch release/v1.3.x but encountered conflicts

Cherry-pick of 146f723 has failed:

On branch mergify/bp/release/v1.3.x/pr-1142
Your branch is up to date with 'origin/release/v1.3.x'.

You are currently cherry-picking commit 146f723.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   immutable_tree.go
	modified:   mutable_tree.go
	modified:   nodedb_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   nodedb.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

…1142) (cherry picked from commit 146f723) # Conflicts: # nodedb.go

…(backport #1142) (#1144) Co-authored-by: mattac21 <matt@cosmoslabs.io>

…(backport #1142) (#1147) Co-authored-by: mattac21 <matt@cosmoslabs.io>

…(backport #1142) (#1144) Co-authored-by: mattac21 <matt@cosmoslabs.io>

mattac21 added 7 commits April 2, 2026 14:54

fix fast node add/delete race against batch commit

b7a8ca5

store pending removal keys as bytes instead of string

a52028b

comment

67ab098

add test

622095f

comments

f7cd4f9

change mtx to rwmtx, add rlock around fetching latest version

e2e31ef

reduce num new keys and attempts

4e7627d

mattac21 marked this pull request as ready for review April 3, 2026 18:01

use smaller cached version of latestversion instead of potential db scan

212cb6b

mattac21 commented Apr 3, 2026

View reviewed changes

mattac21 requested review from Eric-Warehime, aljo242, swift1337 and vladjdk April 3, 2026 18:12

vladjdk approved these changes Apr 3, 2026

View reviewed changes

swift1337 approved these changes Apr 3, 2026

View reviewed changes

aljo242 approved these changes Apr 3, 2026

View reviewed changes

Eric-Warehime approved these changes Apr 6, 2026

View reviewed changes

mattac21 merged commit 146f723 into master Apr 6, 2026
8 checks passed

mattac21 deleted the ma/debug branch April 6, 2026 14:54

mattac21 added the backport/v1.x label Apr 6, 2026

mergify Bot pushed a commit that referenced this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit (#…

25dc52b

…1142) (cherry picked from commit 146f723) # Conflicts: # nodedb.go

mergify Bot mentioned this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit (backport #1142) #1144

Merged

mergify Bot pushed a commit that referenced this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit (#…

a34b75b

…1142) (cherry picked from commit 146f723) # Conflicts: # nodedb.go

mergify Bot mentioned this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit (backport #1142) #1147

Merged

mattac21 added a commit that referenced this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit …

a456106

…(backport #1142) (#1144) Co-authored-by: mattac21 <matt@cosmoslabs.io>

mattac21 added a commit that referenced this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit …

6eb2fb5

…(backport #1142) (#1147) Co-authored-by: mattac21 <matt@cosmoslabs.io>

mattac21 added a commit that referenced this pull request Apr 6, 2026

fix(nodedb): fix race between updating fast node cache and db commit …

8b230d1

…(backport #1142) (#1144) Co-authored-by: mattac21 <matt@cosmoslabs.io>

metalarm10 mentioned this pull request Apr 7, 2026

Chore(deps): bump cosmos/iavl v1.2.4 to v1.2.8 tokenize-x/tx-chain#118

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(nodedb): fix race between updating fast node cache and db commit#1142

fix(nodedb): fix race between updating fast node cache and db commit#1142
mattac21 merged 8 commits into
masterfrom
ma/debug

mattac21 commented Apr 3, 2026 •

edited

Loading

Uh oh!

mattac21 Apr 3, 2026

Uh oh!

mattac21 Apr 3, 2026 •

edited

Loading

Uh oh!

mattac21 Apr 3, 2026

Uh oh!

Uh oh!

mattac21 commented Apr 6, 2026

Uh oh!

mergify Bot commented Apr 6, 2026 •

edited

Loading

Uh oh!

mattac21 commented Apr 6, 2026

Uh oh!

mergify Bot commented Apr 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

		pendingFastNodeAdditions []*fastnode.Node // Fast nodes to add to cache after batch commit.
		pendingFastNodeRemovals [][]byte // Fast node keys to remove from cache after batch commit.

Conversation

mattac21 commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattac21 Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

mattac21 Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattac21 Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattac21 commented Apr 6, 2026

Uh oh!

mergify Bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mattac21 commented Apr 6, 2026

Uh oh!

mergify Bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

mattac21 commented Apr 3, 2026 •

edited

Loading

mattac21 Apr 3, 2026 •

edited

Loading

mergify Bot commented Apr 6, 2026 •

edited

Loading

mergify Bot commented Apr 6, 2026 •

edited

Loading