Skip to content

Add cache for multi asset id to avoid excessive joins#934

Merged
justinfrevert merged 10 commits into
mainfrom
feat/PM-21995-cache-multi_asset-id
Mar 19, 2026
Merged

Add cache for multi asset id to avoid excessive joins#934
justinfrevert merged 10 commits into
mainfrom
feat/PM-21995-cache-multi_asset-id

Conversation

@justinfrevert

@justinfrevert justinfrevert commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Overview

Cache multi_asset.id to avoid excessive joins:

Make the database do less work by pre-caching join data that is fixed. Perf testing has confirmed this improves the query times under load.

Adds an in-memory cache for multi_asset.id lookups, replacing repeated JOIN multi_asset in db-sync queries with a single cached lookup per (policy, name) pair. This eliminates the multi_asset join from registration, deregistration, asset create/spend, and candidate token queries, reducing query complexity and improving observation performance.

🗹 TODO before merging

  • Ready

📌 Submission Checklist

  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

Please describe any additional testing aside from CI:

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • Node Runtime Update
  • Node Client Update
  • Other:
  • N/A

Links

@github-actions

Copy link
Copy Markdown
Contributor

kics-logo

KICS version: v2.1.19

Category Results
CRITICAL CRITICAL 0
HIGH HIGH 2
MEDIUM MEDIUM 52
LOW LOW 3
INFO INFO 64
TRACE TRACE 0
TOTAL TOTAL 121
Metric Values
Files scanned placeholder 27
Files parsed placeholder 27
Files failed to scan placeholder 0
Total executed queries placeholder 73
Queries failed to execute placeholder 0
Execution time placeholder 11

@justinfrevert justinfrevert marked this pull request as ready for review March 16, 2026 16:04
@justinfrevert justinfrevert requested a review from a team as a code owner March 16, 2026 16:04
Comment thread primitives/mainchain-follower/src/db/multi_asset_cache.rs Outdated
@gilescope gilescope enabled auto-merge March 18, 2026 05:33

@LGLO LGLO left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked at some random range of blocks and the speed up was around 10% but still I think it is worth it.
Added complexity is limited and it pays off in multiple queries.

@gilescope gilescope added this pull request to the merge queue Mar 19, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 19, 2026
@justinfrevert justinfrevert added this pull request to the merge queue Mar 19, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 19, 2026
@justinfrevert justinfrevert added this pull request to the merge queue Mar 19, 2026
Merged via the queue into main with commit 17ad5e8 Mar 19, 2026
50 of 51 checks passed
@justinfrevert justinfrevert deleted the feat/PM-21995-cache-multi_asset-id branch March 19, 2026 02:35
@gilescope gilescope added this to the node-1.0.0 milestone Apr 10, 2026
gilescope added a commit that referenced this pull request Apr 20, 2026
* Add cache for multi asset id to avoid excessive joins

* change file

* fix npm audit

* Apply suggestion from @gilescope

---------

Co-authored-by: Squirrel <giles.cope@shielded.io>
Signed-off-by: Giles Cope <gilescope@gmail.com>
gilescope added a commit that referenced this pull request Apr 20, 2026
…y_hash

Needed because cherry-pick landed in a branch whose .sqlx prepared cache
does not include this query (cached key changed after trailing-whitespace
normalisation). Using the runtime form avoids the compile-time DB check
entirely and matches the pattern used elsewhere in #934.

Signed-off-by: Giles Cope <gilescope@gmail.com>
This was referenced Apr 20, 2026
m2ux added a commit that referenced this pull request Apr 23, 2026
m2ux added a commit that referenced this pull request Apr 23, 2026
@gilescope gilescope mentioned this pull request Apr 27, 2026
15 tasks
gilescope added a commit that referenced this pull request Apr 27, 2026
…y_hash

Needed because cherry-pick landed in a branch whose .sqlx prepared cache
does not include this query (cached key changed after trailing-whitespace
normalisation). Using the runtime form avoids the compile-time DB check
entirely and matches the pattern used elsewhere in #934.

Signed-off-by: Giles Cope <gilescope@gmail.com>
github-merge-queue Bot pushed a commit that referenced this pull request Apr 27, 2026
* Increase performance of cNight db-sync queries.

First query for coarse bounds of tx, tx_out and ma_tx_out tables and use
them in the 'business' queries to reduce size of tables before joins are
made.

Signed-off-by: Giles Cope <gilescope@gmail.com>

* Add tx_in indexes as well

Signed-off-by: Giles Cope <gilescope@gmail.com>

* refactor: use sqlx::query_as! macro for cNight observation queries

Convert the runtime query_as form back to the compile-time macro form
for all queries in cnight_observation.rs. This restores static query
verification against the db-sync schema via the .sqlx prepared cache.

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>

* chore: replace sqlx::query_as! macro with runtime form in get_block_by_hash

Needed because cherry-pick landed in a branch whose .sqlx prepared cache
does not include this query (cached key changed after trailing-whitespace
normalisation). Using the runtime form avoids the compile-time DB check
entirely and matches the pattern used elsewhere in #934.

Signed-off-by: Giles Cope <gilescope@gmail.com>

* feat: add changefile

Signed-off-by: Giles Cope <gilescope@gmail.com>

* fix: should be debug not warn

Signed-off-by: Giles Cope <gilescope@gmail.com>

* refactor: introduce PagedQuery to consolidate cNight observation query args

Replaces the seven repeated arguments (start, end, limit, offset, low_bound,
high_bound + per-call locals) on the four cNight observation query helpers
with a single `PagedQuery<'a>` struct.

Squashed from PR #1365 fixup commits b9497e8 + b146dc2 + b1ffce6
(originally landed as "fix: cargo clippy" / "Apply suggestion").
Also drops the now-unused `QueryBounds` import in the data_source module.

Signed-off-by: Giles Cope <gilescope@gmail.com>

* fix: clippy needless borrow in get_block_by_hash bind

Signed-off-by: Giles Cope <gilescope@gmail.com>

* fix: fixed compilation for try-runtime feature (#1427)

Signed-off-by: Tomasz Bartos <tomasz.bartos@shielded.io>
Signed-off-by: Giles Cope <gilescope@gmail.com>

---------

Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Tomasz Bartos <tomasz.bartos@shielded.io>
Co-authored-by: Lech Głowiak <lech.glowiak@iohk.io>
Co-authored-by: Lech Głowiak <lech.glowiak@shielded.io>
Co-authored-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Co-authored-by: Klapeyron <11329616+Klapeyron@users.noreply.github.com>
@gilescope gilescope mentioned this pull request Apr 27, 2026
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants