graphql-alt: scanning api bloom filter pipelines by henryachen · Pull Request #24788 · MystenLabs/sui

henryachen · 2026-01-05T20:13:14Z

Description

Adds two new indexer pipelines to support transaction scanning queries via bloom filters:

cp_blooms concurrent pipeline: Per-checkpoint bloom filters

Indexes function calls, affected objects, senders, and recipients for each checkpoint
Folds bloom filter if it is sparse up to a certain min size or density
Stored in cp_blooms table with one row per checkpoint

cp_bloom_blocks sequential pipeline: Blocked bloom filters spanning 1000 checkpoints

Splits each 256KB bloom into 128 separate smaller 2KB blocks
Stored in cp_bloom_blocks table with ~128 rows per cp_block (sparse, only non-zero blocks)
ORs bloom blocks on conflicts

Schema changes:

cp_blooms: Per-checkpoint bloom filters
cp_bloom_blocks: Blocked bloom filters with (cp_block_id, bloom_block_index) primary key

Test plan

cargo nextest run -p sui-indexer-alt
cargo nextest run -p sui-indexer-alt-schema

Stack:

#24788 👈
#24900
#25097

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2026-01-05T20:13:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
sui-docs	Ready	Preview, Comment	Jan 30, 2026 0:44am

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
multisig-toolkit	Ignored	Preview	Jan 30, 2026 0:44am
sui-kiosk	Ignored	Preview	Jan 30, 2026 0:44am

crates/sui-indexer-alt-schema/src/blooms/blocked.rs

crates/sui-indexer-alt-schema/src/cp_blooms.rs

crates/sui-indexer-alt-schema/src/cp_bloom_blocks.rs

crates/sui-indexer-alt-schema/src/blooms/hash.rs

crates/sui-indexer-alt-schema/src/cp_bloom_blocks.rs

crates/sui-indexer-alt/src/handlers/cp_bloom_blocks.rs

crates/sui-indexer-alt/src/handlers/cp_blooms.rs

crates/sui-indexer-alt-schema/src/blooms/blocked.rs

crates/sui-indexer-alt-schema/src/blooms/bloom.rs

crates/sui-indexer-alt/src/lib.rs

crates/sui-indexer-alt/src/handlers/cp_bloom_blocks.rs

crates/sui-indexer-alt/src/handlers/cp_blooms.rs

crates/sui-indexer-alt/src/handlers/cp_bloom_blocks.rs

crates/sui-indexer-alt/src/handlers/cp_blooms.rs

## Description Adds two new indexer pipelines to support transaction scanning queries via bloom filters: cp_blooms concurrent pipeline: Per-checkpoint bloom filters - Indexes function calls, affected objects, senders, and recipients for each checkpoint - Folds bloom filter if it is sparse up to a certain min size or density - Stored in cp_blooms table with one row per checkpoint cp_bloom_blocks sequential pipeline: Blocked bloom filters spanning 1000 checkpoints - Splits each 256KB bloom into 128 separate smaller 2KB blocks - Stored in cp_bloom_blocks table with ~128 rows per cp_block (sparse, only non-zero blocks) - ORs bloom blocks on conflicts Schema changes: - cp_blooms: Per-checkpoint bloom filters - cp_bloom_blocks: Blocked bloom filters with (cp_block_id, bloom_block_index) primary key ## Test plan ``` cargo nextest run -p sui-indexer-alt cargo nextest run -p sui-indexer-alt-schema ``` ## Stack: #24788 👈 #24900 #25097 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [x] GraphQL: add bloom filter pipelines for scanning APIs - [ ] CLI: - [ ] Rust SDK: - [ ] Indexing Framework:

## Description - Added scanEvent GraphQL query endpoint for scanning events using bloom filters - Reuse existing scan infrastructure (bloom querying, pagination) from Transaction Scanning - Added e2e tests covering filtering by sender, module, type, and pagination ## Test plan ``` cargo nextest run -p sui-indexer-alt-graphql cargo nextest run -p sui-indexer-alt-graphql -- schema_sdl cargo nextest run -p sui-indexer-alt-graphql --features staging -- schema_sdl cargo nextest run -p sui-indexer-alt-e2e-tests -- graphql/ ``` ## Stack: #24788 #24900 #25097 👈 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] Indexing Framework:

## Description Adds a new transactionsScan query that scans checkpoints using bloom filters to find transactions matching filter criteria. Changes GraphQL (sui-indexer-alt-graphql) - New transactionsScan query endpoint - maxScanLimit service config to limit checkpoints scanned per query - Transaction filter and scan logic in scan.rs Schema (sui-indexer-alt-schema) - `bloom_contains` SQL function for checking probe membership in bloom filters Reader (sui-indexer-alt-reader) - `cp_blooms` loader for batch-loading bloom filter data ## Test plan ``` cargo nextest run -p sui-indexer-alt-graphql graphql_scan_limit_tests cargo nextest run -p sui-indexer-alt-graphql cargo nextest run -p sui-indexer-alt cargo nextest run -p sui-indexer-alt-schema ``` Query Plan: ``` Limit (cost=1911.69..1358704.33 rows=62 width=8) (actual time=14.454..14.836 rows=62 loops=1) Buffers: shared hit=7287 CTE block_bit_probes -> ProjectSet (cost=0.00..8.20 rows=1635 width=18) (actual time=0.004..0.431 rows=1635 loops=1) -> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 rows=1 loops=1) -> Nested Loop (cost=1903.49..1358696.13 rows=62 width=8) (actual time=14.453..14.830 rows=62 loops=1) Buffers: shared hit=7287 -> Limit (cost=1903.06..1903.22 rows=62 width=24) (actual time=14.411..14.414 rows=1 loops=1) Buffers: shared hit=6976 CTE block_lookup -> Nested Loop Left Join (cost=41.30..1723.38 rows=200 width=131) (actual time=0.654..2.899 rows=327 loops=1) Buffers: shared hit=1348 -> HashAggregate (cost=40.88..42.88 rows=200 width=10) (actual time=0.620..0.682 rows=327 loops=1) Group Key: block_bit_probes_1.cp_block_index, block_bit_probes_1.bloom_idx Batches: 1 Memory Usage: 61kB -> CTE Scan on block_bit_probes block_bit_probes_1 (cost=0.00..32.70 rows=1635 width=10) (actual time=0.000..0.159 rows=1635 loops=1) -> Index Scan using cp_bloom_blocks_pkey on cp_bloom_blocks bb (cost=0.42..8.40 rows=1 width=131) (actual time=0.006..0.006 rows=1 loops=327) Index Cond: ((cp_block_index = block_bit_probes_1.cp_block_index) AND (bloom_block_index = block_bit_probes_1.bloom_idx)) Buffers: shared hit=1343 -> Sort (cost=179.69..180.17 rows=192 width=24) (actual time=14.410..14.412 rows=1 loops=1) Sort Key: ((block_bit_probes.cp_block_index * '1000'::bigint)) Sort Method: quicksort Memory: 25kB Buffers: shared hit=6976 -> Hash Right Anti Join (cost=48.29..173.01 rows=192 width=24) (actual time=14.392..14.400 rows=5 loops=1) Hash Cond: (p.cp_block_index = block_bit_probes.cp_block_index) Buffers: shared hit=6976 -> Hash Join (cost=7.00..129.78 rows=8 width=8) (actual time=3.224..12.355 rows=1011 loops=1) Hash Cond: ((p.cp_block_index = bl.cp_block_index) AND (p.bloom_idx = bl.bloom_idx)) Join Filter: ((bl.bloom_filter IS NULL) OR (p.bit_mask <> get_byte(bl.bloom_filter, (p.byte_pos % length(bl.bloom_filter))))) Rows Removed by Join Filter: 624 Buffers: shared hit=6976 -> CTE Scan on block_bit_probes p (cost=0.00..32.70 rows=1635 width=18) (actual time=0.000..0.211 rows=1635 loops=1) -> Hash (cost=4.00..4.00 rows=200 width=42) (actual time=3.142..3.142 rows=327 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 93kB Buffers: shared hit=1348 -> CTE Scan on block_lookup bl (cost=0.00..4.00 rows=200 width=42) (actual time=0.656..3.060 rows=327 loops=1) Buffers: shared hit=1348 -> Hash (cost=38.79..38.79 rows=200 width=8) (actual time=1.773..1.774 rows=327 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 21kB -> HashAggregate (cost=36.79..38.79 rows=200 width=8) (actual time=1.668..1.717 rows=327 loops=1) Group Key: block_bit_probes.cp_block_index Batches: 1 Memory Usage: 61kB -> CTE Scan on block_bit_probes (cost=0.00..32.70 rows=1635 width=8) (actual time=0.007..1.162 rows=1635 loops=1) -> Index Scan using cp_blooms_pkey on cp_blooms cb (cost=0.43..21883.74 rows=1 width=8) (actual time=0.040..0.408 rows=62 loops=1) Index Cond: ((cp_sequence_number >= ((block_bit_probes.cp_block_index * '1000'::bigint))) AND (cp_sequence_number <= ((((block_bit_probes.cp_block_index * '1000'::bigint) + '1000'::bigint) - 1)))) Filter: (((get_byte(bloom_filter, (208 % length(bloom_filter))) & 8) = 8) AND ((get_byte(bloom_filter, (5988 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (8084 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (15059 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (8330 % length(bloom_filter))) & 64) = 64) AND ((get_byte(bloom_filter, (10313 % length(bloom_filter))) & 4) = 4)) Rows Removed by Filter: 318 Buffers: shared hit=311 Planning: Buffers: shared hit=78 Planning Time: 2.181 ms Execution Time: 15.060 ms ``` ## Stack: #24788 #24900 👈 #25097 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] Indexing Framework:

## Description Adds a new transactionsScan query that scans checkpoints using bloom filters to find transactions matching filter criteria. Changes GraphQL (sui-indexer-alt-graphql) - New transactionsScan query endpoint - maxScanLimit service config to limit checkpoints scanned per query - Transaction filter and scan logic in scan.rs Schema (sui-indexer-alt-schema) - `bloom_contains` SQL function for checking probe membership in bloom filters Reader (sui-indexer-alt-reader) - `cp_blooms` loader for batch-loading bloom filter data ## Test plan ``` cargo nextest run -p sui-indexer-alt-graphql graphql_scan_limit_tests cargo nextest run -p sui-indexer-alt-graphql cargo nextest run -p sui-indexer-alt cargo nextest run -p sui-indexer-alt-schema ``` Query Plan: ``` Limit (cost=1911.69..1358704.33 rows=62 width=8) (actual time=14.454..14.836 rows=62 loops=1) Buffers: shared hit=7287 CTE block_bit_probes -> ProjectSet (cost=0.00..8.20 rows=1635 width=18) (actual time=0.004..0.431 rows=1635 loops=1) -> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 rows=1 loops=1) -> Nested Loop (cost=1903.49..1358696.13 rows=62 width=8) (actual time=14.453..14.830 rows=62 loops=1) Buffers: shared hit=7287 -> Limit (cost=1903.06..1903.22 rows=62 width=24) (actual time=14.411..14.414 rows=1 loops=1) Buffers: shared hit=6976 CTE block_lookup -> Nested Loop Left Join (cost=41.30..1723.38 rows=200 width=131) (actual time=0.654..2.899 rows=327 loops=1) Buffers: shared hit=1348 -> HashAggregate (cost=40.88..42.88 rows=200 width=10) (actual time=0.620..0.682 rows=327 loops=1) Group Key: block_bit_probes_1.cp_block_index, block_bit_probes_1.bloom_idx Batches: 1 Memory Usage: 61kB -> CTE Scan on block_bit_probes block_bit_probes_1 (cost=0.00..32.70 rows=1635 width=10) (actual time=0.000..0.159 rows=1635 loops=1) -> Index Scan using cp_bloom_blocks_pkey on cp_bloom_blocks bb (cost=0.42..8.40 rows=1 width=131) (actual time=0.006..0.006 rows=1 loops=327) Index Cond: ((cp_block_index = block_bit_probes_1.cp_block_index) AND (bloom_block_index = block_bit_probes_1.bloom_idx)) Buffers: shared hit=1343 -> Sort (cost=179.69..180.17 rows=192 width=24) (actual time=14.410..14.412 rows=1 loops=1) Sort Key: ((block_bit_probes.cp_block_index * '1000'::bigint)) Sort Method: quicksort Memory: 25kB Buffers: shared hit=6976 -> Hash Right Anti Join (cost=48.29..173.01 rows=192 width=24) (actual time=14.392..14.400 rows=5 loops=1) Hash Cond: (p.cp_block_index = block_bit_probes.cp_block_index) Buffers: shared hit=6976 -> Hash Join (cost=7.00..129.78 rows=8 width=8) (actual time=3.224..12.355 rows=1011 loops=1) Hash Cond: ((p.cp_block_index = bl.cp_block_index) AND (p.bloom_idx = bl.bloom_idx)) Join Filter: ((bl.bloom_filter IS NULL) OR (p.bit_mask <> get_byte(bl.bloom_filter, (p.byte_pos % length(bl.bloom_filter))))) Rows Removed by Join Filter: 624 Buffers: shared hit=6976 -> CTE Scan on block_bit_probes p (cost=0.00..32.70 rows=1635 width=18) (actual time=0.000..0.211 rows=1635 loops=1) -> Hash (cost=4.00..4.00 rows=200 width=42) (actual time=3.142..3.142 rows=327 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 93kB Buffers: shared hit=1348 -> CTE Scan on block_lookup bl (cost=0.00..4.00 rows=200 width=42) (actual time=0.656..3.060 rows=327 loops=1) Buffers: shared hit=1348 -> Hash (cost=38.79..38.79 rows=200 width=8) (actual time=1.773..1.774 rows=327 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 21kB -> HashAggregate (cost=36.79..38.79 rows=200 width=8) (actual time=1.668..1.717 rows=327 loops=1) Group Key: block_bit_probes.cp_block_index Batches: 1 Memory Usage: 61kB -> CTE Scan on block_bit_probes (cost=0.00..32.70 rows=1635 width=8) (actual time=0.007..1.162 rows=1635 loops=1) -> Index Scan using cp_blooms_pkey on cp_blooms cb (cost=0.43..21883.74 rows=1 width=8) (actual time=0.040..0.408 rows=62 loops=1) Index Cond: ((cp_sequence_number >= ((block_bit_probes.cp_block_index * '1000'::bigint))) AND (cp_sequence_number <= ((((block_bit_probes.cp_block_index * '1000'::bigint) + '1000'::bigint) - 1)))) Filter: (((get_byte(bloom_filter, (208 % length(bloom_filter))) & 8) = 8) AND ((get_byte(bloom_filter, (5988 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (8084 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (15059 % length(bloom_filter))) & 32) = 32) AND ((get_byte(bloom_filter, (8330 % length(bloom_filter))) & 64) = 64) AND ((get_byte(bloom_filter, (10313 % length(bloom_filter))) & 4) = 4)) Rows Removed by Filter: 318 Buffers: shared hit=311 Planning: Buffers: shared hit=78 Planning Time: 2.181 ms Execution Time: 15.060 ms ``` ## Stack: MystenLabs#24788 MystenLabs#24900 👈 MystenLabs#25097 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] Indexing Framework:

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 5, 2026 20:13 — with GitHub Actions Inactive

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2026 02:26 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2026 02:28 View deployment

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2026 06:45 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2026 06:47 View deployment

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2026 07:45 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2026 07:46 View deployment

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2026 08:05 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2026 08:06 View deployment

henryachen marked this pull request as ready for review January 7, 2026 15:43

henryachen requested a review from a team as a code owner January 7, 2026 15:43

henryachen requested review from amnn, emmazzz, evan-wall-mysten, nickvikeras, tpham-mysten and wlmyng January 7, 2026 15:43

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2026 15:43 — with GitHub Actions Inactive

amnn reviewed Jan 7, 2026

View reviewed changes

vercel bot deployed to Preview – sui-docs January 8, 2026 19:19 View deployment

vercel bot deployed to Preview – sui-docs January 9, 2026 18:17 View deployment

vercel bot deployed to Preview – sui-docs January 12, 2026 17:40 View deployment

henryachen mentioned this pull request Jan 12, 2026

graphql-alt: scan transactions api #24900

Merged

8 tasks

emmazzz reviewed Jan 13, 2026

View reviewed changes

amnn reviewed Jan 13, 2026

View reviewed changes

henryachen force-pushed the hc/dvx-1290-scanning-api-write branch from b1ce241 to 7a0f2d4 Compare January 14, 2026 06:10

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 14, 2026 06:10 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 14, 2026 06:14 View deployment

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 16, 2026 00:22 — with GitHub Actions Inactive

henryachen added 12 commits January 29, 2026 16:01

safer cp_bloom handling

0e62097

update docs for hasher

b947db2

use bytes instead of bits and some code cleanup

c2894b4

address some more comments

3af5b43

remove checkpoint bounds from blocked blooms

55d99f4

generic bloom filter types, and some test cleanup

e4ffd24

fixup

9e3ab1b

address comments for code cleanup

bce5d17

doc cleanup and better variable names

1fae25e

try to clarify bloom skips

568d65e

refactor bloom skip and add tests

718d31b

add more doc comments and better naming for bytea_or sql function

bcfadfe

henryachen force-pushed the hc/dvx-1290-scanning-api-write branch from 85986bb to bcfadfe Compare January 30, 2026 00:40

henryachen temporarily deployed to sui-typescript-aws-kms-test-env January 30, 2026 00:40 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 30, 2026 00:44 View deployment

henryachen requested review from amnn and wlmyng January 30, 2026 02:05

henryachen merged commit e984ad8 into main Jan 30, 2026
72 of 77 checks passed

henryachen deleted the hc/dvx-1290-scanning-api-write branch January 30, 2026 18:28

henryachen mentioned this pull request Feb 5, 2026

(pick #24788+25194) indexer-alt: bloom filter pipelines + support for ConsensusAddressOwner #25310

Merged

8 tasks

henryachen mentioned this pull request Mar 5, 2026

graphql-alt: scan apis #25717

Open

16 tasks

Conversation

henryachen commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test plan

Stack:

Release notes

Uh oh!

vercel bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

henryachen commented Jan 5, 2026 •

edited

Loading

vercel bot commented Jan 5, 2026 •

edited

Loading