Skip to content

Fetch data on origin on subscription#865

Merged
3alpha merged 8 commits intomainfrom
fetch-data-from-origin-on-subscription
Oct 2, 2025
Merged

Fetch data on origin on subscription#865
3alpha merged 8 commits intomainfrom
fetch-data-from-origin-on-subscription

Conversation

@3alpha
Copy link
Copy Markdown
Collaborator

@3alpha 3alpha commented Sep 30, 2025

Usage related changes

Resolves #742.

Development related changes

Checklist:

  • Checked out the contribution guidelines
  • Applied formatting - ./scripts/format.sh
  • No linter errors - ./scripts/clippy_check.sh
  • No unused dependencies - ./scripts/check_unused_deps.sh
  • No spelling errors - ./scripts/check_spelling.sh
  • Performed code self-review
  • Rebased to the latest commit of the target branch (or merged it into my branch)
    • Once you make the PR reviewable, please avoid force-pushing
  • Updated the docs if needed - ./website/README.md
  • Linked the issues resolvable by this PR - linking info
  • Updated the tests if needed; all passing - execution info

Summary by CodeRabbit

  • New Features

    • Subscriptions are fork-aware: head and event subscriptions now surface origin blocks/events when forking is enabled.
    • Origin event fetching supports pagination/chunking for complete histories.
  • Changes

    • Block hashing algorithm changed to Poseidon, altering hashes of newly produced blocks.
    • Improved block-range validation to handle mixed origin/local histories.
  • Tests

    • Added integration tests for fork-aware head and event subscriptions (start-from-origin and by-hash).

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Sep 30, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

  • Replaces Pedersen with Poseidon for StarknetBlock hashing in HashProducer, updating imports and using Poseidon::hash_array with adjusted input ordering.
  • Renames and narrows scope of origin event helper: get_origin_events -> pub(crate) async fn fetch_origin_events_chunk(...) with expanded parameters (continuation_token, address, keys, chunk_size) and chunking support.
  • Makes websocket subscription logic origin-aware: adds origin/local block-header helpers, extends validated block range to return origin range, adds origin heads/events batch-fetching with pagination/continuation-token handling, and integrates origin data into subscribe_new_heads and subscribe_events flows while preserving local fallback behavior.
  • Adds a From implementation converting from starknet_rs_core::types::ResourcePrice.
  • Adds new integration tests covering forked devnet subscriptions for blocks (by number and by hash) and events, validating origin-included notifications, sequencing, and empty-block handling after forking.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Out of Scope Changes Check ⚠️ Warning The update in crates/starknet-devnet-core/src/blocks/mod.rs replacing Pedersen hashing with Poseidon is unrelated to origin data subscription behavior and falls outside the scope defined by issue #742. This core cryptographic change does not impact WebSocket subscriptions and was not mentioned in the objectives. Please remove the standalone hashing engine change from this pull request or split it into a separate PR so that the subscription-focused changes remain isolated and reviewable.
Description Check ⚠️ Warning The pull request description includes the required headings and checklist but lacks substantive detail in the Usage related changes section—it only references the issue without describing how consumer behavior is impacted—and the Development related changes section remains blank. Please expand the Usage related changes section to explain how subscriptions now return origin fork data and prevent BLOCK_NOT_FOUND errors, and populate the Development related changes section with any CI, testing, or tooling updates required for origin-aware subscriptions; also update the checklist to reflect any necessary documentation or issue linking.
✅ Passed checks (3 passed)
Check name Status Explanation
Title Check ✅ Passed The title “Fetch data on origin on subscription” clearly references the main change of integrating origin data retrieval into subscription flows and concisely describes the feature being added. It directly relates to the pull request’s core functionality without including extraneous details or file lists.
Linked Issues Check ✅ Passed The pull request implements origin-aware block and event fetching for WebSocket subscriptions, updates the validated block range logic to include origin ranges, and adds integration tests exercising forks as described in issue #742. These changes fulfill the requirement to surface origin blocks (up to 1024 behind) and gracefully handle blocks outside that window.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@3alpha
Copy link
Copy Markdown
Collaborator Author

3alpha commented Oct 1, 2025

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Oct 1, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@3alpha 3alpha marked this pull request as ready for review October 1, 2025 19:26
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
crates/starknet-devnet-server/src/api/endpoints_ws.rs (1)

59-102: Verify block number stability when querying origin.

Past review comments raised a concern about using block numbers vs hashes when querying the origin. If the origin chain has been reorganized or pruned, block number N on the origin might represent a different block than expected. Consider whether BlockId::Number should be restricted or validated when fetching from origin.

Based on past review comments.

🧹 Nitpick comments (7)
crates/starknet-devnet-types/src/rpc/block.rs (1)

134-138: Straightforward conversion looks correct

Field-to-field mapping matches the types; no ownership pitfalls. Consider also implementing From<&starknet_rs_core::types::ResourcePrice> for flexibility, but not required.

crates/starknet-devnet-core/src/blocks/mod.rs (1)

411-429: Hash inputs/order are explicit; timestamp zeroing is intentional — add a brief doc/test

The array ordering is clear and stable; zeroing timestamp for reproducibility is sensible. Two small asks:

  • Document this rationale at the function (or type) level to avoid regressions.
  • Add a unit test asserting stable hash for identical headers across runs (esp. timestamp).

Optional micro‑refactor for readability:

-        let hash = Poseidon::hash_array(&[
+        // Compose Poseidon inputs in a single place for clarity.
+        let hash_inputs = [
             felt!(self.header.block_header_without_hash.block_number.0), // block number
             self.header.block_header_without_hash.state_root.0,          // global_state_root
             *self.header.block_header_without_hash.sequencer.0.key(),    // sequencer_address
             Felt::ZERO,                                                  /* block_timestamp;
                                                                           * would normally be
                                                                           * felt!(self.header.
                                                                           * timestamp.0), but
                                                                           * is modified to enable replicability
                                                                           * in re-execution on
                                                                           * loading on dump */
             felt!(self.transaction_hashes.len() as u64), // transaction_count
             Felt::ZERO,                                  // transaction_commitment
             Felt::ZERO,                                  // event_count
             Felt::ZERO,                                  // event_commitment
             Felt::ZERO,                                  // protocol_version
             Felt::ZERO,                                  // extra_data
             self.header.block_header_without_hash.parent_hash.0, // parent_block_hash
-        ]);
+        ];
+        let hash = Poseidon::hash_array(&hash_inputs);
crates/starknet-devnet-server/src/api/endpoints.rs (1)

461-506: Make prefix handling explicit; consider an origin call timeout

  • Use strip_prefix to express intent and avoid accidental multi-strip semantics.
  • Wrap the origin fetch in a timeout to prevent request-thread starvation if the origin stalls (duration can be crate-configurable).
-        let origin_continuation_token = continuation_token
-            .map(|token| token.trim_start_matches(CONTINUATION_TOKEN_ORIGIN_PREFIX).to_string());
+        let origin_continuation_token = continuation_token
+            .as_deref()
+            .and_then(|t| t.strip_prefix(CONTINUATION_TOKEN_ORIGIN_PREFIX))
+            .map(str::to_string);

Optional (illustrative):

// let origin_events_chunk = tokio::time::timeout(Duration::from_secs(10), origin_caller
//     .starknet_client
//     .get_events(...))
//     .await
//     .map_err(|_| ApiError::StarknetDevnetError(Error::UnexpectedInternalError { msg: "Origin events timeout".into() }))??
//     .into();
tests/integration/test_subscription_to_events.rs (1)

556-629: Good end‑to‑end coverage of forked origin + local events

Solid flow: origin emission, fork, subscribe, then local emissions using origin account to avoid nonce clash. Optional: also assert finality_status for origin and fork emissions to tighten guarantees.

tests/integration/test_subscription_to_blocks.rs (2)

261-331: Block notifications across fork boundaries look correct

Nice sequencing: origin 0..n, expected empty block after fork, pre‑subscription fork blocks, then live blocks. Optional: assert parent_hash linkage for a sampled pair to detect ordering regressions.


332-392: Subscription starting by origin block hash works; assertions are clear

Covers mid‑range start and post‑fork progression. Optional: include an assertion that block hashes are unique and strictly increase by number for the notified sequence.

crates/starknet-devnet-server/src/api/endpoints_ws.rs (1)

176-187: Consider parallel fetching for better performance.

The method fetches origin block headers sequentially, which could be slow for large ranges (up to 1024 blocks). Consider using parallel requests (e.g., with futures::future::join_all) to improve performance, especially since the headers are pre-fetched before notifying subscribers.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 34ee6e5 and a4f15c1.

📒 Files selected for processing (6)
  • crates/starknet-devnet-core/src/blocks/mod.rs (2 hunks)
  • crates/starknet-devnet-server/src/api/endpoints.rs (1 hunks)
  • crates/starknet-devnet-server/src/api/endpoints_ws.rs (7 hunks)
  • crates/starknet-devnet-types/src/rpc/block.rs (1 hunks)
  • tests/integration/test_subscription_to_blocks.rs (1 hunks)
  • tests/integration/test_subscription_to_events.rs (1 hunks)
🔇 Additional comments (5)
crates/starknet-devnet-core/src/blocks/mod.rs (1)

12-12: Hashing engine switched to Poseidon — confirm backward compatibility

This is a wide-impact change. Please confirm implications for:

  • Existing dumps/snapshots: can older Pedersen-hashed blocks be reloaded or migrated?
  • Interop with any external tooling expecting prior hash derivation.
    If incompatible, note it in release notes and consider a controlled migration path.
crates/starknet-devnet-server/src/api/endpoints_ws.rs (4)

2-13: LGTM! Imports support origin-aware functionality.

The new imports are necessary for fetching and converting block data from the forking origin.


104-117: LGTM! Local block header retrieval is correctly implemented.

Proper error handling for rejected blocks and missing blocks.


207-242: LGTM! Origin headers are correctly pre-fetched and notified.

The implementation properly handles the boundary between origin and local blocks, pre-fetching all origin headers before notifying to ensure consistency.


343-379: LGTM! Pagination logic is correctly implemented.

The method properly handles continuation tokens to fetch all events from the origin. The naming similarity with get_origin_events has already been noted in past reviews.

Based on past review comments.

@3alpha
Copy link
Copy Markdown
Collaborator Author

3alpha commented Oct 1, 2025

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Oct 1, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@3alpha 3alpha merged commit b07aa33 into main Oct 2, 2025
2 checks passed
@3alpha 3alpha deleted the fetch-data-from-origin-on-subscription branch October 2, 2025 10:45
@3alpha 3alpha added the bug Something isn't working label Oct 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Content from forking origin not available in websocket subscription

2 participants