Skip to content

perf(l1): defer KZG blob proof verification from P2P to mempool insertion#6150

Merged
ilitteri merged 10 commits into
mainfrom
perf/defer-kzg-to-mempool
Feb 10, 2026
Merged

perf(l1): defer KZG blob proof verification from P2P to mempool insertion#6150
ilitteri merged 10 commits into
mainfrom
perf/defer-kzg-to-mempool

Conversation

@ilitteri

@ilitteri ilitteri commented Feb 6, 2026

Copy link
Copy Markdown
Collaborator

Motivation

eBPF profiling on Hoodi steady-state (main branch, 5-minute capture at 49 Hz) showed ~51% of CPU samples in BLS12-381/KZG blob proof verification during P2P transaction handling. The expensive verify_kzg_proof_batch/verify_cell_kzg_proof_batch calls in PooledTransactions::validate_requested() run for every blob tx received from every peer, before the mempool deduplication check. When the same blob transaction is received from N peers, KZG runs N times — all but the first are redundant.

Description

Add validate_cheap() to BlobsBundle that performs all structural checks (max blobs per block, version/fork compatibility, lengths, versioned hash SHA256 verification) but skips the KZG proof verification. Use it in validate_requested() so the P2P layer only does cheap validation.

Full KZG verification still happens in add_blob_transaction_to_pool(), which checks for duplicates before validating KZG:

1. P2P: validate_cheap() — structural checks only (fast)
2. Mempool: dedup check — if already in pool, return early (no KZG)
3. Mempool: validate() — full KZG only for new unique blobs

This means KZG runs exactly once per unique blob transaction, regardless of how many peers send it.

eBPF profiling results (simultaneous 5-min captures, same block range)

Both servers running Hoodi on Ryzen 9 9950X3D, profiled during the same 5-minute window (same blocks, same P2P traffic):

Metric Baseline (office-2, main) Test (office-3, this branch) Change
Total CPU samples 2,692 675 -74.9%
KZG samples (absolute) 1,365 180 -86.8%
KZG % of total 50.7% 26.7% -24.0pp

KZG is pure CPU (BLS12-381 pairings), so the RAM difference between servers (126 GB vs 62 GB) does not affect this comparison. The remaining ~27% KZG is the irreducible minimum: each unique blob tx must be validated once at mempool insertion.

EXPB benchmark results (no regression)

Synthetic engine_newPayload benchmarks on ethrex-office-5, comparing against fresh main baselines at the same commit:

Fast scenario (main #253 vs ours #247):

Metric main This branch Change
latency avg 26.23ms 26.33ms +0.4%
latency med 22.30ms 22.83ms +2.4%
latency p95 49.98ms 50.02ms +0.1%
mgas avg 775.78 766.31 -1.2%

Gigablocks scenario (main #254 vs ours #248):

Metric main This branch Change
latency avg 754.51ms 755.88ms +0.2%
latency med 721.06ms 720.65ms -0.1%
latency p95 1150.00ms 1160.00ms +0.9%
mgas avg 1547.14 1545.62 -0.1%

All metrics within noise (<2.5%), confirming no regression in block execution performance. This is expected since the optimization only affects the P2P blob validation path, not engine_newPayload processing.

Test plan

  • Node runs correctly on Hoodi (processing blocks at chain speed, ~1.6 Ggas/s)
  • Simultaneous eBPF profiling confirms -86.8% KZG sample reduction
  • EXPB fast benchmark: no regression (latency avg +0.4%, mgas avg -1.2%)
  • EXPB gigablocks benchmark: no regression (latency avg +0.2%, mgas avg -0.1%)
  • Verify blob transactions still propagate correctly through the network

…rtion

eBPF profiling on main showed ~38% of CPU time spent in BLS12-381/KZG
proof verification during P2P blob transaction handling. The expensive
verify_kzg_proof_batch/verify_cell_kzg_proof_batch calls in
validate_requested() run for every blob tx received from every peer,
before the mempool dedup check.

Add validate_cheap() to BlobsBundle that performs all structural checks
(max blobs, version/fork, lengths, versioned hash SHA256) but skips the
KZG proof verification. Use it in validate_requested() so the P2P layer
only does cheap validation. Full KZG verification still happens in
add_blob_transaction_to_pool() after dedup, ensuring correctness while
avoiding redundant cryptographic work for the same blob seen from
multiple peers.
Copilot AI review requested due to automatic review settings February 6, 2026 18:27
@github-actions github-actions Bot added L1 Ethereum client performance Block execution throughput and performance in general labels Feb 6, 2026
@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

Review Summary

This PR introduces a validate_cheap method for blob bundle validation in P2P contexts, deferring expensive KZG verification to mempool insertion. The implementation is generally sound but has a few issues.

Issues Found

1. Missing KZG Feature Check (Security/Correctness)

File: crates/common/types/blobs_bundle.rs:192
The method is marked with #[cfg(feature = "c-kzg")] but the validation logic doesn't actually use KZG. However, the method references CELLS_PER_EXT_BLOB which is only defined under the same feature flag. This creates a potential compilation issue if the feature flag is removed.

2. Incomplete Validation (Security)

File: crates/common/types/blobs_bundle.rs:192-227
The validate_cheap method skips cryptographic verification entirely, which is the stated purpose. However, it should still validate that:

  • Commitments are valid points on the BLS12-381 curve
  • Proofs are valid points on the BLS12-381 curve
  • The basic structure is valid (lengths, etc.)

The current implementation only checks structural consistency, which is appropriate for the "cheap" validation, but the method name could be clearer about what it does/doesn't validate.

3. Redundant Validation (Performance)

File: crates/common/types/blobs_bundle.rs:222-224
The call to validate_blob_commitment_hashes computes the commitment hashes and compares them, which involves cryptographic operations. While not as expensive as full KZG verification, this might still be more expensive than intended for a "cheap" validation.

4. Error Handling Consistency (Code Quality)

File: crates/common/types/blobs_bundle.rs:215-220
The validation logic for version/fork compatibility could be clearer. Consider:

match (self.version, fork) {
    (0, fork) if fork >= Fork::Osaka => return Err(BlobsBundleError::InvalidBlobVersionForFork),
    (v, fork) if v != 0 && fork < Fork::Osaka => return Err(BlobsBundleError::InvalidBlobVersionForFork),
    _ => {}
}

Suggestions

  1. Rename the method to better reflect its purpose: validate_structure or validate_without_kzg would be clearer than validate_cheap.

  2. Consider the performance impact of validate_blob_commitment_hashes - if this involves significant computation, it might defeat the purpose of "cheap" validation.

  3. Add documentation clarifying exactly what this method validates and what it defers.

  4. Consider adding a debug assertion to ensure the method isn't accidentally used where full validation is required.

The change in transactions.rs to use validate_cheap instead of validate is appropriate for the P2P context described in the PR.


Automated review by Kimi (Moonshot AI) · custom prompt

@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Now I have a complete understanding of the PR. Let me compile my review:

PR Review: Defer KZG blob proof verification from P2P to mempool insertion

Summary

This optimization defers expensive KZG proof verification from the P2P layer to mempool insertion, achieving an 85.5% reduction in KZG-related CPU samples. The approach is sound and the implementation is correct.

Positive Aspects

Solid Performance Improvement: The profiling data shows clear impact - reducing redundant KZG verification for duplicate blob transactions received from multiple peers.

Security Architecture: The validation flow is correct:

  1. P2P: validate_cheap() - structural checks only
  2. Mempool: deduplication check at blockchain.rs:1993
  3. Mempool: full validate() with KZG at blockchain.rs:1999

Code Quality: The validate_cheap() function correctly replicates all non-cryptographic checks from validate().

Issues & Suggestions

1. Missing Operator Precedence Parentheses (Critical for correctness)

Location: blobs_bundle.rs:210

The condition has ambiguous operator precedence:

if self.version == 0 && fork >= Fork::Osaka || self.version != 0 && fork < Fork::Osaka {

This evaluates as: (self.version == 0 && fork >= Fork::Osaka) || (self.version != 0 && fork < Fork::Osaka)

While this appears correct, it's error-prone and hard to verify. The same issue exists in the original validate() at line 147.

Recommendation: Add explicit parentheses for clarity:

if (self.version == 0 && fork >= Fork::Osaka) || (self.version != 0 && fork < Fork::Osaka) {

2. Code Duplication (Maintainability concern)

Location: blobs_bundle.rs:192-225

The validate_cheap() function duplicates 30+ lines from validate(). This creates maintenance burden - any change to validation logic must be applied in both places.

Recommendation: Consider refactoring to reduce duplication:

pub fn validate(&self, tx: &super::EIP4844Transaction, fork: super::Fork) 
    -> Result<(), BlobsBundleError> {
    // Call cheap validation first
    self.validate_cheap(tx, fork)?;
    
    // Then do expensive KZG verification
    if self.version != 0 {
        use ethrex_crypto::kzg::verify_cell_kzg_proof_batch;
        if !verify_cell_kzg_proof_batch(&self.blobs, &self.commitments, &self.proofs)? {
            return Err(BlobsBundleError::BlobToCommitmentAndProofError);
        }
    } else {
        use ethrex_crypto::kzg::verify_kzg_proof_batch;
        if !verify_kzg_proof_batch(&self.blobs, &self.commitments, &self.proofs)? {
            return Err(BlobsBundleError::BlobToCommitmentAndProofError);
        }
    }
    Ok(())
}

This ensures they stay in sync and makes the optimization explicit.

3. Missing Test Coverage (Nice to have)

Location: No tests for validate_cheap()

While existing tests cover the full validate() path, there are no explicit tests for validate_cheap() behavior.

Recommendation: Add tests to verify:

  • validate_cheap() passes for valid blobs (without KZG verification)
  • validate_cheap() rejects invalid structural issues (wrong lengths, version mismatches, versioned hash mismatches)
  • validate_cheap() is faster than validate() (benchmark test)

4. Minor: Comment Clarity

Location: blobs_bundle.rs:187-190

The comment is good but could mention that this is a performance optimization specifically.

Suggestion:

/// Performance optimization: Validates blob bundle structure without expensive 
/// KZG cryptographic verification. Used in P2P validation where full KZG is 
/// deferred to mempool insertion (after dedup check), avoiding redundant proof 
/// verification for the same blob tx received from multiple peers.

Ethereum-Specific Considerations

EIP-4844 Compliance: The structural validation correctly checks all EIP-4844 requirements except cryptographic proof verification, which is intentionally deferred.

Fork Handling: Properly handles Osaka fork blob version transition (version 0 vs version 1).

Versioned Hash Verification: Correctly validates SHA256-based versioned hashes in validate_blob_commitment_hashes().

Verdict

Approve with suggestions. The optimization is sound and achieves significant performance improvements. The security model is correct - all transactions eventually undergo full KZG verification before entering the mempool. The main concern is code duplication which should be addressed to improve maintainability.

Priority fixes:

  1. Add parentheses for operator precedence clarity (both in new and old code)
  2. Refactor to eliminate code duplication between validate() and validate_cheap()

Automated review by Claude (Anthropic) · custom prompt

@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

Findings

  • Potential peer-penalty regression / DoS vector: validate_requested now uses validate_cheap, so invalid KZG proofs are no longer caught during request validation. Those errors only surface later in add_blob_transaction_to_pool, where they’re logged and ignored (no disconnect/penalty). This allows a peer to spam many unique blob txs with valid commitments but invalid proofs and avoid disconnection while still forcing expensive KZG verification once per unique tx. Consider disconnecting/penalizing on BlobToCommitmentAndProofError from add_blob_transaction_to_pool, or keep full KZG validation for requested responses if that’s the intended enforcement point. crates/networking/p2p/rlpx/eth/transactions.rs:244 (callsite) and crates/blockchain/blockchain.rs:1999 (full validation in mempool add).
  • Maintainability: validate_cheap duplicates most of validate’s structural checks; keeping them in sync across forks/versions is error-prone. Consider extracting shared “structural validation” into a helper and letting validate call it, then only add the KZG proof checks. crates/common/types/blobs_bundle.rs:127 and crates/common/types/blobs_bundle.rs:187.
  • Readability/correctness risk: the fork/version guard uses mixed &&/|| without parentheses, which is easy to misread and could be accidentally changed incorrectly later. Prefer explicit grouping. crates/common/types/blobs_bundle.rs:147 and crates/common/types/blobs_bundle.rs:210.

If the above is intentional (especially the peer-penalty change), then the core logic looks consistent and the performance win is clear.


Automated review by OpenAI Codex · custom prompt

@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

Lines of code report

Total lines added: 31
Total lines removed: 0
Total lines changed: 31

Detailed view
+--------------------------------------------------------+-------+------+
| File                                                   | Lines | Diff |
+--------------------------------------------------------+-------+------+
| ethrex/crates/common/types/blobs_bundle.rs             | 472   | +12  |
+--------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/rlpx/connection/server.rs | 1069  | +16  |
+--------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/rlpx/eth/transactions.rs  | 332   | +3   |
+--------------------------------------------------------+-------+------+

@greptile-apps

greptile-apps Bot commented Feb 6, 2026

Copy link
Copy Markdown

Greptile Overview

Greptile Summary

This PR optimizes blob transaction handling by deferring expensive KZG cryptographic proof verification from P2P receipt to mempool insertion, reducing redundant computation when the same blob transaction is received from multiple peers.

Key changes:

  • Added validate_cheap() method to BlobsBundle that performs all structural validation (max blobs check, empty check, fork compatibility, length validation, SHA256 versioned hash verification) but skips KZG proof verification
  • Modified P2P layer's validate_requested() to use validate_cheap() instead of validate()
  • Full KZG validation still occurs in add_blob_transaction_to_pool() after the dedup check, ensuring each unique blob is validated exactly once

Performance impact:
The profiling data shows a 67.8% reduction in total CPU samples and 85.5% reduction in KZG-related samples. The remaining ~15% KZG overhead represents the irreducible minimum (one validation per unique blob transaction).

Security analysis:
The change is safe because:

  1. All cheap structural checks are preserved in P2P validation
  2. Full KZG verification happens at mempool insertion (blockchain.rs:1999) after dedup check (blockchain.rs:1993)
  3. Invalid blobs are rejected before entering the mempool
  4. The optimization only eliminates redundant KZG verification for duplicate transactions from multiple peers

The implementation is clean, minimal, and well-documented with clear comments explaining the optimization strategy.

Confidence Score: 5/5

  • This PR is safe to merge - it's a well-designed performance optimization with no security or correctness issues
  • The implementation correctly preserves all structural validation (bounds, fork compatibility, length checks, SHA256 hash verification) in P2P while only deferring the expensive KZG cryptographic proof verification to mempool insertion. The dedup check in add_blob_transaction_to_pool (line 1993) happens before full validation (line 1999), ensuring KZG runs exactly once per unique blob transaction. The change is minimal, focused, and maintains security invariants.
  • No files require special attention

Important Files Changed

Filename Overview
crates/common/types/blobs_bundle.rs Added validate_cheap() method that performs structural validation without expensive KZG cryptographic verification - correctly implements all checks except KZG proof verification
crates/networking/p2p/rlpx/eth/transactions.rs Changed P2P validation to use validate_cheap() instead of validate() - defers expensive KZG proof verification to mempool insertion

Sequence Diagram

sequenceDiagram
    participant Peer
    participant P2P as P2P Layer<br/>(validate_requested)
    participant Mempool
    participant Validation as Full Validation<br/>(add_blob_transaction_to_pool)
    
    Note over Peer,Validation: Before this PR: KZG runs N times for N peers
    
    Peer->>P2P: Send blob transaction
    P2P->>P2P: validate_cheap()<br/>(structural checks only)
    Note right of P2P: Max blobs check<br/>Empty check<br/>Fork compatibility<br/>Length checks<br/>SHA256 hash verification<br/>❌ Skip KZG proof
    
    P2P->>Mempool: Add to mempool
    Mempool->>Mempool: Check if tx already exists
    alt Transaction already in pool
        Mempool-->>P2P: Return early (no KZG!)
    else New transaction
        Mempool->>Validation: validate() full KZG check
        Note right of Validation: ✅ KZG proof verification<br/>(expensive!)
        Validation-->>Mempool: Valid
        Mempool->>Mempool: Add transaction
    end
    
    Note over Peer,Validation: Result: KZG runs exactly once per unique blob tx
Loading

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces redundant CPU-heavy KZG verification during P2P blob-transaction handling by deferring full proof verification until mempool insertion, where duplicates are filtered first.

Changes:

  • Add BlobsBundle::validate_cheap() to perform structural/versioned-hash checks while skipping KZG proof verification.
  • Switch P2P PooledTransactions::validate_requested() to use validate_cheap() instead of full validate() for EIP-4844 txs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
crates/networking/p2p/rlpx/eth/transactions.rs Uses cheap blob bundle validation during P2P “response matches request” validation.
crates/common/types/blobs_bundle.rs Introduces validate_cheap() to validate blob bundle structure without running KZG verification.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 244 to 247
for tx in &self.pooled_transactions {
if let P2PTransaction::EIP4844TransactionWithBlobs(itx) = tx {
itx.blobs_bundle.validate(&itx.tx, fork)?;
itx.blobs_bundle.validate_cheap(&itx.tx, fork)?;
}

Copilot AI Feb 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching P2P validation from validate() to validate_cheap() means invalid KZG proofs will no longer cause validate_requested() to error, so the peer will not be disconnected in server.rs for sending bad proofs. Those invalid blobs will instead be rejected later in add_blob_transaction_to_pool() and silently ignored here, which can weaken DoS protection (a peer can repeatedly send unique/invalid blob txs and keep the connection). Consider either (a) disconnecting/penalizing the peer when mempool insertion fails specifically due to KZG verification, and/or (b) caching recently-seen invalid blob tx hashes (TTL) to avoid repeated expensive KZG work for the same invalid tx.

Copilot uses AI. Check for mistakes.
Comment on lines +199 to +220
let max_blobs = max_blobs_per_block(fork);
let blob_count = self.blobs.len();

if blob_count > max_blobs {
return Err(BlobsBundleError::MaxBlobsExceeded);
}

if blob_count == 0 {
return Err(BlobsBundleError::BlobBundleEmptyError);
}

if self.version == 0 && fork >= Fork::Osaka || self.version != 0 && fork < Fork::Osaka {
return Err(BlobsBundleError::InvalidBlobVersionForFork);
}

if blob_count != self.commitments.len()
|| (self.version == 0 && blob_count != self.proofs.len())
|| (self.version != 0 && blob_count * CELLS_PER_EXT_BLOB != self.proofs.len())
|| blob_count != tx.blob_versioned_hashes.len()
{
return Err(BlobsBundleError::BlobsBundleWrongLen);
};

Copilot AI Feb 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate_cheap() duplicates most of the structural validation logic in validate(). This increases the risk that future changes update one path but not the other (e.g., fork/version rules, length checks, max blob count). Consider factoring the shared structural checks into a single internal helper (or have validate() call validate_cheap() first, then run the KZG verification) so both paths stay consistent.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, maybe call validate_cheap inside validate?

Comment on lines +187 to +225
/// Validates blob bundle structure without expensive KZG cryptographic verification.
/// Used in P2P validation where full KZG is deferred to mempool insertion
/// (after dedup check), avoiding redundant proof verification for the same
/// blob tx received from multiple peers.
#[cfg(feature = "c-kzg")]
pub fn validate_cheap(
&self,
tx: &super::EIP4844Transaction,
fork: super::Fork,
) -> Result<(), BlobsBundleError> {
use super::CELLS_PER_EXT_BLOB;

let max_blobs = max_blobs_per_block(fork);
let blob_count = self.blobs.len();

if blob_count > max_blobs {
return Err(BlobsBundleError::MaxBlobsExceeded);
}

if blob_count == 0 {
return Err(BlobsBundleError::BlobBundleEmptyError);
}

if self.version == 0 && fork >= Fork::Osaka || self.version != 0 && fork < Fork::Osaka {
return Err(BlobsBundleError::InvalidBlobVersionForFork);
}

if blob_count != self.commitments.len()
|| (self.version == 0 && blob_count != self.proofs.len())
|| (self.version != 0 && blob_count * CELLS_PER_EXT_BLOB != self.proofs.len())
|| blob_count != tx.blob_versioned_hashes.len()
{
return Err(BlobsBundleError::BlobsBundleWrongLen);
};

self.validate_blob_commitment_hashes(&tx.blob_versioned_hashes)?;

Ok(())
}

Copilot AI Feb 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate_cheap() is new behavior that is now relied on by the P2P path, but there are no unit tests covering it (existing tests only exercise validate()). Add tests that assert (1) validate_cheap() rejects the same structural issues as validate() (empty bundle, wrong lengths, wrong fork/version), and (2) validate_cheap() succeeds for a bundle with correct structure but invalid proofs/blobs where validate() fails (to ensure KZG is actually skipped).

Copilot uses AI. Check for mistakes.
…cated

structural checks. Add explicit parentheses for operator precedence clarity
on the version/fork condition.
version-0 and version-1 KZG verification paths
@ilitteri ilitteri requested a review from iovoid February 6, 2026 19:37
@github-actions

github-actions Bot commented Feb 6, 2026

Copy link
Copy Markdown

Benchmark Block Execution Results Comparison Against Main

Command Mean [s] Min [s] Max [s] Relative
base 63.193 ± 0.087 63.052 63.350 1.00
head 63.246 ± 0.168 63.061 63.667 1.00 ± 0.00

@azteca1998 azteca1998 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but please review copilot's comments since they address potential security concerns.

@github-project-automation github-project-automation Bot moved this to In Review in ethrex_l1 Feb 9, 2026
Previously, when add_blob_transaction_to_pool failed due to invalid KZG
proofs, the error was silently logged and swallowed — the peer stayed
connected and could keep sending invalid blobs (DoS vector).

Now, BlobsBundleError is returned from PooledTransactions::handle() and
caught in handle_incoming_message(), which disconnects the peer with
SubprotocolError, matching the existing pattern for validate_requested
and BlockRangeUpdate validation failures.
Cover the new validate_cheap() method with 6 test cases:
- accepts valid blobs
- rejects empty bundles
- rejects wrong lengths (commitment/blob count mismatch)
- rejects version/fork mismatch (version-0 on Osaka)
- rejects wrong versioned hashes
- passes with invalid KZG proofs (key test: structural checks pass
  while full validate() correctly fails on KZG verification)
@ilitteri ilitteri added this pull request to the merge queue Feb 10, 2026
Merged via the queue into main with commit 819977d Feb 10, 2026
61 of 65 checks passed
@ilitteri ilitteri deleted the perf/defer-kzg-to-mempool branch February 10, 2026 01:34
@github-project-automation github-project-automation Bot moved this from Todo to Done in ethrex_performance Feb 10, 2026
@github-project-automation github-project-automation Bot moved this from In Review to Done in ethrex_l1 Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-code-assisted L1 Ethereum client performance Block execution throughput and performance in general

Projects

Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants