Store blobs locally post fusaka + disk reader#4007
Merged
Conversation
The Ethereum Fusaka fork introduced /eth/v1/beacon/blobs as a replacement
for the legacy /eth/v1/beacon/blob_sidecars endpoint. Nitro's getBlobs()
function fetches from the new endpoint but was not saving blobs to disk.
The legacy blobSidecars() function saves blobs in "Version 0" format with
full metadata (commitments, proofs, block roots). The new endpoint returns
only blob data, requiring a minimal storage format optimized for hash-based
lookups.
Additionally, there was no ReadBlobsFromDisk() function - blobs could be
saved but not read back. This commit adds the reader for testing and to
enable a future beacon endpoint emulator that we may add.
Storage Format V1 (new):
- Stores blobs as versioned hash -> blob map
- File: {blob-directory}/{slot}
- Format: {"version": 1, "data": {"0xHASH": "0xBLOB", ...}}
- Used by getBlobs() when fetching from /eth/v1/beacon/blobs
Storage Format V0 (legacy, backward compatible):
- Stores blob_sidecars array with full metadata
- Format: {"data": [{"blob": "0x...", "kzg_commitment": "0x...",
"kzg_proof": "0x...", ...}]}
- Used by blobSidecars() when fetching from legacy endpoint
- Still readable for backward compatibility
ReadBlobsFromDisk() (new):
- Reads blobs from disk, transparently handling V0 or V1 format
- Auto-detects version via presence of "version" field
- Validates blob integrity:
* V1: Computes commitment -> versioned hash, verifies match to key
* V0: Computes commitment from blob, verifies match to stored commitment
- No migration needed - both formats coexist
Renamed functions for clarity:
- saveBlobDataToDisk -> saveBlobsV0ToDisk (legacy format)
- Added saveBlobsV1ToDisk (new format)
Testing improvements:
- Updated createTestBlobs() to use production blobs.EncodeBlobs()
- All tests now use real KZG blobs with valid commitments
- Added TestReadBlobsV1ValidationFailure and TestReadBlobsV0ValidationFailure
- Renamed existing tests to indicate format (e.g., TestSaveBlobsV0ToDisk)
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds disk storage and reading functionality for blobs fetched from the new Fusaka-era /eth/v1/beacon/blobs endpoint. The changes introduce a new storage format (V1) optimized for hash-based lookups while maintaining backward compatibility with the legacy format (V0), and implement a unified reader that handles both formats with validation.
Key Changes:
- Added
saveBlobsV1ToDisk()to store blobs as versioned hash → blob maps for new endpoint - Implemented
ReadBlobsFromDisk()with automatic format detection and validation for both V0 and V1 formats - Updated
getBlobs()to persist fetched blobs to disk in V1 format
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| util/headerreader/blob_client.go | Implements V1 storage format, disk reader with validation, and integrates disk saving into getBlobs() |
| util/headerreader/blob_client_test.go | Adds comprehensive tests for both storage formats, validation failures, and format detection |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // createTestBlobs creates test blobs and their versioned hashes | ||
| func createTestBlobs(count int) ([]kzg4844.Blob, []common.Hash, error) { | ||
| testData := make([]byte, count*blobs.BlobEncodableData) | ||
| r := rand.New(rand.NewSource(1)) |
There was a problem hiding this comment.
Using deprecated rand.New() with rand.Source. In Go 1.20+, use rand.NewSource directly or the new ChaCha8 source. Consider using r := rand.New(rand.NewChaCha8(rand.NewSource(1))) for better randomness characteristics.
Suggested change
| r := rand.New(rand.NewSource(1)) | |
| r := rand.New(rand.NewChaCha8(rand.NewSource(1))) |
| // Save blobs to disk in version 1 format if blobDirectory is configured | ||
| if b.blobDirectory != "" { | ||
| if err := saveBlobsV1ToDisk(output, computedHashes, slot, b.blobDirectory); err != nil { | ||
| return nil, err |
There was a problem hiding this comment.
The error returned from saveBlobsV1ToDisk lacks context about which operation failed. Consider wrapping the error with additional context: return nil, fmt.Errorf("failed to save blobs to disk for slot %d: %w", slot, err)
Suggested change
| return nil, err | |
| return nil, fmt.Errorf("failed to save blobs to disk for slot %d: %w", slot, err) |
| if err != nil { | ||
| return fmt.Errorf("unable to marshal blobs into JSON: %w", err) | ||
| } | ||
|
|
There was a problem hiding this comment.
[nitpick] File permissions 0600 make the file readable/writable only by the owner, which is appropriate for blob data. However, consider if the blob directory itself has appropriate permissions to prevent unauthorized access.
Suggested change
| // Ensure the blob directory exists with secure permissions | |
| if err := os.MkdirAll(blobDirectory, 0700); err != nil { | |
| return fmt.Errorf("failed to create blob directory: %w", err) | |
| } |
Contributor
❌ 4 Tests Failed:
View the top 3 failed tests by shortest run time
📣 Thoughts on this report? Let Codecov know! | Powered by Codecov |
gligneul
approved these changes
Nov 14, 2025
Contributor
|
Looks good, but I still have the feeling Nitro is the wrong place to cache this. |
Member
Author
|
@gligneul wrote:
Agree. I'd like to build a separate service, but for now this change is just to make sure we maintain existing functionality. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Ethereum Fusaka fork introduced /eth/v1/beacon/blobs as a replacement for the legacy /eth/v1/beacon/blob_sidecars endpoint. Nitro's getBlobs() function fetches from the new endpoint but was not saving blobs to disk.
The legacy blobSidecars() function saves blobs in "Version 0" format with full metadata (commitments, proofs, block roots). The new endpoint returns only blob data, requiring a minimal storage format optimized for hash-based lookups.
Additionally, there was no ReadBlobsFromDisk() function - blobs could be saved but not read back. This commit adds the reader for testing and to enable a future beacon endpoint emulator that we may add.
Storage Format V1 (new):
Storage Format V0 (legacy, backward compatible):
ReadBlobsFromDisk() (new):
Renamed functions for clarity:
Testing improvements:
NIT-4113