Add support for external datastore as source of historical ledger data by urvisavla · Pull Request #437 · stellar/stellar-rpc

urvisavla · 2025-05-16T06:22:09Z

What

This PR addresses #425 - adding support for handling getLedgers requests that fall outside the rpc's local ledger retention window by proxies out of range requests to the datastore.

Adds support for serving historical ledgers from a datastore when the requested range isn’t available locally.
Uses BufferedStorageBackend to read from the datastore.
The gen-config-file command now includes default datastore_config and buffered_storage_backend_config sections (commented out by default).

# Buffered storage backend configuration for reading ledgers from the datastore.
# [buffered_storage_backend_config]
  # buffer_size = 100
  # num_workers = 10
  # retry_limit = 3
  # retry_wait = "30s"

# External datastore configuration including type, bucket name and schema.
# [datastore_config]
  # type = "GCS"

  # [datastore_config.params]
    # destination_bucket_path = "path_to_bucket"

  # [datastore_config.schema]
    # files_per_partition = 64000
    # ledgers_per_file = 1

Adds a new flag serve_ledger_from_datastore. The flag used instead of just relying on the presence of a datastore config because same datastore config can in future be also be used for ingestion from a datastore, so this flag control over which behavior is enabled.
Includes basic unit tests.

Why

#425

Known limitations

Currently, this assumes that the datastore has all ledgers starting from genesis. This is because there's no way yet to query datastore what range of ledgers it actually has. That limitation will be fixed once this stellar/go#5498) is done. We will then also use the real range to fill in the latestLedger and oldestLedger fields more accurately.
The datastore_config and buffered_storage_backend_config can only be set through the config file and not with command-line flags. That’s because these settings can be complex or different depending on the type of datastore being used (like S3 or GCS). Supporting them in the config file keeps things simpler and more flexible.

Shaptic

Just a lil' drive by review 🚗

Shaptic · 2025-05-16T22:55:10Z

cmd/stellar-rpc/internal/config/options.go

 			DefaultValue: 5 * time.Second,
 		},
+		{
+			Name:         "serve-ledgers-from-datastore",


Isn't this implied by the presence of a datastore config?

reason

Adds a new flag serve_ledger_from_datastore. The flag used instead of just relying on the presence of a datastore config because same datastore config can in future be also be used for ingestion from a datastore, so this flag control over which behavior is enabled.

cmd/stellar-rpc/internal/config/options.go

cmd/stellar-rpc/internal/datastore/ledger_reader.go

tamirms · 2025-05-28T13:55:03Z

cmd/stellar-rpc/internal/methods/get_ledgers.go

 	return protocol.GetLedgersResponse{
-		Ledgers:               ledgers,
+		Ledgers: ledgers,
+		//	TODO: update these fields using ledger range from datastore


I think we should consider using separate fields for the ledger range belonging to the data store. I think it would be useful to know which ledgers are available directly from rpc and which ledgers require access to the data store.

Can you describe a client use case which benefits from the breakout of ledgers?

This pushes forward an internal aspect of RPC, totally understand it given the possible difference of meta content mentioned. It seems like this introduces additional work for clients to handle and possibly some confusion.

Ideally, the outcome from sourcing other backends for ledger metadata by RPC should be transitive at the client API layer, i.e. data provided by RPC is from the network, nothing more?

One thought..Can the potential difference of ledger close meta between RPC or datastores be mitigated if Galexie were to always enable all captive core flags that affect ledger close meta? In that way a datastore would always provide the superset of meta avoiding missing data cases when interleaved with meta from any other source.

tamirms · 2025-05-28T14:12:44Z

Now that there are several stellar-core flags (diagnostic events, classic cap-67 events, etc) which affect the contents of ledger close meta, I think we should consider the scenario where ledgers in the datastore are inconsistent with the ledgers served locally from rpc.

This issue can be addressed separate from this PR but I think we'll need to introduce some way to determine which features are enabled in the ledgers found from the data lake vs the rpc captive core. https://github.com/stellar/go/issues/5507 could be relevant

tamirms

there are 2 issues that imo warrant further design discussions but I think those can be addressed separately from this PR

mollykarcher · 2025-05-29T18:37:55Z

Now that there are several stellar-core flags (diagnostic events, classic cap-67 events, etc) which affect the contents of ledger close meta, I think we should consider the scenario where ledgers in the datastore are inconsistent with the ledgers served locally from rpc.

This is a really good point. Another way to solve this might be, that if you have a datastore configured as your backend, RPC's ledger retention window gets automatically set to 0, so RPC is effectively just acting as a lightweight facade/API layer on top of the data lake. I think I would be more comfortable with doing that if getTransactions/getEvents were also being served from the data lake though. Though there is precedence in other ecosystems (at least, ethereum) for some endpoints not working on when a node is running in archival mode.

… available ledger range from the datastore.

…ed config

…astore latency.

urvisavla · 2025-06-03T05:22:14Z

I'm going to merge the PR but I've tried to capture all the open items from the PR in this issue #453. Please add anything I might have missed.

urvisavla force-pushed the getledgers-from-datastore branch 2 times, most recently from 9f70442 to 83c48d4 Compare May 16, 2025 20:57

Shaptic reviewed May 16, 2025

View reviewed changes

urvisavla force-pushed the getledgers-from-datastore branch 27 times, most recently from 84e5f50 to 33ab837 Compare May 18, 2025 20:03

urvisavla force-pushed the getledgers-from-datastore branch 2 times, most recently from 16e5297 to abba4e4 Compare May 26, 2025 23:28

urvisavla requested review from 2opremio and tamirms May 27, 2025 21:39

tamirms reviewed May 28, 2025

View reviewed changes

tamirms approved these changes May 28, 2025

View reviewed changes

urvisavla mentioned this pull request May 29, 2025

Implement a UnifiedLedgerReader in RPC to handle ledger fetching from both db and datalake #451

Open

urvisavla added 16 commits June 1, 2025 23:16

Add support for external datastore as source of historical ledger data

5a20fc4

Fix integration test

5615e75

fix linter warning

c866a23

Fix config unit test

93ae73e

Add happy path unit test for fetchLedgers

1277325

Fix linter error

0f969da

another attempt at fixing linter error

e35f0bd

Add more unit tests

8b1531f

add GetAvailableLedgerRange to datastore LedgerReader to retrieve the…

7297506

… available ledger range from the datastore.

Close datastore

0256e1d

Add default datastore and buffered storage backend values for generat…

2cb121c

…ed config

Minor refactoring

3325286

Add unit test for datastore LedgerReader

00f6207

Address review comments. Add unit test for default config.

adf71e6

Add integration test for GetLedgers using fake GCS server

534e6aa

Disable datastore retries. Increase getLedgers timeout to 10s for dat…

6b62f20

…astore latency.

urvisavla force-pushed the getledgers-from-datastore branch from 2f0397a to 6b62f20 Compare June 2, 2025 06:16

urvisavla mentioned this pull request Jun 3, 2025

Epic: RPC datastore integration improvements #453

Open

4 tasks

urvisavla merged commit aa8fc36 into stellar:protocol-23 Jun 3, 2025
10 of 13 checks passed

urvisavla deleted the getledgers-from-datastore branch June 3, 2025 05:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for external datastore as source of historical ledger data#437

Add support for external datastore as source of historical ledger data#437
urvisavla merged 16 commits intostellar:protocol-23from
urvisavla:getledgers-from-datastore

urvisavla commented May 16, 2025 •

edited

Loading

Uh oh!

Shaptic left a comment

Uh oh!

Shaptic May 16, 2025

Uh oh!

urvisavla May 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tamirms May 28, 2025

Uh oh!

sreuland May 29, 2025

Uh oh!

tamirms commented May 28, 2025

Uh oh!

tamirms left a comment

Uh oh!

mollykarcher commented May 29, 2025

Uh oh!

urvisavla commented Jun 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

urvisavla commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Known limitations

Uh oh!

Shaptic left a comment

Choose a reason for hiding this comment

Uh oh!

Shaptic May 16, 2025

Choose a reason for hiding this comment

Uh oh!

urvisavla May 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tamirms May 28, 2025

Choose a reason for hiding this comment

Uh oh!

sreuland May 29, 2025

Choose a reason for hiding this comment

Uh oh!

tamirms commented May 28, 2025

Uh oh!

tamirms left a comment

Choose a reason for hiding this comment

Uh oh!

mollykarcher commented May 29, 2025

Uh oh!

urvisavla commented Jun 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

urvisavla commented May 16, 2025 •

edited

Loading