Issue: bitcoin build-tx fails with "Too many object IDs requested" when handling large number of UTXOs
Summary
rooch bitcoin build-tx command fails when trying to build a transaction with a large number of UTXOs (e.g., 800+ UTXOs). The error message "Too many object IDs requested" is misleading - the actual root cause is that UTXOSelector::load_utxos() triggers 429 Too Many Requests errors when fetching UTXO pages in a loop, and there's no retry mechanism to handle rate limiting.
Version
Rooch version: latest (main branch)
Commit: unknown (working on main branch)
Steps to Reproduce
- Have a Bitcoin address with 800+ UTXOs (e.g.,
bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt)
- Run
rooch bitcoin build-tx without specifying -i parameters (expecting automatic UTXO selection)
rooch bitcoin build-tx \
-s bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt \
-o <receiver>:<amount> \
--fee-rate 1 \
--output-file tx.psbt
- Command fails with error: "Too many object IDs requested"
Expected Behavior
The command should:
- Automatically load all UTXOs for the address across multiple pages
- Handle rate limiting (429 errors) gracefully by retrying with exponential backoff
- Successfully build a transaction with all available UTXOs
Actual Behavior
The command fails with "Too many object IDs requested" error.
Investigation
- The
UTXOSelector::load_utxos() method in crates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116 implements pagination correctly
- However, when loading many pages rapidly (e.g., 18 pages for 864 UTXOs), it triggers RPC rate limiting (HTTP 429)
- There is no retry mechanism to handle 429 errors
- The pagination loop in
load_utxos() works correctly for small numbers of pages but fails under rate limiting
Code Analysis
From utxo_selector.rs:80-116:
async fn load_utxos(&mut self) -> Result<()> {
let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
if !has_next_page {
return Ok(());
}
let utxo_page = self
.client
.rooch
.query_utxos(
UTXOFilterView::owner(self.sender.clone()),
next_cursor.map(Into::into),
None, // No limit specified
Some(false),
)
.await?; // ❌ No retry on 429 errors
// ... process UTXOs ...
}
Problem: The await? propagates any 429 error immediately without retry.
Comparison with Working Code
The balance command in crates/rooch/src/commands/account/commands/balance.rs:187-219 successfully queries all UTXOs using a loop:
async fn get_total_utxo_value(...) {
let mut total_value: u64 = 0;
let mut cursor: Option<IndexerStateIDView> = None;
loop { // ✅ Uses loop for pagination
let page = client
.rooch
.query_utxos(
UTXOFilterView::Owner(address.clone()),
cursor.map(Into::into),
Some(MAX_RESULT_LIMIT),
None,
)
.await?; // But also lacks retry on 429
// ... accumulate value ...
if !page.has_next_page {
break;
}
cursor = page.next_cursor;
}
}
The balance command works because it likely doesn't hit the rate limit as quickly (or MAX_RESULT_LIMIT reduces the number of requests).
Root Cause
- Missing retry logic:
load_utxos() doesn't handle HTTP 429 (Too Many Requests) errors
- Rate limiting: RPC server returns 429 when too many requests are made in quick succession
- No backoff: No delay between requests to avoid triggering rate limits
Proposed Solution
Add retry logic with exponential backoff for 429 errors in load_utxos() method.
Option 1: Retry with Exponential Backoff (Recommended)
async fn load_utxos(&mut self) -> Result<()> {
let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
if !has_next_page {
return Ok(());
}
let mut retry_count = 0;
const MAX_RETRIES: u32 = 5;
loop {
let utxo_page = self
.client
.rooch
.query_utxos(
UTXOFilterView::owner(self.sender.clone()),
next_cursor.map(Into::into),
None,
Some(false),
)
.await;
match utxo_page {
Ok(page) => {
debug!("loaded utxos: {:?}", page.data.len());
let minimal_non_dust = self.sender.script_pubkey().minimal_non_dust();
for utxo_view in page.data {
let utxo = &utxo_view.value;
if !self.skip_seal_check && skip_utxo(&utxo_view, minimal_non_dust) {
continue;
}
if utxo_view.metadata.owner_bitcoin_address.is_none() {
debug!(
"Can not recognize the owner of UTXO {}, metadata: {:?}, skip.",
utxo.outpoint(),
utxo_view.metadata
);
continue;
}
self.candidate_utxos.push_front(utxo_view.into());
}
self.loaded_page = Some((page.next_cursor, page.has_next_page));
return Ok(());
}
Err(e) => {
// Check if it's a 429 error
let error_msg = e.to_string().to_lowercase();
if error_msg.contains("429") || error_msg.contains("too many requests") || error_msg.contains("rate limit") {
retry_count += 1;
if retry_count > MAX_RETRIES {
bail!("Too many 429 errors when loading UTXOs, gave up after {} retries", MAX_RETRIES);
}
// Exponential backoff: wait 2^retry_count seconds
let wait_secs = 2u64.pow(retry_count);
debug!("Got 429 error, retry {} after waiting {} seconds", retry_count, wait_secs);
tokio::time::sleep(tokio::time::Duration::from_secs(wait_secs)).await;
continue;
} else {
// Other errors - propagate
return Err(e);
}
}
}
}
}
Option 2: Add Delay Between Requests (Simpler)
async fn load_utxos(&mut self) -> Result<()> {
let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
if !has_next_page {
return Ok(());
}
let utxo_page = self
.client
.rooch
.query_utxos(
UTXOFilterView::owner(self.sender.clone()),
next_cursor.map(Into::into),
None,
Some(false),
)
.await?;
// Add small delay to avoid rate limiting
tokio::time::sleep(tokio::time::Duration::from_millis(200)).await;
// ... rest of the method
}
Related Code
crates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116 - load_utxos() method
crates/rooch/src/commands/bitcoin/transaction_builder.rs:92-153 - build() method that calls select_utxos()
crates/rooch/src/commands/account/commands/balance.rs:187-219 - get_total_utxo_value() for reference (similar pagination pattern)
Additional Context
Test Case
Address: bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt
- UTXO count: 864
- Total value: 33,059,413 satoshi (0.33059413 BTC)
- Pages needed: ~18 pages (50 UTXOs per page)
When load_utxos() tries to load all 18 pages in quick succession, the RPC server returns 429 errors after a few pages.
Verification
The UTXO data on Rooch matches mempool.space exactly:
- Both show 864 UTXOs
- Both show 33,059,413 satoshi total value
This confirms the issue is not with data integrity, but with the request rate.
Priority
Medium - This is a usability issue that affects users with large numbers of UTXOs. The pagination logic works correctly but needs resilience against rate limiting.
Checklist
Issue:
bitcoin build-txfails with "Too many object IDs requested" when handling large number of UTXOsSummary
rooch bitcoin build-txcommand fails when trying to build a transaction with a large number of UTXOs (e.g., 800+ UTXOs). The error message "Too many object IDs requested" is misleading - the actual root cause is thatUTXOSelector::load_utxos()triggers 429 Too Many Requests errors when fetching UTXO pages in a loop, and there's no retry mechanism to handle rate limiting.Version
Rooch version: latest (main branch)
Commit: unknown (working on main branch)
Steps to Reproduce
bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt)rooch bitcoin build-txwithout specifying-iparameters (expecting automatic UTXO selection)Expected Behavior
The command should:
Actual Behavior
The command fails with "Too many object IDs requested" error.
Investigation
UTXOSelector::load_utxos()method incrates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116implements pagination correctlyload_utxos()works correctly for small numbers of pages but fails under rate limitingCode Analysis
From
utxo_selector.rs:80-116:Problem: The
await?propagates any 429 error immediately without retry.Comparison with Working Code
The
balancecommand incrates/rooch/src/commands/account/commands/balance.rs:187-219successfully queries all UTXOs using a loop:The balance command works because it likely doesn't hit the rate limit as quickly (or MAX_RESULT_LIMIT reduces the number of requests).
Root Cause
load_utxos()doesn't handle HTTP 429 (Too Many Requests) errorsProposed Solution
Add retry logic with exponential backoff for 429 errors in
load_utxos()method.Option 1: Retry with Exponential Backoff (Recommended)
Option 2: Add Delay Between Requests (Simpler)
Related Code
crates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116-load_utxos()methodcrates/rooch/src/commands/bitcoin/transaction_builder.rs:92-153-build()method that callsselect_utxos()crates/rooch/src/commands/account/commands/balance.rs:187-219-get_total_utxo_value()for reference (similar pagination pattern)Additional Context
Test Case
Address:
bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjtWhen
load_utxos()tries to load all 18 pages in quick succession, the RPC server returns 429 errors after a few pages.Verification
The UTXO data on Rooch matches mempool.space exactly:
This confirms the issue is not with data integrity, but with the request rate.
Priority
Medium - This is a usability issue that affects users with large numbers of UTXOs. The pagination logic works correctly but needs resilience against rate limiting.
Checklist