Skip to content

bitcoin build-tx fails with "Too many object IDs requested" when handling large number of UTXOs #3934

@jolestar

Description

@jolestar

Issue: bitcoin build-tx fails with "Too many object IDs requested" when handling large number of UTXOs

Summary

rooch bitcoin build-tx command fails when trying to build a transaction with a large number of UTXOs (e.g., 800+ UTXOs). The error message "Too many object IDs requested" is misleading - the actual root cause is that UTXOSelector::load_utxos() triggers 429 Too Many Requests errors when fetching UTXO pages in a loop, and there's no retry mechanism to handle rate limiting.

Version

Rooch version: latest (main branch)
Commit: unknown (working on main branch)

Steps to Reproduce

  1. Have a Bitcoin address with 800+ UTXOs (e.g., bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt)
  2. Run rooch bitcoin build-tx without specifying -i parameters (expecting automatic UTXO selection)
    rooch bitcoin build-tx \
      -s bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt \
      -o <receiver>:<amount> \
      --fee-rate 1 \
      --output-file tx.psbt
  3. Command fails with error: "Too many object IDs requested"

Expected Behavior

The command should:

  1. Automatically load all UTXOs for the address across multiple pages
  2. Handle rate limiting (429 errors) gracefully by retrying with exponential backoff
  3. Successfully build a transaction with all available UTXOs

Actual Behavior

The command fails with "Too many object IDs requested" error.

Investigation

  1. The UTXOSelector::load_utxos() method in crates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116 implements pagination correctly
  2. However, when loading many pages rapidly (e.g., 18 pages for 864 UTXOs), it triggers RPC rate limiting (HTTP 429)
  3. There is no retry mechanism to handle 429 errors
  4. The pagination loop in load_utxos() works correctly for small numbers of pages but fails under rate limiting

Code Analysis

From utxo_selector.rs:80-116:

async fn load_utxos(&mut self) -> Result<()> {
    let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
    if !has_next_page {
        return Ok(());
    }
    let utxo_page = self
        .client
        .rooch
        .query_utxos(
            UTXOFilterView::owner(self.sender.clone()),
            next_cursor.map(Into::into),
            None,  // No limit specified
            Some(false),
        )
        .await?;  // ❌ No retry on 429 errors

    // ... process UTXOs ...
}

Problem: The await? propagates any 429 error immediately without retry.

Comparison with Working Code

The balance command in crates/rooch/src/commands/account/commands/balance.rs:187-219 successfully queries all UTXOs using a loop:

async fn get_total_utxo_value(...) {
    let mut total_value: u64 = 0;
    let mut cursor: Option<IndexerStateIDView> = None;

    loop {  // ✅ Uses loop for pagination
        let page = client
            .rooch
            .query_utxos(
                UTXOFilterView::Owner(address.clone()),
                cursor.map(Into::into),
                Some(MAX_RESULT_LIMIT),
                None,
            )
            .await?;  // But also lacks retry on 429

        // ... accumulate value ...

        if !page.has_next_page {
            break;
        }
        cursor = page.next_cursor;
    }
}

The balance command works because it likely doesn't hit the rate limit as quickly (or MAX_RESULT_LIMIT reduces the number of requests).

Root Cause

  1. Missing retry logic: load_utxos() doesn't handle HTTP 429 (Too Many Requests) errors
  2. Rate limiting: RPC server returns 429 when too many requests are made in quick succession
  3. No backoff: No delay between requests to avoid triggering rate limits

Proposed Solution

Add retry logic with exponential backoff for 429 errors in load_utxos() method.

Option 1: Retry with Exponential Backoff (Recommended)

async fn load_utxos(&mut self) -> Result<()> {
    let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
    if !has_next_page {
        return Ok(());
    }

    let mut retry_count = 0;
    const MAX_RETRIES: u32 = 5;

    loop {
        let utxo_page = self
            .client
            .rooch
            .query_utxos(
                UTXOFilterView::owner(self.sender.clone()),
                next_cursor.map(Into::into),
                None,
                Some(false),
            )
            .await;

        match utxo_page {
            Ok(page) => {
                debug!("loaded utxos: {:?}", page.data.len());
                let minimal_non_dust = self.sender.script_pubkey().minimal_non_dust();
                for utxo_view in page.data {
                    let utxo = &utxo_view.value;
                    if !self.skip_seal_check && skip_utxo(&utxo_view, minimal_non_dust) {
                        continue;
                    }
                    if utxo_view.metadata.owner_bitcoin_address.is_none() {
                        debug!(
                            "Can not recognize the owner of UTXO {}, metadata: {:?}, skip.",
                            utxo.outpoint(),
                            utxo_view.metadata
                        );
                        continue;
                    }
                    self.candidate_utxos.push_front(utxo_view.into());
                }
                self.loaded_page = Some((page.next_cursor, page.has_next_page));
                return Ok(());
            }
            Err(e) => {
                // Check if it's a 429 error
                let error_msg = e.to_string().to_lowercase();
                if error_msg.contains("429") || error_msg.contains("too many requests") || error_msg.contains("rate limit") {
                    retry_count += 1;
                    if retry_count > MAX_RETRIES {
                        bail!("Too many 429 errors when loading UTXOs, gave up after {} retries", MAX_RETRIES);
                    }

                    // Exponential backoff: wait 2^retry_count seconds
                    let wait_secs = 2u64.pow(retry_count);
                    debug!("Got 429 error, retry {} after waiting {} seconds", retry_count, wait_secs);
                    tokio::time::sleep(tokio::time::Duration::from_secs(wait_secs)).await;
                    continue;
                } else {
                    // Other errors - propagate
                    return Err(e);
                }
            }
        }
    }
}

Option 2: Add Delay Between Requests (Simpler)

async fn load_utxos(&mut self) -> Result<()> {
    let (next_cursor, has_next_page) = self.loaded_page.unwrap_or((None, true));
    if !has_next_page {
        return Ok(());
    }

    let utxo_page = self
        .client
        .rooch
        .query_utxos(
            UTXOFilterView::owner(self.sender.clone()),
            next_cursor.map(Into::into),
            None,
            Some(false),
        )
        .await?;

    // Add small delay to avoid rate limiting
    tokio::time::sleep(tokio::time::Duration::from_millis(200)).await;

    // ... rest of the method
}

Related Code

  • crates/rooch/src/commands/bitcoin/utxo_selector.rs:80-116 - load_utxos() method
  • crates/rooch/src/commands/bitcoin/transaction_builder.rs:92-153 - build() method that calls select_utxos()
  • crates/rooch/src/commands/account/commands/balance.rs:187-219 - get_total_utxo_value() for reference (similar pagination pattern)

Additional Context

Test Case

Address: bc1prcajaj9n7e29u4dfp33x3hcf52yqeegspdpcd79pqu4fpr6llx4sugkfjt

  • UTXO count: 864
  • Total value: 33,059,413 satoshi (0.33059413 BTC)
  • Pages needed: ~18 pages (50 UTXOs per page)

When load_utxos() tries to load all 18 pages in quick succession, the RPC server returns 429 errors after a few pages.

Verification

The UTXO data on Rooch matches mempool.space exactly:

  • Both show 864 UTXOs
  • Both show 33,059,413 satoshi total value

This confirms the issue is not with data integrity, but with the request rate.

Priority

Medium - This is a usability issue that affects users with large numbers of UTXOs. The pagination logic works correctly but needs resilience against rate limiting.

Checklist

  • I've searched for existing issues
  • I've included steps to reproduce
  • I've identified the root cause
  • I've proposed a solution

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions