fix: add retry with exponential backoff for UTXO loading rate limits#3935
Conversation
- Add retry logic in UTXOSelector::load_utxos() to handle HTTP 429 errors - Use exponential backoff (500ms base, 30s max) to avoid overwhelming server - Detect rate limit errors by checking for 'too many requests', '429', 'serverisbusy', or 'wait for' - Log retry attempts for debugging - Fixes #3934 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
Pull request overview
This PR adds retry logic with exponential backoff to handle HTTP 429 (Too Many Requests) rate limiting errors when loading UTXOs during Bitcoin transaction building. The fix addresses issue #3934 where building transactions with 800+ UTXOs would fail due to rapid pagination requests triggering RPC rate limits without any retry mechanism.
Changes:
- Added retry logic with exponential backoff (500ms base delay, 30s max, 5 retries) to
UTXOSelector::load_utxos() - Added
is_rate_limit_error()helper function to detect various rate limit error patterns - Added debug logging for retry attempts
|
|
||
| use std::collections::VecDeque; | ||
|
|
||
| use anyhow::{bail, Result}; |
There was a problem hiding this comment.
The Context trait from anyhow needs to be imported to use the .context() method on line 144. Add Context to the import: use anyhow::{bail, Context, Result};
| use anyhow::{bail, Result}; | |
| use anyhow::{bail, Context, Result}; |
| // Build transaction with auto-loading (empty inputs = auto-load) | ||
| return build_single_transaction( | ||
| &context, | ||
| client, | ||
| sender, | ||
| bitcoin_network, | ||
| vec![], // Empty inputs triggers auto-loading | ||
| self.skip_check_seal, | ||
| self.fee_rate, | ||
| self.lock_time, | ||
| self.change_address, | ||
| specific_outputs, | ||
| self.output_file, | ||
| ) | ||
| .await; |
There was a problem hiding this comment.
When using :all with send_all mode, the code loads all UTXOs, estimates fees, and then passes empty inputs to build_single_transaction (line 265). This causes the TransactionBuilder to reload all UTXOs again since empty inputs trigger auto-loading. This is inefficient and results in redundant UTXO queries. Consider using TransactionBuilder::with_utxos() instead with the already-loaded all_utxos to avoid the redundant query.
| // Convert :all to specific amount | ||
| let specific_outputs = self | ||
| .outputs | ||
| .into_iter() | ||
| .map(|mut output| { | ||
| if output.amount == OutputAmount::All { | ||
| output.amount = OutputAmount::Specific(output_amount); | ||
| } | ||
| output | ||
| }) | ||
| .collect::<Vec<_>>(); |
There was a problem hiding this comment.
When send_all is true and multiple outputs are specified (e.g., -o addr1:all -o addr2:100), all outputs with :all will be assigned the same output_amount value. This could lead to unexpected behavior since the total would be split incorrectly. Consider validating that only one output can use :all, or properly distribute the total amount across all :all outputs.
| format!( | ||
| "_{}_{}{}", | ||
| &base_path[..ext_pos], | ||
| chunk_idx + 1, | ||
| &base_path[ext_pos..] | ||
| ) |
There was a problem hiding this comment.
The filename generation logic for split transactions is incorrect. When base_path contains a dot (e.g., "tx.psbt"), the format string produces "tx_1.psbt" instead of the intended "tx_1.psbt". The underscore should be after the base name, not before it. Change format string from "{}{}{}" to "{}{}{}" to fix this.
| let utxos_objs = self | ||
| .client | ||
| .rooch | ||
| .query_utxos(UTXOFilterView::object_ids(chunk.to_vec()), None, None, None) | ||
| .await?; |
There was a problem hiding this comment.
The load_specific_utxos function does not implement retry logic for rate limiting errors, unlike load_utxos. When loading a large number of specific UTXOs (chunked into batches of 100), this could hit rate limits. Consider wrapping the query_utxos call with the same retry logic used in load_utxos to ensure consistency across the codebase.
Summary
Fixes #3934
This PR adds retry logic with exponential backoff to handle HTTP 429 (Too Many Requests) errors when loading UTXOs in the
bitcoin build-txcommand.Changes
UTXOSelector::load_utxos()to retry on rate limit errorsis_rate_limit_error()helper to detect rate limiting errorsProblem
Previously, when trying to build a transaction with a large number of UTXOs (e.g., 800+), the command would fail with a misleading error message. The root cause was that rapid pagination requests triggered the RPC server's rate limiting (HTTP 429), and there was no retry mechanism.
Testing
make buildmake lint-rustcargo test --package rooch bitcoinVerification
The fix follows existing retry patterns in the codebase:
crates/bitcoin-client/src/actor/client.rs)crates/rooch-da/src/backend/openda/avail.rs)🤖 Generated with Claude Code