feat(experimental): add bidi stream retry manager. #1632

Pulkit0110 · 2025-11-25T09:26:27Z

Add bidi stream retry manager. It will the main class that would be responsible for retries.
Integrate MRD with bidi stream retry manager and reads resumption strategy
Added conformance tests for all error scenarios and smart resumption for bidi reads
Refactored MRD's download_ranges() method, which will now use retry manager's execute() method to send the requests
Modified the logic to parse and handle the BidiReadObjectRedirectedError.
Modified cloud build configuration to also pull the latest changes of the current PR.
Formatted few more files using black formatter

gemini-code-assist · 2025-11-25T09:26:46Z

Summary of Changes

Hello @Pulkit0110, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a foundational component for enhancing the reliability of asynchronous bidirectional streaming operations within the google-cloud-storage library. By providing a dedicated retry manager, it enables robust handling of transient failures, ensuring that streaming operations can automatically recover and continue without manual intervention. This change is crucial for improving the overall stability and user experience of experimental asynchronous features.

Highlights

New Bidirectional Stream Retry Manager: Introduced _BidiStreamRetryManager, a new class responsible for managing retry logic for asynchronous bidirectional streaming operations. This manager integrates with a _BaseResumptionStrategy to handle state management during retries and uses google.api_core.retry.AsyncRetry for defining retry policies.
Robust Retry Mechanism: The _BidiStreamRetryManager's execute method orchestrates the retry loop, allowing for automatic re-attempting of streaming operations upon retriable exceptions. It supports resuming operations from a previous state and updating state based on responses, ensuring resilience against transient network issues or service unavailability.
Comprehensive Unit Tests: Added a new test file test_bidi_stream_retry_manager.py with extensive unit tests for the _BidiStreamRetryManager. These tests cover various scenarios, including successful execution, retries on initial and mid-stream failures, handling of non-retriable errors, and proper behavior when retry deadlines are exceeded.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a _BidiStreamRetryManager for handling retries in bidi streaming operations, along with comprehensive unit tests. The overall implementation is solid and well-tested. I have a few suggestions to improve type safety and maintainability in bidi_stream_retry_manager.py, such as adding a missing type hint, making another one more specific, and pointing out the use of an internal attribute from google-api-core which could be a future maintenance risk. Please see my detailed comments.

google/cloud/storage/_experimental/asyncio/retry/bidi_stream_retry_manager.py

chandra-siri · 2025-12-24T08:59:18Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a robust retry manager for bidi streams, which is a significant enhancement for experimental async downloads. The implementation is well-structured, using a strategy pattern to separate the generic retry loop from the read-specific resumption logic. The addition of conformance tests to validate the retry behavior under various fault-injection scenarios is excellent. My review includes a few suggestions to improve maintainability and logging practices, such as replacing print statements with the logging module and addressing code duplication in error handling.

google/cloud/storage/_experimental/asyncio/async_multi_range_downloader.py

google/cloud/storage/_experimental/asyncio/retry/reads_resumption_strategy.py

chandra-siri

General comments
- convert print into log statements.
- reduce code duplicates
- add comments where ever there's non-trivial logic
In the internal bug attach screenshots of perf results with and without this change.

poc_bidi_retry_final.py

google/cloud/storage/_experimental/asyncio/retry/bidi_stream_retry_manager.py

google/cloud/storage/_media/requests/download.py

google/cloud/storage/_experimental/asyncio/async_multi_range_downloader.py

google/cloud/storage/_experimental/asyncio/retry/reads_resumption_strategy.py

chandra-siri · 2025-12-24T13:16:24Z

Since it's a large change, can you make the description a little detailed. - https://www.conventionalcommits.org/en/v1.0.0/#examples

google/cloud/storage/_experimental/asyncio/async_read_object_stream.py

google/cloud/storage/_experimental/asyncio/retry/bidi_stream_retry_manager.py

PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v0.7.0 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91 <details><summary>google-cloud-storage: 3.8.0</summary> ## [3.8.0](v3.7.0...v3.8.0) (2026-01-13) ### Features * expose persisted size in mrd (#1671) ([0e2961b](0e2961be)) * implement "append_from_file" (#1686) ([1333c95](1333c956)) * compute chunk wise checksum for bidi_writes (#1675) ([139390c](139390cb)) * flush the last chunk in append method (#1699) ([89bfe7a](89bfe7a5)) * add write resumption strategy (#1663) ([a57ea0e](a57ea0ec)) * add bidi stream retry manager. (#1632) ([d90f0ee](d90f0ee0)) * make flush size configurable (#1677) ([f7095fa](f7095faf)) ### Bug Fixes * no state lookup while opening bidi-write stream ([2d5a7b1](2d5a7b16)) * no state lookup while opening bidi-write stream (#1636) ([2d5a7b1](2d5a7b16)) * close write object stream always (#1661) ([4a609a4](4a609a4b)) * add system test for opening with read_handle (#1672) ([6dc711d](6dc711da)) </details> --------- Co-authored-by: Victor Chudnovsky <vchudnov@google.com>

Pulkit0110 requested review from a team as code owners November 25, 2025 09:26

blunderbuss-gcf bot assigned chandra-siri Nov 25, 2025

product-auto-label bot added size: l Pull request size is large. api: storage Issues related to the googleapis/python-storage API. labels Nov 25, 2025

gemini-code-assist bot reviewed Nov 25, 2025

View reviewed changes

Pulkit0110 requested a review from chandra-siri November 25, 2025 09:36

Pulkit0110 force-pushed the bidi-reads-retry4 branch 2 times, most recently from a3795a3 to 4f1aa96 Compare December 1, 2025 04:40

Pulkit0110 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 1, 2025

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 1, 2025

chandra-siri added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 3, 2025

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 3, 2025

product-auto-label bot added size: m Pull request size is medium. size: l Pull request size is large. and removed size: l Pull request size is large. size: m Pull request size is medium. labels Dec 4, 2025

Pulkit0110 force-pushed the bidi-reads-retry4 branch from 8295ed3 to 5027ed4 Compare December 9, 2025 06:10

Pulkit0110 force-pushed the bidi-reads-retry4 branch from 5027ed4 to 96ac8ce Compare December 16, 2025 10:30

Pulkit0110 requested a review from a team as a code owner December 16, 2025 10:30

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Dec 16, 2025

Pulkit0110 force-pushed the bidi-reads-retry4 branch 2 times, most recently from c8da9ac to bea02a5 Compare December 23, 2025 05:43

Pulkit0110 added 5 commits December 24, 2025 05:53

feat(experimental): Add read resumption strategy

4b3d646

add unit tests

1f51e75

resolving comments

53b3350

feat(experimental): Add bidi stream retry manager

2772634

minor change

798d38b

Pulkit0110 added 12 commits December 24, 2025 05:53

integrate retry logic with the MRD

85fc4a7

test scripts

c5333a8

add error handling for open method

df35f02

minor changes

65b3dfd

remove unused imports

2b4c347

update cloudbuild to pull the PR changes

682b92f

rename bidi_reads_test as per the other test files

5f59efa

remove unused import

5a4c168

add conftest modules in noxfile

aad4c2f

minor changes

89017c0

add smarter resumption scenario in integration tests

673c7a1

minor fix

b82987f

gemini-code-assist bot reviewed Dec 24, 2025

View reviewed changes

resolving comments

efe9c1c

Pulkit0110 force-pushed the bidi-reads-retry4 branch from 32d7d6a to efe9c1c Compare December 24, 2025 10:32

chandra-siri requested changes Dec 24, 2025

View reviewed changes

resolving comments

f926a9b

Pulkit0110 requested a review from chandra-siri December 25, 2025 15:56

chandra-siri requested changes Dec 26, 2025

View reviewed changes

google/cloud/storage/_experimental/asyncio/async_read_object_stream.py Show resolved Hide resolved

google/cloud/storage/_experimental/asyncio/retry/bidi_stream_retry_manager.py Outdated Show resolved Hide resolved

add more unit tests

080e593

Pulkit0110 requested a review from chandra-siri December 26, 2025 09:29

chandra-siri approved these changes Dec 26, 2025

View reviewed changes

Pulkit0110 merged commit d90f0ee into googleapis:main Dec 26, 2025
17 checks passed

release-please bot mentioned this pull request Dec 23, 2025

chore(main): release 3.8.0 #1669

Closed

vchudnov-g mentioned this pull request Jan 12, 2026

chore: librarian release pull request: 20260112T135255Z #1701

Closed

parthea mentioned this pull request Jan 13, 2026

chore: librarian release pull request: 20260113T144551Z #1704

Merged

release-please bot mentioned this pull request Jan 13, 2026

chore(main): release 3.8.0 #1706

Closed

feat(experimental): add bidi stream retry manager. #1632

feat(experimental): add bidi stream retry manager. #1632

Uh oh!

Conversation

Pulkit0110 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Nov 25, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chandra-siri commented Dec 24, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chandra-siri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chandra-siri commented Dec 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pulkit0110 commented Nov 25, 2025 •

edited

Loading