Support for abort/cancel in rust w/ idiomatic support for python/go/typescript#2331
Closed
dexhorthy wants to merge 12 commits intoBoundaryML:canaryfrom
Closed
Support for abort/cancel in rust w/ idiomatic support for python/go/typescript#2331dexhorthy wants to merge 12 commits intoBoundaryML:canaryfrom
dexhorthy wants to merge 12 commits intoBoundaryML:canaryfrom
Conversation
- Added Cancelled variant to LLMResponse enum - Integrated stream-cancel Tripwire for cancellation checks - Updated orchestrators (sync and streaming) to accept optional Tripwire - Handled Cancelled variant in all match statements across 8 files - Prepared foundation for language-specific abort handler bridges Currently passing None for cancellation tokens - ready for language bridges in Phase 2-4. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Implemented context cancellation propagation from Go to Rust: CFFI Layer: - Added cancel_function_call extern function for Go to call - Integrated stream-cancel crate with Tripwire for cancellation - Used tokio::select! to handle cancellation at CFFI layer - Track active operations with DashMap for cancellation Go Client: - Added CancelFunctionCall export function - Updated callbacks to call cancel on context.Done() - Fixed streaming template to pass user context directly - Added wrapper functions for dynamic library loading Testing: - Created comprehensive manual test suite - All cancellation tests passing: ✓ Context cancellation (~100ms) ✓ Streaming cancellation (~50ms) ✓ Timeout cancellation (~200ms) ✓ Minimal goroutine leaks The implementation allows Go applications to cancel in-flight BAML operations using standard context.Context patterns, properly cleaning up resources. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…tion Implemented early cancellation detection pattern for Go language support: - Modified runtime.go to monitor context.Done() immediately on function call - Removed redundant late cancellation from callbacks.go - Updated test suite to use correct BAML client functions - All tests passing with responsive cancellation (~100ms response time) Key improvement: Cancellation now happens immediately when Go context is cancelled, not waiting for data callbacks from Rust. This provides much more responsive abort handling. Updated implementation plan with Phase 2 completion and clear instructions for Phase 3 (TypeScript) implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Implement AbortController for Python and TypeScript clients - Add cancellation support via tripwires in Rust runtime - Update all generated client code with abort handler APIs - Add comprehensive tests for abort functionality - Clean up disabled test files
|
@dexhorthy is attempting to deploy a commit to the Boundary Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
thanks for PR! working on taking this over the finish line! #2357 |
4 tasks
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Aug 26, 2025
…ons) (#2357) Implement AbortController in python, typescript, go, and also wasm. Also implement cancel functionality for run test in wasm and add docs for py/ts/go. --- First 80% of this work done by @dexhorthy and @hellovai in #2331 : Built in close collaboration with @hellovai who sat in my apartment and shouted at claude with me all day *Note: plan/Research artifacts in thoughts/ can be removed once they get put in a good location, leaving them in for now* ## What problem(s) was I solving? Previously, once a BAML function call was initiated, there was no way to cancel it. This meant: - Long-running LLM operations couldn't be stopped if no longer needed - Retry chains would continue even after the user navigated away - Resources were wasted on unnecessary LLM calls - Poor user experience when operations needed to be cancelled ## What user-facing changes did I ship? Added abort/cancellation support for all BAML operations using native language patterns: ### TypeScript ```typescript const controller = new AbortController(); const promise = b.ExtractName("John Doe", { abortController: controller }); // Cancel after 100ms setTimeout(() => controller.abort(), 100); try { await promise; } catch (e) { if (e instanceof BamlAbortError) { console.log("Operation cancelled"); } } ``` ### Python ```python from baml_py import AbortController controller = AbortController() task = b.ExtractName("John Doe", baml_options={"abort_controller": controller}) # Cancel after 100ms await asyncio.sleep(0.1) controller.abort() try: await task except Exception as e: if "abort" in str(e).lower(): print("Operation cancelled") ``` ### Go ```go ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel() _, err := b.ExtractName(ctx, "John Doe") if err != nil && strings.Contains(err.Error(), "context canceled") { fmt.Println("Operation cancelled") } ``` ## How I implemented it - Added cancellation support at the orchestrator level in Rust runtime using `stream-cancel` crate's `Tripwire` mechanism - Created language-specific bindings: - **TypeScript**: Native `AbortController` support via NAPI bridge - **Python**: Custom `AbortController` class exposed via PyO3 - **Go**: `context.Context` cancellation with early detection goroutines - Implemented early cancellation detection that monitors signals immediately when functions are called - Added new `LLMResponse::Cancelled` variant to handle cancelled operations - All changes are backward compatible - existing code works unchanged ## How to verify it ### Run the tests: ```bash # Python tests cd integ-tests/python && uv run pytest tests/test_abort_handlers.py -v # TypeScript tests cd integ-tests/typescript && npm test -- tests/abort-handlers.test.ts # Go tests (requires CFFI rebuild) cd engine/language_client_cffi && cargo build cd integ-tests/go && go test -v -run TestAbortHandler ``` ### Test status: - [x] Python abort handler tests (7 tests) - All passing - [x] TypeScript abort handler tests (11 tests) - All passing - [x] Go abort handler tests (7 tests) - All passing - [x] I have ensured `make check test` passes Also re-enabled 6 previously disabled test files (PDF, video, OpenAI responses, expression functions) that are now compatible. ## Description for the changelog Added comprehensive abort handler support for TypeScript, Python, and Go, allowing cancellation of in-flight LLM operations through native language cancellation patterns. --------- Co-authored-by: dexhorthy <dexter@humanlayer.dev> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: hellovai <vbv@boundaryml.com>
cg-jl
pushed a commit
that referenced
this pull request
Aug 28, 2025
…ons) (#2357) Implement AbortController in python, typescript, go, and also wasm. Also implement cancel functionality for run test in wasm and add docs for py/ts/go. --- First 80% of this work done by @dexhorthy and @hellovai in #2331 : Built in close collaboration with @hellovai who sat in my apartment and shouted at claude with me all day *Note: plan/Research artifacts in thoughts/ can be removed once they get put in a good location, leaving them in for now* ## What problem(s) was I solving? Previously, once a BAML function call was initiated, there was no way to cancel it. This meant: - Long-running LLM operations couldn't be stopped if no longer needed - Retry chains would continue even after the user navigated away - Resources were wasted on unnecessary LLM calls - Poor user experience when operations needed to be cancelled ## What user-facing changes did I ship? Added abort/cancellation support for all BAML operations using native language patterns: ### TypeScript ```typescript const controller = new AbortController(); const promise = b.ExtractName("John Doe", { abortController: controller }); // Cancel after 100ms setTimeout(() => controller.abort(), 100); try { await promise; } catch (e) { if (e instanceof BamlAbortError) { console.log("Operation cancelled"); } } ``` ### Python ```python from baml_py import AbortController controller = AbortController() task = b.ExtractName("John Doe", baml_options={"abort_controller": controller}) # Cancel after 100ms await asyncio.sleep(0.1) controller.abort() try: await task except Exception as e: if "abort" in str(e).lower(): print("Operation cancelled") ``` ### Go ```go ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel() _, err := b.ExtractName(ctx, "John Doe") if err != nil && strings.Contains(err.Error(), "context canceled") { fmt.Println("Operation cancelled") } ``` ## How I implemented it - Added cancellation support at the orchestrator level in Rust runtime using `stream-cancel` crate's `Tripwire` mechanism - Created language-specific bindings: - **TypeScript**: Native `AbortController` support via NAPI bridge - **Python**: Custom `AbortController` class exposed via PyO3 - **Go**: `context.Context` cancellation with early detection goroutines - Implemented early cancellation detection that monitors signals immediately when functions are called - Added new `LLMResponse::Cancelled` variant to handle cancelled operations - All changes are backward compatible - existing code works unchanged ## How to verify it ### Run the tests: ```bash # Python tests cd integ-tests/python && uv run pytest tests/test_abort_handlers.py -v # TypeScript tests cd integ-tests/typescript && npm test -- tests/abort-handlers.test.ts # Go tests (requires CFFI rebuild) cd engine/language_client_cffi && cargo build cd integ-tests/go && go test -v -run TestAbortHandler ``` ### Test status: - [x] Python abort handler tests (7 tests) - All passing - [x] TypeScript abort handler tests (11 tests) - All passing - [x] Go abort handler tests (7 tests) - All passing - [x] I have ensured `make check test` passes Also re-enabled 6 previously disabled test files (PDF, video, OpenAI responses, expression functions) that are now compatible. ## Description for the changelog Added comprehensive abort handler support for TypeScript, Python, and Go, allowing cancellation of in-flight LLM operations through native language cancellation patterns. --------- Co-authored-by: dexhorthy <dexter@humanlayer.dev> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: hellovai <vbv@boundaryml.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built in close collaboration with @hellovai who sat in my apartment and shouted at claude with me all day
Note: plan/Research artifacts in thoughts/ can be removed once they get put in a good location, leaving them in for now
What problem(s) was I solving?
Previously, once a BAML function call was initiated, there was no way to cancel it. This meant:
What user-facing changes did I ship?
Added abort/cancellation support for all BAML operations using native language patterns:
TypeScript
Python
Go
How I implemented it
stream-cancelcrate'sTripwiremechanismAbortControllersupport via NAPI bridgeAbortControllerclass exposed via PyO3context.Contextcancellation with early detection goroutinesLLMResponse::Cancelledvariant to handle cancelled operationsHow to verify it
Run the tests:
Test status:
make check testpassesAlso re-enabled 6 previously disabled test files (PDF, video, OpenAI responses, expression functions) that are now compatible.
Description for the changelog
Added comprehensive abort handler support for TypeScript, Python, and Go, allowing cancellation of in-flight LLM operations through native language cancellation patterns.