Skip to content

Support for abort/cancel in rust w/ idiomatic support for python/go/typescript#2331

Closed
dexhorthy wants to merge 12 commits intoBoundaryML:canaryfrom
dexhorthy:abort-handlers
Closed

Support for abort/cancel in rust w/ idiomatic support for python/go/typescript#2331
dexhorthy wants to merge 12 commits intoBoundaryML:canaryfrom
dexhorthy:abort-handlers

Conversation

@dexhorthy
Copy link
Copy Markdown
Contributor

@dexhorthy dexhorthy commented Aug 17, 2025

Built in close collaboration with @hellovai who sat in my apartment and shouted at claude with me all day

Note: plan/Research artifacts in thoughts/ can be removed once they get put in a good location, leaving them in for now

What problem(s) was I solving?

Previously, once a BAML function call was initiated, there was no way to cancel it. This meant:

  • Long-running LLM operations couldn't be stopped if no longer needed
  • Retry chains would continue even after the user navigated away
  • Resources were wasted on unnecessary LLM calls
  • Poor user experience when operations needed to be cancelled

What user-facing changes did I ship?

Added abort/cancellation support for all BAML operations using native language patterns:

TypeScript

const controller = new AbortController();
const promise = b.ExtractName("John Doe", { 
  abortController: controller 
});

// Cancel after 100ms
setTimeout(() => controller.abort(), 100);

try {
  await promise;
} catch (e) {
  if (e instanceof BamlAbortError) {
    console.log("Operation cancelled");
  }
}

Python

from baml_py import AbortController

controller = AbortController()
task = b.ExtractName("John Doe", 
  baml_options={"abort_controller": controller})

# Cancel after 100ms
await asyncio.sleep(0.1)
controller.abort()

try:
  await task
except Exception as e:
  if "abort" in str(e).lower():
    print("Operation cancelled")

Go

ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()

_, err := b.ExtractName(ctx, "John Doe")
if err != nil && strings.Contains(err.Error(), "context canceled") {
  fmt.Println("Operation cancelled")
}

How I implemented it

  • Added cancellation support at the orchestrator level in Rust runtime using stream-cancel crate's Tripwire mechanism
  • Created language-specific bindings:
    • TypeScript: Native AbortController support via NAPI bridge
    • Python: Custom AbortController class exposed via PyO3
    • Go: context.Context cancellation with early detection goroutines
  • Implemented early cancellation detection that monitors signals immediately when functions are called
  • Added new LLMResponse::Cancelled variant to handle cancelled operations
  • All changes are backward compatible - existing code works unchanged

How to verify it

Run the tests:

# Python tests
cd integ-tests/python && uv run pytest tests/test_abort_handlers.py -v

# TypeScript tests  
cd integ-tests/typescript && npm test -- tests/abort-handlers.test.ts

# Go tests (requires CFFI rebuild)
cd engine/language_client_cffi && cargo build
cd integ-tests/go && go test -v -run TestAbortHandler

Test status:

  • Python abort handler tests (7 tests) - All passing
  • TypeScript abort handler tests (11 tests) - All passing
  • Go abort handler tests (7 tests) - All passing
  • I have ensured make check test passes

Also re-enabled 6 previously disabled test files (PDF, video, OpenAI responses, expression functions) that are now compatible.

Description for the changelog

Added comprehensive abort handler support for TypeScript, Python, and Go, allowing cancellation of in-flight LLM operations through native language cancellation patterns.

dexhorthy and others added 4 commits August 16, 2025 15:53
- Added Cancelled variant to LLMResponse enum
- Integrated stream-cancel Tripwire for cancellation checks
- Updated orchestrators (sync and streaming) to accept optional Tripwire
- Handled Cancelled variant in all match statements across 8 files
- Prepared foundation for language-specific abort handler bridges

Currently passing None for cancellation tokens - ready for language bridges in Phase 2-4.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implemented context cancellation propagation from Go to Rust:

CFFI Layer:
- Added cancel_function_call extern function for Go to call
- Integrated stream-cancel crate with Tripwire for cancellation
- Used tokio::select! to handle cancellation at CFFI layer
- Track active operations with DashMap for cancellation

Go Client:
- Added CancelFunctionCall export function
- Updated callbacks to call cancel on context.Done()
- Fixed streaming template to pass user context directly
- Added wrapper functions for dynamic library loading

Testing:
- Created comprehensive manual test suite
- All cancellation tests passing:
  ✓ Context cancellation (~100ms)
  ✓ Streaming cancellation (~50ms)
  ✓ Timeout cancellation (~200ms)
  ✓ Minimal goroutine leaks

The implementation allows Go applications to cancel in-flight BAML operations
using standard context.Context patterns, properly cleaning up resources.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
…tion

Implemented early cancellation detection pattern for Go language support:
- Modified runtime.go to monitor context.Done() immediately on function call
- Removed redundant late cancellation from callbacks.go
- Updated test suite to use correct BAML client functions
- All tests passing with responsive cancellation (~100ms response time)

Key improvement: Cancellation now happens immediately when Go context is cancelled,
not waiting for data callbacks from Rust. This provides much more responsive abort handling.

Updated implementation plan with Phase 2 completion and clear instructions for
Phase 3 (TypeScript) implementation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Implement AbortController for Python and TypeScript clients
- Add cancellation support via tripwires in Rust runtime
- Update all generated client code with abort handler APIs
- Add comprehensive tests for abort functionality
- Clean up disabled test files
@vercel
Copy link
Copy Markdown

vercel Bot commented Aug 17, 2025

@dexhorthy is attempting to deploy a commit to the Boundary Team on Vercel.

A member of the Team first needs to authorize it.

@dexhorthy dexhorthy changed the title Abort handlers Support for abort/cancel in rust w/ idiomatic support for python/go/typescript Aug 17, 2025
@sxlijin
Copy link
Copy Markdown
Contributor

sxlijin commented Aug 21, 2025

thanks for PR! working on taking this over the finish line! #2357

@sxlijin sxlijin closed this Aug 21, 2025
github-merge-queue Bot pushed a commit that referenced this pull request Aug 26, 2025
…ons) (#2357)

Implement AbortController in python, typescript, go, and also wasm. Also
implement cancel functionality for run test in wasm and add docs for
py/ts/go.

---

First 80% of this work done by @dexhorthy and @hellovai in #2331 :

Built in close collaboration with @hellovai who sat in my apartment and
shouted at claude with me all day

*Note: plan/Research artifacts in thoughts/ can be removed once they get
put in a good location, leaving them in for now*

## What problem(s) was I solving?

Previously, once a BAML function call was initiated, there was no way to
cancel it. This meant:
- Long-running LLM operations couldn't be stopped if no longer needed
- Retry chains would continue even after the user navigated away
- Resources were wasted on unnecessary LLM calls
- Poor user experience when operations needed to be cancelled

## What user-facing changes did I ship?

Added abort/cancellation support for all BAML operations using native
language patterns:

### TypeScript
```typescript
const controller = new AbortController();
const promise = b.ExtractName("John Doe", { 
  abortController: controller 
});

// Cancel after 100ms
setTimeout(() => controller.abort(), 100);

try {
  await promise;
} catch (e) {
  if (e instanceof BamlAbortError) {
    console.log("Operation cancelled");
  }
}
```

### Python
```python
from baml_py import AbortController

controller = AbortController()
task = b.ExtractName("John Doe", 
  baml_options={"abort_controller": controller})

# Cancel after 100ms
await asyncio.sleep(0.1)
controller.abort()

try:
  await task
except Exception as e:
  if "abort" in str(e).lower():
    print("Operation cancelled")
```

### Go
```go
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()

_, err := b.ExtractName(ctx, "John Doe")
if err != nil && strings.Contains(err.Error(), "context canceled") {
  fmt.Println("Operation cancelled")
}
```

## How I implemented it

- Added cancellation support at the orchestrator level in Rust runtime
using `stream-cancel` crate's `Tripwire` mechanism
- Created language-specific bindings:
  - **TypeScript**: Native `AbortController` support via NAPI bridge
  - **Python**: Custom `AbortController` class exposed via PyO3
- **Go**: `context.Context` cancellation with early detection goroutines
- Implemented early cancellation detection that monitors signals
immediately when functions are called
- Added new `LLMResponse::Cancelled` variant to handle cancelled
operations
- All changes are backward compatible - existing code works unchanged

## How to verify it

### Run the tests:
```bash
# Python tests
cd integ-tests/python && uv run pytest tests/test_abort_handlers.py -v

# TypeScript tests  
cd integ-tests/typescript && npm test -- tests/abort-handlers.test.ts

# Go tests (requires CFFI rebuild)
cd engine/language_client_cffi && cargo build
cd integ-tests/go && go test -v -run TestAbortHandler
```

### Test status:
- [x] Python abort handler tests (7 tests) - All passing
- [x] TypeScript abort handler tests (11 tests) - All passing
- [x] Go abort handler tests (7 tests) - All passing
- [x] I have ensured `make check test` passes

Also re-enabled 6 previously disabled test files (PDF, video, OpenAI
responses, expression functions) that are now compatible.

## Description for the changelog

Added comprehensive abort handler support for TypeScript, Python, and
Go, allowing cancellation of in-flight LLM operations through native
language cancellation patterns.

---------

Co-authored-by: dexhorthy <dexter@humanlayer.dev>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: hellovai <vbv@boundaryml.com>
cg-jl pushed a commit that referenced this pull request Aug 28, 2025
…ons) (#2357)

Implement AbortController in python, typescript, go, and also wasm. Also
implement cancel functionality for run test in wasm and add docs for
py/ts/go.

---

First 80% of this work done by @dexhorthy and @hellovai in #2331 :

Built in close collaboration with @hellovai who sat in my apartment and
shouted at claude with me all day

*Note: plan/Research artifacts in thoughts/ can be removed once they get
put in a good location, leaving them in for now*

## What problem(s) was I solving?

Previously, once a BAML function call was initiated, there was no way to
cancel it. This meant:
- Long-running LLM operations couldn't be stopped if no longer needed
- Retry chains would continue even after the user navigated away
- Resources were wasted on unnecessary LLM calls
- Poor user experience when operations needed to be cancelled

## What user-facing changes did I ship?

Added abort/cancellation support for all BAML operations using native
language patterns:

### TypeScript
```typescript
const controller = new AbortController();
const promise = b.ExtractName("John Doe", { 
  abortController: controller 
});

// Cancel after 100ms
setTimeout(() => controller.abort(), 100);

try {
  await promise;
} catch (e) {
  if (e instanceof BamlAbortError) {
    console.log("Operation cancelled");
  }
}
```

### Python
```python
from baml_py import AbortController

controller = AbortController()
task = b.ExtractName("John Doe", 
  baml_options={"abort_controller": controller})

# Cancel after 100ms
await asyncio.sleep(0.1)
controller.abort()

try:
  await task
except Exception as e:
  if "abort" in str(e).lower():
    print("Operation cancelled")
```

### Go
```go
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()

_, err := b.ExtractName(ctx, "John Doe")
if err != nil && strings.Contains(err.Error(), "context canceled") {
  fmt.Println("Operation cancelled")
}
```

## How I implemented it

- Added cancellation support at the orchestrator level in Rust runtime
using `stream-cancel` crate's `Tripwire` mechanism
- Created language-specific bindings:
  - **TypeScript**: Native `AbortController` support via NAPI bridge
  - **Python**: Custom `AbortController` class exposed via PyO3
- **Go**: `context.Context` cancellation with early detection goroutines
- Implemented early cancellation detection that monitors signals
immediately when functions are called
- Added new `LLMResponse::Cancelled` variant to handle cancelled
operations
- All changes are backward compatible - existing code works unchanged

## How to verify it

### Run the tests:
```bash
# Python tests
cd integ-tests/python && uv run pytest tests/test_abort_handlers.py -v

# TypeScript tests  
cd integ-tests/typescript && npm test -- tests/abort-handlers.test.ts

# Go tests (requires CFFI rebuild)
cd engine/language_client_cffi && cargo build
cd integ-tests/go && go test -v -run TestAbortHandler
```

### Test status:
- [x] Python abort handler tests (7 tests) - All passing
- [x] TypeScript abort handler tests (11 tests) - All passing
- [x] Go abort handler tests (7 tests) - All passing
- [x] I have ensured `make check test` passes

Also re-enabled 6 previously disabled test files (PDF, video, OpenAI
responses, expression functions) that are now compatible.

## Description for the changelog

Added comprehensive abort handler support for TypeScript, Python, and
Go, allowing cancellation of in-flight LLM operations through native
language cancellation patterns.

---------

Co-authored-by: dexhorthy <dexter@humanlayer.dev>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: hellovai <vbv@boundaryml.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants