feat(core): intelligent streaming retry with continuation and multi-step enhancements#6286
feat(core): intelligent streaming retry with continuation and multi-step enhancements#6286
Conversation
4a70901 to
d2a6f53
Compare
d2a6f53 to
cd013e5
Compare
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as duplicate.
This comment was marked as duplicate.
a1fbf93 to
192cdf6
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Automated patch version bump from production release
…r extraction - Add retry logic for XML parsing with exponential backoff (1s, 2s, 4s... capped at 8s) - Add summary generation retry with graceful fallback message - Add parameter extraction from multi-step decision template - Add formatActionsWithParams() for LLM parameter schema information - Add bounds checking for retry counts (1-10 range) - Add comprehensive tests for retry and parameter scenarios
Enhanced JSON parameter parsing to ensure only non-null objects are accepted, with appropriate logging for invalid types. Updated related test and provider logic to match stricter validation and improved code consistency.
Added explicit checks and warnings for array-type parameters in both DefaultMessageService and multi-step tests. Metadata properties in accumulated state are now prefixed with underscores to avoid collisions with action parameters.
Improves logging granularity and clarity for multi-step message processing, including step-by-step info, provider/action execution, and summary generation. Adds streaming support for summary output in multi-step mode and ensures streaming contexts are correctly managed for both single-shot and multi-step flows. Minor code formatting and consistency improvements applied.
Replaces 'state' with 'accumulatedState' in provider and action calls to ensure correct state is used. Adds logic to prevent duplicate streaming of summary output to the user on retry attempts, tracking streaming status with 'hasStreamedToUser'.
Marks streaming as started before sending content to prevent duplicate streams on retries in DefaultMessageService. Updates action parameter formatting to handle invalid definitions and provide defaults for missing type or description.
Updates submodule from 797ad22 (init commit) to 4a33c0c (Add ElizaOS scenario testing rules) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…entries - Add array check to prevent malformed parameter output - Filter invalid param entries upfront to avoid dangling headers
- Added methods to StreamingContext for managing streaming state, including reset, getStreamedText, and isComplete. - Improved DefaultMessageService to utilize intelligent retry logic for streaming, allowing for continuation prompts when text extraction is incomplete. - Updated tests to validate new streaming context features and ensure correct behavior during retries. - Enhanced IStreamExtractor interface with reset and flush methods for better state management.
- Removed redundant '[MULTI-STEP]' prefix from logging messages in DefaultMessageService to enhance clarity and readability. - Updated various log statements to maintain consistency while preserving essential information about the multi-step processing flow. - other: fix docs
Replace ACTIONS_REGEX with indexOf-based extraction in detectResponseStrategy. This eliminates potential polynomial-time regex matching on malicious inputs.
Ensures continuation response is properly streamed to user in single-shot mode, matching the behavior of multi-step continuation at line 1862.
…nt results Previously, calling getStreamedText() multiple times would return different results because flush() empties the buffer. Now the flushed content is accumulated into streamedText, ensuring consistent results across calls.
…elizaOS/eliza into feature/multi-step-enhancements # Conflicts: # packages/core/src/services/default-message-service.ts
|
Claude finished @wtfsayo's task in 1m 23s —— View job PR Review - Multi-Step Workflow EnhancementsTodo List:
SummaryThis PR adds robust retry logic and parameter extraction to multi-step workflows. The implementation is well-structured with proper validation, exponential backoff, and comprehensive test coverage. ✅ Verified - All Previous Issues Addressed
✅ Implementation QualityRetry Logic:
Parameter Extraction:
Streaming Infrastructure:
✅ Test Coverage
✅ TemplateThe 💡 Minor Non-Blocking Suggestions
Recommendation: ✅ APPROVEThe implementation is solid, all previous review concerns have been addressed, proper TypeScript types are used throughout, tests are comprehensive using |
There was a problem hiding this comment.
Pull request overview
This pull request enhances multi-step workflows with retry logic, parameter extraction, and improved error handling while also optimizing database operations and refactoring initialization logic.
Changes:
- Added configurable retry logic with exponential backoff for XML parsing failures in multi-step workflows
- Implemented parameter extraction and passing from LLM decisions to actions via
<parameters>XML key - Optimized database queries to reduce N+1 patterns and added idempotent upsert operations
- Refactored runtime initialization to parallelize database operations and improve startup performance
Reviewed changes
Copilot reviewed 56 out of 57 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/services/default-message-service.ts | Core retry logic, parameter extraction, streaming context improvements, and intelligent continuation handling |
| packages/core/src/prompts.ts | Updated multi-step template to include parameters key and actionsWithParams provider |
| packages/plugin-bootstrap/src/providers/actions.ts | Added formatActionsWithParams function and actionsWithParams provider output |
| packages/plugin-bootstrap/src/tests/multi-step.test.ts | Comprehensive tests for retry logic and parameter extraction |
| packages/core/src/utils/streaming.ts | Added streaming retry state APIs and createStreamingContext helper |
| packages/core/src/runtime.ts | Parallelized initialization and added ensureEntity helper |
| packages/plugin-sql/src/base.ts | Added idempotent upsert operations with onConflictDoNothing |
| packages/server/src/api/index.ts | Added toRouteRequest converter for plugin route handlers |
| packages/server/src/services/message.ts | Removed duplicate error handling for ensureWorldExists/ensureRoomExists |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Normalize action: empty string or whitespace-only should be treated as undefined | ||
| const rawAction = | ||
| typeof parsedStep.action === 'string' ? parsedStep.action.trim() : undefined; | ||
| const action = rawAction && rawAction.length > 0 ? rawAction : undefined; |
There was a problem hiding this comment.
The action normalization logic converts empty/whitespace strings to undefined, but the prompt template instructs the LLM to use empty string "" when no action is needed. This creates a disconnect - the LLM is told to output "" but the code treats it as undefined. Consider either updating the prompt to say 'omit the action tag' or accepting "" as a valid 'no action' signal.
| return []; | ||
| } | ||
| }, | ||
| // Enable streaming for action execution in multi-step mode |
There was a problem hiding this comment.
The comment states 'Enable streaming for action execution in multi-step mode', but the code passes the streaming callback unconditionally (not checking if we're in multi-step mode). The comment should either clarify this is always enabled regardless of mode, or the code should conditionally pass it only in multi-step mode.
| // Enable streaming for action execution in multi-step mode | |
| // Pass through optional streaming callback for action execution (used by multi-step mode) |
| ).filter( | ||
| ([, paramDef]) => | ||
| paramDef !== null && paramDef !== undefined && typeof paramDef === 'object' |
There was a problem hiding this comment.
The validation checks if paramDef is a non-null object, but ActionParameter is defined as an interface with specific properties (type, description, required). Consider adding stricter validation to ensure paramDef has at least the 'type' property, or checking if it conforms to the ActionParameter shape.
| ).filter( | |
| ([, paramDef]) => | |
| paramDef !== null && paramDef !== undefined && typeof paramDef === 'object' | |
| ).filter(([ , paramDef ]) => | |
| paramDef !== null && | |
| paramDef !== undefined && | |
| typeof paramDef === 'object' && | |
| 'type' in paramDef && | |
| typeof (paramDef as ActionParameter).type === 'string' |
packages/core/src/prompts.ts
Outdated
| "action" Name of the action to execute after providers return (empty string "" if setting isFinish: true or no action needed). | ||
| "parameters" JSON object with exact parameter names and values for the action. Use empty object {} if the action has no parameters or no action is selected. |
There was a problem hiding this comment.
The instructions say to use empty string "" for action when no action is needed, but also say to use empty object {} for parameters. This is inconsistent - consider using the same pattern (either both null/undefined or both empty string/object) for clarity.
Updated prompt instructions for clarity on 'action' and 'parameters' usage. Improved action parameter validation in the bootstrap plugin to ensure 'type' is present and a string. Cleaned up provider timeout logic and comments in the default message service.
|
Claude finished @wtfsayo's task in 1m 59s —— View job PR Review - Multi-Step Workflow EnhancementsTodo List:
SummaryThis PR adds robust retry logic, parameter extraction, and streaming improvements to multi-step workflows. The implementation is well-structured with proper validation, exponential backoff, and comprehensive test coverage. ✅ Verified - All Previous Issues Addressed
✅ Implementation QualityRetry Logic:
Parameter Extraction:
Streaming Infrastructure:
✅ Test Coverage (Comprehensive)
✅ Template StructureThe 💡 Minor Suggestions (Non-Blocking)
|
- Fix test helper metadata to use `_source`/`_timestamp` to match production code that uses underscore-prefixed properties to avoid collision with legitimate action parameters - Update streaming callback comment for clarity (used by multi-step mode) - Apply prettier formatting fixes
|
Claude finished @wtfsayo's task in 2m 4s —— View job PR Review - Multi-Step Workflow EnhancementsTodo List:
SummaryThis PR adds intelligent streaming retry logic, parameter extraction, and multi-step workflow improvements. The implementation is solid with proper validation throughout. ✅ Verified - All Requirements Met
✅ All Previous Review Issues Addressed
✅ Implementation QualityRetry Logic (
Parameter Extraction (
Streaming Infrastructure (
Actions Provider (
💡 Minor Non-Blocking Observations
|
Summary
This PR adds intelligent streaming retry logic inspired by Anthropic's partial response recovery, along with multi-step workflow enhancements.
Key Features
🔄 Intelligent Streaming Retry Logic
<text>extraction completed (closing tag found), uses streamed text directly instead of retryingStreamingContextmethods (reset(),getStreamedText(),isComplete()) enable smart retry decisions🛠️ Streaming Infrastructure
createStreamingContext(): New helper to create streaming contexts with retry state management built-increateStreamingRetryState(): Tracks streamed text and completion state for retry logicXmlTagExtractor.flush(): Returns buffered content when streams end unexpectedly🔒 Security Fix
indexOf()for actions tag detection to prevent catastrophic backtracking📋 Multi-step Enhancements
Changes by File
packages/core/src/services/default-message-service.tspackages/core/src/utils/streaming.tscreateStreamingContext(),createStreamingRetryState(),flush()methods, ReDoS fixpackages/core/src/streaming-context.tsStreamingContextinterface with retry methodspackages/core/src/types/streaming.tsIStreamingRetryStateinterfacepackages/core/src/prompts.tspackages/plugin-bootstrap/src/providers/actions.tsformatActionsWithParams()for LLM tool callingpackages/plugin-bootstrap/src/__tests__/multi-step.test.tsHow Retry Logic Works
Test Plan
Note
docker-build.ts).chat.tsxadding DM channel creation/selection/deletion, URL-based channel routing, safer send/retry flows, media parsing/attachment handling, auto-scroll improvements, confirmation dialogs, and responsive/floating sidebar layout for mobile/desktop.AgentRuntime(plugins, initialize, models/actions, settings/overrides, embedding dimension) and streaming context isolation/ALS behavior.multiStepDecisionTemplate,multiStepSummaryTemplate) to clarify provider/action selection, parameters, and finish criteria.Written by Cursor Bugbot for commit e2bf8c7. This will update automatically on new commits. Configure here.