Skip to content

FlowPilot + Agents#402

Merged
felix-schultz merged 24 commits intoalphafrom
dev
Nov 26, 2025
Merged

FlowPilot + Agents#402
felix-schultz merged 24 commits intoalphafrom
dev

Conversation

@felix-schultz
Copy link
Copy Markdown
Member

This pull request introduces several improvements and updates across the desktop app, focusing on dependency upgrades, UI enhancements for the storage config page, and new backend capabilities for chat interactions. The most significant changes include upgrading major dependencies (notably Next.js and React), adding a streaming chat method to the board state, and improving the layout and loading experience for the storage configuration section.

Dependency and Build System Updates:

  • Upgraded next to 16.0.3, react and react-dom to 19.2.0, and synchronized related type packages and ESLint config versions in package.json, also adding explicit overrides to ensure version consistency. [1] [2] [3] [4]
  • Updated Next.js config to enable the React Compiler, add turbopackFileSystemCacheForDev, and clean up experimental options for improved build performance and development experience. [1] [2]

Backend and API Enhancements:

  • Added a new flowpilot_chat method to the BoardState class, enabling streaming chat interactions with support for token streaming, model selection, and run context. [1] [2] [3]
  • Updated Rust dependencies in Cargo.toml to enable the rmcp feature in rig-core and added the rmcp crate with multiple features for enhanced protocol support.

UI and Layout Improvements:

  • Improved the storage configuration layout to use a full-height flex layout and display a skeleton loader while loading, enhancing the user experience for storage-related pages. [1] [2] [3] [4]
  • Wrapped the application sidebar and related components in an ExecutionEngineProviderComponent to provide execution context throughout the app. [1] [2]

Schema File Maintenance:

  • Reordered properties in several schema definitions in desktop-schema.json to improve consistency, but with no functional changes. [1] [2] [3] [4] [5] [6] [7]

felix-schultz and others added 24 commits November 17, 2025 22:32
- Updated the LinkFloatingToolbar component to apply a non-selectable style to the insert and edit divs using React's useMemo for performance optimization.

chore(package): update dependencies

- Upgraded Next.js to version 16.0.3.
- Updated React and React DOM to version 19.2.0.
- Updated TypeScript types for React and React DOM to the latest versions.
- Updated eslint-config-next to version 16.0.3.
- added blog article about agent configs
- added MCP compatibility
- Introduced `PushReasoningNode`, `PushStepNode`, `PushTextToStepNode`, and `RemoveStepNode` to handle reasoning and step management in chat events.
- Updated `ChatStreamingResponse` to include plan steps and reasoning.
- Enhanced UI components to display execution plans and reasoning effectively.
- Replaced `SimpleMarkdown` with `TextEditor` for rendering markdown content in various components.
- Updated database schema to accommodate new plan step structures and statuses.
- Improved performance by optimizing the rendering of large content in the text editor.
- Implemented `autocomplete` method in `FlowCopilot` component to suggest nodes based on user input.
- Added `autocomplete` method to `IBoardState` interface for backend integration.
- Created `CatalogProvider` trait and `DesktopCatalogProvider` struct for managing node suggestions.
- Integrated `autocomplete` command in Tauri's command handling.
- Updated `FlowBoard` to utilize `FlowCopilot` for enhanced user interaction.
- Introduced new `copilot.rs` file in Rust backend to handle autocomplete logic.
- Added necessary dependencies in `Cargo.toml` for async operations and error handling.
- Export additional types from flow-copilot in flow index.
- Update ScrollArea component to accept viewportRef and onScroll props.
- Implement focus node transformation in TextEditor for handling focus:// links.
- Modify safeDeserialize function to include focus node transformation.
- Add onFocusNode prop to TextEditor for handling focus node clicks.
- Improve error handling in useCommandExecution for command execution failures.
- Update IBoardState interface to include flowpilot_chat method with ChatMessage history.
- Implement flowpilot_chat method in EmptyBoardState to throw an error if called.
- Added a new hook `useCopilotCommands` to handle various board commands such as adding nodes, connecting pins, and managing variables and comments.
- Introduced a mapping system for node references and pin IDs to facilitate command execution.
- Implemented error handling and logging for command execution failures.
- Enhanced the command execution flow with a structured approach for adding nodes and managing connections.

fix: improve keyboard shortcuts to invalidate and refetch board data

- Updated the `useKeyboardShortcuts` hook to include a helper function for invalidating and refetching board data after undo/redo actions and node placements.
- Ensured consistent state updates across the application by replacing direct calls to `board.refetch()` with the new `invalidateBoard` function.

feat: define types for copilot commands and responses

- Created a new schema file `copilot.ts` to define types for agent interactions, chat messages, and board commands.
- Structured command types for operations like adding/removing nodes, connecting pins, and managing variables and comments.
- Added interfaces for copilot responses and suggestions to standardize communication with the copilot system.
- Added FlowCopilot functionality to the Traces component, allowing users to interact with suggestions and commands.
- Implemented resizable panels for better UI/UX, enabling users to adjust the log and FlowPilot view.
- Introduced new IRunContext interface to manage run context data across components.
- Updated various components and hooks to accommodate new props and state management for FlowCopilot.
- Refactored log message handling to optimize rendering and state updates.
- Introduced PendingCommandsView for displaying and executing pending commands with animations and tooltips.
- Added PlanStepsView to visualize the steps of a plan with progress tracking and expandable history.
- Created StatusPill component to represent loading phases with icons and labels.
- Defined types for loading phases and command structures in types.ts.
- Implemented utility functions for command icon retrieval, summary generation, and color coding.
- Updated traces component to support a customizable copilot panel.
- Enhanced useCopilotCommands hook to handle new command types including AddPlaceholder.
- Updated schema to include new command types and chat message structures with image support.
@felix-schultz felix-schultz merged commit 0ab5656 into alpha Nov 26, 2025
4 of 14 checks passed
Comment on lines +689 to +693
const updatedComment: IComment = {
...existingComment,
content: cmd.content ?? existingComment.content,
color: cmd.color ?? existingComment.content,
};
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: color field is incorrectly set to existingComment.content instead of existingComment.color when cmd.color is falsy.
Severity: CRITICAL | Confidence: High

🔍 Detailed Analysis

When updating a comment, if cmd.color is falsy, the color field is incorrectly set to existingComment.content instead of existingComment.color. This corrupts the comment's color data with its text content and causes a type mismatch, leading to loss of the original color.

💡 Suggested Fix

Change color: cmd.color ?? existingComment.content to color: cmd.color ?? existingComment.color to ensure the correct fallback.

🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: packages/ui/hooks/use-copilot-commands.tsx#L689-L693

Potential issue: When updating a comment, if `cmd.color` is falsy, the `color` field is
incorrectly set to `existingComment.content` instead of `existingComment.color`. This
corrupts the comment's color data with its text content and causes a type mismatch,
leading to loss of the original color.

Did we get this right? 👍 / 👎 to inform future reviews.
Reference ID: 142498

Comment on lines +189 to +191
if pin_type == PinType::Input || data_type == VariableType::Execution {
continue;
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: execute_tool_call incorrectly maps LLM tool arguments to OUTPUT pins instead of INPUT pins, causing tools to receive wrong inputs.
Severity: CRITICAL | Confidence: High

🔍 Detailed Analysis

The execute_tool_call function incorrectly assumes LLM tool arguments should be set on OUTPUT pins of referenced functions, instead of standard INPUT pins. This semantic mismatch means LLM-generated arguments will not reach the function's actual input parameters, leading to silent failures or incorrect tool execution results.

💡 Suggested Fix

Modify execute_tool_call to map LLM tool arguments to the INPUT pins of the referenced functions, aligning with standard tool calling conventions.

🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: packages/catalog/src/ai/generative/agent/helpers.rs#L189-L191

Potential issue: The `execute_tool_call` function incorrectly assumes LLM tool arguments
should be set on OUTPUT pins of referenced functions, instead of standard INPUT pins.
This semantic mismatch means LLM-generated arguments will not reach the function's
actual input parameters, leading to silent failures or incorrect tool execution results.

Did we get this right? 👍 / 👎 to inform future reviews.
Reference ID: 142498

(p) => `${p.name}:${p.id.slice(0, 8)}`,
),
});
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: nodeIndex increments unconditionally, creating orphaned node references when executeCommand fails to register a placeholder.
Severity: CRITICAL | Confidence: High

🔍 Detailed Analysis

The nodeIndex increments unconditionally even when executeCommand returns a falsy value (e.g., undefined), meaning the placeholder is not registered in nodeReferenceMap. This creates an orphaned node reference, causing subsequent commands that rely on the auto-generated $${nodeIndex} reference to fail silently.

💡 Suggested Fix

Move the nodeIndex++ increment inside the if (result) block to ensure it only increments when a placeholder is successfully registered.

🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: packages/ui/hooks/use-copilot-commands.tsx#L312

Potential issue: The `nodeIndex` increments unconditionally even when `executeCommand`
returns a falsy value (e.g., `undefined`), meaning the placeholder is not registered in
`nodeReferenceMap`. This creates an orphaned node reference, causing subsequent commands
that rely on the auto-generated `$${nodeIndex}` reference to fail silently.

Did we get this right? 👍 / 👎 to inform future reviews.
Reference ID: 142498

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant