Skip to content

Epic/525 implement fully functioning node registry#545

Merged
felix-schultz merged 62 commits intodevfrom
epic/525-implement-fully-functioning-node-registry
Apr 1, 2026
Merged

Epic/525 implement fully functioning node registry#545
felix-schultz merged 62 commits intodevfrom
epic/525-implement-fully-functioning-node-registry

Conversation

@felix-schultz
Copy link
Copy Markdown
Member

This pull request introduces several important improvements and new documentation for the flow-like project, focusing on enhanced developer guidance, stricter coding standards, and expanded backend capabilities. Notably, it adds a comprehensive developer onboarding guide (CLAUDE.md), enforces critical node creation and pin naming rules, updates workspace dependencies to include new backend and library packages, and improves workflow automation for credential management.

Documentation and Developer Guidance:

  • Added a detailed CLAUDE.md onboarding guide covering repository structure, build commands, coding standards, Rust/TypeScript best practices, node creation rules, and code quality requirements.
  • Expanded node creation instructions to clarify WASM node permission declarations, enforce unique input/output pin names, and document update/versioning practices for nodes. [1] [2] [3]
  • Updated general coding standards to prohibit destructive git operations (e.g., stash, reset) without explicit user confirmation.

Workspace and Dependency Updates:

  • Added new backend and library packages to the Rust workspace, including AWS/Kubernetes compiler/executor components, secrets management, code interpreter, and catalog build helper. [1] [2]
  • Updated workspace dependencies to reference new packages and switched to updated versions of copilot-sdk and markitdown for improved protocol support and bug fixes. [1] [2] [3]

Backend and Build Improvements:

  • Enhanced AWS API backend dependencies to support SQS, Sentry, and AWS Secrets integration; adjusted release profile for smaller binary size. [1] [2]
  • Integrated Sentry tracing and secrets provider configuration into AWS API backend initialization for improved observability and secret management. [1] [2]
  • Improved CI workflow to decode and inject Google/Firebase credentials for both macOS and Windows runners, streamlining release automation.

- Implemented a new PublishDialog component for publishing updates to packages.
- Allows users to select version bump type (patch, minor, major) and enter release notes.
- Validates release notes length before enabling the publish button.
- Integrates with existing UI components for a consistent user experience.

feat: create usePackageStoreData hook for package purchasing

- Added usePackageStoreData hook to manage package purchasing and access requests.
- Handles fetching user profile, purchasing packages, and requesting access.
- Provides utility functions for formatting prices and managing access states.

feat: introduce wasm-package-permission utilities

- Created permission helpers for managing package permissions.
- Includes functions to check user roles (owner, maintainer, user, buyer) and manage permission levels.

feat: extend wasm schema with new package and node definitions

- Updated wasm schema to include new interfaces for NodeScores, PinOptions, and Pin.
- Enhanced PackageNodeEntry and RegistryEntry with additional fields for better package management.

feat: enhance registry state with access request management

- Modified IRegistryState interface to include methods for purchasing packages and managing access requests.
- Added support for listing and accepting/rejecting access requests.

feat: implement version management in WASM client

- Updated RegistryClient to handle multiple installed versions of packages.
- Introduced methods for installing specific versions and batch installation of packages.

feat: add cross-compilation support in WASM engine

- Enhanced WasmConfig to support cross-compilation target triples.
- Implemented precompilation for both host and cross-compilation targets.

feat: improve manifest structure for package nodes

- Updated PackageNodeEntry to include optional fields for friendly names, scores, and pin definitions.
- Enhanced documentation and metadata handling for nodes.

refactor: reorganize installed package structure

- Refactored InstalledPackage to manage multiple versions of a package.
- Introduced InstalledVersion struct to encapsulate version-specific details.
- Added Chain of Density post-processing to enhance information density in summaries.
- Introduced EntityTracker to manage and track important entities across summarization chunks.
- Developed Hierarchical summarization to leverage document structure for improved summaries.
- Implemented Hybrid summarization combining Map-Reduce and Refine strategies for efficiency and coherence.
- Created Map-Reduce summarization to process chunks in parallel and reduce them recursively.
- Established a modular structure for summarization strategies in the mod.rs file.
- Enhanced prompts for various summarization tasks, ensuring clarity and adherence to rules.
- Introduced Iterative Refinement summarization for coherent output through sequential processing.
- Developed Sliding Window summarization to maintain a fixed-size memory buffer, preventing context pressure.
- Updated `PackageSummary` struct to include `price` and `visibility` fields with default values.
- Added `auth_token` field to `RegistryConfig` for authenticated API requests.
- Introduced integration tests for new context functions related to schema, image, DB, and models.
- Verified that host-side schema module returns valid JSON schemas and that linker registration succeeds for all new host functions.
- Ensured that WAT-based WASM modules can call host functions without panics and that schema data is correctly returned through the WASM ABI.
- Updated `wasm_package_join_queue.rs`, `wasm_package_purchase.rs`, `wasm_package_review.rs`, `wasm_package_user.rs`, `wasm_package_version.rs`, and `widget.rs` to enhance code formatting and organization.
- Removed unnecessary whitespace and improved the alignment of attributes for better clarity.
- Added missing `impl` blocks for `ActiveModelBehavior` in several entities.
- Refactored the `download.rs`, `index.rs`, `prerun_check.rs`, and `server.rs` routes to streamline access control and package retrieval logic.
- Enhanced error handling and logging in `jwt.rs`, `execute.rs`, and `wasm_loader.rs` for better debugging and maintainability.
- Improved the clarity of comments and documentation throughout the codebase.
…ty, including Sentry integration for error reporting
…ironment variable handling and SQS support; add soft-delete functionality for packages in the registry
…ipulation

- Introduced `setIframeSrc` node to set the source URL of an iframe element.
- Introduced `setIframeSrcdoc` node to set raw HTML content for iframe preview.
- Updated `IframeProps` to include optional `srcdoc` field.
- Modified `Iframe` component to handle both `src` and `srcdoc` properties.
- Updated documentation to reflect changes in iframe component capabilities.
- Updated various ML nodes (Decision Tree, Naive Bayes, SVM, DBSCAN, KMeans, etc.) to use a cached database instance for loading data.
- Added calls to `ensure_flushed()` after loading the cached database to ensure data consistency.
- Modified database access patterns to improve performance and reliability by reading from the cached database.
- Adjusted dataset operations (Sample, Shuffle, Split, Stratified) to ensure proper flushing and access to the source database.
- Enhanced metrics nodes (Accuracy, Confusion Matrix, Regression Metrics) to follow the new database loading pattern.
- Updated prediction and PCA nodes to utilize the cached database and ensure data is flushed before processing.
- Overall improvements to database handling across multiple ML components for better data integrity and performance.
- Updated Cargo.toml to include additional crate types and dependencies for enhanced functionality.
- Modified flow-like.toml to streamline package metadata and permissions, including new node definitions.
- Refactored lib.rs to implement new nodes: GreetingNode and FileWriterNode, with structured input/output pins and enhanced functionality.
- Added tests for new nodes and ensured compliance with linting rules, including checks for duplicate pin names, schema validation, and node descriptions.
- Introduced lint.rs to enforce coding standards and best practices across all nodes.
- Added a new TypeScript file for dexie-tauri adapter that handles large strings and number arrays by offloading them to Tauri's native filesystem.
- Introduced a new tsconfig.json for the frontend package with strict type checking and module resolution settings.
- Updated the executor to use a new API endpoint for progress and event callbacks.
- Enhanced the model provider to track and accumulate usage statistics for model calls.
- Extended the chat default interfaces to include usage statistics for model invocations.
- Implemented a new UsageStats component to display aggregated model usage statistics in the UI.
- Improved the execution engine to deduplicate events based on event IDs.
- Updated the Cargo.toml for the wasm package to enable all cranelift backends for cross-compilation.
@felix-schultz felix-schultz linked an issue Apr 1, 2026 that may be closed by this pull request
@sentry
Copy link
Copy Markdown

sentry bot commented Apr 1, 2026

🚧 Skipped: PR exceeds review size limit.

Please split into smaller PRs and re-run.
Reference ID: 1067174

@sentry
Copy link
Copy Markdown

sentry bot commented Apr 1, 2026

🚧 Skipped: PR exceeds review size limit.

Please split into smaller PRs and re-run.
Reference ID: 1067191

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces significant updates, including the addition of a WASM compiler service for Kubernetes and AWS, improved storage configuration management, and enhanced UI components for package management and developer tools. I have reviewed the code and identified several issues related to unsafe string slicing, missing timeouts for network requests, potential stack overflows in schema resolution, and code duplication. Please address these concerns to improve the robustness and maintainability of the codebase.

@felix-schultz felix-schultz requested a review from Copilot April 1, 2026 21:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements a more complete node/package registry experience across the desktop app and backend deployments, adding compiler infrastructure, admin workflows, and updated developer guidance.

Changes:

  • Desktop: expanded registry/package UX (explore + details), new admin pages, improved navigation/link handling, and additional settings pages.
  • Backend: added Kubernetes + Docker Compose compiler service, AWS compiler/executor targets, secret-store integration, and expanded metrics/monitoring.
  • Docs/tooling: added CLAUDE.md, refined node-creation rules, and updated CI credential decoding.

Reviewed changes

Copilot reviewed 115 out of 1415 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
apps/desktop/components/global-anchor-component.tsx Improves link/window handling for Tauri, including data-href navigables.
apps/desktop/components/auth-provider.tsx Adjusts Tauri auth invalidation ordering to ensure user info first.
apps/desktop/components/app-sidebar.tsx Updates nav structure and adds server-side profile deletion + new window URL resolution.
apps/desktop/components/admin/profile-templates-admin-page.tsx Adds admin UI for browsing/managing profile templates.
apps/desktop/app/widget/page.tsx Adds widget “Events” editing UI and passes action context into editor.
apps/desktop/app/use/page.tsx Replaces in-app Use page implementation with shared UsePageContent.
apps/desktop/app/store/page.tsx Refactors store page loading/error handling and adds app reviews section.
apps/desktop/app/settings/registry/layout.tsx Renames “Installed” tab text to “Owned”.
apps/desktop/app/settings/registry/explore/page.tsx Adds package detail view selection and authenticated detail fetch support.
apps/desktop/app/provider.tsx Moves PostHog config to environment variables.
apps/desktop/app/profile/page.tsx Adds href support to app cards for consistent navigation.
apps/desktop/app/onboarding/page.tsx Filters out deleted profiles during onboarding fetch.
apps/desktop/app/notifications/page.tsx Adds router-based navigation for notification links.
apps/desktop/app/library/packages/publish/page.tsx Converts manifest payload keys to snake_case and expands resource tier options.
apps/desktop/app/library/packages/page.tsx Initializes registry before queries and adds auth token to update checks.
apps/desktop/app/library/config/widgets/page.tsx Improves empty-state copy/styling and tweaks preview overflow behavior.
apps/desktop/app/library/config/visibility-status-switcher.tsx Adds publication request history card and cache invalidation.
apps/desktop/app/library/config/user-storage/page.tsx Adds user storage file operations (reveal/open-with/list-apps).
apps/desktop/app/library/config/templates/page.tsx Improves templates empty-state UX and copy.
apps/desktop/app/library/config/storage/page.tsx Adds storage file operations (reveal/open-with/list-apps).
apps/desktop/app/library/config/runtime-vars/page.tsx Updates runtime vars layout and header copy/styling.
apps/desktop/app/library/config/publication/page.tsx Adds publication review page backed by API query.
apps/desktop/app/library/config/packages/page.tsx Adds app packages settings page wrapper.
apps/desktop/app/library/config/flows/page.tsx Adds board href generation for flows list navigation.
apps/desktop/app/library/config/events/page.tsx Wires hub context into events settings page.
apps/desktop/app/library/config/endpoints/page.tsx Replaces placeholder with real endpoints settings page.
apps/desktop/app/library/config/configuration/page.tsx Improves configuration empty-state and header UX/copy.
apps/desktop/app/library/config/analytics/page.tsx Replaces placeholder with analytics dashboard.
apps/desktop/app/layout.tsx Adjusts query-cache retention and initializes blob offload.
apps/desktop/app/join/page.tsx Adds retry/backoff + better UX for invite join flow.
apps/desktop/app/developer/new/page.tsx Redirects developer wizard to package registry projects tab.
apps/desktop/app/developer/manifest/page.tsx Expands permission options and removes node list section.
apps/desktop/app/debug/streaming/page.tsx Adds a streaming markdown editor debug page.
apps/desktop/app/admin/user/page.tsx Adds admin “user” page wrapper for profile templates view-only mode.
apps/desktop/app/admin/user/edit/page.tsx Adds admin “user edit” page wrapper for managing templates.
apps/desktop/app/admin/solutions/page.tsx Adds summary cards and improves admin solutions layout.
apps/desktop/app/admin/profiles/add/page.tsx Adds create/edit profile template flow backed by admin APIs.
apps/desktop/app/admin/packages/page.tsx Enhances package registry admin list (visibility/status formatting, layout).
apps/desktop/app/admin/governance/requests/page.tsx Adds governance requests list/detail routing page.
apps/desktop/app/admin/governance/page.tsx Adds governance list/detail routing page.
apps/desktop/app/admin/bits/add/llm.tsx Adds hosted provider params editing and provider options.
apps/desktop/.gitignore Ignores google-services.json for desktop app.
apps/desktop/.env.example Updates staging API URL comment.
apps/backend/local/runtime/src/main.rs Simplifies queued job conversion via ExecutionRequest::try_from(job).
apps/backend/local/runtime/Cargo.toml Adjusts release opt-level for runtime.
apps/backend/local/api/src/main.rs Adds secrets store and Sentry-tracing filtering; passes secret config into state.
apps/backend/local/api/Cargo.toml Enables Sentry feature and adds secrets dependency.
apps/backend/kubernetes/sink-trigger/Cargo.toml Adjusts release opt-level.
apps/backend/kubernetes/helm/values.yaml Adds compiler component configuration section.
apps/backend/kubernetes/helm/templates/monitoring/prometheus.yaml Adds compiler scrape config + recording/alert rules.
apps/backend/kubernetes/helm/templates/monitoring/grafana-dashboard-compiler.yaml Adds compiler Grafana dashboard ConfigMap template.
apps/backend/kubernetes/helm/templates/compiler-service.yaml Adds compiler service (http + metrics ports).
apps/backend/kubernetes/helm/templates/compiler-hpa.yaml Adds optional compiler HPA template.
apps/backend/kubernetes/helm/templates/compiler-deployment.yaml Adds compiler deployment template with env/config.
apps/backend/kubernetes/helm/templates/api-deployment.yaml Wires API env vars for external compiler backend.
apps/backend/kubernetes/helm/templates/_helpers.tpl Adds helper labels for compiler component.
apps/backend/kubernetes/executor/Cargo.toml Adjusts release opt-level.
apps/backend/kubernetes/compiler/src/metrics.rs Adds OTEL tracing + Prometheus metrics setup for compiler.
apps/backend/kubernetes/compiler/src/main.rs Adds compiler binary that serves API + separate metrics listener.
apps/backend/kubernetes/compiler/Cargo.toml Adds new Kubernetes compiler package manifest.
apps/backend/kubernetes/api/src/storage.rs Splits storage into content/meta/cdn store constructors.
apps/backend/kubernetes/api/src/main.rs Switches to CDN store and passes optional secrets config parameter.
apps/backend/kubernetes/api/src/config.rs Adds cdn/meta bucket config fields + env fallback resolution.
apps/backend/kubernetes/api/Cargo.toml Adjusts release opt-level.
apps/backend/kubernetes/.env.example Adds WASM compilation env variables.
apps/backend/docker-compose/sink-services/Cargo.toml Adjusts release opt-level.
apps/backend/docker-compose/runtime/src/main.rs Uses ExecutionRequest::try_from(job) and preserves job_id logging.
apps/backend/docker-compose/runtime/Cargo.toml Adjusts release opt-level.
apps/backend/docker-compose/monitoring/prometheus/prometheus.yml Adds compiler scrape configuration.
apps/backend/docker-compose/monitoring/grafana/dashboards/compiler.json Adds compiler Grafana dashboard.
apps/backend/docker-compose/flow-like.config.example.json Enables wasm_registry in example config.
apps/backend/docker-compose/docker-compose.yml Adds compiler service and wires compilation env vars into API.
apps/backend/docker-compose/compiler/Dockerfile Adds compiler build/docker image recipe.
apps/backend/docker-compose/api/src/storage.rs Splits storage constructors into content/meta/cdn.
apps/backend/docker-compose/api/src/main.rs Uses CDN store and passes optional secrets config parameter.
apps/backend/docker-compose/api/Cargo.toml Adjusts release opt-level.
apps/backend/docker-compose/.env.example Documents compilation backend + compiler settings.
apps/backend/aws/media-transformer/Cargo.toml Adjusts release opt-level.
apps/backend/aws/file-tracker/src/main.rs Switches SeaORM import source and adds entity module.
apps/backend/aws/file-tracker/src/event_handler.rs Moves entities to local module and uses SeaORM crate directly.
apps/backend/aws/file-tracker/src/entity.rs Adds SeaORM entity definitions for app/user tables/enums.
apps/backend/aws/file-tracker/Cargo.toml Removes flow-like-api dependency and adds sea-orm dependency.
apps/backend/aws/executor/src/main.rs Uses current-thread Tokio runtime flavor for Lambda executor.
apps/backend/aws/executor/Cargo.toml Switches to server-execute features and adjusts release opt-level.
apps/backend/aws/executor-ecs/src/main.rs Adds ECS task executor that resolves/executes dispatch payloads.
apps/backend/aws/executor-ecs/Cargo.toml Adds ECS executor package manifest.
apps/backend/aws/executor-async/src/main.rs Uses current-thread Tokio runtime flavor for async executor lambda.
apps/backend/aws/executor-async/src/execution.rs Uses shared DispatchPayload and ExecutionRequest::try_from(payload).
apps/backend/aws/executor-async/Cargo.toml Switches to server-execute features and adjusts release opt-level.
apps/backend/aws/event-bridge/src/main.rs Simplifies handler input type and drops CloudWatchEvent wrapper.
apps/backend/aws/event-bridge/Dockerfile Updates build image + workspace copy strategy.
apps/backend/aws/event-bridge/Cargo.toml Removes aws_lambda_events dependency and adjusts release opt-level.
apps/backend/aws/compiler-lambda/src/main.rs Adds SQS-driven compiler lambda with deadline-aware behavior.
apps/backend/aws/compiler-lambda/Cargo.toml Adds compiler lambda package manifest.
apps/backend/aws/compiler-ecs/src/main.rs Adds ECS compiler task with remote job resolution and diagnostics.
apps/backend/aws/compiler-ecs/Cargo.toml Adds compiler ECS package manifest.
apps/backend/aws/api/src/main.rs Adds secret store (SSM+env), improves Sentry breadcrumb filtering, passes secret config into state.
apps/backend/aws/api/Cargo.toml Enables SQS/Sentry/Secrets features and adjusts release opt-level.
Cargo.toml Adds new workspace members and dependencies (compiler/secrets/etc.) and updates git deps.
CLAUDE.md Adds developer onboarding + coding standards and critical rules.
.github/workflows/alpha-release.yml Decodes Google/Firebase credentials for multiple OS runners.
.github/instructions/node-creation.instructions.md Expands node creation guidance (permissions, pin naming, versioning).
.github/instructions/general-coding.instructions.md Adds explicit guidance against destructive git operations.

@codacy-production
Copy link
Copy Markdown

codacy-production bot commented Apr 1, 2026

Not up to standards ⛔

🔴 Issues 27 minor . 3 medium . 69 medium . 1 high

Alerts:
⚠ 100 issues (≤ 0 issues of at least minor severity)

Results:
100 new issues

Category Results
BestPractice 3 medium
Security 1 high
CodeStyle 27 minor
Complexity 69 medium

View in Codacy

🟢 Metrics 9664 complexity . 1928 duplication

Metric Results
Complexity 9664
Duplication 1928

View in Codacy

TIP This summary will be updated as you push new changes. Give us feedback

…ve embed detection logic and app category formatting
@sentry
Copy link
Copy Markdown

sentry bot commented Apr 1, 2026

🚧 Skipped: PR exceeds review size limit.

Please split into smaller PRs and re-run.
Reference ID: 1067328


while (i < lines.length) {
const line = lines[i];
const match = line.match(/^:::(\w+)(?:\s+(.+))?$/);

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
This
regular expression
that depends on
library input
may run slow on strings starting with ':::a ' and with many repetitions of ' '.
result = get_type_schema(type_name)
if result is not None:
return json.loads(result)
except Exception:
]
return AIMessage(content=content, tool_calls=tool_calls)
return AIMessage(content=content)
except (json.JSONDecodeError, TypeError):

try:
exec(compile(code, "<flow_code>", "exec"), ns)
except SystemExit:
import types as _types
if origin is _types.UnionType:
return True
except AttributeError:
@sentry
Copy link
Copy Markdown

sentry bot commented Apr 1, 2026

🚧 Skipped: PR exceeds review size limit.

Please split into smaller PRs and re-run.
Reference ID: 1067395

- Added `#[default]` attribute to enum variants in `styles.rs`, removing the need for manual Default implementations.
- Updated various enums including `PageSize`, `Orientation`, `TextAlignment`, `ParagraphStyle`, `ShapeType`, `PageNumberPosition`, and `Rotation`.
- Improved code readability by using `if let` syntax for conditional checks in multiple files, including `chart_data_agent.rs`, `instantiate_widget.rs`, and others.
- Enhanced error handling and logging in `aws.rs` and `sync_node_schema.rs`.
- Cleaned up unnecessary code and improved performance in `execution.rs`, `context.rs`, and `node.rs`.
- General code style improvements across multiple files for consistency and clarity.
@sentry
Copy link
Copy Markdown

sentry bot commented Apr 1, 2026

🚧 Skipped: PR exceeds review size limit.

Please split into smaller PRs and re-run.
Reference ID: 1067533

@felix-schultz felix-schultz merged commit be61fe1 into dev Apr 1, 2026
13 of 19 checks passed
@felix-schultz felix-schultz deleted the epic/525-implement-fully-functioning-node-registry branch April 1, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement fully functioning Node-Registry

3 participants