Skip to content

adding small improvements to upload data to catalog#348

Merged
Edwardvaneechoud merged 1 commit intomainfrom
feature/add-feedback-for-table-adding
Mar 15, 2026
Merged

adding small improvements to upload data to catalog#348
Edwardvaneechoud merged 1 commit intomainfrom
feature/add-feedback-for-table-adding

Conversation

@Edwardvaneechoud
Copy link
Copy Markdown
Owner

This pull request introduces significant improvements to catalog table registration and materialization, mainly focusing on restricting direct registration to Parquet files in the frontend, improving user feedback, and making the backend materialization process more robust and isolated. Additionally, it removes legacy migration logic from the database initialization.

Frontend and backend changes are grouped below:

Frontend: Catalog Table Registration and User Experience

  • Restricts direct catalog table registration via the UI to only Parquet files, updating allowed file types, hints, and file extension handling. Users are now guided to use Flows for other formats (CSV, Excel, etc.) [1]], [2]]).
  • Improves user feedback in the registration modal: disables the register button during submission, shows a loading state, and displays success/error messages using ElMessage instead of alert [1]], [2]], [3]], [4]], [5]]).
  • Ensures catalog data is loaded before node data is fetched in the CatalogReader component, improving data consistency and preventing race conditions [1]], [2]], [3]]).

Backend: Catalog Table Materialization

  • Refactors the /catalog/materialize endpoint to perform file reading and Parquet writing in a separate subprocess, improving reliability and resource isolation. The subprocess returns schema and metadata through a queue; errors are reported more robustly [1]], [2]], [3]], [4]]).
  • Adds a new materialize_catalog_table_task function to handle file conversion and metadata extraction in the subprocess, supporting CSV, Parquet, and Excel files ([flowfile_worker/flowfile_worker/funcs.pyR437-R479]).

Other

These changes collectively make catalog table registration safer, clearer to users, and more robust in handling different file types and potential errors.

@netlify
Copy link
Copy Markdown

netlify bot commented Mar 15, 2026

Deploy Preview for flowfile-wasm canceled.

Name Link
🔨 Latest commit db5fbcb
🔍 Latest deploy log https://app.netlify.com/projects/flowfile-wasm/deploys/69b68ca11df385000878f664

@Edwardvaneechoud Edwardvaneechoud merged commit 2dbc0d5 into main Mar 15, 2026
25 checks passed
@Edwardvaneechoud Edwardvaneechoud deleted the feature/add-feedback-for-table-adding branch March 15, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant