feat(V2): rewrite XLSX import/export with metadata-driven V2 implementation#1538
Merged
Conversation
…tation V2 XLSX import was non-functional — the V1 utility functions (tableDataFromXlsx, collapseTablesData) could not match V2 model names or foreign keys. V2 export omitted child records because associations were missing. This commit: - Add missing hasMany associations to ProjectV2 (validations, verifications, projectMethodologies, stakeholderProjects) - Create src/utils/v2-xls.js with metadata-driven helpers that derive schema from Sequelize associations at runtime - Rewrite ProjectV2/UnitV2 updateFromXLS to use new V2 utilities - Fix export controllers to include all child associations when xls=true - Add integration tests that verify actual staging (INSERT/UPDATE), singular/plural sheet name support, and Excel export
- Deduplicate Sequelize includes in project controller when xls=true and columns also request child associations (avoids duplicate alias error) - Replace naive replace(/s$/, '') singularization with naiveSingular() that handles -ies → -y (e.g. projectMethodologies → projectMethodology) - Restore unitSerialId derivation from block range via prepareXlsRow hook on UnitV2, called generically from stageV2XlsRecords - Fix org creation test to skip PENDING record when polling for the finalized V1 organization - Rewrite audit tests to use baseline-delta counts and future timestamps so background sync activity cannot cause false failures
- Restore child record JSON parsing (locations, estimations, ratings, coBenefits) and cadTrustProjectId backfilling in updateProjectPropertiesV2, which is used by CSV batch upload - Guard parseV2Xlsx against ragged rows where trailing cells are missing, preventing undefined values from overwriting existing fields during update staging
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Blank trailing rows in XLSX files were parsed as objects with all-null values, causing stageV2XlsRecords to generate random UUIDs and stage INSERT records with empty payloads. Add an isEmptyRow guard at the top of both parent and child staging loops.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hasManyassociations toProjectV2forValidationV2,VerificationV2,ProjectMethodologyV2, andStakeholderProjectV2— enablesfindAll({ include: [...] })to load all child records for XLS exportsrc/utils/v2-xls.jswith metadata-driven helpers (buildXlsSchema,createV2Xls,parseV2Xlsx,stageV2XlsRecords) that derive XLS schema at runtime from Sequelize associations — no separate config to maintainProjectV2.updateFromXLSandUnitV2.updateFromXLSnow use the new V2-specific utilities instead of V1'stableDataFromXlsx/collapseTablesDatawhich silently failed on V2 model namesxls=trueand usecreateV2Xlsfor proper sheet namingDesign decisions
src/utils/xls.js) is untouched — V1 and V2 have independent XLS implementationsstatic xlsSheetNameonProjectV2andUnitV2; everything else is derived from existing Sequelize metadatagetAssociatedModels()is unchanged — newhasManyassociations only affect Sequelizeincludequeries, not staging/commit flowTest plan
.xlsxwith correct content-disposition headersGET ?xls=true, re-import the downloaded file viaPUT /xlsxNote
Medium Risk
Changes the V2 XLSX import/export pipeline (sheet naming, association includes, staging upserts), which can affect data staging correctness and exported file structure; impact is limited to V2 endpoints and utilities.
Overview
Reworks V2 XLSX import/export for
ProjectV2andUnitV2to be metadata-driven. Controllers now export viacreateV2Xlsand, whenxls=true, automatically include allHasManychild associations (avoiding duplicate includes when specific columns also request associations).Rewrites V2 XLSX import to stage records consistently.
ProjectV2.updateFromXLSandUnitV2.updateFromXLSswitch from V1 parsing/collapsing helpers to newparseV2Xlsx+stageV2XlsRecords, which supports singular/plural sheet names, stages parent and child rows independently withStagingV2.upsert, and allows model-specific row prep (e.g. derivingunitSerialId).Expands model associations and tests.
ProjectV2adds missinghasManyrelationships (e.g. validations/verifications/etc.) and setsxlsSheetNameon V2 models; integration tests are updated to assert stagingINSERTvsUPDATE, tolerate background audit noise via baselines, and stabilize org-creation polling.Written by Cursor Bugbot for commit bd5191a. This will update automatically on new commits. Configure here.