Description

Story: https://breedinginsight.atlassian.net/browse/BI-1760

This bug occurs when an Excel or CSV file with null/empty columns is uploaded. Such a file can be created in Excel by selecting cells and pressing delete to clear their contents, rather than deleting columns. Because the template allows dynamic columns, the Experiment & Observations upload is where users would most likely encounter this bug.

It can be observed with BreedBase or the BrAPI Java Test Server, as it is a file parsing bug, not a BrAPI/BreedBase bug.

This PR fixes the bug with the following behavior:

1. for Excel and CSV file uploads with fully empty columns (neither header nor data), silently drop those columns,
2. for Excel and CSV file uploads with non-empty columns that are missing headers, show an appropriate error message to the user, as they may want to fix the file by adding the missing headers.

Unit tests and test files were added, see FileUtilUnitTest.parseCsvRemoveAllNullColumns() and FileUtilUnitTest.parseExcelRemoveAllNullColumns().

Potential Issues

My implementation of removeNullColumns in FileUtil.java, which is used to read CSVs:

1. depends on the (undocumented?) way tablesaw handles columns with missing headers (see comment, and tablesaw implementation), and if that changes in future versions of tablesaw, this would break, but the failing unit test would reveal that;
2. would result in a legitimate column being accidentally dropped if the following conditions are met: it is the nth column, it is named Cn, and it has no data (besides the header).

Suggestions for a more robust implementation are welcome, but this wasn't my first or favored approach 😉.

Dependencies

No dependencies.

Testing

To test, I recommend going to an empty program, importing Germplasm using RILS and Parents.xls, and Ontology using bi_lettuce_ontology_v1.xlsx.

Then try importing Experiments & Observations using SCY RIL exp truncated missing header.xls or SCY RIL exp truncated missing header.csv. This should result in a descriptive error message being displayed.

Finally, try importing Experiments & Observations using SCY RIL exp truncated.xls or SCY RIL exp truncated.csv which have empty/null columns. This would result in the "Duplicate column names" error before this fix (the duplicate name was the empty string), but should now succeed, as empty/null columns are dropped silently.

Checklist:

• I have performed a self-review of my own code
• I have tested my code and ensured it meets the acceptance criteria of the story
• I have tested that my code works with both the brapi-java-server and BreedBase
• I have create/modified unit tests to cover this change
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to documentation
• I have run TAF: https://github.com/Breeding-Insight/taf/actions/runs/4852361130

Files:
RILS and Parents.xls
bi_lettuce_ontology_v1.xlsx
SCY RIL exp truncated.xls
SCY RIL exp truncated.csv
SCY RIL exp truncated missing header.xls
SCY RIL exp truncated missing header.csv