Skip to content

BI-1881 - Store Parent UUIDs in Germplasm AdditionalInfo#290

Merged
mlm483 merged 9 commits intodevelopfrom
bug/BI-1881-mlm483
Sep 22, 2023
Merged

BI-1881 - Store Parent UUIDs in Germplasm AdditionalInfo#290
mlm483 merged 9 commits intodevelopfrom
bug/BI-1881-mlm483

Conversation

@mlm483
Copy link
Contributor

@mlm483 mlm483 commented Sep 19, 2023

Description

Story: https://breedinginsight.atlassian.net/browse/BI-1881

Previously, BrAPIGermplasmDAO::processGermplasmForDisplay required parent BrAPIGermplasm objects to be available in order to add the pedigreeByUUID and pedigreeByName keys to additionalInfo on each BrAPIGermplasm. This required some hacks, passing all germplasm in the program for example.

To make the code more efficient and avoid having to pass all program germplasm into processGermplasmForDisplay, I made changes to

  • store male/female parent UUIDs on each BrAPIGermplasm in additionalInfo on create,
  • read the parent UUIDs in processGermplasmForDisplay to build the required pedigree string ("/"),
  • wrote a SQL script to update Germplasm records in BreedBase to store the parental UUIDs in additionalInfo, that's in this PR on breedbase_configs.

Note: because the BrAPI Java TestServer (BJTS) stores additional info as byte arrays of serialized Java objects, it was deemed impractical to make the change in SQL. For the 1 program using the BJTS in production, there are no germplasm records so there is no data to be upgraded. The non-production BJTS instances will be wiped out, it's on the v0.8.1 deploy checklist. If a DeltaBreed instance has the environment variable BRAPI_REFERENCE_SOURCE set to anything other than "breedinginsight.net", that may also need to be updated, see https://breedinginsight.atlassian.net/l/cp/VfZ8uHXS.

Dependencies

If you checkout this branch and run it against BJTS or BreedBase with existing germplasm records, it will break. In the case of BJTS, the only option currently is to wipe out the database. For BreedBase, germplasm records can be upgraded in place by running breedbase-deploy.sh from the bug/BI-1881-mlm483 branch. See this PR for details.

Testing

Steps 1-10 focus on BreedBase, step 11 is for BJTS.

germplasm_with_additional_info (click to expand)
-- Germplasm with additional info.
SELECT 
    germ.stock_id, germ.uniquename AS "stock.uniquename", cvterm.name AS "cvterm.name", stockprop.stockprop_id, stockprop.value, stockprop.rank
FROM 
    public.stock germ
    LEFT JOIN public.stockprop ON stockprop.stock_id = germ.stock_id
    LEFT JOIN cvterm ON cvterm.cvterm_id = stockprop.type_id
WHERE 
    germ.stock_id IN
    (
        SELECT
            stock.stock_id
        FROM 
            stock
            JOIN
            cvterm cvt ON cvt.cvterm_id = stock.type_id AND cvt.name::text = 'accession'::text
    )
    AND cvterm.name = 'stock_additional_info'
    -- Uncomment if you only want to see germplasm from the attached file.
    -- AND germ.uniquename LIKE '1881-%'
ORDER BY germ.uniquename, cvterm.name, stockprop.rank
  1. Checkout the develop branch for both bi-api and breedbase_configs.
  2. Wipe out BJTS and BreedBase (optional, but makes testing cleaner). I like to delete all local containers and volumes.
  3. Start up BreedBase (with breedbase_deploy.sh) and start up all the services needed for DeltaBreed, then run bi-api and bi-web.
  4. Create a new program stored in BreedBase and upload germplasm with various pedigree permutations (example file attached).
  5. Check the additionalInfo doesn't contain "maleParentUUID" or "femaleParentUUID" fields with the germplasm_with_additional_info SQL query above. If those keys are present, you are probably running the bug/BI-1881-mlm483 branch of bi-api. Oops, start over!
  6. Stop bi-api and checkout bug/BI-1881-mlm483. If you restart before performing the next step, you'll see some exceptions saying "Germplasm has a female parent but femaleParentUUID is missing.".
  7. Checkout the bug/BI-1881-mlm483 branch of breedbase_configs and run breedbase_deploy.sh (without the --reinstall flag). The flyway docker container will be pulled, start up, wait for the breedbase_db container to be healthy and then run the migrations in the ./flyway/sql directory.
  8. You can confirm that the data has been updated with the germplasm_with_additional_info SQL above.
  9. Start bi-api and ensure it runs without the exception and germplasm list, detail and pedigree views work as expected.
  10. Upload more germplasm, and use steps 8. and 9. to ensure the running code stores new germplasm with the parental UUIDs correctly.
  11. Create a program in the BJTS and upload germplasm and ensure the upload and list, detail and pedigree views work as expected. You can use the brapi/v2/germplasm endpoint as well to see that the "femaleParentUUID" and/or "maleParentUUID" fields are present in additionalInfo.

Checklist:

  • I have performed a self-review of my own code
  • I have tested my code and ensured it meets the acceptance criteria of the story
  • I have tested that my code works with both the brapi-java-server and BreedBase
  • I have create/modified unit tests to cover this change
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to documentation
  • I have run TAF: https://github.com/Breeding-Insight/taf/actions/runs/6253277974

@github-actions github-actions bot added the bug Something isn't working label Sep 19, 2023
@mlm483 mlm483 marked this pull request as ready for review September 20, 2023 19:40
@mlm483 mlm483 requested review from a team, nickpalladino and timparsons and removed request for a team September 20, 2023 19:40
@mlm483 mlm483 assigned mlm483 and timparsons and unassigned mlm483 Sep 20, 2023
Copy link
Member

@nickpalladino nickpalladino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove these lines from BrAPIGermplasmDAO?

// Need all program germplasm for processGermplasmForDisplay parents pedigree

List<BrAPIGermplasm> germplasm = getRawGermplasm(programId);

no longer need to pass all germplasm in program to processGermplasmForDisplay
@mlm483
Copy link
Contributor Author

mlm483 commented Sep 21, 2023

Can we remove these lines from BrAPIGermplasmDAO?

// Need all program germplasm for processGermplasmForDisplay parents pedigree

List<BrAPIGermplasm> germplasm = getRawGermplasm(programId);

Yes, thank you for catching this! I removed that and tested against BJTS and BreedBase.

made it consistent with how it was before changed for BI-1855
Copy link
Member

@timparsons timparsons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, just some small changes

Comment on lines +115 to +117
if(germplasm.getAdditionalInfo() != null && germplasm.getAdditionalInfo().has(GERMPLASM_RAW_PEDIGREE)
&& !(germplasm.getAdditionalInfo().get(GERMPLASM_RAW_PEDIGREE).isJsonNull())) {
germplasm.setPedigree(germplasm.getAdditionalInfo().get(GERMPLASM_RAW_PEDIGREE).getAsString());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a personal preference, but I prefer to see BrAPIAdditionalInfoFields.<constant> as it tells me when I'm reading the code where the constant is located. If I see a constant as you've changed it to, I'm expecting that constant to be defined in this class.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it back. I could go either way, do we have a convention on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but worth discussing as a group!

uuidPedigreeString = additionalInfo.has(GERMPLASM_FEMALE_PARENT_UUID) ? additionalInfo.get(GERMPLASM_FEMALE_PARENT_UUID).getAsString() : "";
// Throw a descriptive error if femaleParentUUID is absent.
if (!additionalInfo.has(GERMPLASM_FEMALE_PARENT_UUID)) {
log.debug("The germplasm data in your backing BrAPI service needs to be updated: https://github.com/Breeding-Insight/bi-api/pull/290");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be more helpful to have the log message reference the program in error instead of "your"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

uuidPedigreeString += "/" + (additionalInfo.has(GERMPLASM_MALE_PARENT_UUID) ? additionalInfo.get(GERMPLASM_MALE_PARENT_UUID).getAsString() : "");
// Throw a descriptive error if maleParentUUID is absent.
if (!additionalInfo.has(GERMPLASM_MALE_PARENT_UUID)) {
log.debug("The germplasm data in your backing BrAPI service needs to be updated: https://github.com/Breeding-Insight/bi-api/pull/290");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the above review comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

qualified constants, improved log messages
@mlm483 mlm483 requested a review from timparsons September 22, 2023 15:09
@mlm483 mlm483 merged commit 6ae61ef into develop Sep 22, 2023
@mlm483 mlm483 deleted the bug/BI-1881-mlm483 branch September 22, 2023 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants