Skip to content

BI-2043 - Improve Observation Overwrite Performance#329

Merged
mlm483 merged 1 commit intorelease/0.9from
bug/BI-2043
Feb 12, 2024
Merged

BI-2043 - Improve Observation Overwrite Performance#329
mlm483 merged 1 commit intorelease/0.9from
bug/BI-2043

Conversation

@mlm483
Copy link
Contributor

@mlm483 mlm483 commented Feb 1, 2024

Description

Story: BI-2043

ExperimentProcessor::initNewBrapiData was calling fetchOrCreateObservationPIO in a nested loop for each phenotype column for each row, which in turn was calling fetchExistingObservations, an expensive method which resulted in 2 network calls to the backing BrAPI service. I was able to pull the fetchExistingObservations call outside the loops without changing the behavior of the code (except the BrAPI calls are now made just once). Performance is greatly improved.

Dependencies

bi-web: release/0.9
bi-api: bug/BI-2043

Testing

Feel free to use my test files attached below.

  1. Upload an Experiment and Observations file with lots of phenotype columns and observations.
  2. Download the experiment you just uploaded (all environments, any filetype).
  3. Edit the file you downloaded, change the Observation values for a good number of phenotypes (tip: Type a new value in the top cell of a phenotype column and drag down from the corner to the bottom of your file to quickly overwrite the whole column).
  4. Upload the edited file, make sure DeltaBreed displays the preview within 1 minute. Proceed with the overwrite, and ensure the data is saved correctly.

Checklist:

  • I have performed a self-review of my own code
  • I have tested my code and ensured it meets the acceptance criteria of the story
  • I have tested that my code works with both the brapi-java-server and BreedBase
  • I have create/modified unit tests to cover this change
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to documentation
  • I have run TAF: <please include a link to TAF run>

@mlm483 mlm483 changed the base branch from develop to release/0.9 February 1, 2024 21:45
@mlm483 mlm483 requested review from a team, davedrp and nickpalladino and removed request for a team February 1, 2024 21:45
Copy link
Member

@nickpalladino nickpalladino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to update using a file with a single row but when I tried another update with a bunch of rows it was hanging on loading prior to generating the preview table.

shawn_exp_4.xls
shawn_exp_4_update.xls
shawn_exp_4_update_2.xls

@nickpalladino
Copy link
Member

nickpalladino commented Feb 9, 2024

I was looking through some redis stuff and looked like we aren't caching observation units or observations. What do you think about caching those and the impacts on overwrite performance? Maybe a new card?

@mlm483
Copy link
Contributor Author

mlm483 commented Feb 9, 2024

I was looking through some redis stuff and looked like we aren't caching observation units or observations. What do you think about caching those and the impacts on overwrite performance? Maybe a new card?

@nickpalladino I created a v0.10 card to cache Observations and ObservationUnits.

I was able to update using a file with a single row but when I tried another update with a bunch of rows it was hanging on loading prior to generating the preview table.

@nickpalladino I uploaded those files in order, the last file took about 30 seconds to get to preview. Did it take longer than that for you?

@mlm483 mlm483 merged commit d15e597 into release/0.9 Feb 12, 2024
@mlm483 mlm483 deleted the bug/BI-2043 branch February 12, 2024 17:26
@mlm483 mlm483 restored the bug/BI-2043 branch February 12, 2024 17:26
@mlm483 mlm483 deleted the bug/BI-2043 branch October 3, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants