Description

Optimizing memory usage by ~70% in tuning calculations using iterators.

Related Issues

closes #456

Additional Notes

• I couple of dataclasses were added to schemas/tuning to help with passing data between prob_calc and base. TuningPair and TuningProbabilities. The former is particularly helpful with a shift to iterators, as it's no longer possible to return values from the mpi_service queries.
• AlgorithmPass.resolved_label was added to help with type checks, as that guarantees a string will be returned (AlgorithmPass.label, could be null)
• Feature was configured to be "frozen" (meaning once a Feature object is instantiated, it can't be changed). This is useful as we can now use them as keys in our log-odds dicts, rather than the feature string values as the keys. This reduced the number of time we need to convert back and forth between strings and Features.
• tuning/base::tune was split to use 2 sub-functions, run_log_odds and run_rms. With an iterators version, we need to gather true-match and non-match pairs from the database twice. Separating the calculations into two functions makes it clearer as to where the pairs are being used.
• The looping constructs in calculate_and_sort_tuning_scores we essentially reversed. Previously we looped over all the algorithm passes, then we looped over the pairs. Now that we have iterators, that doesn't work, as we can only access the pair once. To accomplish this, we now loop over the pairs, then loop over the algorithm passes. _score_pairs_in_class was renamed to _score_records_in_pair as now it just performs the value on 1 pair over N passes.

Performance Notes

Results from running tuning job tests with a 500k cluster using the default params.

main stats

runtime: 42s
max memory: 665MB

PR-469 stats

runtime: 44s
max memory: 204MB
<--------------------- REMOVE THE LINES BELOW BEFORE MERGING --------------------->

Checklist

Please review and complete the following checklist before submitting your pull request:

• I have ensured that the pull request is of a manageable size, allowing it to be reviewed within a single session.
• I have reviewed my changes to ensure they are clear, concise, and well-documented.
• I have updated the documentation, if applicable.
• I have added or updated test cases to cover my changes, if applicable.
• I have minimized the number of reviewers to include only those essential for the review.

Checklist for Reviewers

Please review and complete the following checklist during the review process:

• The code follows best practices and conventions.
• The changes implement the desired functionality or fix the reported issue.
• The tests cover the new changes and pass successfully.
• Any potential edge cases or error scenarios have been considered.