Add scrambler for load testing#384
Merged
Merged
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #384 +/- ##
=======================================
Coverage 98.51% 98.51%
=======================================
Files 33 33
Lines 1948 1948
=======================================
Hits 1919 1919
Misses 29 29 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
bamader
approved these changes
May 21, 2025
bamader
left a comment
Collaborator
There was a problem hiding this comment.
Everything logically looks good! Just a few comments on cleanup and possible value swaps/imports, but nothing blocking merge once those get addressed.
ericbuckley
reviewed
May 21, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a script to scramble generated records for the purposes of load testing. It re-uses much of the code @bamader wrote for the scrambling the csv of NBS data, and adapts it for the json/PIIRecord data we have. To test out the code, you can run
python3 -m tests.load.scripts.scramble_data --file="./tests/load/assets/test_data.json"and it will spit out a file with scrambled records and the original from thetest_data.jsonfile.Related Issues
Closes #376
Additional Notes
If there is time after load testing to refactor this, there's a good amount of cleanup/refactoring I'd like to do to make this more adaptable to other file types, e.g., the NBS CSV and other CSVs. But, given the priority for load testing (not scrambling), this script does the job and does it quickly.
Background for why we need the script:
We want to be able to seed the test MPI during load testing with a variety of records that may or may not match each other. We plan to:
/linkendpoint with to see how the database performs with a larger MPI