Summary

We need a way to incrementally (or stream) JSON data for load testing since the file containing the 1.5 million (and eventually 5 million) records will be too large to load in memory.

Details

• Look into ijson to incrementally load
• Will need to load 1 cluster at a time to scramble
• Will need to load 100 clusters at a time (and keep track) to seed