Using a dense pivot operation in Polars for tabularizing large datasets consumes excessive memory, as highlighted in this GitHub issue. In response, I developed a sparse version of tabularization in Polars that bypasses traditional pivoting, showing promising results. This approach was tested on a dataset comprising 5,000 patients with 9,354,128 observations and 42,266 unique codes. The implementation can be viewed here.
The memory usage of this sparse Polars method compares favorably to that of a sparse matrix approach, yet it executes significantly faster—taking only 5 seconds as opposed to 250 seconds. Here are the memory profiles for the Polars experiment: 
and for the approach using scipy sparse matrices: 
One limitation of the Polars approach is that it only aggregates codes at times they are present, omitting computations at event times where the code did not occur but is within the rolling window for the event. This could increase memory usage and computational time, casting doubt on whether this method is definitively superior. Simple aggregations like code/count could potentially be implemented using this Colab notebook method, although more complex operations such as value/sum, value/max, value/min, and other arbitrary aggregations may pose greater challenges.
Using a dense pivot operation in Polars for tabularizing large datasets consumes excessive memory, as highlighted in this GitHub issue. In response, I developed a sparse version of tabularization in Polars that bypasses traditional pivoting, showing promising results. This approach was tested on a dataset comprising 5,000 patients with 9,354,128 observations and 42,266 unique codes. The implementation can be viewed here.
The memory usage of this sparse Polars method compares favorably to that of a sparse matrix approach, yet it executes significantly faster—taking only 5 seconds as opposed to 250 seconds. Here are the memory profiles for the Polars experiment:
and for the approach using scipy sparse matrices: 
One limitation of the Polars approach is that it only aggregates codes at times they are present, omitting computations at event times where the code did not occur but is within the rolling window for the event. This could increase memory usage and computational time, casting doubt on whether this method is definitively superior. Simple aggregations like code/count could potentially be implemented using this Colab notebook method, although more complex operations such as value/sum, value/max, value/min, and other arbitrary aggregations may pose greater challenges.