ICYI, here's what I was doing to test that this works (at least for loading RSMTool modeler objects, so I haven't tested the backwards compatibility yet):

❯ ../../../env/bin/rsmtool config_rsmtool.json -f
Output directory: /Users/mmulholland/Documents/rsmtool/rsmtool/examples/rsmtool
WARNING: /Users/mmulholland/Documents/rsmtool/rsmtool/examples/rsmtool already contains a non-empty 'output' directory. Th
e generated report might contain unexpected information from a previous experiment.
Saving configuration file.
Reading in all data from files.
Preprocessing all features.
Pre-processing training and test set features
Saving training and test set data to disk.
Training LinearRegression model.
Running analyses on training set.
Generating training and test set predictions.
Processing train set predictions.
Processing test set predictions.
Scaling the coefficients and saving them to disk
Running prediction analyses.
Starting report generation.
Merging sections
Exporting HTML
WARNING: Alternative text is missing on 8 image(s).
❯ ../../../env/bin/ipython
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:41) [Clang 15.0.7 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import rsmtool

In [2]: m = rsmtool.rsmtool.Modeler.load_from_file("output/ASAP2.model")

In [3]: rsmtool.rsmpredict.fast_predict({"DISCOURSE": 4.93806460126142, "ORGANIZATION": -0.0846667513334603, "GRAMMAR": -0.316793975540994, "MECHANICS": 4.65591397849462}, m)
Out[3]:
{'raw': 3.4671587960793446,
 'scale': 3.4876896893346814,
 'raw_trim': 3.4671587960793446,
 'raw_trim_round': 3,
 'scale_trim': 3.4876896893346814,
 'scale_trim_round': 3}

Hmm, One thing that immediately occurs to me is this – saving a modeler instead of a SKLL model – would be a completely backwards-incompatible change.

At the very least, we would need to modify how rsmpredict and rsmexplain work at the very least since they need access to the model. Also, we would need to change notebook sections that try to inspect the model object. Probably lots of stuff in the documentation too.

Oh, yeah, rsmpredict is being modified now. I started with fast_predict right now as a POC. But the idea would be to have logic that tries to load a Learner object and, if that fails, load a Modeler object (or vice versa).

Ah, okay, Yeah, we can probably just modify Modeler.load_from_file() to return a Modeler instance irrespective of whether the file on disk contains a serialized SKLL model or a serialized modeler. And then the rest should just work?

I'm going to push a change that should make that aspect a little bit clearer.

@desilinguist I think this is ready for the part you were going to handle (converting unit test models).

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.06% 🎉

Comparison is base (2ed3733) 93.21% compared to head (5924248) 93.28%.

@@            Coverage Diff             @@
##             main     #632      +/-   ##
==========================================
+ Coverage   93.21%   93.28%   +0.06%     
==========================================
  Files          33       33              
  Lines        4921     4972      +51     
==========================================
+ Hits         4587     4638      +51     
  Misses        334      334              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Thanks, @blongwill! I believe that I have addressed your concerns. Please take another look!

@mulhod I just updated all the relevant model files. I just realized we haven't yet looked at the documentation. I think there may be few places where the .model file may be described (the outputs generated by rsmtool). Can you please take a look and update all the relevant places?

@desilinguist I have done some grepping around in the documentation. I found one instance so far that needed to be changed. I'll continue digging a bit, though. My first iteration was looking specifically for pieces that touched on literally a .model file output. But, perhaps search for SKLL, etc., would be a good idea too.

Thanks, @blongwill! I believe that I have addressed your concerns. Please take another look!

Looks great Matt! Thanks for making those updates!