Add W&B logging #761
Add W&B logging #761
Conversation
desilinguist
left a comment
There was a problem hiding this comment.
@tamarl08 this looks pretty elegant for the most part! I like that you added what you needed to the learner result dictionaries and then just use those for logging most of the time.
I made some minor suggestions based on the code itself but without testing what the various W&B outputs look like. I will do that next.
|
@tamarl08 any idea why gitlab is failing and Azure is passing? |
|
Will check. I didn't expect anything to pass! |
|
I am seeing things like this in my run: 
I am also seeing charts with a single data point (e.g., the accuracy values etc.) which aren't useful. I wonder if we can tell W&B to create relevant charts from the summary file on the fly? Let's sit down next week and try to figure out what things are actually valuable to log and how to make a run appear useful right when someone opens it. |
|
This is why I tried to log to summary instead, but I couldn't always get rid of these charts. I'll try some more and let's talk next week. |
…in tests and fix some tests. Some code fixes and improvement.
|
Hello @tamarl08! Thanks for updating this PR.
Comment last updated at 2024-01-26 21:47:38 UTC |
|
@desilinguist @mulhod @damien2012eng @Frost45 See the updated evaluation logging here - look only at the most recent run. Changes to tests are due to: a change I made to the job name/output file names; data added to the result dict; bug fixes in some tests. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #761 +/- ##
==========================================
+ Coverage 95.33% 95.44% +0.11%
==========================================
Files 30 30
Lines 3598 3688 +90
==========================================
+ Hits 3430 3520 +90
Misses 168 168 ☔ View full report in Codecov by Sentry. |
desilinguist
left a comment
There was a problem hiding this comment.
Looks really good! I ran some test experiments and W&B logging now looks much better than before. I made some minor suggestions. I am assuming that you are going to add documentation in a separate MR?
|
Thanks for the review @desilinguist! All suggestions applied. I did add a section in the docs, and also a comment about the output file names. Will add more examples later. |
desilinguist
left a comment
There was a problem hiding this comment.
Just some minor tweaks for the documentation.
Co-authored-by: Nitin Madnani <nmadnani@ets.org>
This PR adds W&B logging for all task types.
Changes include:
Examples:
train: currently only logs the configuration, model file path and train set size (in run summary).
predict: logs configuration, train and test sizes and predictions file
learning curve: logs config, plots and some raw data from the learning curve results (in run summary)
evaluate: logs config and all evaluation metrics and confusion matrix as a chart.
cross_validate: logs evaluation per fold + average of folds
TODO: