Conversation
Make the prediction on the batch in one call of the NN model
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #677 +/- ##
==========================================
+ Coverage 99.56% 99.57% +0.01%
==========================================
Files 87 87
Lines 6158 6157 -1
==========================================
Hits 6131 6131
+ Misses 27 26 -1
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
osma
left a comment
There was a problem hiding this comment.
Looks good, I gave a couple of suggestions for improvement but they could also be done separately.
|
Kudos, SonarCloud Quality Gate passed!
|
|
I rerun the NN ensemble suggest and eval timings and updated the results table after the last change: it seemed to give a few seconds improvement more. |
This adds the
_suggest_batchmethod to ensemble backends.In all ensemble backends (simple, PAV, NN) the suggestions from the source projects are fetched by using batched suggest calls (this is on project level, so whether a backend actually uses batched suggest depends on the backend).
In case of NN ensemble the prediction (now via
_merge_hit_sets_from_sources) is performed on the whole batch of the base suggestions in one call of the NN model. The prediction is made using the model's__call__()method instead ofpredict()as it is the recommended way for "small numbers of inputs that fit in one batch" and it should offer better performance; this should also fix #674.Also the EnsembleOptimizer is made to use batched suggest. Quickly testing the
hyperoptcommand on an ensemble project gives very similar weights and best NDCG scores before and after this PR.This PR improves somewhat the performance of NN ensemble suggest functionality while the results remain (only nearly?) identical (I think there were small differences in suggestion scores on my laptop). I think most of the increase comes from using
__call__of the model instead ofpredict.The below results are from runs at kj-kk using the current Finto AI YSO NN ensemble model (having MLLM, fastText and Omikuji base projects).
suggest
Targeting 6 times
tests/corpora/archaeology/fulltext/*.txt:eval
Targeting 200 documents from
kirjaesittelyt2021/yso/fin/test:With 1 job
With 4 jobs