Conversation
Codecov Report
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
+ Coverage 79.12% 80.84% +1.71%
==========================================
Files 168 168
Lines 32303 32305 +2
==========================================
+ Hits 25560 26117 +557
+ Misses 6743 6188 -555
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Won't this fail if the model has output_attentions or output_hidden_states set to True in their configuration? the logits variable would be made of every output, not only the first one. If these are tuples they can't be used with .detach, for one, but it wouldn't even make sense to take them into account for the loss computation.
This works only if the model outputs values that should only be used to compute the loss, right?
|
Yes this doesn't work if the model has |
LysandreJik
left a comment
There was a problem hiding this comment.
Okay, that sounds good to me. I guess users could override the prediction_step anyway if they didn't want to take the full range of inputs.
|
@sgugger are you saying that the eval in Trainer-backed https://github.com/huggingface/transformers/blob/master/examples/multiple-choice/run_multiple_choice.py is not currently working? |
|
I'm pretty sure he meant |
Oh yes, makes sense now. Thanks @LysandreJik ;) |
|
Yes @LysandreJik reads my thoughts right, sorry about the typo ;-) |
* Allow multiple outputs * Formatting * Move the unwrapping before metrics * Fix typo * Add test for non-supported config options
This allows the
Trainerto properly return predictions when the model has several outputs (for instance, all modelsXxxForMultipleChoice). This should unlock progress in #7032 where the start and end logits are both required.