nested_concat assumes lists are equal in size #25939
Description
System Info
I have stumbled upon a quirk while trying to figure out how to calculate custom metrics.
using a detr model for object detection and a provided trainer with a dataset with that has a smaller last batch I am missing labels in the custom metric input.
the length of batched_labels in the metric is per the length of the last batch (smaller) and isn't merged like the other fields via the
transformers/src/transformers/trainer.py
Line 3237 in 0afa507
for equal sized batches it works fine. (total eval_dataset size is 16, batched automatically into 4's)
Who can help?
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
setup similar to: https://huggingface.co/docs/transformers/main/en/tasks/object_detection#object-detection
def compute_metrics(eval_pred: EvalPrediction):
(loss_dict, logits, pred_boxes, last_hidden_state, encoder_last_hidden_state), batched_labels = eval_pred
# (loss_dict, logits, pred_boxes, last_hidden_state, encoder_last_hidden_state), labels = eval_pred
outputs = DetrObjectDetectionOutput(#loss_dict=loss_dict,
logits=torch.from_numpy(logits),
pred_boxes=torch.from_numpy(pred_boxes),
last_hidden_state=None,
decoder_hidden_states=None,
)
number_of_image_ids_in_each_indecies = [batched_label['image_id'].shape for batched_label in batched_labels]
print(number_of_image_ids_in_each_indecies)
trainer,
trainer = Trainer(
model=model,
args=training_args,
data_collator=collate_fn,
train_dataset=ds_train_augmented,
eval_dataset=ds_val_augmented,
tokenizer=image_processor,
compute_metrics=compute_metrics,
)
Expected behavior
to pad it to the last? to append it to a growing list?
anything but this that retains the data.