System Info

• transformers version: 4.25.0.dev0
• Platform: Linux-4.14.81.bm.22-amd64-x86_64-with-glibc2.17
• Python version: 3.8.13
• Huggingface_hub version: 0.10.0
• PyTorch version (GPU?): 1.12.1+cu116 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@sgugger, @patil-suraj

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I was running the official glue example script transformers/examples/pytorch/text-classification/run_glue_no_trainer.py on STS-B task.

export TASK_NAME=stsb
python run_glue_no_trainer.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --max_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/

The training went well, but on saving the results it raised the error below:

Configuration saved in /tmp/stsb/config.json
Model weights saved in /tmp/stsb/pytorch_model.bin
tokenizer config file saved in /tmp/stsb/tokenizer_config.json
Special tokens file saved in /tmp/stsb/special_tokens_map.json
Traceback (most recent call last):
  File "run_glue_no_trainer.py", line 633, in <module>
    main()
  File "run_glue_no_trainer.py", line 629, in main
    json.dump({"eval_accuracy": eval_metric["accuracy"]}, f)
KeyError: 'accuracy'

Expected behavior

Some of the glue tasks (STS-B, CoLA) don't use "accuracy" as metric. Maybe need to check the metric keys before accessing eval_metric.

BTW, I have noticed that this block of code also appears in lots of other example scripts like multiple-choice, semantic-segmentation, etc. I'm not sure whether those scripts have the same issue.