System Info

Environment: Circle CI image running examples_torch

Who can help?

@muellerzr

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Due to the nature of the issue, unfortunately it cannot be reliably replicated. Examples of this occurring can be found here:

• https://app.circleci.com/pipelines/github/huggingface/transformers/49621/workflows/14c25312-58a5-4b0b-8b41-6c5bec668043/jobs/593213
• https://app.circleci.com/pipelines/gh/huggingface/transformers/49224/workflows/fbae76ab-9259-4695-bb06-475357172587/jobs/589262

Expected behavior

Occasionally, test_run_squad_no_trainer fails on CI runs, even when the PR is not touching code related to the test e.g. I would expect the output of the tested run, result, to be either deterministic or pass reliably when the are otherwise no other changes to the code.