What does this PR do?

This PR set scale_embedding=False in TFSpeech2TextModelTester to avoid inputs_embeds and the PT/TF difference being scaled by 4 - the objective is to keep a low tolerance 1e-5 in the PT/TF test, as we see several times this is a strong safe guard!

(This is not a real bug. It's also similar to #15684, where we got larger differences between PT/TF simply because the model weights are initialized with larger values).

TF: @gante @Rocketknight1
Speech: @patrickvonplaten

More context

Set scale_embedding to False in some TF tests.

Current Speech2TextConfig has default scale_embedding=True. Therefore we have

• self.embed_scale = tf.math.sqrt(float(embed_dim)).
• The tests for speech_to_text has hidden_size=16, and therefore inputs_embeds will be scaled by 4.

Since inputs_embeds here is obtained by some (conv.) layer instead of via look-up table, it contains some tiny difference between PT/TF. This difference is scaled by 4 through self.embed_scale.

This makes TFSpeech2TextModel the only one model that will fail the aggressive PT/TF test introduced in #15839 (with low tolerance 1e-5). More precisely, the output tensors failed are encoder_hidden_states_0 and encoder_hidden_states_1.

Results

With this PR, the tolerance 1e-5 works for all TF models' test_pt_tf_model_equivalence (in #15839), both on GPU / CPU, tested 100 times per model.