Recently @xuhdev enabled bfloat16 type promotion in PyTorch with the exception of bfloat16 x float16 promotion, which I asked him to delay implementing so we could discuss it.

I propose we be consistent with JAX and promote bfloat16 x float16 to fp32:

import jax.numpy as jnp
a = jnp.array(1, dtype=jnp.float16)
b = jnp.array(1, dtype=jnp.bfloat16)
a + b
: DeviceArray(2., dtype=float32)

Other reasonable options for this choice are:

• continue to not support bfloat16 x float16 type promotion and tell users to solve this problem for themselves
• promote to fp16, which is the more precise of the two dtypes but has less dynamic range than bfloat16
• promote to bfloat16, which has the greater dynamic range but is less precise

Being consistent with JAX while not sacrificing the precision of fp16 or the dynamic range of bfloat16 seems like the best option to me.

cc @xuhdev @gchanan @nairbv