RFC-0030: FP8 dtype introduction to PyTorch#51
Conversation
RFC-0030-native-fp8-dtype.md
Outdated
| Since fp8 data type seems to be a natural evolution of currently used fp16/bf16, to reduce computation of big DL models, it’s worth to standardize this type. Few attempts of this were done recently: | ||
|
|
||
| * Nvidia, Arm and Intel - https://arxiv.org/pdf/2209.05433.pdf | ||
| * GraphCore and AMD - https://arxiv.org/pdf/2206.02915.pdf |
There was a problem hiding this comment.
For completeness, these formats are proposed by Graphcore, AMD, and Qualcomm.
There was a problem hiding this comment.
I'll correct it when more comments are there.
|
Curious what the progress for fp8 support looks like? Thanks! |
|
@jakeh-gc , |
|
@australopitek I've been working more on the XLA side. The only activity I've seen in PyTorch was this pytorch/pytorch#97798, which didn't get merged. |
|
Hey! |
|
Hi @australopitek, in your md file, you mentioned that for E5M2 "there are many models that can be trained only with this variant". May I know what models/type of models you are referring to? Also, does your statement mean that those models would not be able to be trained with E4M3? |
|
Hi @timljj , |
|
Any updates? |
|
@maxpain, |
|
Yes I think this one is good. |
This RFC proposes adding 8-bit floating point data types to PyTorch.