DOC: add BFloat16 dtype and BFloat16Tensor (#37051)

mattip · facebook-github-bot · commit c31913671c69 · 2020-05-11T12:44:46.000-07:00
Summary: Related to gh-36318 Mention `bfloat16` dtype and `BFloat16Tensor` in documentation. The real fix would be to implement cpu operations on 16-bit float `half`, and I couldn't help but notice that `torch.finfo(torch.bfloat16).xxx` crashes for `xxx in ['max', 'min', 'eps']` Pull Request resolved: #37051 Differential Revision: D21476851 Pulled By: ngimel fbshipit-source-id: fef601d3116d130d67cd3a5654077f31b699409b
diff --git a/docs/source/tensor_attributes.rst b/docs/source/tensor_attributes.rst
@@ -15,23 +15,31 @@ torch.dtype
 .. class:: torch.dtype
 
 A :class:`torch.dtype` is an object that represents the data type of a
-:class:`torch.Tensor`. PyTorch has eleven different data types:
+:class:`torch.Tensor`. PyTorch has twelve different data types:
 
-========================   ===========================================   ===========================
+========================== ===========================================   ===========================
 Data type                  dtype                                         Legacy Constructors
-========================   ===========================================   ===========================
+========================== ===========================================   ===========================
 32-bit floating point      ``torch.float32`` or ``torch.float``          ``torch.*.FloatTensor``
 64-bit floating point      ``torch.float64`` or ``torch.double``         ``torch.*.DoubleTensor``
 64-bit complex             ``torch.complex64`` or ``torch.cfloat``
-128-bit floating point     ``torch.complex128`` or ``torch.cdouble``
-16-bit floating point      ``torch.float16`` or ``torch.half``           ``torch.*.HalfTensor``
+128-bit complex            ``torch.complex128`` or ``torch.cdouble``
+16-bit floating point [1]_ ``torch.float16`` or ``torch.half``           ``torch.*.HalfTensor``
+16-bit floating point [2]_ ``torch.bfloat16``                            ``torch.*.BFloat16Tensor``
 8-bit integer (unsigned)   ``torch.uint8``                               ``torch.*.ByteTensor``
 8-bit integer (signed)     ``torch.int8``                                ``torch.*.CharTensor``
 16-bit integer (signed)    ``torch.int16`` or ``torch.short``            ``torch.*.ShortTensor``
 32-bit integer (signed)    ``torch.int32`` or ``torch.int``              ``torch.*.IntTensor``
 64-bit integer (signed)    ``torch.int64`` or ``torch.long``             ``torch.*.LongTensor``
 Boolean                    ``torch.bool``                                ``torch.*.BoolTensor``
-========================   ===========================================   ===========================
+========================== ===========================================   ===========================
+
+.. [1] Sometimes referred to as binary16: uses 1 sign, 5 exponent, and 10
+  significand bits. Useful when precision is important.
+
+.. [2] Sometimes referred to as Brain Floating Point: use 1 sign, 8 exponent and 7
+  significand bits. Useful when range is important, since it has the same
+  number of exponent bits as ``float32``
 
 To find out if a :class:`torch.dtype` is a floating point data type, the property :attr:`is_floating_point`
 can be used, which returns ``True`` if the data type is a floating point data type.
diff --git a/docs/source/tensors.rst b/docs/source/tensors.rst
@@ -8,21 +8,33 @@ torch.Tensor
 A :class:`torch.Tensor` is a multi-dimensional matrix containing elements of
 a single data type.
 
-Torch defines nine CPU tensor types and nine GPU tensor types:
+Torch defines 10 tensor types with CPU and GPU variants:
 
-========================   ===========================================   ===========================   ================================
+========================== ===========================================   ============================= ================================
 Data type                  dtype                                         CPU tensor                    GPU tensor
-========================   ===========================================   ===========================   ================================
+========================== ===========================================   ============================= ================================
 32-bit floating point      ``torch.float32`` or ``torch.float``          :class:`torch.FloatTensor`    :class:`torch.cuda.FloatTensor`
 64-bit floating point      ``torch.float64`` or ``torch.double``         :class:`torch.DoubleTensor`   :class:`torch.cuda.DoubleTensor`
-16-bit floating point      ``torch.float16`` or ``torch.half``           :class:`torch.HalfTensor`     :class:`torch.cuda.HalfTensor`
+16-bit floating point [1]_ ``torch.float16`` or ``torch.half``           :class:`torch.HalfTensor`     :class:`torch.cuda.HalfTensor`
+16-bit floating point [2]_ ``torch.bfloat16``                            :class:`torch.BFloat16Tensor` :class:`torch.cuda.BFloat16Tensor`
+32-bit complex             ``torch.complex32``
+64-bit complex             ``torch.complex64``
+128-bit complex            ``torch.complex128`` or ``torch.cdouble``
 8-bit integer (unsigned)   ``torch.uint8``                               :class:`torch.ByteTensor`     :class:`torch.cuda.ByteTensor`
 8-bit integer (signed)     ``torch.int8``                                :class:`torch.CharTensor`     :class:`torch.cuda.CharTensor`
 16-bit integer (signed)    ``torch.int16`` or ``torch.short``            :class:`torch.ShortTensor`    :class:`torch.cuda.ShortTensor`
 32-bit integer (signed)    ``torch.int32`` or ``torch.int``              :class:`torch.IntTensor`      :class:`torch.cuda.IntTensor`
 64-bit integer (signed)    ``torch.int64`` or ``torch.long``             :class:`torch.LongTensor`     :class:`torch.cuda.LongTensor`
 Boolean                    ``torch.bool``                                :class:`torch.BoolTensor`     :class:`torch.cuda.BoolTensor`
-========================   ===========================================   ===========================   ================================
+========================== ===========================================   ============================= ================================
+
+.. [1] 
+  Sometimes referred to as binary16: uses 1 sign, 5 exponent, and 10
+  significand bits. Useful when precision is important at the expense of range.
+.. [2]
+  Sometimes referred to as Brain Floating Point: use 1 sign, 8 exponent and 7
+  significand bits. Useful when range is important, since it has the same
+  number of exponent bits as ``float32``
 
 :class:`torch.Tensor` is an alias for the default tensor type (:class:`torch.FloatTensor`).
 
