[API Proposal]: Future of Numerics and AI - Provide a downlevel System.Numerics.Tensors.TensorPrimitives class #89639
Description
Future of Numerics and AI
.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond. In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, is behind the times. Developers writing .NET applications, services, and libraries currently need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.
.NET Framework Compatible APIs
While there are many reasons for developers to migrate to modern .NET and it is the goto target for many new libraries or applications, there remains many large repos that have been around for years where migration can be non-trivial. And while there are many new language and runtime features which cannot ever work on .NET Framework, there still remains a subset of APIs that would be beneficial to provide and which can help those existing codebases bridge the gap until they are able to successfully complete migration.
It is therefore proposed that an out of band System.Numerics.Tensors package be provided which provides a core set of APIs that are compatible with and appropriate for use on .NET Standard 2.0. There are also two types that would be beneficial to polyfill downlevel as part of this process, System.Half and System.MathF, which will significantly improves the usability of the libraries for common scenarios.
Provide the Tensor class
Note
The following is an extreme simplification meant to give the general premise behind the APIs being exposed. It is intentionally skimming over several concepts and deeper domain specific terminology that is not critical to cover for the design proposal to be reviewed.
At a very high overview, you have scalars, vectors, and matrices. A scalar is largely just a single value T, a vector is a set of scalars, a matrix is a set of vectors. All of these can be loosely thought of as types of tensors and they have various relationships and can be used to build up and represent higher types. That is, you could consider that T is a scalar, or a vector of length 1, or a 1x1 matrix. Likewise, you can then consider that T[] is a vector or unknown length and that T[,] represents a 2-dimensional matrix and so on. All of these can be defined as types of tensors. -- As a note, this terminology is partially where the name for std::vector<T> in C++ (and similar in other languages) comes from. This is also where the general considerations of "vectorization" when writing SIMD or SIMT arise, although they are typically over fixed-length, rather than arbitrary length.
From the mathematical perspective, many of the operations you can do on scalars also apply to the higher types. There can sometimes be limitations or other considerations that can specialize or restrict these operations, but in principle they tend to exist and work mostly the same way. It is therefore generally desirable to support many such operations more broadly. This is particularly prevalent for "element-wise" operations which could be thought of as effectively doing array.Select((x) => M(x)) and where an explicit API can often provide significant performance improvements over the naive approach.
The core set of APIs described below cover the main arithmetic and conversion operators provided for T as well as element-wise operations for the functionality exposed by System.Math/System.MathF. Some design notes include:
- Where the scalar API
Methodwould return abool, the vector API has two overloadsMethodAnyandMethodAll. - APIS will require that developers utilize
Span<T>to access these APIs, overloads takingT[]are not provided - APIs will support the destination buffer and one of the source buffers being the same.
- APIs taking more than 1 operand will support the other operands being a scalar and it indicating the same value is used for every element.
- In order to make usage of the APIs less "clunky", particularly given these are mathematical APIs where building up complex expressions may be more prevalent, it is proposed a slight deviation from the normal API signature is taken. Rather than simply returning the number of elements written to the destination span, it is proposed that the APIs return a
Span<T>instead. This would simply bedestinationsliced to the appropriate length. This still provides the required information on number of elements written, but gives the additional advantage that the result can be immediately passed into the next user of the algorithm without requiring the user to slice or do length checks themselves.
For targeting modern .NET, there will be a separate future proposal detailing a Tensor<T> type. This then matches a similar split we have for other generic types such as Vector<T> and Vector. The non-generic Tensor class holds extension methods, APIs that are only meant to support a particular set of T, and static APIs. While Tensor<T> will hold operators, instance methods, and core properties. The APIs defined for use on .NET Framework are effectively the "workhorse" APIs that Tensor<T> would then delegate to. They more closely resemble the signatures from the BLAS and LAPACK libraries, which are the industry standard baseline for Linear Algebra operations and allow tensor like functionality to be supported for arbitrary memory while allowing modern .NET to provide a type safe and friendlier way to work with such functionality that can simultaneously take advantage of newer language/runtime features, such as static virtuals in interfaces or generic math.
namespace System.Numerics.Tensors;
public static partial class Tensor
{
// Element-wise Arithmetic
public static Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
public static Span<float> Add(ReadOnlySpan<float> left, T right, Span<float> destination);
public static Span<float> Subtract(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
public static Span<float> Subtract(ReadOnlySpan<float> left, T right, Span<float> destination);
public static Span<float> Multiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
public static Span<float> Multiply(ReadOnlySpan<float> left, T right, Span<float> destination); // BLAS1: scal
public static Span<float> Divide(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
public static Span<float> Divide(ReadOnlySpan<float> left, T right, Span<float> destination);
public static Span<float> Negate(ReadOnlySpan<float> values, Span<float> destination);
public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> multiplier, Span<float> destination);
public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T multiplier, Span<float> destination);
public static Span<float> AddMultiply(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> multiplier, Span<float> destination);
public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> addend, Span<float> destination);
public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T addend, Span<float> destination); // BLAS1: axpy
public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> addend, Span<float> destination);
public static Span<float> Exp(ReadOnlySpan<float> x, Span<float> destination);
public static Span<float> Log(ReadOnlySpan<float> x, Span<float> destination);
public static Span<float> Cosh(ReadOnlySpan<float> x, Span<float> destination);
public static Span<float> Sinh(ReadOnlySpan<float> x, Span<float> destination);
public static Span<float> Tanh(ReadOnlySpan<float> x, Span<float> destination);
// Vector Arithmetic
// A measure of similarity between two non-zero vectors of an inner product space.
// It is widely used in natural language processing and information retrieval.
public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);
// A measure of distance between two points in a Euclidean space.
// It is widely used in mathematics, engineering, and machine learning.
public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);
// A mathematical operation that takes two vectors and returns a scalar.
// It is widely used in linear algebra and machine learning.
public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y); // BLAS1: dot
// A mathematical operation that takes a vector and returns a unit vector in the same direction.
// It is widely used in linear algebra and machine learning.
public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2
// A function that takes a collection of real numbers and returns a probability distribution.
// It is widely used in machine learning and deep learning.
public static float SoftMax(ReadOnlySpan<float> x);
// A function that takes a real number and returns a value between 0 and 1.
// It is widely used in machine learning and deep learning.
public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);
public static float Max(ReadOnlySpan<float> value);
public static float Min(ReadOnlySpan<float> value);
public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value); // BLAS1: iamax
public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);
public static float Sum(ReadOnlySpan<float> value);
public static float SumOfSquares(ReadOnlySpan<float> value);
public static float SumOfMagnitudes(ReadOnlySpan<float> value); // BLAS1: asum
public static float Product(ReadOnlySpan<float> value);
public static float ProductOfSums(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
public static float ProductOfDifferences(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
// Vector Conversions
public static Span<Half> ConvertToHalf(ReadOnlySpan<float> value, Span<Half> destination);
public static Span<float> ConvertToSingle(ReadOnlySpan<Half> value, Span<float> destination);
}Polyfill System.Half in Microsoft.Bcl.Half
System.Half is a core interchange type for AI scenarios, often being used to minify the storage impact for the tens of thousands to millions of data points that need to be stored. It is not, however, as frequently used for computation.
We initially exposed this type in .NET 5 purely as an interchange type and it is therefore lacking the arithmetic operators. These operators were later added in .NET 6/7 as a part of the generic math initiative. For .NET Framework, the interchange surface area should be sufficient and will follow the general guidance required for polyfills that they meet the initial shape we shipped, even if the version it shipped on is no longer in support (that is, the .NET Standard 2.0 surface area needs to remain compatible with .NET 5, even though .NET 5 is out of support).
namespace System;
public readonly struct Half : IComparable, IComparable<Half>, IEquatable<Half>, IFormattable
{
public static Half Epsilon { get; }
public static Half MaxValue { get; }
public static Half MinValue { get; }
public static Half NaN { get; }
public static Half NegativeInfinity { get; }
public static Half PositiveInfinity { get; }
public static bool operator ==(Half left, Half right);
public static bool operator !=(Half left, Half right);
public static bool operator >(Half left, Half right);
public static bool operator >=(Half left, Half right);
public static bool operator <(Half left, Half right);
public static bool operator <=(Half left, Half right);
public static explicit operator double(Half value);
public static explicit operator Half(float value);
public static explicit operator Half(double value);
public static explicit operator float(Half value);
public int CompareTo(Half other);
public int CompareTo(object? other);
public bool Equals(Half other);
public override bool Equals(object? other);
public override int GetHashCode();
public static bool IsFinite(Half value);
public static bool IsInfinity(Half value);
public static bool IsNaN(Half value);
public static bool IsNegative(Half value);
public static bool IsNegativeInfinity(Half value);
public static bool IsNormal(Half value);
public static bool IsPositiveInfinity(Half value);
public static bool IsSubnormal(Half value);
public static Half Parse(string s);
public static Half Parse(string s, NumberStyles style);
public static Half Parse(string s, IFormatProvider? provider);
public static Half Parse(string s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
public static Half Parse(ReadOnlySpan<char> s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
public override string ToString();
public string ToString(string? format);
public string ToString(string? format, IFormatProvider? provider);
public string ToString(IFormatProvider? provider);
public bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format = default, IFormatProvider? provider = default);
public static bool TryParse(string? s, out Half result);
public static bool TryParse(string? s, NumberStyles style, IFormatProvider? provider, out Half result);
public static bool TryParse(ReadOnlySpan<char> s, out Half result);
public static bool TryParse(ReadOnlySpan<char> s, NumberStyles style, IFormatProvider? provider, out Half result);
}Polyfill System.MathF in Microsoft.Bcl.MathF
System.MathF was added in .NET Core 2.0 to provide float support that had parity with the double support in System.Math. Given that float is the core computational type used in AI scenarios, many downlevel libraries currently provide their own internal wrappers around System.Math. .NET ships several such shims for its own scenarios and the proposed System.Numerics library would be no exception. As such, it would be beneficial to simply provide this functionality officially and allow such targets to remove their shims. This simplifies their experience and may give additional performance or correctness over the naive approach.
namespace System;
public static class MathF
{
public const float E = 2.71828183f;
public const float PI = 3.14159265f;
public static float Abs(float x);
public static float Acos(float x);
public static float Asin(float x);
public static float Atan(float x);
public static float Atan2(float y, float x);
public static float Cos(float x);
public static float Sin(float x);
public static float Tan(float x);
public static float Cosh(float x);
public static float Sinh(float x);
public static float Tanh(float x);
public static float Ceiling(float x);
public static float Floor(float x);
public static float Truncate(float x);
public static float Exp(float x);
public static float Log(float x);
public static float Log(float x, float y);
public static float Log10(float x);
public static float IEEERemainder(float x, float y);
public static float Max(float x, float y);
public static float Min(float x, float y);
public static float Pow(float x, float y);
public static float Round(float x);
public static float Round(float x, int digits);
public static float Round(float x, int digits, MidpointRounding mode);
public static float Round(float x, MidpointRounding mode);
public static int Sign(float x);
public static float Sqrt(float x);
}