What changes were proposed in this pull request?

Modify the calculation formula of Pearson correlation.

Why are the changes needed?

Spark uses the formula ck / sqrt(xMk * yMk) to calculate the Pearson Correlation Coefficient. If xMk and yMk are very small, it can lead to double multiplication overflow, resulting in a denominator of 0. This leads to an Infinity result in the calculation.

For example, when calculating the correlation for the same columns a and b in the following table, the result will be Infinity, but the correlation for identical columns should be 1.0 instead.

scala> val tinyDouble = Seq(1e-200, 1e-200, 1e-100)
tinyDouble: Seq[Double] = List(1.0E-200, 1.0E-200, 1.0E-100)

scala> val df3 = tinyDouble.zip(tinyDouble).toDF("a", "b")
df3: org.apache.spark.sql.DataFrame = [a: double, b: double]

scala> df3.stat.corr("a", "b", "pearson")
res0: Double = Infinity

Modifying the formula to ck / sqrt(xMk) / sqrt(yMk) can indeed solve this issue and improve the stability of the calculation. The benefit of this modification is that it splits the square root of the denominator into two parts: sqrt(xMk) and sqrt(yMk). This helps avoid multiplication overflow or cases where the product of extremely small values becomes zero.

Does this PR introduce any user-facing change?

The precision of the decimal last digit in some results may change. For example, the corr result for two columns that used to be equal to 1.0 may now become 0.9999999999999999.

How was this patch tested?

Add a unit test case in DataFrameStatSuite.

Was this patch authored or co-authored using generative AI tooling?

No