[SPARK-36754][SQL] ArrayIntersect handle duplicated Double.NaN and Float.NaN by AngersZhuuuu · Pull Request #33995 · apache/spark

AngersZhuuuu · 2021-09-14T14:41:26Z

What changes were proposed in this pull request?

For query

select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double)))

This returns [NaN], but it should return [].
This issue is caused by OpenHashSet can't handle Double.NaN and Float.NaN too.
In this pr fix this based on #33955

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

ArrayIntersect won't show equal NaN value

How was this patch tested?

Added UT

…at.Nan

…oat.NaN

SparkQA · 2021-09-14T15:28:39Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47768/

dongjoon-hyun · 2021-09-14T19:33:17Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

-                hsResult.add(elem)
+              if (isNaN(elem)) {
+                if (hs.containsNaN() && !hsResult.containsNaN()) {
+                  arrayBuffer += elem


For this, let's wait a little bit for the decision at the first PR.

https://github.com/apache/spark/pull/33955/files#r708570515

SparkQA · 2021-09-14T19:40:25Z

Test build #143265 has finished for PR 33995 at commit 4189d71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T15:58:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47811/

SparkQA · 2021-09-15T19:42:37Z

Test build #143308 has finished for PR 33995 at commit 64afef9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-09-16T06:07:15Z

ping @cloud-fan

SparkQA · 2021-09-17T05:57:47Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47904/

SparkQA · 2021-09-17T07:46:55Z

Test build #143396 has finished for PR 33995 at commit a9e6205.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-09-17T12:59:46Z

ping @cloud-fan

SparkQA · 2021-09-17T14:06:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47919/

SparkQA · 2021-09-17T14:16:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47919/

SparkQA · 2021-09-17T18:03:35Z

Test build #143412 has finished for PR 33995 at commit 85f9d9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class WriterBucketSpec(

…oat.NaN ### What changes were proposed in this pull request? For query ``` select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double))) ``` This returns [NaN], but it should return []. This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too. In this pr fix this based on #33955 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? ArrayIntersect won't show equal `NaN` value ### How was this patch tested? Added UT Closes #33995 from AngersZhuuuu/SPARK-36754. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2fc7f2f) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2021-09-20T08:52:58Z

thanks, merging to master/3.2/3.1/3.0!

karenfeng · 2021-09-27T20:47:12Z

For clarification @AngersZhuuuu: the PR description says:

For query
select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double)))
This returns [NaN], but it should return [].

Is this the right way around? It seems like we now correctly return [NaN], but previously incorrectly returned [].

AngersZhuuuu · 2021-09-28T08:01:25Z

For clarification @AngersZhuuuu: the PR description says:
For query
select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double)))
This returns [NaN], but it should return [].
Is this the right way around? It seems like we now correctly return [NaN], but previously incorrectly returned [].

Oh， sorry for the mistake. Correct is we should return [NaN]

…oat.NaN ### What changes were proposed in this pull request? For query ``` select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double))) ``` This returns [NaN], but it should return []. This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too. In this pr fix this based on apache#33955 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? ArrayIntersect won't show equal `NaN` value ### How was this patch tested? Added UT Closes apache#33995 from AngersZhuuuu/SPARK-36754. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2fc7f2f) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

AngersZhuuuu added 3 commits September 14, 2021 21:24

[SPARK-36742][SQL] ArrayDistinct handle duplicated Double.NaN and Flo…

1b39da5

…at.Nan

Update collectionOperations.scala

0ac9924

[SPARK-36754][SQL] ArrayIntersect handle duplicated Double.NaN and Fl…

4189d71

…oat.NaN

github-actions bot added the SQL label Sep 14, 2021

Update CollectionExpressionsSuite.scala

63763df

dongjoon-hyun reviewed Sep 14, 2021

View reviewed changes

AngersZhuuuu added 5 commits September 15, 2021 22:22

Merge branch 'master' into SPARK-36741

55f230d

Update collectionOperations.scala

7bc972f

Merge branch 'master' into SPARK-36741

f5c5452

Merge branch 'master' into SPARK-36754

2fe60cd

Update collectionOperations.scala

64afef9

AngersZhuuuu added 13 commits September 16, 2021 16:05

refactor

2478eb4

follow comment

202cf4e

Update SQLOpenHashSet.scala

5546a55

update

72870f6

follow comment

389c9fd

Update collectionOperations.scala

d8e80fc

update

d0a914c

Update SQLOpenHashSet.scala

6194de6

Merge branch 'SPARK-36741' into SPARK-36754

df380c3

update

217934b

Update collectionOperations.scala

f97b838

Update collectionOperations.scala

ac52229

Update collectionOperations.scala

a9e6205

AngersZhuuuu marked this pull request as draft September 17, 2021 05:43

Merge branch 'master' into SPARK-36754

85f9d9d

AngersZhuuuu marked this pull request as ready for review September 17, 2021 12:54

cloud-fan approved these changes Sep 17, 2021

View reviewed changes

cloud-fan approved these changes Sep 20, 2021

View reviewed changes

cloud-fan closed this in 2fc7f2f Sep 20, 2021

Conversation

AngersZhuuuu commented Sep 14, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

dongjoon-hyun Sep 14, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

AngersZhuuuu commented Sep 16, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

AngersZhuuuu commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

cloud-fan commented Sep 20, 2021

Uh oh!

karenfeng commented Sep 27, 2021

Uh oh!

AngersZhuuuu commented Sep 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants