Skip to content

[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN and Float.NaN#34003

Closed
AngersZhuuuu wants to merge 4 commits intoapache:masterfrom
AngersZhuuuu:SPARK-36702-FOLLOWUP
Closed

[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN and Float.NaN#34003
AngersZhuuuu wants to merge 4 commits intoapache:masterfrom
AngersZhuuuu:SPARK-36702-FOLLOWUP

Conversation

@AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented Sep 15, 2021

What changes were proposed in this pull request?

According to #33955 (comment) use normalized NaN

Why are the changes needed?

Use normalized NaN for duplicated NaN value

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exiting UT

@github-actions github-actions bot added the SQL label Sep 15, 2021
@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Sep 15, 2021

ping @dongjoon-hyun @cloud-fan

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47797/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47797/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143296 has finished for PR 34003 at commit 13deceb.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47799/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47800/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47801/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47800/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47801/

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143294 has finished for PR 34003 at commit 1c7ad8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Sep 15, 2021

ping @cloud-fan

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143297 has finished for PR 34003 at commit e467ff3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 15, 2021

Test build #143298 has finished for PR 34003 at commit 73e3145.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented Sep 15, 2021

thanks, merging to master/3.2/3.1!

@cloud-fan cloud-fan closed this in 6380859 Sep 15, 2021
cloud-fan pushed a commit that referenced this pull request Sep 15, 2021
@AngersZhuuuu @cloud-fan
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
e641556 
…and Float.NaN

### What changes were proposed in this pull request?
According to #33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes #34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan pushed a commit that referenced this pull request Sep 15, 2021
@AngersZhuuuu @cloud-fan
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
4a266f0 
…and Float.NaN

### What changes were proposed in this pull request?
According to #33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes #34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Sep 15, 2021

+1, LGTM. Thank you!

fishcus pushed a commit to fishcus/spark that referenced this pull request Jan 12, 2022
@AngersZhuuuu @fishcus
[SPARK-36702][SQL][FOLLOWUP] ArrayUnion handle duplicated Double.NaN …
29b38d3 
…and Float.NaN

### What changes were proposed in this pull request?
According to apache#33955 (comment) use normalized  NaN

### Why are the changes needed?
Use normalized NaN for duplicated NaN value

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Exiting UT

Closes apache#34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 6380859)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AngersZhuuuu @SparkQA @cloud-fan @dongjoon-hyun