Skip to content

Optimize isnan (2-5x faster) #12889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
alamb merged 3 commits into from
Oct 15, 2024
Merged

Optimize isnan (2-5x faster) #12889

alamb merged 3 commits into from
Oct 15, 2024

Conversation

@simonvandel
Copy link
Contributor

@simonvandel simonvandel commented Oct 12, 2024 •
edited
Loading

Which issue does this PR close?

Closes #.

Rationale for this change

Apply same idea from #12881 to isnan.

What changes are included in this PR?

  • Add bench for is_nan
  • Optimize implementation

Are these changes tested?

Yes, with existing tests.

Are there any user-facing changes?

Yes, faster is_nan function.

isnan f32 array: 1024   time:   [794.46 ns 795.95 ns 797.51 ns]
                        change: [-60.752% -60.453% -60.230%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

isnan f64 array: 1024   time:   [694.70 ns 698.25 ns 702.33 ns]
                        change: [-66.246% -66.029% -65.767%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe

isnan f32 array: 4096   time:   [2.5207 µs 2.5282 µs 2.5358 µs]
                        change: [-64.245% -64.126% -64.015%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

isnan f64 array: 4096   time:   [2.0779 µs 2.0812 µs 2.0847 µs]
                        change: [-74.140% -74.042% -73.950%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

isnan f32 array: 8192   time:   [4.9547 µs 4.9730 µs 4.9971 µs]
                        change: [-74.219% -74.059% -73.933%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

isnan f64 array: 8192   time:   [4.0478 µs 4.0562 µs 4.0652 µs]
                        change: [-82.993% -82.947% -82.900%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates functions Changes to functions implementation labels Oct 12, 2024
jonahgao
jonahgao reviewed Oct 12, 2024
View reviewed changes
datafusion/physical-expr/src/math_expressions.rs
use datafusion_common::Result;

/// Isnan SQL function
pub fn isnan(args: &[ArrayRef]) -> Result<ArrayRef> {
Copy link
Member

@jonahgao jonahgao Oct 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this function is unused, and the real one is in datafusion/functions/src/math/nans.rs.

Copy link
Contributor Author

@simonvandel simonvandel Oct 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely right.
Changing the right function does indeed give a speed-up.

I have left out the cleanup of the isnan of math_expressions.rs out of this PR now.

Copy link
Member

@jonahgao jonahgao Oct 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #12917 to remove math_expressions.rs

@github-actions github-actions bot removed the physical-expr Changes to the physical-expr crates label Oct 12, 2024
@simonvandel simonvandel changed the title Cleanup isnan using BooleanArray::from_unary Optimize isnan (2-5x faster) Oct 12, 2024
jonahgao
jonahgao approved these changes Oct 14, 2024
View reviewed changes
Copy link
Member

@jonahgao jonahgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @simonvandel

@jonahgao jonahgao mentioned this pull request Oct 14, 2024
Merged
@alamb
Copy link
Contributor

alamb commented Oct 14, 2024

Thanks @simonvandel and @jonahgao -- I merged this PR up to resolve a conflict

@alamb alamb mentioned this pull request Oct 14, 2024
Merged
@alamb alamb merged commit 1936774 into apache:main Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jonahgao jonahgao jonahgao approved these changes

Assignees

No one assigned

Labels

functions Changes to functions implementation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@simonvandel @alamb @jonahgao