BUG: fix `_selected_real_kind_func` return values for macOS on arm64 by dhomeier · Pull Request #21785 · numpy/numpy

dhomeier · 2022-06-17T01:42:19Z

np.f2py.crackfortran._selected_real_kind_func was only considering aarch64 as a machine value for determining Fortran's selected_real_kind values on these CPUs, but on macOS M1 platform.machine() returns arm64 (in system /usr/bin/python3 (3.8.9) and any Python 3.9-3.11 build I have tested).
This PR adds the latter to the list of processors, fixing #21765. I have also extended the return range to 33, which matches the Fortran values on arm64 and x86_64 (and thus possibly works for all higher numbers, for which –1 is returned); giving this a try here by extending the test to range(40).
Tested on macOS 12.4 with the new ports for gcc 11.3 and 12.1 documented in iains/gcc-darwin-arm64#91 and iains/gcc-darwin-arm64#92.

rgommers · 2022-06-17T04:45:42Z

Thanks @dhomeier. The ppc64le job on TravisCI doesn't seem to like this:

______________________________ TestKind.test_all _______________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x7f25100ea4c0>

    def test_all(self):

        selectedrealkind = self.module.selectedrealkind

        selectedintkind = self.module.selectedintkind

    

        for i in range(40):

            assert selectedintkind(i) == selected_int_kind(

                i

            ), f"selectedintkind({i}): expected {selected_int_kind(i)!r} but got {selectedintkind(i)!r}"

    

        for i in range(40):

>           assert selectedrealkind(i) == selected_real_kind(

                i

            ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"

E           AssertionError: selectedrealkind(32): expected 16 but got -1

E           assert -1 == 16

E            +  where -1 = <fortran function selectedrealkind>(32)

E            +  and   16 = selected_real_kind(32)

i          = 32

selectedintkind = <fortran function selectedintkind>

selectedrealkind = <fortran function selectedrealkind>

self       = <numpy.f2py.tests.test_kind.TestKind object at 0x7f25100ea4c0>

../../builds/venv/lib/python3.8/site-packages/numpy-1.24.0.dev0+275.g4fead45a2-py3.8-linux-ppc64le.egg/numpy/f2py/tests/test_kind.py:24: AssertionError

dhomeier · 2022-06-17T12:06:26Z

E AssertionError: selectedrealkind(32): expected 16 but got -1

Yes, apparently for ppc64le the limit should be 31, not 33.
So I could either limit the tests to range(32), or add another case for 'ppc'; question is of course, does this apply for ppc64le only (unlikely IMO), ppc64 or ppc in general? And what about power...?

I still tend to the second option, as it would at least correct some more cases that have just not been covered by tests before. Or might try and investigate what the kinds are on other CPUs.

rgommers · 2022-06-17T12:13:32Z

Adding ppc64le as a special case and updating something else if a new test failure is reported would be fine with me. If you can select any 64-bit PowerPC platform that works too. I don't think ppc64 is a valid platform identifier though?

dhomeier · 2022-06-17T12:33:48Z

As it only checks for startswith that would cover all PowerPC[64] platforms – probably having the same precision, see e.g. OpenFAST/openfast#502 (comment); although this would seem to fall short of the IEEE 754 specification.
The else case btw. already implies that everything else is an Intel-compatible CPU with 80bit support, which likely is not very robust either (it does not look like this test is run in the armv7-linux job).

dhomeier · 2022-06-17T18:18:23Z

As an alternative I have made an xfailing version in the spirit of it is unlikely to matter if the data type provides 31 or 33 digits precision (given that previously everything beyond 20 digits was marked as unavailable).
I still very much doubt that other arm variants, or more likely, anything but x86* and ia64, will support the real(kind=10) variant. Looking at the modification history through dc4fd64, #8659, #11725, #12672, #13095 etc. it rather appears that those had simply become the default (in 3dfe0e1), and other machines were just added to the exceptions as they popped up.
But for arm I don't understand the setup of armv7_simd_test enough to make an F90 available there for a trial run.
There is also still a ticket for this in https://github.com/numpy/numpy/projects/13#card-55307692

rgommers · 2022-06-21T08:27:34Z

I tested this on arm64 macOS, and it shows two failures:

__________________________________________________ TestKind.test_real ___________________________________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x17e464af0>

    def test_real(self):
        """
        Test (processor-dependent) `real` kind_func for real numbers
        of up to 31 digits precision (extended/quadruple).
        """
        selectedrealkind = self.module.selectedrealkind

        for i in range(32):
>           assert selectedrealkind(i) == selected_real_kind(
                i
            ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"
E           AssertionError: selectedrealkind(16): expected 16 but got -1
E           assert -1 == 16
E            +  where -1 = <fortran function selectedrealkind>(16)
E            +  and   16 = selected_real_kind(16)

i          = 16
selectedrealkind = <fortran function selectedrealkind>
self       = <numpy.f2py.tests.test_kind.TestKind object at 0x17e464af0>

numpy/f2py/tests/test_kind.py:32: AssertionError
_____________________________________________ TestKind.test_quad_precision ______________________________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x17e219280>

    @pytest.mark.xfail(platform.machine().lower().startswith("ppc"),
                       reason="Some PowerPC may not support full IEEE 754 precision")
    def test_quad_precision(self):
        """
        Test kind_func for quadruple precision [`real(16)`] of 32+ digits .
        """
        selectedrealkind = self.module.selectedrealkind

        for i in range(32, 40):
>           assert selectedrealkind(i) == selected_real_kind(
                i
            ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"
E           AssertionError: selectedrealkind(32): expected 16 but got -1
E           assert -1 == 16
E            +  where -1 = <fortran function selectedrealkind>(32)
E            +  and   16 = selected_real_kind(32)

i          = 32
selectedrealkind = <fortran function selectedrealkind>
self       = <numpy.f2py.tests.test_kind.TestKind object at 0x17e219280>

dhomeier · 2022-06-21T12:26:23Z

      assert selectedrealkind(i) == selected_real_kind(
            i
        ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"
E AssertionError: selectedrealkind(16): expected 16 but got -1
E assert -1 == 16
E + where -1 = (16)
E + and 16 = selected_real_kind(16)

i = 16

Wow. This must be failing on main and 1.23rc as well?
Which Fortran compiler version is that?

rgommers · 2022-06-21T13:09:36Z

% echo $FC
/Users/rgommers/mambaforge/envs/numpy-dev/bin/arm64-apple-darwin20.0.0-gfortran

$ gfortran --version
GNU Fortran (GCC) 11.0.1 20210403 (experimental)

There's one failure on main, and I think it's been introduced fairly recently:

___________________________________________________ TestKind.test_all ___________________________________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x11792d580>

    def test_all(self):
        selectedrealkind = self.module.selectedrealkind
        selectedintkind = self.module.selectedintkind

        for i in range(40):
            assert selectedintkind(i) == selected_int_kind(
                i
            ), f"selectedintkind({i}): expected {selected_int_kind(i)!r} but got {selectedintkind(i)!r}"

        for i in range(20):
>           assert selectedrealkind(i) == selected_real_kind(
                i
            ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"
E           AssertionError: selectedrealkind(16): expected 10 but got -1
E           assert -1 == 10
E            +  where -1 = <fortran function selectedrealkind>(16)
E            +  and   10 = selected_real_kind(16)

i          = 16
selectedintkind = <fortran function selectedintkind>
selectedrealkind = <fortran function selectedrealkind>
self       = <numpy.f2py.tests.test_kind.TestKind object at 0x11792d580>

numpy/f2py/tests/test_kind.py:24: AssertionError

dhomeier · 2022-06-21T15:48:09Z

I guess darwin-arm64 has not been tested for that long anyway.
That gcc is still a bit older than the one from https://github.com/fxcoudert/gcc/releases/tag/gcc-11.2.0-arm-20211201 I tested first, so probably rather far from a stable release. But in a way that drives home a thought from my previous comment, that there is no guarantee the compiler supports any extended or quad-precision real at all, so perhaps the default kind for p >= 16 should rather be -1. And that it does depend on the compiler as well as on the CPU – I just found out that current ifort, of all things, provides no real(kind=10) at all!

seberg · 2023-01-20T16:09:06Z

@HaoZeke any chance you have time to check this out? Since a couple of platforms have these issues including M1 setups and debian arm builds.

Will close reopen, to see retrigger CI, since this has been a while (it was all green).

HaoZeke · 2023-04-15T20:10:00Z

The arm64 addition has already made its way into main but the rest of the PR would be good to have too.

seberg · 2023-04-16T07:46:30Z

Thanks @HaoZeke in that case, it sounds like we should give this a try. I guess if something is off, it should be noisy and easy to fix.

github-actions bot added the 00 - Bug label Jun 17, 2022

dhomeier force-pushed the arm64_real_kind branch 3 times, most recently from b77ce35 to d688510 Compare June 17, 2022 16:39

HaoZeke added the component: numpy.f2py label Aug 9, 2022

seberg closed this Jan 20, 2023

seberg reopened this Jan 20, 2023

dhomeier added 2 commits April 15, 2023 20:08

BUG: include macOS arm64 machine() value in _selected_real_kind_func

7aa6d6d

DNM: test some more selected_real_kind results [skip azurepipelines]

0bee070

HaoZeke force-pushed the arm64_real_kind branch from d688510 to 0bee070 Compare April 15, 2023 20:09

seberg merged commit a578b70 into numpy:main Apr 16, 2023

HaoZeke mentioned this pull request Apr 16, 2023

MAINT: Update conftest for newer hypothesis versions #23596

Merged

charris mentioned this pull request Apr 21, 2023

MAINT: Update conftest for newer hypothesis versions #23636

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

BUG: fix `_selected_real_kind_func` return values for macOS on arm64#21785

BUG: fix `_selected_real_kind_func` return values for macOS on arm64#21785
seberg merged 2 commits intonumpy:mainfrom
dhomeier:arm64_real_kind

dhomeier commented Jun 17, 2022

Uh oh!

rgommers commented Jun 17, 2022

Uh oh!

dhomeier commented Jun 17, 2022

Uh oh!

rgommers commented Jun 17, 2022

Uh oh!

dhomeier commented Jun 17, 2022 •

edited

Loading

Uh oh!

dhomeier commented Jun 17, 2022 •

edited

Loading

Uh oh!

rgommers commented Jun 21, 2022

Uh oh!

dhomeier commented Jun 21, 2022

Uh oh!

rgommers commented Jun 21, 2022

Uh oh!

dhomeier commented Jun 21, 2022

Uh oh!

seberg commented Jan 20, 2023

Uh oh!

HaoZeke commented Apr 15, 2023

Uh oh!

seberg commented Apr 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

dhomeier commented Jun 17, 2022

Uh oh!

rgommers commented Jun 17, 2022

Uh oh!

dhomeier commented Jun 17, 2022

Uh oh!

rgommers commented Jun 17, 2022

Uh oh!

dhomeier commented Jun 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhomeier commented Jun 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Jun 21, 2022

Uh oh!

dhomeier commented Jun 21, 2022

Uh oh!

rgommers commented Jun 21, 2022

Uh oh!

dhomeier commented Jun 21, 2022

Uh oh!

seberg commented Jan 20, 2023

Uh oh!

HaoZeke commented Apr 15, 2023

Uh oh!

seberg commented Apr 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dhomeier commented Jun 17, 2022 •

edited

Loading

dhomeier commented Jun 17, 2022 •

edited

Loading