BUG: f2py cannot handle non-ascii characters in comments since 1.24 #22871
Description
Describe the issue:
F2py works fine in version 1.23 when an utf8-encoded Fortran file has non-ascii characters in comments, but gives an error in 1.24. The error message recommends installing chardet but it does not seem to fix the issue.
Reproduce the code example:
$ cat test.f90
subroutine foo(x)
real(8), intent(in) :: x
print*, x ! Écrit à l'écran la valeur de x
end subroutine
$ conda create -n numpy1.23 numpy=1.23
[...]
$ conda create -n numpy1.24 numpy=1.24
[...]
$ conda activate numpy1.23
$ f2py -m test test.f90
Reading fortran codes...
Reading file 'test.f90' (format:free)
Post-processing...
Block: test
Block: foo
Post-processing (stage 2)...
Building modules...
Building module "test"...
Generating possibly empty wrappers"
Maybe empty "test-f2pywrappers.f"
Constructing wrapper function "foo"...
foo(x)
Wrote C/API module "test" to file "./testmodule.c"
$ conda activate numpy1.24
$ f2py -m test test.f90
Reading fortran codes...
Traceback (most recent call last):
[...]
Error message:
Traceback (most recent call last):
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 393, in readfortrancode
l = fin.readline()
^^^^^^^^^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/fileinput.py", line 292, in readline
line = self._readline()
^^^^^^^^^^^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/fileinput.py", line 372, in _readline
return self._readline()
^^^^^^^^^^^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 59: ordinal not in range(128)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/mancellin/mambaforge/envs/numpy1.24/bin/f2py", line 10, in <module>
sys.exit(main())
^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 704, in main
run_main(sys.argv[1:])
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 441, in run_main
postlist = callcrackfortran(files, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 342, in callcrackfortran
postlist = crackfortran.crackfortran(files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 3298, in crackfortran
readfortrancode(files, crackline)
File "/opt/mancellin/mambaforge/envs/numpy1.24/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 395, in readfortrancode
raise Exception(
Exception: readfortrancode: reading test.f90#0 failed with
'ascii' codec can't decode byte 0xc3 in position 59: ordinal not in range(128).
It is likely that installing chardet package will help f2py determine the input file encoding correctly.
Runtime information:
import sys, numpy; print(numpy.version); print(sys.version)
1.24.0
3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:40) [GCC 10.4.0]
Context for the issue:
I can't build the Fortran dependency of my package with Numpy 1.24. See also capytaine/capytaine#273