bpo-36515: Disable unaligned memory access in _sha3 on ARM #25927

gpshead · 2021-05-05T18:52:06Z

Contributed-By: Matthias Klose

https://bugs.python.org/issue36515

Automerge-Triggered-By: GH:tiran

tiran

Could you please try HAVE_ALIGNED_REQUIRED from our configure script?

bedevere-bot · 2021-05-05T18:56:26Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

gpshead · 2021-05-05T20:34:34Z

Could you please try HAVE_ALIGNED_REQUIRED from our configure script?

Done. Looking at configure.ac, our check for that probably passes on aarch64 because it allows them even if suboptimal. That's an optimization problem though, this should fix the crasher which is the real problem.

miss-islington · 2021-05-05T20:55:33Z

@gpshead: Status check is done, and it's a success ✅ .

miss-islington · 2021-05-05T20:55:36Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.9.
🐍🍒⛏🤖

miss-islington · 2021-05-05T20:55:38Z

Sorry @gpshead, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker da5c808fb50d34bc2e180d9481706072f33025da 3.10

…25927) Contributed-By: Matthias Klose Automerge-Triggered-By: GH:tiran (cherry picked from commit da5c808) Co-authored-by: Gregory P. Smith <greg@krypto.org>

bedevere-bot · 2021-05-05T20:55:50Z

GH-25928 is a backport of this pull request to the 3.9 branch.

miss-islington · 2021-05-05T21:41:12Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

bedevere-bot · 2021-05-05T21:41:21Z

GH-25930 is a backport of this pull request to the 3.10 branch.

Contributed-By: Matthias Klose Automerge-Triggered-By: GH:tiran (cherry picked from commit da5c808) Co-authored-by: Gregory P. Smith <greg@krypto.org>

bedevere-bot · 2021-05-05T22:39:37Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.10 has failed when building commit 3b2a45f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/681/builds/13) and take a look at the build logs.
Check if the failure is related to this commit (3b2a45f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/681/builds/13

Failed tests:

test_importlib

Failed subtests:

test_multiprocessing_pool_circular_import - test.test_importlib.test_threaded_import.ThreadedImportTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

408 tests OK.

10 slowest tests:

test_unparse: 4 min 26 sec
test_concurrent_futures: 4 min 15 sec
test_tokenize: 3 min 24 sec
test_multiprocessing_spawn: 3 min 1 sec
test_lib2to3: 2 min 47 sec
test_unicodedata: 2 min 19 sec
test_multiprocessing_forkserver: 2 min
test_capi: 1 min 57 sec
test_asyncio: 1 min 57 sec
test_pickle: 1 min 24 sec

1 test failed:
test_importlib

18 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_importlib

Total duration: 30 min 50 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/util.py", line 450, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 90 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-rkrzg6sf': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ezp2g6c2': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-uytwt3ti': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0qwoz24w': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-u_ao3s40': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-yy9ae8sd': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ysyng4c3': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ckhazxwj': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-c0_0buj1': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ki86w7rg': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-kiu2822n': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-fow1lmye': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-xry2bksl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-r6jk5zb9': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-rk9jbjng': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-jptom_14': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---

…25927) (GH-25928) Contributed-By: Matthias Klose Automerge-Triggered-By: GH:tiran (cherry picked from commit da5c808) Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Gregory P. Smith <greg@krypto.org>

Disable unaligned memory access on ARM.

Loading status checks…

7f45dae

gpshead added type-bugfix needs backport to 3.9 needs backport to 3.10 labels May 5, 2021

gpshead requested a review from tiran as a code owner May 5, 2021

bedevere-bot added the awaiting core review label May 5, 2021

the-knights-who-say-ni added the CLA signed label May 5, 2021

tiran requested changes May 5, 2021

View changes

bedevere-bot added awaiting changes and removed awaiting core review labels May 5, 2021

Use HAVE_ALIGNED_REQUIRED instead.

Loading status checks…

588aaf8

tiran approved these changes May 5, 2021

View changes

bedevere-bot added awaiting merge and removed awaiting changes labels May 5, 2021

tiran added the 🤖 automerge label May 5, 2021

bedevere-bot removed the awaiting merge label May 5, 2021

miss-islington self-assigned this May 5, 2021

bedevere-bot removed the needs backport to 3.9 label May 5, 2021

gpshead added needs backport to 3.10 and removed needs backport to 3.10 labels May 5, 2021

bedevere-bot removed the needs backport to 3.10 label May 5, 2021

python / cpython

bpo-36515: Disable unaligned memory access in _sha3 on ARM #25927

bpo-36515: Disable unaligned memory access in _sha3 on ARM #25927

gpshead commented May 5, 2021 •

edited by miss-islington

tiran left a comment

bedevere-bot commented May 5, 2021

gpshead commented May 5, 2021

miss-islington commented May 5, 2021

miss-islington commented May 5, 2021

miss-islington commented May 5, 2021

bedevere-bot commented May 5, 2021

miss-islington commented May 5, 2021

bedevere-bot commented May 5, 2021

bedevere-bot commented May 5, 2021

python / cpython

Sponsor python/cpython

bpo-36515: Disable unaligned memory access in _sha3 on ARM #25927

bpo-36515: Disable unaligned memory access in _sha3 on ARM #25927

Conversation

gpshead commented May 5, 2021 • edited by miss-islington

tiran left a comment

bedevere-bot commented May 5, 2021

gpshead commented May 5, 2021

miss-islington commented May 5, 2021

miss-islington commented May 5, 2021

miss-islington commented May 5, 2021

bedevere-bot commented May 5, 2021

miss-islington commented May 5, 2021

bedevere-bot commented May 5, 2021

bedevere-bot commented May 5, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gpshead commented May 5, 2021 •

edited by miss-islington