Skip to content

EbuildPhase/EbuildMetadataPhase: async check_locale#1267

Merged
gentoo-bot merged 8 commits intogentoo:masterfrom
zmedico:bug_923841_async_check_locale
Feb 21, 2024
Merged

EbuildPhase/EbuildMetadataPhase: async check_locale#1267
gentoo-bot merged 8 commits intogentoo:masterfrom
zmedico:bug_923841_async_check_locale

Conversation

@zmedico
Copy link
Copy Markdown
Member

@zmedico zmedico commented Feb 13, 2024

Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase/EbuildMetadataPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread.

Note that this series of changes causes access to the portdbapi doebuild_settings attribute to no longer be serialized via the EbuildMetadataPhase _start method. As a result, exclusive access to config instances needs to be guaranteed in some other way to avoid triggering problems (see bug 924319), such as by maintaining a config pool or by serializing access via an asyncio.Lock instance.

Bug: https://bugs.gentoo.org/923841
Bug: https://bugs.gentoo.org/924319

@zmedico zmedico marked this pull request as draft February 13, 2024 06:23
@zmedico zmedico force-pushed the bug_923841_async_check_locale branch 2 times, most recently from e1f5b70 to 40f7b68 Compare February 13, 2024 09:08
@zmedico zmedico marked this pull request as ready for review February 13, 2024 09:36
@zmedico zmedico changed the title Bug 923841 async check_locale EbuildPhase/EbuildMetadataPhase: async check_locale Feb 13, 2024
@zmedico zmedico force-pushed the bug_923841_async_check_locale branch from 40f7b68 to d44ee6b Compare February 13, 2024 10:03
@zmedico zmedico force-pushed the bug_923841_async_check_locale branch 2 times, most recently from d77392f to add913e Compare February 14, 2024 19:11
The egencache usage in ResolverPlayground that was used to trigger
bug 924319 triggered a pickling error for AuxdbTestCase.test_anydbm
with the multiprocessing spawn start method, so fix the anydbm
cache module to omit the unpicklable database object from pickled
state, and regenerate it after unpickling.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
Use a deallocate_config future to release self.settings when
it is no longer needed. It's necessary to manage concurrency
since commit c95fc64 because mutation of self.settings
is no longer limited to the EbuildMetadataPhase _start method,
where exclusive access was guaranteed within the main thread.

Add support to the isAlive() method to detect when the
EbuildMetadataPhase has started but the pid is not yet
available (due to async_check_locale usage from commit
c95fc64). This can be used to check if an
EbuildMetadataPhase instance has been successfully started
so that it can be relied upon to set the result of the
deallocate_config future.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
For EbuildMetadataPhase consumers like MetadataRegen and
depgraph, store a pool of config instances in a config_pool
list, and return instaces to the list when the deallocate_config
future is done.

Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico <zmedico@gentoo.org>
Make the ResolverPlayground _create_ebuild_manifests method
call egencache --jobs, which reliably triggers the KeyError
from bug 924319 for multiple tests:

lib/portage/tests/bin/test_doins.py::DoIns::testDoInsFallback Exception in callback EbuildMetadataPhase._async_start_done(<Task finishe...Error('EAPI')>)
handle: <Handle EbuildMetadataPhase._async_start_done(<Task finishe...Error('EAPI')>)>
Traceback (most recent call last):
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "lib/_emerge/EbuildMetadataPhase.py", line 154, in _async_start_done
    future.cancelled() or future.result()
                          ^^^^^^^^^^^^^^^
  File "lib/_emerge/EbuildMetadataPhase.py", line 130, in _async_start
    retval = portage.doebuild(
             ^^^^^^^^^^^^^^^^^
  File "lib/portage/package/ebuild/doebuild.py", line 1030, in doebuild
    doebuild_environment(
  File "lib/portage/package/ebuild/doebuild.py", line 519, in doebuild_environment
    eapi = mysettings.configdict["pkg"]["EAPI"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "lib/portage/util/__init__.py", line 1684, in __getitem__
    return UserDict.__getitem__(self, item_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/portage/cache/mappings.py", line 175, in __getitem__
    return self.data[key]
           ~~~~~~~~~^^^^^
KeyError: 'EAPI'

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
Wrap asyncio.Lock for compatibility with python 3.9 where the
deprecated loop parameter is required in order to avoid "got
Future <Future pending> attached to a different loop" errors.

The pordbapi async_aux_get method can use asyncio.Lock to
serialize access to its doebuild_settings attribute in order
to prevent issues like bug 924319.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
For the portdbapi async_aux_get method, there is not a very
good place to store a config pool, so instead use asyncio.Lock
to manage access to the portdbapi doebuild_settings attribute
when using the main event loop in the main thread. For other
threads, clone a config instance since we do not have a
thread-safe config pool. This cloning is expensive, but since
portage internals do not trigger this case, it suffices for now
(an AssertionError ensures that internals do not trigger it).
For the main event loop running in the main thread, performance
with the asyncio.Lock should not be significantly different to
performance prior to commit c95fc64, since check_locale
results are typically cached and before there was only a single
shared doebuild_settings instance with access serialized via
the EbuildMetadataPhase _start method.

Update async_aux_get callers to use asyncio.ensure_future on
the returned coroutine when needed, since it used to return
a future instead of a coroutine, and sometimes a future is
needed for add_done_callback usage.

In the portdbapi async_fetch_map method, fix a broken reference
to "future" which should have been "aux_get_future", an error
discovered while testing this patch.

Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico <zmedico@gentoo.org>
Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico <zmedico@gentoo.org>
Change config.environ() check_locale calls to async_check_locale
calls in the EbuildPhase _async_start method in order to eliminate
synchronous waiting for child processes in the main event loop
thread.

Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico <zmedico@gentoo.org>
@zmedico zmedico force-pushed the bug_923841_async_check_locale branch from add913e to 18cdb63 Compare February 21, 2024 15:28
@gentoo-bot gentoo-bot merged commit 18cdb63 into gentoo:master Feb 21, 2024
@zmedico zmedico deleted the bug_923841_async_check_locale branch February 21, 2024 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants