Skip to content

Catch other error variant of missing icu dependency#18261

Closed
MadLittleMods wants to merge 2 commits intodevelopfrom
madlittlemods/optional-icu
Closed

Catch other error variant of missing icu dependency#18261
MadLittleMods wants to merge 2 commits intodevelopfrom
madlittlemods/optional-icu

Conversation

@MadLittleMods
Copy link
Contributor

@MadLittleMods MadLittleMods commented Mar 22, 2025

Catch other error variant of missing icu dependency

$ poetry run synapse_homeserver --config-path homeserver.yaml
The "poetry.dev-dependencies" section is deprecated and will be removed in a future version. Use "poetry.group.dev.dependencies" instead.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys; from importlib import import_module; sys.argv = ['/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.13/bin/synapse_homeserver', '--config-path', 'homeserver.yaml']; sys.exit(import_module('synapse.app.homeserver').main())
                                                                                                                                                                                                               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/eric/Documents/github/element/synapse/synapse/app/homeserver.py", line 43, in <module>
    from synapse.app import _base
  File "/home/eric/Documents/github/element/synapse/synapse/app/_base.py", line 71, in <module>
    from synapse.events.auto_accept_invites import InviteAutoAccepter
  File "/home/eric/Documents/github/element/synapse/synapse/events/auto_accept_invites.py", line 28, in <module>
    from synapse.module_api import EventBase, ModuleApi, run_as_background_process
  File "/home/eric/Documents/github/element/synapse/synapse/module_api/__init__.py", line 60, in <module>
    from synapse.handlers.auth import (
    ...<7 lines>...
    )
  File "/home/eric/Documents/github/element/synapse/synapse/handlers/auth.py", line 63, in <module>
    from synapse.api.ratelimiting import Ratelimiter
  File "/home/eric/Documents/github/element/synapse/synapse/api/ratelimiting.py", line 28, in <module>
    from synapse.storage.databases.main import DataStore
  File "/home/eric/Documents/github/element/synapse/synapse/storage/__init__.py", line 40, in <module>
    from synapse.storage.databases import Databases
  File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/__init__.py", line 27, in <module>
    from synapse.storage.databases.main.events import PersistEventsStore
  File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/main/__init__.py", line 86, in <module>
    from .user_directory import UserDirectoryStore
  File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/main/user_directory.py", line 42, in <module>
    import icu
  File "/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.13/lib/python3.13/site-packages/icu/__init__.py", line 37, in <module>
    from ._icu_ import *
ImportError: libicui18n.so.75: cannot open shared object file: No such file or directory

I'm not sure why I ran into this error as I think I have icu installed on my system:

Manjaro Linux (arch-based):

$ pamac search icu --installed
lib32-icu  76.1-1                                                                                                     multilib
    International Components for Unicode library (32 bit)
harfbuzz-icu  10.4.0-1                                                                                                   extra
    OpenType text shaping engine - ICU integration
icu  76.1-1                                                                                                               core
    International Components for Unicode library

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct
    (run the linters)

@MadLittleMods MadLittleMods marked this pull request as ready for review March 24, 2025 14:38
@MadLittleMods MadLittleMods requested a review from a team as a code owner March 24, 2025 14:38
@reivilibre
Copy link
Contributor

Not a big fan of merging this; this will conceal errors when you thought you had ICU installed but actually it's not installed properly. (Ironically such as yours)

The problem in your case is that the underlying ICU library on your system has been updated to a new version, but you need to rebuild the Python package, because the ICU library has changed major version and is not ABI compatible with the previous one.

To do so, you might have to clear your Python wheel cache and then reinstall the PyICU package.

@MadLittleMods
Copy link
Contributor Author

These kinds of problems shouldn't exist. The fact that Poetry can't figure out the right thing to do is such a big fail. Perhaps, it doesn't have enough info given how tenuous dynamically linked libraries are as far as I can tell.

I can't seem to get the PyICU (icu) package to rebuild and get into the correct state. I'm using Poetry. I don't see anything in our development docs about how to manage the wheels specifically.

Commands I tried:

$ poetry cache clear --all .
$ poetry env list
$ poetry env remove matrix-synapse-xCtC9ulO-py3.13
$ poetry install --extras all

Tried removing them all:

$ poetry env remove --all
$ poetry install --extras all

$ poetry env remove --all
$ poetry install --extras all --no-cache
$ rm -rf ./matrix_synapse.egg-info

Since I'm not working on or even using search, I don't really care whether I have the icu feature in Synapse. This is just the from-source development copy that I start-up sometimes to manually poke and prod.


Other posts that mention libicui18n.so in the Manjaro stable update forum posts:

@reivilibre
Copy link
Contributor

Yeah it's super painful. And definitely there's a fail somewhere in the package management system that this doesn't get figured out (but I refer back to many statements of 'Python Packaging is broken').

I think in the past for me, clearing ~/.cache/pypoetry was the answer, though sometimes I think ~/.cache/pip also needed to go.

@MadLittleMods
Copy link
Contributor Author

MadLittleMods commented Mar 26, 2025

I think in the past for me, clearing ~/.cache/pypoetry was the answer, though sometimes I think ~/.cache/pip also needed to go.

Great tip! Works perfectly now 🌄 Thank you @reivilibre! I'll add this to our docs ⏩

$ rm -rf ~/.cache/pypoetry
$ poetry env remove --all
$ poetry install --extras all

# Works!
$ poetry run synapse_homeserver --config-path homeserver.yaml

Poetry is crazy for their built-in commands not handling this. It looks like they store wheels in ~/.cache/pypoetry/artifacts but that doesn't get cleared from either of the following commands. So the manual rm -rf ~/.cache/pypoetry actually helps to clear away the wheel artifacts so we get to rebuild them again.

  • poetry cache clear --all . -> Only clears ~/.cache/pypoetry/cache
  • poetry env remove --all -> Only clears ~/.cache/pypoetry/virtualenvs

@MadLittleMods
Copy link
Contributor Author

Created #18283 to add this tip to our docs ✅

Created python-poetry/poetry#10304 to add this to the Poetry issue tracker ⏩

MadLittleMods added a commit that referenced this pull request Mar 26, 2025
As shared by @reivilibre,
#18261 (comment)

Relevant Poetry issue around how this should be handled by them:
python-poetry/poetry#10304
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants