Non UTF-8 encoding in terminal breaks poetry commands #8603
Description
- Poetry version: 1.6.1 and 1.5.0
- Python version: 3.11.2
- OS version and name: Raspberry PI OS (
Linux raspberrypi 6.1.0-rpi4-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.54-1+rpt2 (2023-10-05) aarch64 GNU/Linux) - pyproject.toml: Might be any valid and simple project.
- I am on the latest stable Poetry version, installed using a recommended method.
- I have searched the issues of this repo and believe that this is not a duplicate.
- [ x I have consulted the FAQ and blog for any relevant entries or release notes.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
(Maybe similar issue: #6784)
Hi 😄
I was developing a project for a Raspberry Pi for with a recent Raspberry Pi OS (installed last week - see above). On my development machine (Ubuntu 20.04 LTS) everything worked fine, I could use poetry to add and remove dependencies etc. poetry install did work too.
However, once I migrated it to the Raspberry Pi, I couldn't do poetry install and it silently exited without installing anything. Because the default poetry installation coming from the apt sources is too old (v1.3.2), I used a new virtualenv to get the newest Poetry (v1.6.1).
python3 -m venv venv
source venv/bin/activate
python3 -m pip install poetry
poetry --version # yields: Poetry (version 1.6.1)That seems to be working.
However, when I now cd into my project and do poetry install -vvv, the output is just:
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 11 installs, 2 updates, 0 removals, 4 skipped
Nothing happens, but poetry errors out with exit-code 1.
When I manually attempt to add a dependency again, e.g. poetry add click -vvv, I finally get a traceback:
Using virtualenv: /home/pi/dev/critical-vibration-experiment2/venv
The following packages are already present in the pyproject.toml and will be skipped:
Stack trace:
13 venv/lib/python3.11/site-packages/cleo/application.py:327 in run
325|
326| try:
> 327| exit_code = self._run(io)
328| except BrokenPipeError:
329| # If we are piped to another process, it may close early and send a
12 venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
188| self._load_plugins(io)
189|
> 190| exit_code: int = super()._run(io)
191| return exit_code
192|
11 venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
429| io.input.interactive(interactive)
430|
> 431| exit_code = self._run_command(command, io)
432| self._running_command = None
433|
10 venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
471|
472| if error is not None:
> 473| raise error
474|
475| return terminate_event.exit_code
9 venv/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
455|
456| if command_event.command_should_run():
> 457| exit_code = command.run(io)
458| else:
459| exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
8 venv/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
115| io.input.validate()
116|
> 117| return self.execute(io) or 0
118|
119| def merge_application_definition(self, merge_args: bool = True) -> None:
7 venv/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
59|
60| try:
> 61| return self.handle()
62| except KeyboardInterrupt:
63| return 1
6 venv/lib/python3.11/site-packages/poetry/console/commands/add.py:152 in handle
150|
151| if existing_packages:
> 152| self.notify_about_existing_packages(existing_packages)
153|
154| packages = [name for name in packages if name not in existing_packages]
5 venv/lib/python3.11/site-packages/poetry/console/commands/add.py:301 in notify_about_existing_packages
299| )
300| for name in existing_packages:
> 301| self.line(f" \u2022 {name}")
302| self.line(self._hint_update_packages)
303|
4 venv/lib/python3.11/site-packages/cleo/commands/command.py:237 in line
235| styled = f"<{style}>{text}" if style else text
236|
> 237| self._io.write_line(styled, verbosity=verbosity)
238|
239| def line_error(
3 venv/lib/python3.11/site-packages/cleo/io/io.py:52 in write_line
50| type: OutputType = OutputType.NORMAL,
51| ) -> None:
> 52| self._output.write_line(messages, verbosity=verbosity, type=type)
53|
54| def write(
2 venv/lib/python3.11/site-packages/cleo/io/outputs/output.py:86 in write_line
84| type: Type = Type.NORMAL,
85| ) -> None:
> 86| self.write(messages, new_line=True, verbosity=verbosity, type=type)
87|
88| def write(
1 venv/lib/python3.11/site-packages/cleo/io/outputs/output.py:107 in write
105| message = strip_tags(self._formatter.format(message))
106|
> 107| self._write(message, new_line=new_line)
108|
109| def flush(self) -> None:
UnicodeEncodeError
'latin-1' codec can't encode character '\u2022' in position 2: ordinal not in range(256)
at venv/lib/python3.11/site-packages/cleo/io/outputs/stream_output.py:78 in _write
74| def _write(self, message: str, new_line: bool = False) -> None:
75| if new_line:
76| message += "\n"
77|
> 78| self._stream.write(message)
79| self._stream.flush()
80|
81| def _has_color_support(self) -> bool:
82| # Follow https://no-color.org/
So something encoding related. And as suspected, in contrast to a standard Ubuntu where the terminal encoding is "utf-8", the encoding on a Raspberry Pi terminal (or SSH session) is:
python3 -c "import sys; print(sys.stdout.encoding)"
# output: iso8859-1I tried to work around this issue successfully by using
export PYTHONIOENCODING=utf-8The output is a bit broken (the fancy progress char animations don't work exactly like they should), but at least it installs stuff again.
Wild guessing from the traceback I would assume that the cleo package has no encoding fallback, or poetry does not use cleo with an explicit encoding fallback specified (or some non-strict encoding parameter).
I also tried this with a clean new poetry project after poetry init and the same effect appears on my Raspberry Pi. So I highly doubt this is pyproject.toml related (but if you still need a pyproject.toml example please feel free to ask for it 😁).
So to clarify, there are 2 bugs here:
- For
poetry install, the encoding bug is silently swallowed and no traceback is visible. At least there is a non-zero return code. - The actual bug that poetry doesn't work at all (or at least
poetry addandpoetry install) when the output terminal has a non-UTF8 encoding (there might be other encodings that work different from UTF-8, but just for the sake of simplicity).