Skip to content

[fix] set utf-8 as default encoding for RotatingFileHandler#2510

Merged
liiight merged 1 commit intodevelopfrom
fix/logging_crash
Nov 28, 2019
Merged

[fix] set utf-8 as default encoding for RotatingFileHandler#2510
liiight merged 1 commit intodevelopfrom
fix/logging_crash

Conversation

@liiight
Copy link
Copy Markdown
Member

@liiight liiight commented Nov 27, 2019

Motivation for changes:

Fix crashes with some non ascii data. To reproduce:

test_task:
  series:
    - בדיקה
Output 

/Users/orcarmi/PycharmProjects/Flexget/venv/bin/python /Users/orcarmi/PycharmProjects/Flexget/flexget_vanilla.py --debug --test execute --tasks test_task
2019-11-28 00:44 DEBUG    manager                       Figuring out config load paths
2019-11-28 00:44 DEBUG    manager                       Adding virtualenv path
2019-11-28 00:44 DEBUG    manager                       Found config: /Users/orcarmi/PycharmProjects/Flexget/config.yml
2019-11-28 00:44 DEBUG    manager                       Config file /Users/orcarmi/PycharmProjects/Flexget/config.yml selected
2019-11-28 00:44 DEBUG    manager                       sys.defaultencoding: utf-8
2019-11-28 00:44 DEBUG    manager                       sys.getfilesystemencoding: utf-8
2019-11-28 00:44 DEBUG    manager                       flexget detected io encoding: utf-8
2019-11-28 00:44 DEBUG    manager                       os.path.supports_unicode_filenames: True
2019-11-28 00:44 INFO     manager                       Test mode, creating a copy from database ...
2019-11-28 00:44 INFO     manager                       Test database created
2019-11-28 00:44 DEBUG    plugin                        Trying to load plugins from: ['/Users/orcarmi/PycharmProjects/Flexget/plugins', '/Users/orcarmi/PycharmProjects/Flexget/flexget/plugins']
2019-11-28 00:44 DEBUG    plugin                        Plugin `memusage` requires plugin `ext lib `guppy`` to load.
2019-11-28 00:44 DEBUG    plugin                        Trying to load components from: ['/Users/orcarmi/PycharmProjects/Flexget/components', '/Users/orcarmi/PycharmProjects/Flexget/flexget/components']
2019-11-28 00:44 DEBUG    plugin                        Plugins took 1.14 seconds to load. 298 plugins in registry.
2019-11-28 00:44 DEBUG    manager                       Connecting to: sqlite:////Users/orcarmi/PycharmProjects/Flexget/test-config.sqlite
2019-11-28 00:44 DEBUG    manager                       config_name: config
2019-11-28 00:44 DEBUG    manager                       config_base: /Users/orcarmi/PycharmProjects/Flexget
2019-11-28 00:44 DEBUG    manager                       New config data loaded.
2019-11-28 00:44 DEBUG    schema                        updating flexget version 3.0.11.dev in db
2019-11-28 00:44 DEBUG    parsing                       setting default movie parser to internal. (options: {'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x113f74780>, 'guessit': <flexget.components.parsing.parsers.parser_guessit.ParserGuessit object at 0x113f747b8>})
2019-11-28 00:44 DEBUG    parsing                       setting default series parser to internal. (options: {'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x113f74780>, 'guessit': <flexget.components.parsing.parsers.parser_guessit.ParserGuessit object at 0x113f747b8>})
2019-11-28 00:44 DEBUG    cron_env                      Encoding utf-8 stored
2019-11-28 00:44 DEBUG    util.simple_persistence                 setting key terminal_encoding value 'utf-8'
2019-11-28 00:44 DEBUG    task_queue                    task queue shutdown requested
2019-11-28 00:44 VERBOSE  task_queue                    There are 1 tasks to execute. Shutdown will commence when they have completed.
2019-11-28 00:44 INFO     ipc.rpyc                      server started on [127.0.0.1]:55051
2019-11-28 00:44 DEBUG    task          test_task       executing test_task
2019-11-28 00:44 DEBUG    status        test_task       Adding new task test_task
--- Logging error ---
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py", line 994, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 90-94: ordinal not in range(128)
Call stack:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task_queue.py", line 45, in run
    self.current_task.execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 78, in wrapper
    return func(self, *args, **kw)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 695, in execute
    self._execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 661, in _execute
    self.__run_task_phase(phase)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 487, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 520, in __run_plugin
    result = method(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/components/series/series.py", line 1128, in on_task_start
    normalize_series_name(series_name),
Message: 'adding series `%s` `%s` into db (on_task_start)'
Arguments: ('בדיקה', 'בדיקה')
--- Logging error ---
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py", line 994, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 90-94: ordinal not in range(128)
Call stack:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task_queue.py", line 45, in run
    self.current_task.execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 78, in wrapper
    return func(self, *args, **kw)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 695, in execute
    self._execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 661, in _execute
    self.__run_task_phase(phase)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 487, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 520, in __run_plugin
    result = method(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/components/series/series.py", line 1130, in on_task_start
    log.debug('adding series `%s` into db (on_task_start)', series_name)
Message: 'adding series `%s` into db (on_task_start)'
Arguments: ('בדיקה',)
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       adding series `בדיקה` `בדיקה` into db (on_task_start)
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       adding series `בדיקה` into db (on_task_start)
--- Logging error ---
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py", line 994, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 103-107: ordinal not in range(128)
Call stack:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task_queue.py", line 45, in run
    self.current_task.execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 78, in wrapper
    return func(self, *args, **kw)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 695, in execute
    self._execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 661, in _execute
    self.__run_task_phase(phase)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 487, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 520, in __run_plugin
    result = method(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/components/series/series.py", line 1136, in on_task_start
    log.debug('-> added `%s`', db_series)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
--- Logging error ---
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py", line 994, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 94-98: ordinal not in range(128)
Call stack:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task_queue.py", line 45, in run
    self.current_task.execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 78, in wrapper
    return func(self, *args, **kw)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 695, in execute
    self._execute()
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 661, in _execute
    self.__run_task_phase(phase)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 487, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/task.py", line 520, in __run_plugin
    result = method(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
  File "/Users/orcarmi/PycharmProjects/Flexget/flexget/components/series/series.py", line 1140, in on_task_start
    log.debug('connecting series `%s` to task `%s`', db_series.name, task.name)
Message: 'connecting series `%s` to task `%s`'
Arguments: ('בדיקה', 'test_task')
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       -> added `<Series(id=3,name=בדיקה)>`
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       connecting series `בדיקה` to task `test_task`
2019-11-28 00:44 WARNING  task          test_task       Task doesn't have any input plugins, you should add (at least) one!
2019-11-28 00:44 DEBUG    backlog.db    test_task       0 entries purged from backlog
2019-11-28 00:44 WARNING  details       test_task       Task didn't produce any entries. This is likely due to a mis-configured or non-functional input.
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       series on_task_metainfo took 0.0036880000000001356 to parse
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       processing series took 0.00021100000000018326
2019-11-28 00:44 DEBUG    urlrewriter   test_task       Checking 0 entries
2019-11-28 00:44 VERBOSE  details       test_task       Summary - Accepted: 0 (Rejected: 0 Undecided: 0 Failed: 0)
2019-11-28 00:44 WARNING  task          test_task       Task doesn't have any output plugins, you should add (at least) one!
2019-11-28 00:44 DEBUG    flexget.components.series.series test_task       on_task_learn
2019-11-28 00:44 DEBUG    util.simple_persistence test_task       Flushing simple persistence for task test_task to db.
2019-11-28 00:44 DEBUG    task_queue                    task queue shut down
2019-11-28 00:44 INFO     ipc.rpyc                      listener closed
2019-11-28 00:44 DEBUG    util.simple_persistence                 Flushing simple persistence for task None to db.
2019-11-28 00:44 DEBUG    manager                       Shutting down
2019-11-28 00:44 INFO     manager                       Removed test database
2019-11-28 00:44 DEBUG    manager                       Removed /Users/orcarmi/PycharmProjects/Flexget/.test-config-lock

Process finished with exit code 0

Detailed changes:

  • Added utf-8 as the default encoder for RotatingFileHandler

@gazpachoking
Copy link
Copy Markdown
Member

gazpachoking commented Nov 28, 2019

@liiight Makes sense. I'm a bit confused why we've never seen this issue before.

@liiight liiight merged commit 0ebc04c into develop Nov 28, 2019
@liiight
Copy link
Copy Markdown
Member Author

liiight commented Nov 28, 2019

Maybe I removed it by mistake with v3.0.0? I'm AFK so can't check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@liiight @gazpachoking