Flexget killed by OOM killer after removing a long-running task from config #2781
Description
Actual behaviour:
When starting flexget with daemon start -d or database cleanup, flexget in couple seconds uses all system non-used memory (~700M) then fills free swap (~200M) till:
Nov 10 00:20:02 kernel: [24807224.304623] Out of memory: Kill process 12643 (flexget) score 511 or sacrifice child
Nov 10 00:20:02 kernel: [24807224.304701] Killed process 12643 (flexget) total-vm:819652kB, anon-rss:689904kB, file-rss:0kB, shmem-rss:0kB
Nov 10 00:20:02 kernel: [24807224.456867] oom_reaper: reaped process 12643 (flexget), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
database vacuum works fine.
Steps to reproduce & notes:
- Task local had been removed from config quite some days ago.
- This was a task that had been running for year(s)
Just start flexget daemon in foreground or background mode or use database cleanup
I noticed Purging obsolete status data for task local was common between daemon start & database cleanup, so I tried re-adding the task --> no attempt to clean data --> Flexget runs fine.
Config:
tasks:
local:
priority: 2
csv:
url: file:///scripts/txts/flex.csv
values:
title: 1
url: 2
download: /srv/torrent/
rtorrent_magnet: /srv/torrent
template:
- glopr
templates:
glopr:
regexp:
accept:
- lots of regex
from: title
content_size:
max: 15360
strict: no
content_filter:
require_mainfile: no
exec:
auto_escape: yes
on_output:
for_accepted:
Log:
CLICK ME
2020-11-10 00:16:43 DEBUG manager Figuring out config load paths
2020-11-10 00:16:43 DEBUG manager Adding virtualenv path
2020-11-10 00:16:43 DEBUG manager Found config: .flexget/config.yml
2020-11-10 00:16:43 DEBUG manager Config file .flexget/config.yml selected
2020-11-10 00:16:43 DEBUG manager sys.defaultencoding: utf-8
2020-11-10 00:16:43 DEBUG manager sys.getfilesystemencoding: utf-8
2020-11-10 00:16:43 DEBUG manager flexget detected io encoding: utf-8
2020-11-10 00:16:43 DEBUG manager os.path.supports_unicode_filenames: False
2020-11-10 00:16:43 DEBUG plugin Trying to load plugins from: ['.flexget/plugins', 'flexget/lib/pyt
hon3.7/site-packages/flexget/plugins']
2020-11-10 00:16:43 DEBUG plugin Plugin `memusage` requires plugin `ext lib `guppy`` to load.
2020-11-10 00:16:43 DEBUG plugin Trying to load components from: ['.flexget/components', 'flexget/l
ib/python3.7/site-packages/flexget/components']
2020-11-10 00:16:44 DEBUG plugin Plugins took 1.43 seconds to load. 307 plugins in registry.
2020-11-10 00:16:44 DEBUG manager Connecting to: sqlite:///.flexget/db-config.sqlite
2020-11-10 00:16:44 DEBUG manager config_name: config
2020-11-10 00:16:44 DEBUG manager config_base: .flexget
2020-11-10 00:16:45 DEBUG manager New config data loaded.
2020-11-10 00:16:45 DEBUG schema current flexget version already exist in db 3.1.82
2020-11-10 00:16:45 DEBUG parsing setting default movie parser to internal. (options: {'guessit': <flexget.components.parsing.parser
s.parser_guessit.ParserGuessit object at 0x7fd31c5f4c50>, 'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x7fd31c5f
4cc0>})
2020-11-10 00:16:45 DEBUG parsing setting default series parser to internal. (options: {'guessit': <flexget.components.parsing.parse
rs.parser_guessit.ParserGuessit object at 0x7fd31c5f4c50>, 'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x7fd31c5
f4cc0>})
2020-11-10 00:16:45 INFO manager Daemonizing...
2020-11-10 00:16:45 DEBUG manager Removed .flexget/.config-lock
2020-11-10 00:16:45 DEBUG manager Lockfile .flexget/.config-lock not found
2020-11-10 00:16:45 INFO manager Daemonize complete. New PID: 12233
2020-11-10 00:16:45 DEBUG irc No irc connections defined in the config
2020-11-10 00:16:45 INFO scheduler Starting scheduler
2020-11-10 00:16:45 INFO apscheduler.scheduler Scheduler started
2020-11-10 00:16:45 INFO apscheduler.scheduler Resumed scheduler job processing
2020-11-10 00:16:45 DEBUG apscheduler.scheduler Looking for jobs to run
2020-11-10 00:16:45 INFO apscheduler.executors.default Running job "Subsplease (trigger: interval[0:01:00], next run at: 2020-11-09 23:16
:50 CET)" (scheduled at 2020-11-10 00:15:50.121873+01:00)
2020-11-10 00:16:45 DEBUG scheduler executing tasks: ['Subsplease']
2020-11-10 00:16:45 INFO manager Subsplease Running database cleanup.
2020-11-10 00:16:45 DEBUG scheduler task finished executing: Subsplease
2020-11-10 00:16:45 INFO apscheduler.executors.default Running job "Nyaa (trigger: interval[0:03:00], next run at: 2020-11-09 23:18:50 CE
T)" (scheduled at 2020-11-10 00:15:50.135839+01:00)
2020-11-10 00:16:45 DEBUG scheduler executing tasks: ['Nyaa']
2020-11-10 00:16:45 INFO ipc.rpyc server started on [127.0.0.1]:37763
2020-11-10 00:16:45 DEBUG scheduler task finished executing: Nyaa
2020-11-10 00:16:45 INFO apscheduler.executors.default Running job "TokyoTosho (trigger: interval[0:21:00], next run at: 2020-11-09 23:29
:36 CET)" (scheduled at 2020-11-10 00:11:36.331716+01:00)
2020-11-10 00:16:45 DEBUG scheduler executing tasks: ['TokyoTosho']
2020-11-10 00:16:45 DEBUG scheduler task finished executing: TokyoTosho
2020-11-10 00:16:45 DEBUG apscheduler.scheduler Next wakeup is due at 2020-11-10 00:16:50.121873+01:00 (in 4.782760 seconds)
2020-11-10 00:16:45 DEBUG seen.db Subsplease TODO: Disabled because of ticket #1321
2020-11-10 00:16:45 VERBOSE status.db Subsplease Purging obsolete status data for task local
2020-11-10 00:16:50 DEBUG apscheduler.scheduler Looking for jobs to run
2020-11-10 00:16:50 WARNING apscheduler.scheduler Execution of job "Subsplease (trigger: interval[0:01:00],next run at: 2020-11-10 00:16:50
CET)" skipped: maximum number of running instances reached (1)
2020-11-10 00:16:50 DEBUG apscheduler.scheduler Next wakeup is due at 2020-11-10 00:17:50.121873+01:00 (in 59.938270 seconds)
<killed>
---OR---
2020-11-10 10:41:29 DEBUG manager Figuring out config load paths
2020-11-10 10:41:29 DEBUG manager Adding virtualenv path
2020-11-10 10:41:29 DEBUG manager Found config: .flexget/config.yml
2020-11-10 10:41:29 DEBUG manager Config file .flexget/config.yml selected
2020-11-10 10:41:29 DEBUG manager sys.defaultencoding: utf-8
2020-11-10 10:41:29 DEBUG manager sys.getfilesystemencoding: utf-8
2020-11-10 10:41:29 DEBUG manager flexget detected io encoding: utf-8
2020-11-10 10:41:29 DEBUG manager os.path.supports_unicode_filenames: False
2020-11-10 10:41:29 DEBUG plugin Trying to load plugins from: ['.flexget/plugins', 'flexget/lib/python3.7/site-packages/flexget/plugins']
2020-11-10 10:41:30 DEBUG plugin Plugin `memusage` requires plugin `ext lib `guppy`` to load.
2020-11-10 10:41:30 DEBUG plugin Trying to load components from: ['.flexget/components', 'flexget/lib/python3.7/site-packages/flexget/components']
2020-11-10 10:41:31 DEBUG plugin Plugins took 1.33 seconds to load. 307 plugins in registry.
2020-11-10 10:41:31 DEBUG manager Connecting to: sqlite:///.flexget/db-config.sqlite
2020-11-10 10:41:31 DEBUG manager config_name: config
2020-11-10 10:41:31 DEBUG manager config_base: .flexget
2020-11-10 10:41:31 DEBUG manager New config data loaded.
2020-11-10 10:41:31 DEBUG schema current flexget version already exist in db 3.1.87
2020-11-10 10:41:31 DEBUG parsing setting default movie parser to internal. (options: {'guessit': <flexget.components.parsing.parsers.parser_guessit.ParserGuessit object at 0x7fb092d7cc50>, 'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x7fb092d7ccc0>})
2020-11-10 10:41:31 DEBUG parsing setting default series parser to internal. (options: {'guessit': <flexget.components.parsing.parsers.parser_guessit.ParserGuessit object at 0x7fb092d7cc50>, 'internal': <flexget.components.parsing.parsers.parser_internal.ParserInternal object at 0x7fb092d7ccc0>})
2020-11-10 10:41:31 INFO manager Running database cleanup.
2020-11-10 10:41:31 DEBUG seen.db TODO: Disabled because of ticket #1321
2020-11-10 10:41:31 VERBOSE status.db Purging obsolete status data for task local
<killed>
Additional information:
- FlexGet version: 3.1.82 & 3.1.87
- Python version: Python 3.7.3
- sqlite size: 156M
- Installation method: virtualenv pip
- Using daemon (yes/no): yes
- OS and version: Debian Buster