Skip to content

Fix and Upgrade anidb_list.py#2220

Merged
paranoidi merged 10 commits intoFlexget:developfrom
XVicarious:anidb_fix
Oct 15, 2018
Merged

Fix and Upgrade anidb_list.py#2220
paranoidi merged 10 commits intoFlexget:developfrom
XVicarious:anidb_fix

Conversation

@XVicarious
Copy link
Copy Markdown
Contributor

@XVicarious XVicarious commented Oct 9, 2018 •
edited
Loading

Motivation for changes:

The AniDB list plugin was simply broken. You would get a 403 because of the user-agent.

Detailed changes:

  • Change the default User-Agent to something realistic
  • "mode" parameter allow you to get a specific wishlist as opposed to the entire thing (Add wishlist category support in anidb_list #1031 old, but relevant)
  • "pass" parameter to allow the user to keep their wishlist marked as private (see AniDB Settings > Permissions > Guest Password to enable the password)
  • Add support for OVAs as they were always skipped over because they don't fall under "shows" or "movies"
  • If there is more than one page of wishlist items, it will now load those as well

Addressed issues:

Implemented feature requests:

Config usage if relevant (new plugin or updated schema):

anidb_list:
  user_id: (no changes)
  type: (added new option "ovas")
  strip_dates: (no changes)
  mode: (all, undefined, watch, get, blacklist, buddy)
  pass: (special password set in AniDB Settings > Permissions > Guest Password)

Log and/or tests output (preferably both):

2018-10-09 17:01 DEBUG    manager                       Figuring out config load paths
2018-10-09 17:01 DEBUG    manager                       Found config: /home/xvicarious/repos/Flexget/config.yml
2018-10-09 17:01 DEBUG    manager                       Config file /home/xvicarious/repos/Flexget/config.yml selected
2018-10-09 17:01 DEBUG    manager                       sys.defaultencoding: utf-8
2018-10-09 17:01 DEBUG    manager                       sys.getfilesystemencoding: utf-8
2018-10-09 17:01 DEBUG    manager                       flexget detected io encoding: utf-8
2018-10-09 17:01 DEBUG    manager                       os.path.supports_unicode_filenames: False
2018-10-09 17:01 INFO     manager                       Test mode, creating a copy from database ...
2018-10-09 17:01 INFO     manager                       Test database created
2018-10-09 17:01 DEBUG    plugin                        Trying to load plugins from: ['/home/xvicarious/repos/Flexget/plugins', '/home/xvicarious/repos/Flexget/flexget/plugins']
2018-10-09 17:01 DEBUG    plugin                        ('Plugin `%s` requires `%s` to load.', 'memusage', 'ext lib `guppy`')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('addic7ed = subliminal.providers.addic7ed:Addic7edProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('legendastv = subliminal.providers.legendastv:LegendasTVProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('opensubtitles = subliminal.providers.opensubtitles:OpenSubtitlesProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('podnapisi = subliminal.providers.podnapisi:PodnapisiProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('shooter = subliminal.providers.shooter:ShooterProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('subscenter = subliminal.providers.subscenter:SubsCenterProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('thesubdb = subliminal.providers.thesubdb:TheSubDBProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('metadata = subliminal.refiners.metadata:refine')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('omdb = subliminal.refiners.omdb:refine')
2018-10-09 17:01 DEBUG    stevedore.extension                 found extension EntryPoint.parse('tvdb = subliminal.refiners.tvdb:refine')
2018-10-09 17:01 DEBUG    plugin                        Plugins took 1.00 seconds to load. 289 plugins in registry.
2018-10-09 17:01 DEBUG    manager                       Connecting to: sqlite:////home/xvicarious/repos/Flexget/test-config.sqlite
2018-10-09 17:01 DEBUG    manager                       config_name: config
2018-10-09 17:01 DEBUG    manager                       config_base: /home/xvicarious/repos/Flexget
2018-10-09 17:01 DEBUG    manager                       New config data loaded.
2018-10-09 17:01 DEBUG    schema                        current flexget version already exist in db 2.14.25.dev
2018-10-09 17:01 DEBUG    parsing                       setting default movie parser to internal. (options: {'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x7f62161cada0>, 'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x7f62161cadd8>})
2018-10-09 17:01 DEBUG    parsing                       setting default series parser to internal. (options: {'guessit': <flexget.plugins.parsers.parser_guessit.ParserGuessit object at 0x7f62161cada0>, 'internal': <flexget.plugins.parsers.parser_internal.ParserInternal object at 0x7f62161cadd8>})
2018-10-09 17:01 DEBUG    cron_env                      Encoding utf-8 stored
2018-10-09 17:01 DEBUG    util.simple_persistence                 setting key terminal_encoding value 'utf-8'
2018-10-09 17:01 DEBUG    task_queue                    task queue shutdown requested
2018-10-09 17:01 INFO     ipc.rpyc                      server started on [127.0.0.1]:41139
2018-10-09 17:01 DEBUG    task          test_anidb      executing test_anidb
2018-10-09 17:01 DEBUG    input_cache   test_anidb      cache name: anidb_list_69cc009c836780ff69b2105a4c09a8a7 (has: )
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Retrieving AniDB list: mywishlist:all
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Requesting: http://anidb.net/perl-bin/animedb.pl?show=mywishlist&uid={redacted}&pass={redacted}
2018-10-09 17:01 DEBUG    utils.requests test_anidb      GETing URL http://anidb.net/perl-bin/animedb.pl?show=mywishlist&uid={redacted}&pass={redacted} with args () and kwargs {'headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebkit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}, 'allow_redirects': True, 'timeout': 30}
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      .hack//Sign id is 24
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Anima Yell! id is 13756
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Beelzebub-jou no Oki ni Mesu Mama. id is 14025
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Ben-Tou id is 8292
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Binbougami ga! id is 8888
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Chuu Bra!! id is 7124
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Conception id is 14068
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Date a Live id is 8808
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Death March kara Hajimaru Isekai Kyousoukyoku id is 12603
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Demi-chan wa Kataritai id is 12374
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Devidol! id is 14389
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Dog Days id is 8206
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Fairy Tail (2018) id is 13295
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Gaikotsu Shoten`in Honda-san id is 13440
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Gakuen Basara id is 14001
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Gintama id is 3468
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Goblin Slayer id is 13831
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Hashiri Tsuzukete Yokattatte. id is 14349
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Himote House id is 13629
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Ikkitousen id is 618
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Irozuku Sekai no Ashita kara id is 13925
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Jingai-san no Yome id is 14174
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Kaguya-sama wa Kokurasetai: Tensai-tachi no Ren`ai Zunousen id is 14111
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Kimi ga Aruji de Shitsuji ga Ore de: They Are My Noble Masters id is 5235
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Kishuku Gakkou no Juliet id is 13911
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Requesting: http://anidb.net/perl-bin/animedb.pl?show=mywishlist&uid={redactd}&pass={redacted}&orderby.pri=0.2&orderby.name=1.1&h=1&page=1
2018-10-09 17:01 DEBUG    utils.requests test_anidb      GETing URL http://anidb.net/perl-bin/animedb.pl?show=mywishlist&uid={redacted}&pass={redacted}&orderby.pri=0.2&orderby.name=1.1&h=1&page=1 with args () and kwargs {'headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebkit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}, 'allow_redirects': True, 'timeout': 30}
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      KissXSis (2010) id is 7215
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Kore ga Watashi no Goshujin-sama id is 2621
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Manyuu Hiken-chou id is 8338
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Muteki Kanban Musume id is 4381
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Nanatsu no Bitoku id is 13599
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Ore ga Suki na no wa Imouto Dakedo Imouto ja Nai id is 13534
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Otona no Bouguya-san id is 14412
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Release the Spyce id is 13760
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Renkin San-kyuu Magical? Pokaan id is 4144
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      School Rumble id is 2165
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Seishun Buta Yarou wa Bunny Girl Senpai no Yume o Minai id is 13900
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Senran Kagura: Shinovi Master - Tokyo Youma Hen id is 13341
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Sin: Nanatsu no Taizai id is 12324
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Sono Toki, Kanojo wa. id is 14339
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Sora to Umi no Aida id is 13312
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      SSSS.Gridman id is 13260
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Stella Jogakuin Koutouka C3-bu id is 9620
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Suzumiya Haruhi no Yuuutsu (2009) id is 6367
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Sword Art Online: Alicization id is 13493
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Tonari no Kyuuketsuki-san id is 13966
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Uchi no Maid ga Uzasugiru! id is 14026
2018-10-09 17:01 DEBUG    anidb_list    test_anidb      Yagate Kimi ni Naru id is 14053
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      Entry does not match the requested type
2018-10-09 17:01 VERBOSE  anidb_list    test_anidb      No more pages on the wishlist.
2018-10-09 17:01 DEBUG    input_cache   test_anidb      storing to cache anidb_list_69cc009c836780ff69b2105a4c09a8a7 47 entries
2018-10-09 17:01 DEBUG    input_cache   test_anidb      Storing cache anidb_list_69cc009c836780ff69b2105a4c09a8a7 to database.
2018-10-09 17:01 WARNING  task          test_anidb      Task doesn't have any input plugins, you should add (at least) one!
2018-10-09 17:01 DEBUG    backlog       test_anidb      0 entries purged from backlog
2018-10-09 17:01 WARNING  details       test_anidb      Task didn't produce any entries. This is likely due to a mis-configured or non-functional input.
2018-10-09 17:01 DEBUG    series        test_anidb      series on_task_metainfo took 0.006308999999999898 to parse
2018-10-09 17:01 DEBUG    series        test_anidb      processing series took 0.005649999999999711
2018-10-09 17:01 DEBUG    urlrewriter   test_anidb      Checking 0 entries
2018-10-09 17:01 VERBOSE  details       test_anidb      Summary - Accepted: 0 (Rejected: 0 Undecided: 0 Failed: 0)
2018-10-09 17:01 WARNING  task          test_anidb      Task doesn't have any output plugins, you should add (at least) one!
2018-10-09 17:01 DEBUG    series        test_anidb      on_task_learn
2018-10-09 17:01 DEBUG    util.simple_persistence test_anidb      Flushing simple persistence for task test_anidb to db.
2018-10-09 17:01 DEBUG    task_queue                    task queue shut down
2018-10-09 17:01 INFO     ipc.rpyc                      server has terminated
2018-10-09 17:01 INFO     ipc.rpyc                      listener closed
2018-10-09 17:01 DEBUG    util.simple_persistence                 Flushing simple persistence for task None to db.
2018-10-09 17:01 DEBUG    manager                       Shutting down
2018-10-09 17:01 INFO     manager                       Removed test database
2018-10-09 17:01 DEBUG    manager                       Removed /home/xvicarious/repos/Flexget/.test-config-lock

Extra Thoughts:

This plugin should probably be renamed to "anidb_wishlist", as that is what it is fetching. The AniDB list (called "mylist") is the list of shows you've seen or reported having copies of.

Brian Maurer added 4 commits October 1, 2018 21:12
Fix 403 error due to AniDB not liking the headers
2e3cc29
Add mode and pass parameters for AniDB
06320e2
Move the base anidb url to an instance variable
7690e67 
Add OVA type
Rearrange modes in index order for more simple usage
Add user_agent property just in case AniDB blocks this one
Move generation of the URL to its own private method
If there is more than one wishlist page, it will now fetch all of them
Fix line length
de89d02
cvium
cvium requested changes Oct 11, 2018
View reviewed changes
cvium
cvium reviewed Oct 12, 2018
View reviewed changes
cvium
cvium reviewed Oct 12, 2018
View reviewed changes
paranoidi
paranoidi requested changes Oct 13, 2018
View reviewed changes
@paranoidi
Copy link
Copy Markdown
Member

paranoidi commented Oct 13, 2018

Very nicely done, small corrections and we can merge this for sure!

cvium
cvium reviewed Oct 15, 2018
View reviewed changes
flexget/plugins/input/anidb_list.py
task_header = {
'User-Agent': self.default_user_agent
}
task_headers = task.requests.headers
Copy link
Copy Markdown
Contributor

@cvium cvium Oct 15, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be best to clone task.requests.headers so it doesn't set the user agent for the entire task. I should've mentioned this sooner, sorry.

@paranoidi
Copy link
Copy Markdown
Member

paranoidi commented Oct 15, 2018

Thank you for your contributions!

@paranoidi paranoidi merged commit 0f2619e into Flexget:develop Oct 15, 2018
@XVicarious XVicarious mentioned this pull request Oct 21, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cvium cvium cvium requested changes

@paranoidi paranoidi paranoidi requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AniDB Input Plugin is Broken

3 participants

@XVicarious @paranoidi @cvium