Skip to content

npo_watchlist: Rewrite, after big changes to npo.nl#1334

Merged
cvium merged 4 commits intoFlexget:developfrom
jeroenl:npo_new_layout
Sep 10, 2016
Merged

npo_watchlist: Rewrite, after big changes to npo.nl#1334
cvium merged 4 commits intoFlexget:developfrom
jeroenl:npo_new_layout

Conversation

@jeroenl
Copy link
Copy Markdown
Contributor

@jeroenl jeroenl commented Aug 14, 2016

Motivation for changes:

npo.nl completely changed the set-up of its watchlist, requiring a rewrite of the plug-in.

Detailed changes:

Now creates entries for:

  • individual episodes put on watchlist
  • all episodes for series on watchlist (since max_episode_age_days)

Config usage if relevant (new plugin or updated schema):

            npo_watchlist:
              email: aaaa@bbb.nl
              password: xxx
              remove_accepted: yes
              max_episode_age_days: 7
            accept_all: yes
            exec:
              fail_entries: yes
              auto_escape: yes
              on_output:
                for_accepted:
                  - download-npo -o "path/to/directory/{{series_name_plain}}" -f "{serie_titel} - {datum} {aflevering_titel} ({episode_id})" -t {{url}}

Now creates entries for:
- individual episodes put on watchlist
- all episodes for series on watchlist (since max_episode_age_days)
(forgot to finish a sentence)
profile_response = self._get_profile(task, config)
profile_page = BeautifulSoup(profile_response.content, 'html5lib')
response = self._get_page(task, config, 'https://mijn.npo.nl/profiel/kijklijst')
page = BeautifulSoup(response.content, 'html5lib')
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

If 'remove_accepted' is set to 'yes', the plugin will delete accepted entries from the watchlist after download
is complete.
is complete.
If 'max_episode_age_days' is set (and not 0), entries will only be generated for episodes broadcast in the last x days.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codacy complains about a trailing whitespace after the dot. Please remove it.

@cvium cvium merged commit f68f485 into Flexget:develop Sep 10, 2016
@cvium
Copy link
Copy Markdown
Contributor

cvium commented Sep 10, 2016

You mind updating the wiki?

@jeroenl jeroenl deleted the npo_new_layout branch September 24, 2016 14:35
@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Sep 24, 2016

@cvium Done: http://flexget.com/Plugins/npo_watchlist

@mfonville
Copy link
Copy Markdown
Contributor

@jeroenl sorry for contacting you in this PR, but I would like to contact you about the npo_watchlist plugin in combination with tvdb.

I am currently having trouble to make the episodes found via npo_watchlist to be matched with tvdb (using thetvdb_lookup), to fill-in the episode information (especially the season and episode id).
Would it be possible to either provide guidance how to create such a set-up, or to make this more easy within npo_watchlist itself?

I believe technically it should be possible to use the seriesname + airdate to find the correct episode.

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Oct 17, 2016

Hey @mfonville, I tried that earlier but stopped because most programs are not listed in those databases. Happy to try and help though... can you tell me what series/program you are trying to do this with?

@mfonville
Copy link
Copy Markdown
Contributor

@jeroenl I am trying some popular programs that are maintained (partly by me) in tvdb, like "Zondag met Lubach", "Tegenlicht" (can sometimes only be found if querying for "Backlight"), "Keuringsdienst van Waarde", "Koefnoen" and "Toren C"

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Oct 18, 2016

@mfonville With #1452 I got it working with Koefnoen.

For Zondag met Lubach, the issue is that thetvdb_lookup searches by the English title, which is apparently Sunday with Lubach. You can try to modify thetvdb_lookup to allow specifying the language. Alternatively, you can have the title changed to the English title, using a combination of regexp and set.

@mfonville
Copy link
Copy Markdown
Contributor

@jeroenl thanks very much for implementing that!
Concerning the searching via thetvdb_lookup, could you give an example config of how I could use the 'set' command (might be useful to add the documentation in general)?

I gave api/plugins/tvdb_lookup.py a short look to see what can be done about passing a language argument, I would assume that from line 180 there would have to be an extra 'language' parameter be passed, it is already defined in the class.
But probably it needs some other updates throughout the code too, like https://github.com/Flexget/Flexget/blob/develop/flexget/plugins/internal/api_tvdb.py#L155 and passing the language variable in plugins/metainfo/thetvdb_lookup.py.

I am not acquainted with the FlexGet codebase at all (trying to make this set-up was my first endeavour).

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Oct 18, 2016

@mfonville I'm not familiar enough with the tvdb plugin to be able to say how to do that. Perhaps the developer of that plug-in can help? Or try it out, see if you can make it work (that's how I got started with this one).

The simple solution is with regexp+set, or actually if+set should also work. See http://flexget.com/Plugins/if

This seems to work (at least for ZmL):

    if:
      - series_name == 'Zondag met Lubach':
          set:
            series_name: Sunday with Lubach
      - series_name == 'Tegenlicht':
          set:
            series_name: Backlight

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Oct 18, 2016

@mfonville By the way, out of curiosity: do you maintain the episode data on TVDB manually? Update it every time there's a new episode?

@mfonville
Copy link
Copy Markdown
Contributor

@jeroenl thanks for these examples!

Yes, I do update them manually, but luckily I am not alone and often other users contribute for these series too. Often I also just process them a bit batch-like, i.e. Tegenlicht secretly shares the name of their upcoming episodes via their MeetUp schedule X-D

Were you thinking about automatically crawling NPO to submit the info to TVDB?

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Oct 18, 2016

No just curious, I think I'm fine without tvdb. I suppose it's partially
possible, but it wouldn't give you anything (for flexget). Npo.nl doesn't
have series and episode numbers for example. It does have title,
description and date, but this plugin already extracts those.

Op di 18 okt. 2016 23:34 schreef Maarten Fonville <notifications@github.com

:

@jeroenl https://github.com/jeroenl thanks for these examples!

Yes, I do update them manually, but luckily I am not alone and often other
users contribute for these series too. Often I also just process them a bit
batch-like, i.e. Tegenlicht secretly shares the name of their upcoming
episodes via their MeetUp schedule X-D

Were you thinking about automatically crawling NPO to submit the info to
TVDB?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1334 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATeWBqg6hFYzEvTThSkw94SsUtir8Reks5q1TtagaJpZM4Jj7OX
.

@mfonville
Copy link
Copy Markdown
Contributor

The big benefit for me (and why I add them) is that TVDB information is supplied to trakt.tv, and I use trakt to keep track of which episodes I have seen :-)

Some series of NPO have some kind of series and episode data, when they were released on DVD, and that this pattern has been continued with new broadcasts added to TVDB.
For series that don't have this info, TVDB allows to register episodes just by date too, I believe, but I am not really acquainted with that feature.

@mfonville
Copy link
Copy Markdown
Contributor

@gazpachoking @liiight
Sorry for tagging you guys, as major contributors of tvdb_lookup could you reflect on #1334 (comment) and how hard it would be to add language-specific search (or the 'all languages'-style) to the tvdb_lookup plugin?

@liiight
Copy link
Copy Markdown
Member

liiight commented Oct 19, 2016

np, but please open another issue for that

@mfonville
Copy link
Copy Markdown
Contributor

@liiight thanks for looking into it, I made this new issue: #1453
If you need any extra information, just let me know :-)

@mfonville
Copy link
Copy Markdown
Contributor

@jeroenl I contacted NPO.nl recently and asked them if they could include the inLanguage in their XML scheme.
And they said they have now added it for their next version, that will go live at 24 november, so it would be great if we could extract the language information at that point using the npo_watchlist plugin to pass it on to the lookup plugins.

Do you think that would be possible? :)

@jeroenl
Copy link
Copy Markdown
Contributor Author

jeroenl commented Dec 13, 2016

@mfonville I hadn't heard of any XML scheme. Can you give an example? Perhaps in a separate issue, to stop spamming the others on this discussion.

@mfonville
Copy link
Copy Markdown
Contributor

mfonville commented Dec 13, 2016

@jeroenl I implemented it and it got merged in PR #1529

You might want to take a look at the test file I added, if you want to expand those for the original functions you developed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants