Feature/Torrentleech v5 API by carlba · Pull Request #2121 · Flexget/Flexget

carlba · 2018-04-11T08:36:27Z

Motivation for changes:

The last PR I made used the Torrentleech V4 API as a temporary fix to get the plugin working. This could stop working any time. Implementing support for the new API will futureproof the plugin

Detailed changes:

Reverted back the changes in The torrentleech output plugin is temporarily using the V4 api #2117
Rewrote the plugin so it interfaces directly with the torrentleech backend endpoint.

Addressed issues:

Fixes The torrentleech plugin is not working with their V5 api #2115

Notes:

This is the first real PR I make to this project so if I have made any mistakes feel free to point them out and I will gladly fix them :)

…the V4 api (#2117)" This reverts commit ff2f130

…ture/torrentleech_v5_api

carlba · 2018-04-11T08:45:00Z

What would be a good User-Agent to use? The default FlexGet one didn't work so I tried with a couple of others and the default User-Agent of cURL seems to work.

cvium · 2018-04-11T08:46:58Z

flexget/plugins/sites/torrentleech.py

        """
        Search for name from torrentleech.
        """
+        task.requests.headers.update({'User-Agent': 'curl/7.54.0'})


This changes the user agent for all plugins in the same task run. Not ideal.

Agreed when I tested in the beginning I thought setting it globally was needed. But when I retested now it wasn't. I also didn't realise it messed with the requests of the other plugins used in the task.

Will fix.

Unfortunately my initial tests where correct. The User-Agent has to be set globally otherwise the task will fail when using the download output plugin flexget.plugins.output.download.download_entry. When it tries to get the file it fails if the User-Agent is not set to curl/7.54.0 or something else supported.

Setting it globally affects other plugins so that is not good. I suppose the headers could be passed through the entry entry['download_headers']= {'User-Agent': 'curl/7.54.0'} much like auth is handled in the flexget.plugins.output.download.download_entry method.

Another approach would be to create a custom download_auth like so:

class TorrentleechHeader(AuthBase): def __call__(self, request): request.headers['User-Agent'] = 'curl/7.54.0' return request

@cvium @liiight What do you think is best?

cvium · 2018-04-11T08:47:49Z

flexget/plugins/sites/torrentleech.py

+            url = ('https://www.torrentleech.org/torrents/browse/list/query/' +
                   quote(query.encode('utf-8')) + filter_url)
-            log.debug('Using %s as torrentleech search url' % url)
+            log.debug('Using {} as torrentleech search url'.format(url))


Change to log.debug('Using %s as torrentleech search url', url)

cvium · 2018-04-11T08:49:08Z

flexget/plugins/sites/torrentleech.py

-                entry['content_size'] = parse_filesize(size.group(0))
+                # construct download URL
+                torrent_url = 'https://www.torrentleech.org/rss/download/{}/{}/{}'.format(
+                    torrent['fid'], rss_key, torrent['filename'])


Is the linebreak necessary? We allow up to 120 chars on one line.

I will adjust my IDE accordingly :)

This particular line is still above 120 chars though.

cvium · 2018-04-11T08:49:40Z

flexget/plugins/sites/torrentleech.py

+                entry['torrent_leeches'] = torrent['leechers']
+                entry['search_sort'] = torrent_availability(entry['torrent_seeds'],
+                                                            entry['torrent_leeches'])
+                entry['content_size'] = torrent['size']


Which unit is the size in?

It is in bytes I realized now that is should be in MiB.

That is why we have a utility function to do the heavy lifting.

Yes I did of course look at the util but it didn't seem like it was a perfect fit since it is meant to handle strings not integers.

I will use the util according to your other comment.

liiight · 2018-04-11T08:46:10Z

flexget/plugins/sites/torrentleech.py

+            url = ('https://www.torrentleech.org/torrents/browse/list/query/' +
                   quote(query.encode('utf-8')) + filter_url)
-            log.debug('Using %s as torrentleech search url' % url)
+            log.debug('Using {} as torrentleech search url'.format(url))


leave the logger to do its own interpolation:

log.debug('Using %s as torrentleech search url', url)

Good point having gone through have the python logger actually works your suggestion makes more sense :)

liiight · 2018-04-11T08:47:33Z

flexget/plugins/sites/torrentleech.py

+            # urllib.quote will crash if the unicode string has non ascii characters,
+            # so encode in utf-8 beforehand
+
+            url = ('https://www.torrentleech.org/torrents/browse/list/query/' +


i don't think you meant to make the url a tuple

It's not a tuple.

This will not make it into a tuple it is just a method to construct a multiline string.

liiight · 2018-04-11T08:50:34Z

flexget/plugins/sites/torrentleech.py

+                entry['torrent_leeches'] = torrent['leechers']
+                entry['search_sort'] = torrent_availability(entry['torrent_seeds'],
+                                                            entry['torrent_leeches'])
+                entry['content_size'] = torrent['size']


we have a util that converts size to our requested format:

Flexget/flexget/utils/tools.py

Line 441 in c616922

def parse_filesize(text_size, si=True):

The util is according to it's docstring meant to parse a string containing a size to a MiB float. The value returned from the torrentleach API is a int of bytes.

Yes, so just append 'b' to the int. Not the nicest way and I've been wanting to change the function signature, but it's also very low priority.

Alright that works for me! Will fix

cvium · 2018-04-11T13:22:01Z

flexget/plugins/output/download.py

+
        try:
-            response = task.requests.get(url, auth=auth, raise_status=False)
+            response = task.requests.get(url, auth=auth, raise_status=False, headers=headers)


This overwrites any header set using the header plugin. None should not be the default.

No it doesn't override the custom headers specified by the headers plugin. The default value of the headers parameter in the requests library is actually None so it has no affect. But I changed the method to retrieve the custom headers configured in the headers plugin from the config anyway.

task.requests is a requests Session instance, so it may or may not contain headers set by other plugins. The default should be task.requests.headers.

Yes but if you pass in headers=None to the get request it will not override the existing headers. Any existing headers are merged with headers passed in to the function.

I did change to task.requests.headers because it is a lot easier to understand and generally nicer.

Thanks for all the patience with the PR btw :)

cvium · 2018-04-11T17:15:28Z

flexget/plugins/sites/torrentleech.py

+
+                # construct download URL
+                torrent_url = 'https://www.torrentleech.org/rss/download/{}/{}/{}'.format(torrent['fid'], rss_key,
+                                                                                          torrent['filename'])


I don't think this line break is necessary though

The limit was 120 chars? Because the line is over that limit.

I think my mono-space font is missing.

I just use my IDE ruler :)

carlba · 2018-04-12T15:29:44Z

@cvium @liiight Is there anything further I need to fix with the PR?

carlba added 3 commits April 10, 2018 14:42

Revert "[change] The torrentleech output plugin is temporarily using …

eb11036

…the V4 api (#2117)" This reverts commit ff2f130

[change] The torrentleech output plugin uses the V5 API

a4625c8

Merge branch 'develop' of https://github.com/Flexget/Flexget into fea…

b9f58e0

…ture/torrentleech_v5_api

cvium reviewed Apr 11, 2018

View reviewed changes

liiight reviewed Apr 11, 2018

View reviewed changes

carlba added 4 commits April 11, 2018 12:11

Correct use of string interpolation in log.debug statements

ef98ea8

Get the correct content size for the entry

8f9380a

Corrected logging again

acec0a8

Passing headers through the entry['download_headers'] key

06385b2

cvium reviewed Apr 11, 2018

View reviewed changes

carlba added 2 commits April 11, 2018 16:02

Default headers are read from config.

88863f9

Changed default headers according to recommendation

23e96e1

cvium reviewed Apr 11, 2018

View reviewed changes

cvium merged commit a3f01e4 into Flexget:develop Apr 12, 2018

carlba deleted the feature/torrentleech_v5_api branch November 20, 2018 17:18

Conversation

carlba commented Apr 11, 2018

Motivation for changes:

Detailed changes:

Addressed issues:

Notes:

Uh oh!

carlba commented Apr 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlba Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlba Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cvium Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlba Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cvium Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlba Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

carlba Apr 11, 2018 •

edited

Loading

carlba Apr 11, 2018 •

edited

Loading

cvium Apr 11, 2018 •

edited

Loading

carlba Apr 11, 2018 •

edited

Loading

cvium Apr 11, 2018 •

edited

Loading

carlba Apr 11, 2018 •

edited

Loading