@gazpachoking sorry for poking you, can you please take a look at this one?

Thanks for this. A couple things I noticed:

• I don't think it will break out of the pagination loop when we hit our entry limit as-is. It will keep looping through the extra pages doing nothing.
• I'm wondering if we should have a limit by default for the popular/trending lists... My only hesitation there is that it might be confusing that those get limited by default but other lists don't. I'm also not sure what a reasonable default number would be.

1. You're absolutely right — it will not break the pagination loop, but it's kind-a fine in current implementation, because processing of gathered data is done in another loop, and there is a chance that not all items would be converted to flexget's entries (see https://github.com/Flexget/Flexget/blob/develop/flexget/components/trakt/trakt_list.py#L264 and https://github.com/Flexget/Flexget/blob/develop/flexget/components/trakt/trakt_list.py#L324).

2. I think the proper way for now is not to set any default limits, but just amend wiki page for trakt_list, to add a note that popular/trending lists are huge, and it's reasonable to set a limit when using them.

How many entries do the popular/trending lists normally have?

Right now it's 778 movies for each popular and trending. But when I've started my changes — there were something like 1300 items in each of them.

Cool, I'm good with this as is then. We should definitely update the wiki page with the limit option and mention it's probably wanted in any situation using popular/trending lists. If we end up deciding there should be a default limit later we can always add it. Pagination optimization probably isn't worth it given that we request 1000 items per page.

Thanks! I've updated the trakt_list wiki.

There is currently 160,121 items in the popular movies list, with the 2 second limit, that's over 5 minutes to fetch the list. Would be really nice if it broke out of the pagination loop. Should I put in a separate issue?

2019-05-08 14:34 VERBOSE  trakt_list    Auto-Movies     Retrieving `movies` list `popular`
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, u'timeout': 30}
2019-05-08 14:34 DEBUG    trakt_list    Auto-Movies     Response is paginated. Number of items: 160121, number of pages: 161.0
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.91 seconds until next request to trakt.tv
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, 'params': {u'limit': 1000, u'page': 1}, u'timeout': 30}
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.95 seconds until next request to trakt.tv
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, 'params': {u'limit': 1000, u'page': 2}, u'timeout': 30}
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.96 seconds until next request to trakt.tv
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, 'params': {u'limit': 1000, u'page': 3}, u'timeout': 30}
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.94 seconds until next request to trakt.tv
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, 'params': {u'limit': 1000, u'page': 4}, u'timeout': 30}
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.96 seconds until next request to trakt.tv
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     GETing URL https://api.trakt.tv/movies/popular with args () and kwargs {'allow_redirects': True, 'params': {u'limit': 1000, u'page': 5}, u'timeout': 30}
2019-05-08 14:34 DEBUG    utils.requests Auto-Movies     Waiting 1.96 seconds until next request to trakt.tv

160k! Holy shit! Please, create another issue, I'll be able to take a look at this in a few weeks. Probably it'll require some refactoring.

Ohhh. Yeah, let's make a separate issue since this is merged. If there are that many in the list, I reverse my stance on both of those points above. I think we should both stop the pagination loop early, as well as institute a default limit when either of these big lists are specified.