Skip to content

html plugin: add stop_when_404 increment option#3240

Merged
paranoidi merged 1 commit intoFlexget:developfrom
haegrr:stop-when-404
Jul 25, 2022
Merged

html plugin: add stop_when_404 increment option#3240
paranoidi merged 1 commit intoFlexget:developfrom
haegrr:stop-when-404

Conversation

@haegrr
Copy link
Copy Markdown
Contributor

@haegrr haegrr commented Nov 20, 2021 •
edited
Loading

Motivation for changes:

Some sites will return 404 at pagination stop. Currently that causes the task to abort due to an uncaught HTTPError.

Detailed changes:

When stop_when_404 is set to false, the HTTPError thrown by requests will bubble up (default and existing behaviour) - aborting the task. If it is set to true, the HTTPError will be caught and iteration will halt as if an empty page was returned and stop_when_empty was set to true.

Addressed issues:

N/A

Implemented feature requests:

N/A

Config usage if relevant (new plugin or updated schema):

html:
  url: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page={{i}}
  increment:
    from: 1
    stop_when_404: true
    entries_count: 999

Log and/or tests output (preferably both):

2021-11-20 14:22:15 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=1
2021-11-20 14:22:15 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:15 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=2
2021-11-20 14:22:16 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:16 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=3
2021-11-20 14:22:16 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:16 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=4
2021-11-20 14:22:17 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:17 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=5
2021-11-20 14:22:17 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:17 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=6
2021-11-20 14:22:18 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:18 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=7
2021-11-20 14:22:18 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:18 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=8
2021-11-20 14:22:19 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:19 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=9
2021-11-20 14:22:19 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:19 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=10
2021-11-20 14:22:20 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:20 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=11
2021-11-20 14:22:20 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:20 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=12
2021-11-20 14:22:21 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:21 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=13
2021-11-20 14:22:21 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:21 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=14
2021-11-20 14:22:22 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:22 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=15
2021-11-20 14:22:22 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:22 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=16
2021-11-20 14:22:23 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:23 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=17
2021-11-20 14:22:23 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:23 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=18
2021-11-20 14:22:24 VERBOSE  html          giant-bomb-unprofessional-fridays Response: 200 (OK)
2021-11-20 14:22:24 VERBOSE  html          giant-bomb-unprofessional-fridays Requesting: https://www.giantbomb.com/api/show-episode/19801/?display_type=1&display_order=0&page=19
2021-11-20 14:22:25 VERBOSE  details       giant-bomb-unprofessional-fridays Produced 353 entries.

@haegrr
html plugin: add stop_when_404 increment option
81cdd28 
Some sites will return 404 at pagination stop. When 'stop_when_404'
is set to 'false', the HTTPError thrown by requests will bubble up
(default and existing behaviour) - aborting the task. If it is set
to 'true', the HTTPError will be caught and iteration will halt as
if an empty page was returned and 'stop_when_empty' was set to 'true'.
@paranoidi
Copy link
Copy Markdown
Member

paranoidi commented Nov 23, 2021

Hmm, would it make more sense to include this behavior in stop_when_empty instead of making new option?
In general I very much like the idea of handling 404 better.

@haegrr
Copy link
Copy Markdown
Contributor Author

haegrr commented Nov 23, 2021 •
edited
Loading

Hmm, would it make more sense to include this behavior in stop_when_empty instead of making new option?

I guess that depends in part on how much you care about getting feedback on unexpected 404s. It could be that remote changes break your config and since stop_when_empty is true by default, you wouldn't be explicitly made aware that it is an error (by which I mean that the task, as I understand it, wouldn't fail - just stop producing entries). That's my reasoning, at least.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 23, 2022

This PR is stale because it has been open 150 days with no activity. Remove stale label or comment or this will be closed in 60 days.

@github-actions github-actions bot added the Stale label Apr 23, 2022
@UkrainianHulk
Copy link
Copy Markdown

UkrainianHulk commented May 15, 2022

+1 for this feature

@github-actions github-actions bot removed the Stale label May 15, 2022
@paranoidi paranoidi merged commit e97fdd4 into Flexget:develop Jul 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@haegrr @paranoidi @UkrainianHulk