Skip to content

plugins.bloomberg: update/fix headers to avoid bot detection #4919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bastimeyer merged 1 commit into from
Nov 2, 2022
Merged

plugins.bloomberg: update/fix headers to avoid bot detection #4919

bastimeyer merged 1 commit into from
Nov 2, 2022

Conversation

@mkbloke
Copy link
Member

@mkbloke mkbloke commented Nov 2, 2022 •
edited
Loading

As far as my testing goes, all the previously added headers seem to be unnecessary. Adding the Accept-Encoding header back seems to trigger bot detection, possibly because with requests it only includes gzip and deflate, but not br, which Firefox also includes. Removing the header entirely seems to work.

$ streamlink https://www.bloomberg.com/live
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live
Available streams: 272p (worst), 360p, 720p_alt, 720p, 1080p (best)

$ streamlink https://www.bloomberg.com/news/videos/2022-10-26/rishi-sunak-and-the-uk-economy-showdown
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/news/videos/2022-10-26/rishi-sunak-and-the-uk-economy-showdown
Available streams: 100k (worst), 176p, 272p, 360p, 540p, 720p_alt, 720p, 1080p (best)

closes #4918

@mkbloke mkbloke added the plugin issue A Plugin does not work correctly label Nov 2, 2022
@bastimeyer bastimeyer merged commit fc19ae5 into streamlink:master Nov 2, 2022
@mkbloke mkbloke deleted the bloomberg branch November 2, 2022 20:16
@bastimeyer
Copy link
Member

bastimeyer commented Nov 2, 2022

possibly because with requests it only includes gzip and deflate, but not br

Brotli does only get added by urllib3 if the brotlicffi or brotli package is installed, hence why we're installing brotli for the CI test runners:

The bot detection triggers regardless whether brotli is available or not, just checked it. No idea why removing the header works... 😕

@bastimeyer bastimeyer mentioned this pull request Nov 2, 2022
Closed
4 tasks
@mkbloke
Copy link
Member Author

mkbloke commented Nov 2, 2022

Ah, I just read your comment here after writing: #4918 (comment)

Well, that's very strange indeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

plugin issue A Plugin does not work correctly

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

plugins.bloomberg: Always get Could not Find JSON data error from today

2 participants

@mkbloke @bastimeyer