Skip to content

Fix youtube support and use requests for downloads. #1043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
auouymous merged 1 commit into from
May 24, 2021
Merged

Fix youtube support and use requests for downloads. #1043

auouymous merged 1 commit into from
May 24, 2021

Conversation

@auouymous
Copy link
Member

@auouymous auouymous commented May 20, 2021

The get_video_info URL no longer exists and instead of throwing a 404 not found error, http_request() was returning an empty page which threw a no formats found error. The new requests code will throw the correct error if this happens in the future.

The player response data is now fetched from the watch URL due to the removal of get_video_info.

Error messages have been shortened by removing video ID, which could be removed from the code in the future.

The code has been tested but I want to wait a couple days before merging to make sure nothing breaks.

@auouymous auouymous added the wip label May 20, 2021
@auouymous auouymous mentioned this pull request May 20, 2021
Closed
@tpikonen
Copy link
Contributor

tpikonen commented May 21, 2021

Streaming from youtube is broken, but this does not fix it for me. I get redirected to consent.youtube.com and then get

1621585164.076277 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): www.youtube.com:443
1621585164.227633 [urllib3.connectionpool] DEBUG: https://www.youtube.com:443 "GET /watch?v=XXXXXXX HTTP/1.1" 302 0
1621585164.231223 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): consent.youtube.com:443
1621585164.390938 [urllib3.connectionpool] DEBUG: https://consent.youtube.com:443 "GET /m?continue=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXXXXXXX&gl=FI&m=0&pc=yt&uxe=23983172&hl=fi&src=1 HTTP/1.1" 302 0
1621585164.451144 [urllib3.connectionpool] DEBUG: https://consent.youtube.com:443 "GET /ml?continue=https://www.youtube.com/watch?v%3DXXXXXXX&gl=FI&hl=fi&pc=yt&uxe=23983172&src=1 HTTP/1.1" 200 None
1621585164.464587 [gpodder.gtkui.main] ERROR: Error in playback!
Traceback (most recent call last):
  File "/home/user/src/gpodder/master-worktree/src/gpodder/gtkui/main.py", line 2151, in playback_episodes
    self.playback_episodes_for_real(episodes)
  File "/home/user/src/gpodder/master-worktree/src/gpodder/gtkui/main.py", line 2083, in playback_episodes_for_real
    filename = episode.get_playback_url(self.config, allow_partial)
  File "/home/user/src/gpodder/master-worktree/src/gpodder/model.py", line 532, in get_playback_url
    url = registry.download_url.resolve(config, self.url, self, allow_partial)
  File "/home/user/src/gpodder/master-worktree/src/gpodder/registry.py", line 31, in resolve
    result = resolver(item, *args)
  File "/home/user/src/gpodder/master-worktree/src/gpodder/youtube.py", line 186, in youtube_real_download_url
    res, duration = get_real_download_url(episode.url, allow_partial, fmt_ids)
  File "/home/user/src/gpodder/master-worktree/src/gpodder/youtube.py", line 209, in get_real_download_url
    raise YouTubeError('No ytInitialPlayerResponse found')
gpodder.youtube.YouTubeError: No ytInitialPlayerResponse found

There probably needs to be some cookies set.

The gPodder youtube plugin is nice, but getting into the obfuscation arms race with youtube leads to gPodder reimplementing functionality in youtube-dl, which is kind of pointless...

@auouymous
Copy link
Member Author

auouymous commented May 21, 2021

Is that an age-gated video? Age-gated content is not supported by gpodder, you can use youtube-dl to download it but not stream.

Does the following live stream episode work?

Study & Coffee: Chill Study Beats Music Radio - Cozy Jazzhop Lounge Music
https://www.youtube.com/feeds/videos.xml?channel_id=UCJhjE7wbdYAae1G25m0tHAA

@tpikonen
Copy link
Contributor

tpikonen commented May 21, 2021

The video in question was not age-restricted. I get the same response from your example. The redirection asks permission to set cookies and this is likely not done outside of EU.

@auouymous
Copy link
Member Author

auouymous commented May 21, 2021

Ah, and the error also happens when downloading the video with manage_downloads turned off?

@tpikonen
Copy link
Contributor

tpikonen commented May 21, 2021

Yes, the error message is exactly the same.

@auouymous
Copy link
Member Author

auouymous commented May 21, 2021

Does that fix the issue?

@tpikonen
Copy link
Contributor

tpikonen commented May 21, 2021

No, there is still the same redirection.

@auouymous
Fix youtube support and use requests for downloads.
7177834 
The `get_video_info` URL no longer exists (without html5=1) and instead
of throwing a 404 not found error, http_request() was returning an empty
page which threw a no formats found error. The new requests code will
throw the correct error if this happens in the future.

The player response data is still fetched from `get_video_info` but will
fallback to the `watch` URL if `get_video_info` is eventually removed.
The `watch` URL will fail for anyone in Europe due to it redirecting to
a GDPR cookie consent page.

Error messages have been shortened by removing video ID, which could be
removed from the code in the future.
@auouymous
Copy link
Member Author

auouymous commented May 21, 2021

It seems the old page is still reachable with a different query string. I fixed the patch to use the old page and fall back to the new page if it should fail again in the future. A fix for the cookie consent will need to be found in case the old page gets removed, I think youtube-dl uses the API.

@tpikonen
Copy link
Contributor

tpikonen commented May 21, 2021

The example episode now streams, but does not download, which I suppose is how it should work.

@auouymous
Copy link
Member Author

auouymous commented May 21, 2021

The example episode is a never ending live stream. Can you verify other non-livestream content downloads and streams with the patch and manage_downloads=False?

@TinoJ
Copy link

TinoJ commented May 22, 2021

Hi, I notices the problem yesterday and are happy to find the issue listed here. Just tried the patch and it works at the moment. :)
Best, Tino

@tpikonen
Copy link
Contributor

tpikonen commented May 23, 2021

Yeah, downloading and streaming works with this patch and without the ytdl extension.

@auouymous auouymous merged commit a2fc81e into gpodder:master May 24, 2021
@auouymous auouymous deleted the improve-youtube branch May 24, 2021 01:05
@auouymous
Copy link
Member Author

auouymous commented May 24, 2021

Thanks for testing.

@ghost
Copy link

ghost commented May 26, 2021

how to I add the Patch? is there instructions somewhere?

@18928172992817182
Copy link
Contributor

18928172992817182 commented Jun 18, 2021

Hello,

I am an EU pleb and encountered this TODO: # TODO: watch URL does not work in europe due to GDPR cookie consent

I have made a solution that uses the HTMLParser to parse out the form Google wants submitted on the consent page and then I use that data to create the final URL needed to simulate the user clicking the approve button.

Even though the form submission is a post on that page I have seen that the only thing Google seems to do after the post is to redirect it to a GET request with the exact same data. So when I do this with the data extracted from the form everything works for me.

18928172992817182@3003c8a

I am totally new to open source code contributions but this should probably be useful for other EU plebs :)

@tpikonen
Copy link
Contributor

tpikonen commented Jun 18, 2021

Looks promising, but could you submit a separate pull request? Just make a new branch with the commit, push it to github and you should see an URL which allows you to make the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

wip

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@auouymous @tpikonen @TinoJ @18928172992817182