Skip to content

Conversation

@elelay
Copy link
Member

@elelay elelay commented Jul 11, 2020

No description provided.

elelay added 2 commits July 10, 2020 16:49
now a requests.Response is returned instead of the file-like object from urllib.
Fixed all usages of util.urlopen: it simplifies getting json, text encoding detection.
In particular feedcore (responsible for fetching feeds) is simplified.

This is a first pass and could benefit from better usage of the requests api
(Sessions for instance, to keep connection pools)

TODO: download.py
@auouymous
Copy link
Member

src/gpodder/feedcore.py:192:56: E226 missing whitespace around arithmetic operator

I applied these patches and they seem to work with limited testing, I'll let you know if they break in the next few days. Do you know the requests package has at least 6 dependencies? This one change doubles the minimum dependencies needed to run gpodder.

@elelay
Copy link
Member Author

elelay commented Jul 12, 2020

Thanks for your testing and feedback: it's good to know nothing is broken (yet)!

Yes, it adds dependencies but urllib is lagging in features for the core thing in gPodder (http).
requests will let us move forward and solve past and upcomming struggles (without reimplementing it ourself on top of urllib).

requests is a thin wrapper around urllib3. It does the right thing for:

It depends on:

  • chardet for guessing the encoding when it's not specified (handy in autodiscovery mode when looking for links to the feed in html page)
  • idna for correctly handling internationalized domain names (apparently broken in python standard library, I don't know anything about that)
  • urllib3 for the actual http(s) implementation
  • certifi for a bundle of ssl trust roots (I already use it for the windows port)

Optional dependencies:

  • PySocks for SOCKS proxies (not yet decided if I include it in packages)
  • brotlipy to support brotli compression

I'm not concerned about those dependencies because urllib3 is already used in recent versions of pip, so it's likely already installed on Linux and the size of python deps is dwarfed by the whole gtk3 stack in macOS/Windows packages.

The next area I'll port to requests is download.py.

@auouymous
Copy link
Member

I know this is needed. Gentoo's urllib3 also requires PySocks, pyopenssl and cryptography, is that not the case for the windows/mac builds?

@elelay
Copy link
Member Author

elelay commented Jul 12, 2020

Example internationalized domain name: https://日本語.jp/
It fails in master branch with encoding error and still fails in request branch, but with invalid html (expected because it's not a feed, but it was able to be downloaded)...

@elelay
Copy link
Member Author

elelay commented Jul 12, 2020

Gentoo's urllib3 also requires PySocks, pyopenssl and cryptography, is that not the case for the windows/mac builds?

AFAICT pyopenssl and cryptography are only needed to enable SNI verification in python2 (https://github.com/urllib3/urllib3/blob/eee53a69e1af019da18635d6974f893308db0ada/src/urllib3/contrib/pyopenssl.py and https://github.com/psf/requests/blob/251f73f65d46ff5de64b8ae2e6c6fe38c1a28808/requests/__init__.py#L96).

Pysocks should indeed be installed for SOCKS proxy support

@elelay
Copy link
Member Author

elelay commented Jul 14, 2020

I overlooked requests's encoding detection: it doesn't use chardet automatically (one has to call response.apparent_encoding to get it) and falls back to ISO-8859-1 if content-type is text/* and the server doesn't specify the charset.

So all my happy changes from shady decode to response.text must be reconsidered.

@elelay
Copy link
Member Author

elelay commented Jul 14, 2020

Fixed for feedcore in d59bba5, but escapist.py, youtube.py, etc. need to be fixed

elelay added 2 commits July 18, 2020 15:04
bytes for xml, utf-8 fallback for html, known encoding if available
@auouymous
Copy link
Member

Downloading youtube episodes works fine, but subscribing fails.
https://www.youtube.com/c/securitynow

1595530245.964432 [gpodder.gtkui.main] ERROR: Subscription error: 'str' object has no attribute 'headers'
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/gpodder/gtkui/main.py", line 2471, in thread_proc
    max_episodes=self.config.max_episodes_per_feed)
  File "/usr/lib/python3.7/site-packages/gpodder/model.py", line 1389, in load_podcast
    max_episodes)
  File "/usr/lib/python3.7/site-packages/gpodder/model.py", line 977, in load
    tmp.update(max_episodes)
  File "/usr/lib/python3.7/site-packages/gpodder/model.py", line 1153, in update
    result = self.feed_fetcher.fetch_channel(self, max_episodes)
  File "/usr/lib/python3.7/site-packages/gpodder/model.py", line 207, in fetch_channel
    return self.fetch(url, channel.http_etag, channel.http_last_modified, max_episodes=max_episodes)
  File "/usr/lib/python3.7/site-packages/gpodder/feedcore.py", line 197, in fetch
    ad.feed(util.response_text(stream.text))
  File "/usr/lib/python3.7/site-packages/gpodder/util.py", line 2206, in response_text
    if 'charset=' in response.headers.get('content-type'):
AttributeError: 'str' object has no attribute 'headers'

@elelay
Copy link
Member Author

elelay commented Jul 23, 2020

you're right, thanks!
It's fixed now

@elelay
Copy link
Member Author

elelay commented Aug 2, 2020

@auouymous I have further work to push (unittests, fix lint, retry rss download) but it's not quite ready yet...

@auouymous
Copy link
Member

@elelay Okay, all patches are working fine so far.

@auouymous
Copy link
Member

The isort -rc removal causes the following warnings for me.

WARNING: Unable to parse file share due to [Errno 21] Is a directory: '/home/github-auouymous/gpodder/share'
WARNING: Unable to parse file src/gpodder due to [Errno 21] Is a directory: '/home/github-auouymous/gpodder/src/gpodder'
WARNING: Unable to parse file tools due to [Errno 21] Is a directory: '/home/github-auouymous/gpodder/tools'

@elelay
Copy link
Member Author

elelay commented Sep 7, 2020

Hi, please try with isort >= 5.0.0.

@auouymous
Copy link
Member

Gentoo doesn't have 5.0+ yet and the ebuild for 4.3.21 won't successfully build the 5.0 branch.

@auouymous
Copy link
Member

Gentoo doesn't have 5.0+ yet and the ebuild for 4.3.21 won't successfully build the 5.0 branch.

@elelay Gentoo now has isort 5.7 if you want to revert 2f6ad17.

@elelay
Copy link
Member Author

elelay commented Mar 28, 2021

@auouymous done, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants