processImg: skip data URI#13034
Conversation
This is the underlying reason that The Verge + download full article fails, see <koreader#12953 (comment)>.
| if src:sub(1,5) == "data:" then | ||
| logger.dbg("NewsDownloader: skipping data URI") | ||
| return nil |
There was a problem hiding this comment.
Please donc infect wikipedia.lua with references to NewsDownloaded :)
What happens when we meet a data: ? It just let it as is and it works ? How did it behave previously ?
html = html:gsub("(<%s*img [^>]*>)", processImg) just let it as-is when it returns nil ? Or it removes it completely ?
Just asking, this feels right if it lets it untouched.
There was a problem hiding this comment.
My point is to leave it untouched (as opposed to converting it, let alone trying to download it — which will fail just as hard on Wikipedia). But you're right, it should probably be img_tag for that purpose and not nil.
There was a problem hiding this comment.
No, nil is correct. That's unchanged. I thought I had tested it and I had but I wasn't sure anymore. :-)
There was a problem hiding this comment.
Good.
So just update logger.dbg("NewsDownloader:... to how any other logger.dbg in wikipedia.lua looks :)
There was a problem hiding this comment.
Ohhh now I see what you were getting at. I'm not sure how that got in there, haha
|
Incidentally, something like this might not be a bad idea either. But perhaps it's better to stick a pcall around the actual download instead. diff --git a/plugins/newsdownloader.koplugin/epubdownloadbackend.lua b/plugins/newsdownloader.koplugin/epubdownloadbackend.lua
index 68e6dda44..cfeadcd97 100644
--- a/plugins/newsdownloader.koplugin/epubdownloadbackend.lua
+++ b/plugins/newsdownloader.koplugin/epubdownloadbackend.lua
@@ -381,6 +381,11 @@ function EpubDownloadBackend:createEpub(epub_path, html, url, include_images, me
end
end
end
+ local parsed_url = socket_url.parse(src)
+ if not (parsed_url and parsed_url.scheme and parsed_url.host) then
+ logger.dbg("skipping unparsable URL", src)
+ return nil
+ end
cur_image = {
imgid = imgid,
imgpath = imgpath, |
This is the underlying reason that The Verge + download full article fails, see <koreader#12953 (comment)>.
This is the underlying reason that The Verge + download full article fails, see #12953 (comment).
This change is