Skip to content

util.urlEncode: encode parentheses and quote#14718

Merged
Frenzie merged 5 commits into
koreader:masterfrom
emdioh:WebDav_UrlEncode
Jan 31, 2026
Merged

util.urlEncode: encode parentheses and quote#14718
Frenzie merged 5 commits into
koreader:masterfrom
emdioh:WebDav_UrlEncode

Conversation

@emdioh

@emdioh emdioh commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

Improved util.urlEncode:

  • only leave unencoded RFC3986 unreserved characters
  • additionally accept a list of characters to preserve unencoded, e.g. "/" in file paths. This allowed to use util.urlEncode instead of WebDavApi.urlEncode

Removed WebDavApi.urlEncode and called the improved util.urlEncode directly.


This change is Reviewable

- only leave unencoded RFC3986 unreserved characters
- additionally accept a list of characters to preserve unencoded, e.g. "/" in file paths

Remove WebDavApi.urlEncode and call the improved util.urlEncode directly.
Comment thread frontend/util.lua Outdated
end
preserve_chars = preserve_chars or ""
local regexp_base = "^%w%-%._~"
local regexp = string.format("([^%s%s])", regexp_base, preserve_chars)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubled ^.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please change the variable name to pattern since it's a Lua pattern, not a regular expression.

@emdioh

emdioh commented Dec 13, 2025

Copy link
Copy Markdown
Contributor Author

This CL does in fact change the behaviour of urlEncode: to make it compliant with RFC 3986 it will encode some characters that the test expects not to be encoded.
I think that a better compliance is a gain (in fact, the non-compliance is what broke my download) but let me know what you think.

@Frenzie

Frenzie commented Dec 13, 2025

Copy link
Copy Markdown
Member

It sounds like you're trying to change a function meant for encoding whole well-formed URIs into one that's meant to assembling string values into a URI. That is, without providing further context as to why that would be desirable my thoughts would not be in favor and I'd suggest splitting this function off into one named something like urlEncodeComponent.

urlEncode could simply call that with an exception for the characters of A–Z a–z 0–9 - _ . ! ~ * ' ( ).

Then the behavior would be as one expects from the function name and it ought to align with the tests.

@emdioh

emdioh commented Dec 13, 2025

Copy link
Copy Markdown
Contributor Author

Context is in the linked bug. WebDavAP.urlEncode() does not encode ()' which makes download fail.
I originally just patched that but then I noticed that WebDavAPI.urlEncode() is a copy of util.urlEncode() that preserves the / character. After doing this, WebDavApi.urlEncode() was superfluous and I removed it.
RFC3986 lists the non-reserved characters that should not be encoded:

unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

Which means that ( ) ' ! * should in fact be %-encoded.
To make util.urlEncode() compliant but allow for the cases in which not all the characters must be encoded I took the same approach that python urllib.parse.quote() has, allowing to specify a set of characters to preserve.
The broken test comes not from the 'preserved' characters, but from the improved adherence to the RFC3986 standard.
I hope this clarifies the context.
I'll fix the tests of course but first let's understand if you're interested in making urlEncode more compliant.

@Frenzie

Frenzie commented Dec 14, 2025

Copy link
Copy Markdown
Member

tl;dr FTP and OPDS will need a fair bit of testing prior to merging, and some investigation into third-party plugin usage may be warranted as well. Purely theoretically it should be okay.


After doing this, WebDavApi.urlEncode() was superfluous and I removed it.

That's what I'm getting at. Speaking only of WebDAV isn't context but a lack of context.

Note that the Python function has the different purpose I indicated.

https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote

By default, this function is intended for quoting the path section of a URL.

urlEncode and urlEncodeComponent/Path/Query are two different things conceptually. But it seems this function was always the latter everywhere, so it doesn't matter in that sense. I'd expect it to behave more or less like this based on the name.

I'm not at all convinced your WebDAV server isn't buggy. ;-) I didn't give your original issue too much thought, and besides which it was limited and contained to WebDAV, but the existing tests show what amounts to completely normal and expected behavior. In any case we can observe that cURL behaves the same as that Python function when calling its escape utility function.

Unreserved was defined in RFC 2396 as:

Unreserved characters can be escaped without changing the semantics
of the URI, but this should not be done unless the URI is being used
in a context that does not allow the unescaped character to appear.

Emphasis on "this should not be done". All servers were expected to deal with () as is, but also it's fine to escape them.

Then in RFC 3986 that was changed to:

The mark characters that are typically unsafe to decode,
including the exclamation mark ("!"), asterisk ("*"), single-quote
("'"), and open and close parentheses ("(" and ")"), have been moved
to the reserved set in order to clarify the distinction between
reserved and unreserved and, hopefully, to answer the most common
question of scheme designers.

Unsafe in this context seems to mean something like regex/patterns/wildcards. But it's certainly worth noting that it doesn't look like any browser has ever encoded these characters.


Way too much related discussion in:

https://github.com/whatwg/url/issues/369

@emdioh

emdioh commented Dec 14, 2025

Copy link
Copy Markdown
Contributor Author

Yeah I agree that urlEncode is not the right name, but I just took it as it is.
And I am definitely not sure the random Android WebDav server I was using is not buggy :)

Anyway: it seemed that the fail was due to a non compliant quoting and I thought to contribute a fix back, but I don't have the time to commit to a thorough testing for FTP and OPDS or external plugins. I agree that it makes sense, but I can't do it, sorry: it seems that the quick fix wasn't so quick after all ._.

I'm ok with confining the change to WebDavAPI or to just drop the PR and rename the files when the issue arises.

@Frenzie

Frenzie commented Dec 14, 2025

Copy link
Copy Markdown
Member

Like I said I ultimately believe it should be okay; I just needed to put my thoughts and findings in writing for future reference.

I'm not noticing any issues with OPDS (other than it definitely being harder to read with encoded ()). I imagine the same will be true for FTP, but I'd have to set up a server first.

@oleasteo

Copy link
Copy Markdown

FWIW: I remember having had issues with some files via OPDS with my calibre-web server. IIRC, it was parentheses in the book names causing the issue.

@Frenzie Frenzie changed the title Improve util.urlEncode to encode parentheses and quote. Fixes #14715 util.urlEncode: encode parentheses and quote Jan 31, 2026
@Frenzie Frenzie merged commit 4d38174 into koreader:master Jan 31, 2026
2 checks passed
0xstillb pushed a commit to 0xstillb/koreader-thai that referenced this pull request May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants