util.urlEncode: encode parentheses and quote#14718
Conversation
- only leave unencoded RFC3986 unreserved characters - additionally accept a list of characters to preserve unencoded, e.g. "/" in file paths Remove WebDavApi.urlEncode and call the improved util.urlEncode directly.
| end | ||
| preserve_chars = preserve_chars or "" | ||
| local regexp_base = "^%w%-%._~" | ||
| local regexp = string.format("([^%s%s])", regexp_base, preserve_chars) |
There was a problem hiding this comment.
Also please change the variable name to pattern since it's a Lua pattern, not a regular expression.
|
This CL does in fact change the behaviour of urlEncode: to make it compliant with RFC 3986 it will encode some characters that the test expects not to be encoded. |
|
It sounds like you're trying to change a function meant for encoding whole well-formed URIs into one that's meant to assembling string values into a URI. That is, without providing further context as to why that would be desirable my thoughts would not be in favor and I'd suggest splitting this function off into one named something like
Then the behavior would be as one expects from the function name and it ought to align with the tests. |
|
Context is in the linked bug.
Which means that |
|
tl;dr FTP and OPDS will need a fair bit of testing prior to merging, and some investigation into third-party plugin usage may be warranted as well. Purely theoretically it should be okay.
That's what I'm getting at. Speaking only of WebDAV isn't context but a lack of context. Note that the Python function has the different purpose I indicated. https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote
urlEncode and urlEncodeComponent/Path/Query are two different things conceptually. But it seems this function was always the latter everywhere, so it doesn't matter in that sense. I'd expect it to behave more or less like this based on the name. I'm not at all convinced your WebDAV server isn't buggy. ;-) I didn't give your original issue too much thought, and besides which it was limited and contained to WebDAV, but the existing tests show what amounts to completely normal and expected behavior. In any case we can observe that cURL behaves the same as that Python function when calling its escape utility function. Unreserved was defined in RFC 2396 as:
Emphasis on "this should not be done". All servers were expected to deal with Then in RFC 3986 that was changed to:
Unsafe in this context seems to mean something like regex/patterns/wildcards. But it's certainly worth noting that it doesn't look like any browser has ever encoded these characters. Way too much related discussion in: |
|
Yeah I agree that urlEncode is not the right name, but I just took it as it is. Anyway: it seemed that the fail was due to a non compliant quoting and I thought to contribute a fix back, but I don't have the time to commit to a thorough testing for FTP and OPDS or external plugins. I agree that it makes sense, but I can't do it, sorry: it seems that the quick fix wasn't so quick after all ._. I'm ok with confining the change to WebDavAPI or to just drop the PR and rename the files when the issue arises. |
|
Like I said I ultimately believe it should be okay; I just needed to put my thoughts and findings in writing for future reference. I'm not noticing any issues with OPDS (other than it definitely being harder to read with encoded |
|
FWIW: I remember having had issues with some files via OPDS with my calibre-web server. IIRC, it was parentheses in the book names causing the issue. |
Improved util.urlEncode:
Removed WebDavApi.urlEncode and called the improved util.urlEncode directly.
This change is