Skip to content

WebDav: Simplify path manipulations to fix wonky heuristics#12038

Merged
NiLuJe merged 3 commits into
koreader:masterfrom
mergen3107:w3_webdav_phantom_fix
Jun 20, 2024
Merged

WebDav: Simplify path manipulations to fix wonky heuristics#12038
NiLuJe merged 3 commits into
koreader:masterfrom
mergen3107:w3_webdav_phantom_fix

Conversation

@mergen3107

@mergen3107 mergen3107 commented Jun 16, 2024

Copy link
Copy Markdown
Contributor

Supersedes #12034
Based on this diff: #12034 (comment)
Closes #12025

@NiLuJe
This diff on top of -191 master caused all folders to disappear from WebDav. I can only see files only.


This change is Reviewable

@mergen3107

Copy link
Copy Markdown
Contributor Author

All folders have empty item_name:

06/16/24-13:15:43 DEBUG LOGG item_fullpath_ =  /Books/ 
06/16/24-13:15:43 DEBUG LOGG item_name_____ =   
06/16/24-13:15:43 DEBUG LOGG is_current_dir =  true 
06/16/24-13:15:43 DEBUG LOGG item_fullpath_ =  /Deus%20Ex%20PC%20Build/ 
06/16/24-13:15:43 DEBUG LOGG item_name_____ =   
06/16/24-13:15:43 DEBUG LOGG is_current_dir =  true 

@mergen3107

mergen3107 commented Jun 16, 2024

Copy link
Copy Markdown
Contributor Author

It's being lost here:

            local item_fullpath = item:match("<[^:]*:href[^>]*>(.*)</[^:]*:href>")
            local item_name = FFIUtil.basename(util.htmlEntitiesToUtf8(util.urlDecode(item_fullpath)))

@mergen3107

mergen3107 commented Jun 16, 2024

Copy link
Copy Markdown
Contributor Author

Narrowed down to FFIUtil.basename:

06/16/24-13:20:54 DEBUG LOGG 1 item_fullpath_________________________________________ =  /Books/ 
06/16/24-13:20:54 DEBUG LOGG 2 util.urlDecode(item_fullpath)_________________________ =  /Books/ 
06/16/24-13:20:54 DEBUG LOGG 3 util.htmlEntitiesToUtf8(util.urlDecode(item_fullpath)) =  /Books/ 
06/16/24-13:20:54 DEBUG LOGG 4 item_name_____________________________________________ =   

@mergen3107

Copy link
Copy Markdown
Contributor Author

(Don't mind underscores in the LOGG var names - I did it for pure alignment :D)

@mergen3107

Copy link
Copy Markdown
Contributor Author

FFIUtil.basename is in C, that's all I can do here for now I guess :D

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

It appears to behave differently than the coreutils basename tool, that's... interesting.

> require("setupkoenv")
> FFIUtil = require("ffi/util")
> item_name = FFIUtil.basename("/Books/")
> print(item_name)

> 
┌─(ROOT@dahlia:pts/0)────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────(/mnt/onboard/.adds/koreader)─┐
└─(0.11:13%:19:27:73%:#)── basename "/Books/"                                                                                                                                                                                                                                                                                                          ──(Sun, Jun 16)─┘
Books

@mergen3107

mergen3107 commented Jun 16, 2024

Copy link
Copy Markdown
Contributor Author

For context, this is from base/util.lua: (see below)

--- Wrapper for C.dirname.
if ffi.os == "Windows" then
    function util.dirname(in_path)
        --[[
        Both PathRemoveFileSpec and dirname will change original input string,
        so we need to make a copy.
        --]]
        local path = ffi.new("char[?]", #in_path + 1, in_path)
        if C.PathRemoveFileSpec(path) then
            return ffi.string(path)
        else
            return path
        end
    end
else
    function util.dirname(in_path)
        local path = ffi.new("char[?]", #in_path + 1, in_path)
        return ffi.string(C.dirname(path))
    end
end

@Frenzie

Frenzie commented Jun 16, 2024

Copy link
Copy Markdown
Member

dirname and basename are two different things

@Frenzie

Frenzie commented Jun 16, 2024

Copy link
Copy Markdown
Member

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

A pure C (POSIX) version behaves correctly, so at least it's not a weird POSIX vs. GNU implementation quirk (as we should be pulling the POSIX one, too).

@mergen3107

Copy link
Copy Markdown
Contributor Author

So, this return ffi.string(C.dirname(path)) should rather be return ffi.string(C.basename(path))?

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

No, you just quoted the wrong bit of FFIUtil ;).

@mergen3107

Copy link
Copy Markdown
Contributor Author

Yep, my bad :D Was looking for ffi.new and got lost (still don't know where to find it).
Here is the correct snippet:

--- Wrapper for C.basename.
if ffi.os == "Windows" then
    function util.basename(path)
        local ptr = ffi.cast("uint8_t *", path)
        return ffi.string(C.PathFindFileNameA(ptr))
    end
else
    function util.basename(in_path)
        -- We have no guarantee of getting a GNU implementation of basename;
        -- POSIX compliant implementations *will* modify input, so, always make a copy.
        local path = ffi.new("char[?]", #in_path + 1, in_path)
        return ffi.string(C.basename(path))
    end
end

@mergen3107

Copy link
Copy Markdown
Contributor Author

Fun fact, KOReader's Terminal Emulator plugin can do basename /Books and results in Books

@Frenzie

Frenzie commented Jun 16, 2024

Copy link
Copy Markdown
Member

If you mean on a device that's probably from busybox; otherwise probably GNU coreutils or equivalent.

@mergen3107

Copy link
Copy Markdown
Contributor Author

Yes, this was on emulator.

@mergen3107

mergen3107 commented Jun 16, 2024

Copy link
Copy Markdown
Contributor Author

It looks all OK until the last operation (this is for /Books/):

06/16/24-14:28:53 DEBUG LOGG FFI path =  cdata<char [?]>: 0x7fb6c7929a68 
06/16/24-14:28:53 DEBUG LOGG FFI C.basename(path) =  cdata<char *>: 0x7fb6c7929a6f

Last operation being ffi.string(C.basename(path)) with yields empty string (I guess).

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

A pure C (POSIX) version behaves correctly, so at least it's not a weird POSIX vs. GNU implementation quirk (as we should be pulling the POSIX one, too).

Except we don't, not since koreader/koreader-base#804, because Android.

And... The GNU version never modifies its argument, and returns the empty string when path has a trailing slash, and in particular also when it is "/".

Raaaah. >_<".

@mergen3107

Copy link
Copy Markdown
Contributor Author

Time for elseif ffi.os == "Android" then?

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member
diff --git a/ffi/util.lua b/ffi/util.lua
index 093fbed7..d4d0a984 100644
--- a/ffi/util.lua
+++ b/ffi/util.lua
@@ -193,10 +193,15 @@ if ffi.os == "Windows" then
     end
 else
     function util.basename(in_path)
-        -- We have no guarantee of getting a GNU implementation of basename;
-        -- POSIX compliant implementations *will* modify input, so, always make a copy.
-        local path = ffi.new("char[?]", #in_path + 1, in_path)
-        return ffi.string(C.basename(path))
+        -- NOTE: While we attempt to prefer the GNU implementation, because it's the only one Android provides,
+        --       we have no guarantee of getting it (e.g., on macOS, or non-glibc Linux systems);
+        --       POSIX compliant implementations *will* modify input, so, always make a copy.
+        -- NOTE: Moreover, the GNU implementation has a few potentially annoying quirks:
+        --       it returns an empty string when path has a trailing slash (as well as for "/").
+        --       So, we'll want to strip trailing slashes, too...
+        local stripped_path = in_path:match(".*[^/]") or "/" -- Ensure basename("/") leads to "/" as per SUSv2
+        local c_path = ffi.new("char[?]", #stripped_path + 1, stripped_path)
+        return ffi.string(C.basename(c_path))
     end
 end

@mergen3107

Copy link
Copy Markdown
Contributor Author

@NiLuJe
Thanks! Emulator checks out well.
I'll go do koreader-base PR then.

Comment thread frontend/apps/cloudstorage/webdavapi.lua Outdated
Comment thread frontend/apps/cloudstorage/webdavapi.lua Outdated
@mergen3107

Copy link
Copy Markdown
Contributor Author

@NiLuJe
Thanks! Tests on Kindle Scribe and emulator passed. All good.

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

Review and testing quite welcome, as I never tested this myself, and I wrote this last "night" while heavily sleep-deprived because insomnia, so, err, here be dragons ;p.

@mergen3107

Copy link
Copy Markdown
Contributor Author

I hope you get better! :D
What are the obvious other places to check basename?
I tried OPDS with Flibusta, no issues.
I am thinking Wikipedia, Online dictionaries, downloading local dictionaries so far.

Anything else?

@NiLuJe

NiLuJe commented Jun 16, 2024

Copy link
Copy Markdown
Member

I'm not too worried about the basename stuff (there are only a few call sites); I was mostly thinking of the actual WebDav stuff ;).

@mergen3107

Copy link
Copy Markdown
Contributor Author

ALright, in that case Yandex.Disk webdav works.

Anyone else using other WebDav clouds?

@mergen3107

Copy link
Copy Markdown
Contributor Author

@NiLuJe
How do I bump the base here (in this PR) so this updates makes to nightly? koreader/koreader-base#1825

@NiLuJe

NiLuJe commented Jun 18, 2024

Copy link
Copy Markdown
Member

With a git client? ;o).

No idea how that's doable without a CLI.

@mergen3107

Copy link
Copy Markdown
Contributor Author

Yes, I am using git on linux :D
I just have never bumped base before.
While on that branch, do I just git pull upstream master and push?

@NiLuJe

NiLuJe commented Jun 18, 2024

Copy link
Copy Markdown
Member

Essentially, yeah.

e.g., cd base && git pull upstream master && cd - && git commit -a

@mergen3107

Copy link
Copy Markdown
Contributor Author

Thanks!
If I did it right, then this PR is ready to merge.

@NiLuJe NiLuJe changed the title WebDav:listFolder - third trial to fix the phantom folders WebDav: Simplify path manipulations to fix wonky heuristics Jun 18, 2024

@NiLuJe NiLuJe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's break all the things \o/

Reviewed 2 of 3 files at r1, 1 of 1 files at r2, 1 of 1 files at r3, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @mergen3107)

@NiLuJe NiLuJe added this to the 2024.06 milestone Jun 18, 2024

@mergen3107 mergen3107 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @mergen3107)

@mergen3107

Copy link
Copy Markdown
Contributor Author

(why does it still say it is waiting for me?)

@NiLuJe

NiLuJe commented Jun 19, 2024

Copy link
Copy Markdown
Member

(why does it still say it is waiting for me?)

It doesn't; you're probably looking at my own review report, which is frozen in time ;).

@mergen3107

Copy link
Copy Markdown
Contributor Author

Someone please merge :D

@NiLuJe NiLuJe merged commit b5a822c into koreader:master Jun 20, 2024
@mergen3107 mergen3107 deleted the w3_webdav_phantom_fix branch June 20, 2024 17:47
Commodore64user pushed a commit to Commodore64user/KOReader_fork that referenced this pull request Aug 2, 2024
…#12038)

This prevents the current directory from appearing in the listing.
0xstillb pushed a commit to 0xstillb/koreader-thai that referenced this pull request May 9, 2026
…#12038)

This prevents the current directory from appearing in the listing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud Storage repetitive phantom folder

3 participants