fix(loader): randomized AppImage path pollutes luac cache #31165#35636
fix(loader): randomized AppImage path pollutes luac cache #31165#35636justinmk merged 1 commit intoneovim:masterfrom
Conversation
7df52cc to
ed0b9b5
Compare
ed0b9b5 to
26c4ad1
Compare
|
Hm, it fails some checks like this: I'm not exactly sure what it doesn't like. Any suggestion how to prevent that? The |
becd3fe to
c069214
Compare
|
I pushed a change, can you confirm it works so we can wrap this up |
Never did it before, how can I build an appimage from my branch? |
|
OK, I managed to repack things using appimage-tool using the patched file. Checking. |
|
Weird, even with fixed file I still see this accumulation: FYI, the way I tested it. Extracted existing appimage release: Replaced Really not sure why it's not helping though. Could be something else is going on? I can try replacing things in the function you suggested for a test. |
|
Nope, we are missing something here. Even moving that fix to the normalize function doesn't help. |
|
Try removing the |
|
Nope, that's not it. It's still using that cursed path in filenames: I think something else is happening besides loader may be? Also, I can confirm that the variable is set if I read this from inside neovim itself Returns binary path. |
|
To double check that the appimage "repack" is actually working, can you try throwing an error in the loader somewhere and confirm that the error is thrown after repack :) |
|
Weird, adding Also, I see my changes in loader.lua that's mounted to /tmp with appimage when neovim is opened. |
|
Just a guess, but confirm that neovim/runtime/lua/vim/loader.lua Line 424 in 2f78ff8 else you are hitting the "standard" path in that if-else, which does not use the custom loader. |
|
I added something like this in But I don't see any exception thrown. I don't get it, is this module even used at all? |
|
i would guess u need to use make to compile as loader.lua may be preload as bytecode |
Where would it come from though? I don't see any luac files in the appimage tree. |
|
how are you starting |
|
I was actually starting re-packed appimage since that should trigger the env variable check. I can try using the above method. |
|
@justinmk: Just got to testing it using your method with providing the modified runtime with the fix. It works! I.e. I unpacked the appimage to Now those previously problematic files are created without the bad prefix like this: So I can confirm that the fix helps. |
|
Nice! Did you try moving the logic to To write a test for this, just add a new
|
|
That's a bit over my head now, I'll have to take a closer look how to make a new test. Do you suggest to add it to some existing unit test file or make a new one? |
|
On doing it in normalize, it does work. I tried this: local function normalize(path)
path = fs.normalize(path, { expand_env = false, _fast = true })
if os.getenv('APPIMAGE') ~= nil then
-- Avoid cache pollution caused by AppImage randomizing the program root. #31165
-- "/tmp/.mount_nvimAmpHPH/usr/share/nvim/runtime" => "/usr/share/nvim/runtime"
path = path:match("(/usr/.*)") or path
end
return path
end |
1b8d6f1 to
321338a
Compare
try an AI to get something started. You can run the test like this:
|
321338a to
19e1b39
Compare
19e1b39 to
0a8c61e
Compare
0a8c61e to
97692bd
Compare
|
I'll look into the test a bit later. |
|
rebase? |
|
I'll refersh it soon, but I didn't have a chance to look into unit tests. I'll have some time for that later in the month. |
97692bd to
f2534e8
Compare
|
We can just merge this for now. Meanwhile, the value of |
|
Successfully created backport PR for |
Problem:
When using the Nvim appimage, `~/.cache/nvim/luac` directory can grow to
250,000+ files.
Example of 2 identical files in `./luac/`:
%2ftmp%2f.mount_nvim.a65Rja0%2fusr%2fshare%2fnvim%2fruntime%2flua%2fvim%2ftreesitter.luac
%2ftmp%2f.mount_nvim.aNpxXgo%2fusr%2fshare%2fnvim%2fruntime%2flua%2fvim%2ftreesitter.luac
Analysis:
The `nvim.appimage` mounts nvim at a different temporary path each time
it is invoked. The naming scheme of these cache files is random, which
defats the purpose of the cache creates N new files on every launch of
nvim.
Steps to reproduce:
1. install `nvim.appimage`
2. `mv ~/.cache/nvim/luac ~/.cache/nvim/luac.backup`
3. `nvim`
4. Observe contents of `~/.cache/nvim/luac/`
5. Close nvim and run `nvim` again
6. Observe contents of `~/.cache/nvim/luac/` and see that new identical
files have been added with a different mount prefix
Solution:
When running from an appimage, trim the random part of the filepaths.
(cherry picked from commit 78bbe53)
Problem:
When using the Nvim appimage, `~/.cache/nvim/luac` directory can grow to
250,000+ files.
Example of 2 identical files in `./luac/`:
%2ftmp%2f.mount_nvim.a65Rja0%2fusr%2fshare%2fnvim%2fruntime%2flua%2fvim%2ftreesitter.luac
%2ftmp%2f.mount_nvim.aNpxXgo%2fusr%2fshare%2fnvim%2fruntime%2flua%2fvim%2ftreesitter.luac
Analysis:
The `nvim.appimage` mounts nvim at a different temporary path each time
it is invoked. The naming scheme of these cache files is random, which
defats the purpose of the cache creates N new files on every launch of
nvim.
Steps to reproduce:
1. install `nvim.appimage`
2. `mv ~/.cache/nvim/luac ~/.cache/nvim/luac.backup`
3. `nvim`
4. Observe contents of `~/.cache/nvim/luac/`
5. Close nvim and run `nvim` again
6. Observe contents of `~/.cache/nvim/luac/` and see that new identical
files have been added with a different mount prefix
Solution:
When running from an appimage, trim the random part of the filepaths.
(cherry picked from commit 78bbe53)
Following is a list of fix/feature commits in this release. See `:help news` in Nvim for release notes. FEATURES -------------------------------------------------------------------------------- - b92e92b lsp: support auto-force escalation in client stop neovim#36430 - 808d973 lsp: warn about unknown filetype neovim#36910 FIXES -------------------------------------------------------------------------------- - 5e7af0b :ls: check for finished terminal properly (neovim#37303) - 6ce7b9b api: autocmds mess up nvim_get_option_value's dummy buffer - c124594 api: buffer overflow in nvim_buf_get_extmarks overlap neovim#37184 - 7f51431 api: crash when moving curwin to other tabpage neovim#35679 - 7896fe2 api: do not allow opening float to closing buffer - 91ebbc6 api: ignore split_disallowed when opening a float - 10a1df2 api: nvim_get_option_value dummy buffer crashes - da825e5 api: on_bytes gets stale data on :substitute neovim#36487 - a9ffdca api: open_win leak from naughty autocommands - 92849ea api: parse_expression crash with ident and curly - 1db945b api: parse_expression crash with unopened ] and node - 79b67ce appimage: wrong $ARCH used by linuxdeploy neovim#36712 - 46011a4 autocmd: fire TabClosed after freeing tab page - a512d43 autocmd: heap UAF with :bwipe in Syntax autocmd - bea500d autocmd: parsing of comma-separated buflocal patterns - 648cf4e autocmd: skip empty comma-separated patterns properly - fa24e04 buffer: defer w_buffer clearing to prevent dict watcher crash neovim#36748 - 9fb49aa buffer: don't allow changedtick watcher to delete buffer (neovim#36764) - cae3c83 buffer: don't reuse 1-line terminal buffer (neovim#37261) - 6f84ea7 buffer: switching buffer should respect jumpoptions+=view (neovim#36969) - 16ca7ce build: disable problematic marktree assert in RelWithDebInfo builds - c469cba channel: unreference list after callback finishes (neovim#37358) - 43f5297 clipboard: tmux clipboard data may be stale neovim#36787 - 0358f37 clipboard: use tmux only in a tmux session (neovim#36603) - 7e99466 eval: 0 should mean current tabpage in gettabvar() (neovim#36891) - 656ff4c events: crash on WinScrolled neovim#35995 - 63c5a10 install: only install "tee" on Windows neovim#36629 - e8c21a8 langmap: assert failure on mapping to char >= 256 (neovim#37291) - 890c257 lsp: check `nvim.lsp.enable` before `doautoall` neovim#36518 - 275c769 lua: don't remove first char of non-file stacktrace source (neovim#37008) - 83c589d lua: relax `vim.wait()` timeout validation (neovim#36907) - bd2317f lua: separate vim.{g,b,w,t} types neovim#37081 - f21c169 lua: vim._with() doesn't save boolean options properly (neovim#37354) - 9acbf51 lua: vim.wait(math.huge) fails neovim#36885 - 5143419 man.lua: :Man slow/hangs if MANPAGER is set neovim#36689 - df9452e man.lua: show_toc condition may cause infinite loop neovim#36979 - 124c182 marks: wrong line('w$', win) with conceal_lines (neovim#37047) - 6ef1b65 normal: assertion failure with "gk" in narrow window (neovim#37444) - 5ca2eb5 remote: remote-ui connect timeout on slow networks neovim#36800 - 2a3cd8d rpc: don't overwrite already received results on error (neovim#37339) - e0fdfd3 scripts: release.sh - ba600c4 session: window sizes not stored with float windows (neovim#37344) - 53090ab statusline: scope truncation bookkeeping - 800118e terminal: :edit should respect 'bufhidden' with exited job (neovim#37301) - 4b41c28 terminal: <Ignore> should be no-op (neovim#37494) - 074d342 terminal: avoid multiple terminals writing to same buffer (neovim#37219) - 2cc7873 terminal: crash when TermClose deletes other buffers - ea87192 terminal: crash when TermClose switches back to terminal buffer - ceed171 terminal: crash with race between buffer close and OSC 2 (neovim#37225) - acc46e1 terminal: handle closing terminal with pending TermRequest (neovim#37227) - bb31e7b terminal: inconsistent mode change when switching buffer (neovim#37215) - 40c974e terminal: restore options properly when switching buffer (neovim#37485) - 46f569a treesitter: use metadata in :EditQuery captures neovim#37116 - 076f799 trust: :trust command on Windows neovim#36509 - d997c8e tutor: escape tutor filename neovim#36539 - fcd0517 ui.open: use "start" instead of deprecated "rundll32" neovim#36731 - 6a507ba vim.fs: abspath(".") returns "/…/." (neovim#36584) - d974c68 vim.fs: root() should always return absolute path neovim#36466 - 91fd4d1 vim.loader: randomized AppImage path pollutes luac cache neovim#35636 - 45cda1b vim.loader: randomized AppImage path pollutes luac cache neovim#36944 - d9631c7 window: crash closing only non-float if autocmds :tabonly (neovim#37218) - f7e2554 window: crash closing split if autocmds close other splits (neovim#37233) - 7a9bced window: disallow closing autocmd window in other tabpage - 88619e1 window: handle closing the only non-float in other tabpage - d38ba7e window: restore b_nwindows if win_close_othertab keeps window - 6338d2d window: win_move_after UAF from naughty autocmds (neovim#37065) - fac7c10 windows: set manifest resource ID to 1 in nvim.rc for MinGW (neovim#36611) BUILD -------------------------------------------------------------------------------- - d0ed06d haiku os support neovim#36639 - a94647b build(windows): restore "tee" on Windows neovim#36627 - 1f93acc build(windows): vendor xxd.c (neovim#36755) REVERTED CHANGES -------------------------------------------------------------------------------- - ae25f69 fix: vim.lsp.omnifunc should not throw away other items VIM PATCHES -------------------------------------------------------------------------------- - b3eab00 229f79c: runtime(yaml): fix wrong order of undo_ftplugin suboptions - 89f8e97 3a324c8: runtime(doc): Fix typo in syntax.txt (neovim#37522) - d1cd79a 64799a5: runtime(doc): clarify the behavior of CTRL-Z - 0978d83 7bc9880: runtime(make): do not automatically indent after a special target - 781da75 8.1.0753: printf format not checked for semsg() (neovim#37248) - 44eae48 9.1.0893: No test that undofile format does not regress (neovim#37193) - 9a50420 9.1.1872: Cmdline history not updated when mapping <Up> and <CR> (neovim#36334) - d1604e0 9.1.1969: Wrong cursor position after formatting with long 'formatprg' (neovim#36918) - fda8d2c 9.1.2028: [security]: Buffer-overflow with incomplete multi-byte chars (neovim#37133) - f96e401 9.1.2055: Division by zero in :file after failing to wipe buffer (neovim#37268) - f8961c3 9.1.2058: b_locked_split is not checked for :sbuffer - 9f2b991 9.1.2066: :wqall! doesn't close a terminal like :qall! does (neovim#37314) - b1fa8f1 9.1.2068: :bd/bw may try to switch to a closing buffer - 600d9f3 9.1.2086: Memory leak when skipping invalid literal dict - 0cc15be 9.1.2087: Crash when using :tabonly in BufUnload - d052d22 9.1.2090: Last buffer not freed with EXITFREE - 537e8d6 9.1.2095: :wqall! doesn't quit when using :quit in BufWritePost - 0b1f5a1 9.1.2105: tests: not enough tests for using plain_vgetc() (neovim#37521) - 0da1e4b 9.1.2107: :normal may change cmdline history (neovim#37523) - a66fce6 98a0cbf: patch 9.1.1971: crash with invalid positional argument 0 in printf() (neovim#36919) - 85404d1 eb732ed: runtime(doc): Wrap overlength lines in uganda.txt (neovim#36550) - a93b5a7 fe8c8b1: runtime(doc): fix outdated :function help - 0706c55 partial:9.1.1955: sort() does not handle large numbers correctly (neovim#36840)
Fixes infinite growth of number of *.luac files in the cache and actual cache file misses caused by randomness of filenames when using AppImage with loader enabled.
fix #31165