Conversation
|
I'm ok with the code, but I wonder if it would better be made into its own file (frontent/livepatch.lua or startuppatch.lua or readerpatch.lua ?) to save 30 lines from the small and essential reader.lua where these 30 lines are a bit distracting :) |
|
+1 for outsourcing the patch thing. For the scripts execution question there is #6297. |
|
@zwim: can we get rid of https://github.com/koreader/koreader/blob/master/platform/android/llapp_main.lua#L19-L65 when this is merged? |
|
@pazos: If we drop https://github.com/koreader/koreader/blob/master/platform/android/llapp_main.lua#L19-L65 completely, we would lose some functionality. I will see if I can move the functionality to At a first glance: I see the main difference, that |
|
At a second glance I have found a minor problem. With the migrating function we can replace base libraries (crengine ...). If we do that later (in |
|
So far only tested on the emulator. Will test on Kobo tomorrow. |
| end | ||
|
|
||
| --- A wrapper to os.execute | ||
| function util.execute(...) |
There was a problem hiding this comment.
We already have a util.execute() in base/ffi/util.lua - so even if this one's argument may be different, this could become confusing.
Also see FileManager:moveFile() in frontend/apps/filemanager/filemanager.lua.
We also have a util.shell_escape(args) used in readerdictionary.lua, which might be better at handling the quoting you do below (or not, dunno :).
Anyway, using the shell like you do here (and like we do elsewhere, because nobody cleaned that up :) is a bit odd and ugly, when Lua has a os.rename().
| -- Apply developer patches | ||
| require("livepatch") | ||
|
|
There was a problem hiding this comment.
You know more than us about what kind of stuff can be in your scripts.always/afterupdate, and if they are fine being executed that late.
But I feel the Android patch.lua might be used to tweak the framebuffer dimensions (@pazos ? @NiLuJe ) , and I guess it needs to be run before reader.lua - or may be at its very top. So, you might need to stage in live.patch.lua, that you could import early and run the 0* scripts early, and others later ?
Or local livepatch = require("livepatch") at top and have it run any existing patch.lua (without moving it) at require time - and for the later stuff, here do livepatch.executeLivePatches()
|
So now we have
In all tree folders there might be shell-scripts, lua patches and the migrate funcitonallity. (Not tested for now if lua-patches and migrate bite each other, without migrate it works.) The contents of scripts-directories and Te contents of the Tested on the Emulator, Kobo Sage and on my mobile. |
| -- Set up ffi search paths | ||
| require("setupkopaths") | ||
|
|
||
| -- Apply `patches/patch.lua` and execute startup user scripts and perform user data migration | ||
| local livepatch = require("livepatch") | ||
| livepatch.executeScriptsAndMigrate() | ||
|
|
||
| -- Load default settings | ||
| require("defaults") | ||
| local DataStorage = require("datastorage") |
There was a problem hiding this comment.
I'm not sure this split of setupkoenvs into setupkopaths is really needed - as setupkoenv doesn't seem to use anything from the DataStorage stuff (and the loading of defaults.persistent.lua that happens just after what I could select to comment on :) and the require("setupkoenv") that happens just after.
So, I guess you could just move the require("setupkoenv") where you put your require("setupkopaths").
I also guess no patch/livepatch stuff would need to tweak the ffi load stuff hat happens in setupkoenv, and this patching stuff might actually be happy to have the ffi loading stuff available.
There was a problem hiding this comment.
Hmm, the current implementation is as close to the deleted Android one as possible. Splitting setupkoenv in two parts allows:
1.) to patch the search paths (e.g. a custom plugin) and
2.) copy some .so libraries before they are used (e.g. crengine).
(So a really developer, testing thing).
There was a problem hiding this comment.
The 2nd part of setupkoenv.lua does not load any library (I think, @NiLuJe ?), it just redefines ffi.load, so I think this won't hurt if done early if you just move the require("setupkoenv") where you put your require("setupkopaths").
| -- If an older `patch.lua` is found it is moved to `patches/master-patch.lua`. This has to be done here, | ||
| -- as onetime_migration gets called to late. |
There was a problem hiding this comment.
"master-patch.lua" feels a bit unprecise (why "master" ?). "early-patch.lua" would sound more true.
And if we allow multiple "late" patch files, why only a single "early" file ?
Also, dunno about moving an existing patch.lua (I would just run it early if it exists).
There was a problem hiding this comment.
I wanted to bundle all developer scripts and patches in as least places as possible. IMHO an additional patch.lua feels weird.
And I did not want to change existing behaviour, so we already have scripts.always and scripts.afterupdate. But you could complain that patches does not fit in that row. So what about renaming patches-> scripts.late (And I move the early-patch.lua into scripts.always).
Note to self: Check if folder exists, before moving.
There was a problem hiding this comment.
IMHO an additional patch.lua feels weird.
I mean, don't advertise about it, but if there is one (legacy user), run it if found.
But advertise the new common way to do that.
I wanted to bundle all developer scripts and patches in as least places as possible
So do I :)
And I did not want to change existing behaviour
I have no problem with changeing it, current users are rare (mostly us), and will notice it and update it according to the new way/documention :) I don't think we would need to do any migration about them.
(Except for patch.lua that some old users may have added one with viewport stuff specific to their devices, and that could have forgotten about it - so either keep it in its original place and use it if found - or move it as you did.)
If we don't change anything, these directory names will sound inconsistent:
scripts.always/ (ok, always, but late or early ?)
scripts.afterupdate/ (ok, only after update, but always or early ? :)
scripts.late/ (ok, late, but always I guess)
So, we might want to decide about a generic futureproof scheme. Directories (but they would clutter koreader/ directory), convention about filenames (like /etc/rc1.d/ ... rc6.d/ on old Unix) in a single directory (patch/ livepatch/ userscripts/?) ?
There was a problem hiding this comment.
Yep inconsistent. May I propose:
livepatches/early_afterupdate (early and after an update)
livepatches/early (early independend to updates)
livepatches/late (late independend to updates)
There was a problem hiding this comment.
These would be subdirectories ? or filenames ? or filename prefixes ?
Also, livepatches and livepatch.lua may better be named userpatch.lua, to make it obvious it's for handling... users' patches :) (or is the notion of "live" patch to distinguish from manual code patching is important ?)
There was a problem hiding this comment.
Thees would be subdirectories, with files and folders in them.
userpatches/early_afterupdate/ (early and after an update)
userpatches/early/ (early independend to updates)
userpatches/late/ (late independend to updates)
There was a problem hiding this comment.
with files and folders in them.
Felt like overengineering :) but I guess "folders in them" is just for content that should be copied into koreader's own tree, like hyphenation files ? (and you would not want a tweaked translator.lua - to be copied over koreader's one - to lie into userpatches/early/ and be considered a patch itself).
I guess personally, I would have gone with:
file starting with 0-* : early patch
file starting with 1-* : later patch
file starting with 0somethingelse-* afterupdate early patch (no real idea for a prefix :)
anything else: ignore.
So everything is quite visible in that layout, and you can just rename a file (add a NOT_0-test.lua prefix) to have it not considered.
| if lfs.attributes(afterupdate_marker, "mode") == "file" then | ||
| local migrate = lfs.attributes(run_once_scripts .. "/migrate", "mode") ~= nil | ||
| logger.info("after-update: running", migrate and "migration" or "shell scripts") |
There was a problem hiding this comment.
What for do you use this "/migrate/" stuff ? For your german hyphenation test file ? Anything else ?
Dunno if other users than you do use this feature - but may be we could simplify the amount of available tricks :) and just use early and last lua scripts. You could do any move/migration from a Lua script (even if shell scripts might be easier to write for that kind of thing), it would feel cleaner to me.
You could add helpers methods in this livepatch.lua so shell stuff like cp can be easily done in a lua file.
(I'm having a hard time understanding what's possible and what happens when reading this new linvepatch.lua.)
There was a problem hiding this comment.
And if "migrate" stuff only do copy of files from the user dir into koreader distributed dir, I would at least don't use the word "migrate" in the code/comments, and it feels to me you are migrating some files/settings. Just "copy" would work and read clearer.
There was a problem hiding this comment.
Originally I thought that it was you, who wanted and implemented the migrate thing ;) But after studying the old commits I believe @pazos introduced the migrate functionality.
I don't use migrate at all. So, if we want to drop it, I would be glad ;)
There was a problem hiding this comment.
One question concerning the execute function. Now it uses os.execute. Would it be of any benefit to use os.rename and os.remove where possible?
There was a problem hiding this comment.
Pure Lua for clarity, and it would avoid some issues with executing /bin/mv via the shell and the need to escape filenames properly.
|
Changes to |
| early_afterupdate = "0", | ||
| early = "1", | ||
| late = "2", |
There was a problem hiding this comment.
Good idea, looks logical :)
| if mode == "file" and fullpath:match("%.sh$") then -- execute shell scripts | ||
| execute("sh", fullpath, home_dir .. "/koreader", package_dir) | ||
| elseif mode == "file" and fullpath:match("%.lua$") | ||
| and not fullpath:match("early%-patch%.lua$") then -- execute patch-files |
There was a problem hiding this comment.
is that early%-patch%.lua check still needed ? or leftover ?
There was a problem hiding this comment.
a harmless leftover -> to be removed.
|
Do you see any problems with removing the migrate option (IMHO any script or lua-patch can do that if someone really needs this). Is this solution to apply (0000-patch.lua) very early in |
|
@NiLuJe : we need some feedback from you (questions above, and general feeling about this PR). Just pasting here some very old patch.lua I had around, with various experiments of mine, so we can see what kind of trick people could have done (and might still be doing): -- Android Koreader patches
-- Avoid koreader stuck and crashes
require("jit.opt").start("sizemcode=64","maxmcode=512", "recunroll=8", "callunroll=12", "loopunroll=60", "instunroll=16", "tryside=16", "hotexit=160", "hotloop=1792", "maxsnap=160000", "maxside=32000", "maxirconst=160000", "maxrecord=1280000", "maxtrace=320000")
for i=1,1000 do end -- Force allocation of one large segment
-- Remove this plugin
local A = require("android")
A.execute("rm", "-rf", "plugins/backgroundrunner.koplugin")
-- Set CREngine documents background color
local called_once = false
local fba = require("ffi/framebuffer_android")
fba.refreshFullImp_orig = fba.refreshFullImp
fba.refreshFullImp = function(self)
fba.refreshFullImp_orig(self)
if not called_once then
called_once = true
return
end
fba.refreshFullImp = fba.refreshFullImp_orig
local CreDocument = require("document/credocument")
CreDocument.initOrig = CreDocument.init
CreDocument.init = function(self)
CreDocument.initOrig(self)
self._document:setStringProperty("background.color.default", "#eeeeee")
end
end
-- Mais en fait non
fba.refreshFullImp = fba.refreshFullImp_orig
-- Make white be gray
-- local BB = require("ffi/blitbuffer")
-- BB.COLOR_WHITE = BB.Color8(0xCC)
-- OLD, no more needed:
-- fba.init = function(self)
-- self.bb = BB.new(android.screen.width, android.screen.height, BB.TYPE_BBRGB32)
-- self.bb:fill(BB.COLOR_WHITE)
-- self:refreshFull()
-- fba.parent.init(self)
-- end
-- OLD, no more needed:
-- local android = require("android")
-- android.setScreenBrightnessOrig = android.setScreenBrightness
-- android.setScreenBrightness = function()
-- android.setScreenBrightness = android.setScreenBrightnessOrig
-- end |
|
I might have some time to look at it tomorrow evening. If not, ping me again on Sunday ;). |
|
@zwim: I'm sorry. Was so busy this week :p.
Not really, I coded it as a friendlier manner of copying stuff to /data/data/org.koreader.launcher than shell scripts. I would also like the shell scripts patch refactored. I wrote something a few days ago but I had no time to finish it. (and a lot of commits happened since then). I'm hope it is still relevant: |
|
Anyhow, I'll plan to review this on sunday too :) |
Should work with
Yep, we can have *.sh and *.lua patches now.
No need for that, if we the patches are in level 0 or 1. |
pazos
left a comment
There was a problem hiding this comment.
I don't like the current state of affairs (at all!)
Looks utterly complicated for something that should be way simpler: provide different hooks/entry points for developers to pcall stuff and modify the behaviour/logic of the program during runtime execution.
Since I'm going to comment on certain issues that I see on the code I'm trying to put my global impression as a general comment here: it's not a matter of fixing the specific issues but a design decision:
do we need all these priorities, all those hooks and the ability to execute shell scripts buried inside this part of the logic?
| } | ||
|
|
||
| if isAndroid and android.prop.flavor == "fdroid" then | ||
| return userpatch |
There was a problem hiding this comment.
No!. Returning the whole field, will allow us to call the no-op applyPatches even on F-Droid -> see below.
| return userpatch -- live patching is not supported | ||
| end | ||
|
|
||
| logger.info("Live update using package_dir:", package_dir, "; home_dir:", home_dir) |
There was a problem hiding this comment.
I don't want this logged unconditionally each time the program starts on devices/users that don't use this feature.
There was a problem hiding this comment.
Yepp, that's just for testing this PR -> will be removed.
| return os.execute(table.concat(command)) | ||
| end | ||
|
|
||
| --- Run user shell scripts or lua patches |
There was a problem hiding this comment.
what is the purpose of running arbitrary shell scripts?
Now that you have hooks available to run arbitrary lua code sounds easier to just call shell/python or perl scripts from your own lua patches.
There was a problem hiding this comment.
Executing shell scripts is a 4-liner and why not use it. If you want execute patch .... it would be clumsy to write a patch.lua and execute the patch command there.
NiLuJe
left a comment
There was a problem hiding this comment.
Reviewed 8 of 11 files at r5, 3 of 3 files at r10, all commit messages.
Reviewable status: all files reviewed, 17 unresolved discussions (waiting on @Frenzie, @pazos, @poire-z, and @zwim)
reader.lua line 25 at r5 (raw file):
Previously, zwim wrote…
OK, let's wait on @NiLuJe.
Yep, it just sets up the searchpaths for ffi.load.
I have no idea what the original impelmentation was used for, but touching ffi.load seems like a terribad idea ;).
(You can just pass a path to ffi.load if you ever need to load something in a plugin).
reader.lua line 384 at r10 (raw file):
-- Apply exit user patches and execute user scripts userpatch.applyPatches(userpatch.on_exit)
Kind of the same question as @poire-z above:
What's the use-case for those?
Code quote:
-- Apply before_exit patches and execute user scripts
userpatch.applyPatches(userpatch.before_exit)
local reader_retval = exitReader()
-- Apply exit user patches and execute user scripts
userpatch.applyPatches(userpatch.on_exit)|
Same general feeling as @pazos, that it feels a bit cthulhuesque for no good reason. Then again, I'm not the target audience, so that doesn't really help getting into it ;). |
|
|
||
| -- Date at which the last migration snippet was added | ||
| local CURRENT_MIGRATION_DATE = 20220523 | ||
| local CURRENT_MIGRATION_DATE = 20220615 |
There was a problem hiding this comment.
One last update of this date :) and we're good to go.
There was a problem hiding this comment.
That was a long birth. But it looks good ;)
|
For documentation: A small patch to add a menu entry for USBNet can be found here https://gist.github.com/zwim/b0d46fa83d9dbb853324f3e0c17562b5 Works on a Kobo. |
|
Speaking of documentation: this should probably be documented somewhere on the Wiki? ;). |
|
Do you mean the userpatch-feature itself, or a collection of available patches or the USBNet-specific patch? For general documentation of the userpatch option I would ping @offset-torque, I would vote for the Wiki-page. |
|
All of the above? ;).
…On Fri, Oct 14, 2022, 23:24 zwim ***@***.***> wrote:
Do you mean the userpatch-feature itself, or a collection of available
patches or the USBNet-specific patch?
For general documentation of the userpatch option I would ping
@offset-torque <https://github.com/offset-torque>,
for documenting available (untested) userpatches I could imagine a
Wiki-page (maybe witch links to gists for downloading)
and the special USBNet-patch one I don't know the expected audience.
I would vote for the Wiki-page.
—
Reply to this email directly, view it on GitHub
<#9104 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAA3KZWSCG7E3QHRDEFEYK3WDHFQDANCNFSM5WEMR7FQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
:+1 If nobody else wants to to this, I will try to include this in the wiki. Maybe between
|
|
Yep, this was on my radar which I was going to ask about.
Exactly what I imagined. I can explain this feature in the User Guide and direct users to the wiki for checking the available patches. When there is a new patch, its author can add it to the wiki with a short explanation. I was in the process of adding an Advanced Operations section/tag to the User Guide so this feature already has a place to go. |
And there is a special label: |
|
I have done a preliminary Wiki entry. Please feel free to change/update. |
Supersede old android-only patch.lua.
|
For reference and for @kaznelson, regarding https://www.mobileread.com/forums/showthread.php?p=4314627#post4314627 and patching gesturedetector.lua's private module-local variable object koreader/frontend/device/gesturedetector.lua Lines 121 to 129 in af45ec7 I wondered if with some Lua magic we could reach it, and it happens we can, so sharing this useful trick: local GestureDetector = require("device/gesturedetector")
-- local logger = require("logger")
-- local z = debug.getinfo(GestureDetector.newContact)
-- logger.warn(z)
-- Find the Contact object, which is local and private to gesturedetector.lua.
-- But as the public GestureDetector.newContact uses it, we can fetch it from
-- its upvalues!
local Contact
local n = 1
while true do
local name, value = debug.getupvalue(GestureDetector.newContact, n)
if not name then break end
if name == "Contact" then
Contact = value
break
end
n = n + 1
end
if not Contact then return end
-- logger.warn("found:", Contact.getPath)
-- Now we can override that private object method
Contact.getPath = function(self, simple, diagonal, initial_tev)
return "west", 123 -- replace that with your code
end |
|
Oops, I'd forgotten that (Usual caveat that anything from the |
|
(The debug.stuff would just be used once, just when executing the patch to setup things - and indeed, in the things it setups, it better be classic lua, and optimized if run a lot.) |
|
Oh, yeah, it's definitely harmless here, that was just a preemptive warning about its usage in general ;o). |
|
@NiLuJe, @poire-z |
|
For reference, another trick (following the one above about reaching local variables), to allow reaching and user patching plugins: local PluginLoader = require("pluginloader")
-- On each new Reader/FileManager, plugins are dofile()d, and then
-- instantiated thru createPluginInstance: catch there ere.
local patch_plugin_func -- defined below
local orig_PluginLoader_createPluginInstance = PluginLoader.createPluginInstance
PluginLoader.createPluginInstance = function(self, plugin, attr)
local ok, plugin_or_err = orig_PluginLoader_createPluginInstance(self, plugin, attr)
if ok and plugin.name == "your_target_plugin_name" then -- ie. "coverbrowser"
patch_plugin_func(plugin)
end
return ok, plugin_or_err
end
patch_plugin_func = function(plugin)
-- do your patching to the plugin object here
endSome hardcore example combining these 2 tricks at #11838 (comment). |
Supersede old android-only patch.lua.
This PR allows to apply developer patches. See #9058 (comment) and onward.
The
patchesdirectory is not created by default, so a regular user won't see any change.Don't allow patching on FDroid.
This change is