Statistics: fix null terminated strings, add toggle stats#12547
Conversation
@benoit-pierre can you look at it please. |
| if nr_id == 0 then | ||
| authors = authors .. "\0" | ||
| result = stmt:reset():bind(title, authors, self.doc_md5):step() | ||
| nr_id = tonumber(result[1]) | ||
| if nr_id == 0 then | ||
| title = self.data.title | ||
| result = stmt:reset():bind(title, authors, self.doc_md5):step() | ||
| nr_id = tonumber(result[1]) | ||
| end | ||
| end | ||
| end |
There was a problem hiding this comment.
Can we really get a mismatch between authors & titles in terms of NUL termination (e.g., only one of them being "sane")?
Not familiar with the whole thing, but I would just have basically expected a single fallback to result = stmt:reset():bind(title .. '\0', authors .. '\0', self.doc_md5):step()
There was a problem hiding this comment.
I think yes. If document title is absent, we use display_title (filename without extension), if document authors is absent, we use "N/A".
There was a problem hiding this comment.
I've checked once more.
If a property is absent, muPDF returns "" (empty string without null-terminator).
(ADD: it did so even before our fix of null-terminators)
Then we do
koreader/frontend/document/document.lua
Lines 175 to 177 in ca569c1
So, in
doc_props we have nil for empty properties.Then we build
display_title (that doesn't have null-terminator if title is empty).Finally the statistics plugin uses:
koreader/plugins/statistics.koplugin/main.lua
Lines 176 to 177 in ca569c1
Summarizing: old
authors and title in stats had null-terminator if they existed, and didn't have if any of them was empty in the document. We need to check 4 combinations.
| if nr_id == 0 and self.ui.paging then -- old strings are null-terminated | ||
| title = title .. "\0" |
There was a problem hiding this comment.
May be make that comment more verbose:
In the past, title and/or authors strings, got from MuPDF, may have been or not null terminated.
We need to check with all combinations if a book with these null terminated exists, and use it.
Btw, are these auto updated to non-nul-terminated elsewhere ? I see you don't explicitely update them.
There was a problem hiding this comment.
They remain null-terminated.
There was a problem hiding this comment.
No real opinion whether we should fix them or let them be, if they are just getting stored as-is and we find them again, and what the user sees is clean.
Any thought about whay may happen with the cloud storage statistics sync if between devices, and one has that book opened with strings nul terminated, and the other device has it without them?
There was a problem hiding this comment.
I'm not good at sync, but at the first glance
koreader/plugins/statistics.koplugin/main.lua
Lines 3166 to 3171 in ca569c1
it's better to keep the strings in the local db unchanged.
KOReader 2024.11 "Slang"  Some of you on Android may have had dictionary issues. That should be fixed now. Connecting to Wi-Fi should be more reliable on platforms where we manage it ourselves, most notably Kobo (koreader/lj-wpaclient#11). Color is now supported when reflowing (koreader#12085). This month's logo was contributed by @Sonnenfee We'd like to thank all contributors for their efforts. Some highlights since the previous release include: * Certain network features will now work in Flatpak (<flathub/rocks.koreader.KOReader#52>) @kii-chan-reloaded * 'Use left and right keys for page turning' option added (koreader#12174) @Commodore64user * Text editor: enhancement (koreader#12176) @hius07 (1) Speed up adding/deleting chars (by avoiding a call of expensive splitToChars), noticeable on big files. (2) Indicate select mode: wider cursor. * Annotations: more fixes (koreader#12181) @hius07 (1) Allows annotations with equal datetime fields. (2) Allows page bookmarks with equal page numbers. * kopt: color support (koreader#12085) @benoit-pierre * bump base (updated build system) & fix macOS CI (koreader#12187) @benoit-pierre * AltStatusBar/Footer: add the read timer value (koreader#12002) @zwim * bump translations (koreader#12193) @pazos * CRE call cache: don't wrap setViewMode (koreader#12194) @poire-z * bump crengine: various fixes and optimizations (koreader#12200) @poire-z * Add author option for alt status bar (koreader/crengine#575) (koreader#12197) @trash-pandy * Page turns: invert vertical tap zones (koreader#12195) @hius07 * Annotations: pageno honors reference pages and hidden flows (koreader#12214) @hius07 * ToggleSwitch: fix long-press (koreader#12215) @hius07 * SDL: Return true in NetworkMgr:isConnected on !hasWifiToggle platforms (koreader#12217) @NiLuJe * Kobo: Refactor various aspects of the Kaleido/MTK support (koreader#12221) @NiLuJe * ReaderAnnotation: fix doesMatch func (koreader#12231) @hius07 * bump base (koreader#12241) @benoit-pierre * NetworkMgr: Attempt to handle wpa_supplicant rescans better (koreader#12236) @NiLuJe * miscellaneous build system changes (koreader#12242) @benoit-pierre * make: handle old build data directory (koreader#12259) @benoit-pierre * ReaderFooter: improve (koreader#12243) @hius07 * Annotations: fix exporting (koreader#12261) @hius07 * make: install missing `cr3.css`… (koreader#12262) @benoit-pierre * [fix] NetworkMgr: apply HTTP proxy setting on startup (koreader#12260) @Frenzie * Kobo: Track frontlight state more accurately on suspend/resume (koreader#12256) @NiLuJe * Make W-Fi scan more reliable (koreader#12265, koreader/lj-wpaclient#11) @NiLuJe * ReaderAnnotation: fix doesMatch func 2 (koreader#12270) @hius07 * make: fix android update sdcv (koreader#12275) @benoit-pierre * [plugin] Reader footer: check if header resp. footer are available (koreader#12264) @zwim * ReadHistory: On remove, trust the index from FileManagerHistory if possible (koreader#12253) @NiLuJe * ReaderUI: add `seamless` argument to switchDocument (koreader#12290) @ziz57 * Fix early unhighlight after Add to vocab builder (koreader#12288) @poire-z * [CoverBrowser.MosaicMenu] Increase the thickness of the mosaic’s focus underline (koreader#12189) @Commodore64user * doc: update building documentation (koreader#12295) @benoit-pierre * NetworkMgr: fix debug trace (koreader#12297) @benoit-pierre * Panel zoom: Properly handle `renderPage()` not rendering the whole page (koreader#12296) @fbriere * PowerD: Keep track of frontlight state change by interactive callers for suspend/resume purposes (koreader#12283) @NiLuJe * InputDialog and MultiInputDialog: improve enter and edit callback handling, and implement addWidget (koreader#12289) @ziz57 * ReaderSearch: "All text" improve (koreader#12287) @hius07 * Show/Hide Virtual keyboard and more keyboard shortcuts (koreader#12162) @Commodore64user * bump base; mostly thirdparty bumps (koreader#12311) @benoit-pierre * Document: Do not cache panel-zoom tiles to disk and fix their caching and rendering (koreader#12303) @NiLuJe * Disable the legacy ReaderRotation module (koreader#12309) @NiLuJe * bump luajit (koreader#12317) @benoit-pierre * [AutoWarmth] Delay front light actions after resume (koreader#12315) @zwim * Allow numbers in measure text: fixes a crash in table of contents when debugging on (koreader#12310) @zwim * ReaderFooter: Add page-turn item (koreader#12249) @Commodore64user * luajit launcher: out of tree build + don't rebuild luajit (koreader#12285) @benoit-pierre * BookInfo: rating, review (koreader#12313) @hius07 * fix missing android APK asset (koreader#12333) @benoit-pierre * ImageWidget: Don't attempt to pan if we haven't rendered the bb yet (koreader#12331) @NiLuJe * OTM: Unbreak koreader#11999 (koreader#12337) @NiLuJe * Kobo: Refactor suspend in order to be able to catch input events sent during the 2s window of ntx madness (koreader#12330) @NiLuJe * [Menu] Remove title bar buttons from layout on hasSymKey or hasScreenKB (koreader#12328) @Commodore64user * [doc] Build: update Android instructions to openjdk-17-jdk (koreader#12345) @Frenzie * Support 'Ctrl' shortcuts on hasKeyboard (koreader#12347) @Commodore64user * Annotations: fix page numbers (koreader#12324) @hius07 * Bug Fix: DPad conflict on Kindle Keyboard (koreader#12341) @Commodore64user * Nit: Make some cache keys prettier (koreader#12354) @NiLuJe * Dispatcher: minor wording fixes (koreader#12356) @hius07 * kodev rework (koreader#12338) @benoit-pierre * czmq & libzmq: update to 4.2.1 & 4.3.5 respectively (koreader#12350) @benoit-pierre * [TextViewer] Add hamburger menu to FocusManager and Menu key (koreader#12365) @Commodore64user * ci: fix test / coverage (koreader#12367) @benoit-pierre * A couple of tiny fixes (koreader#12359) @NiLuJe * android: tweak update rule (koreader#12370) @benoit-pierre * doc: update Android build instructions (koreader#12368) @benoit-pierre * Display menu preview of italic-only fonts (koreader#12351) @charrarr * ReaderStatus, Book status widget: cleanup (koreader#12343) @hius07 * ci/macos: fix homebrew packages install (koreader#12382) @benoit-pierre * Fix terminal broken cursor and backspace (koreader#12375) @zwim * Screensaver: add option to rotate to fit screen (koreader#12366) @charrarr * Minor keyboard corrections (koreader#12372) @Commodore64user * Touch menu: always go up to correct parent page (koreader#12378) @hius07 * Collections: fix context menu (koreader#12383) @hius07 * PDF: no "Contrast set to" notification on opening (koreader#12391) @hius07 * Closing Bookmark details dialog with X should also refresh list entry (koreader#12390) @fbriere * Add numerical key_codes to event_map on kindle 3 (koreader#12358) @Commodore64user * FocusManager: Fix focus_flags check in moveFocusTo, and deal with the fallout (koreader#12361) @NiLuJe * Various notification, text editor and footnote popup minor fixes (koreader#12406) @poire-z * Bug: Account for local reading order direction adjustments. (koreader#12388) @Commodore64user * android: add custom `adb logcat` formatter (koreader#12385) @benoit-pierre * kodev: improve `kodev prompt` (koreader#12408) @benoit-pierre * kodev: fix `./kodev run android…` (koreader#12407) @benoit-pierre * Readerfont: Current font on top in menu; New fonts from current session on top in font test doc (koreader#12410) @jonnyl2 * Fix a couple of potentially spurious footer refreshes (koreader#12422) @NiLuJe * FileManager: Handle the FlushSettings event (koreader#12425) @NiLuJe * PageBrowser: fix drawing of thicker thumbnail frame on tap (koreader#12432) @poire-z * FileManagerSetDefaults: Handle strings that we default to nil (koreader#12433) @NiLuJe * Input: Ignore ABS_X, ABS_Y & ABS_PRESSURE for snow_protocol (koreader#12427) @NiLuJe * Add color highlight menu (koreader#11044) @smasher816 * Exporter: choose highlight styles to be exported (koreader#12431) @hius07 * ReaderView: Don't crash on pre-koreader#11044 highlights without a color field (koreader#12438) @NiLuJe * Terminal fixes (koreader#12369) @ziz57 * Dispatcher/DeviceListener: Temporarily turn on Accelerometer/GSensor for 5 sec (koreader#12419) @jonnyl2 * fix bug preventing export to joplin; (koreader#12446) @bitterlox * bump base: blitbuffer, android build, crengine (koreader#12451) @poire-z * InputText: Unbreak onTapTextBox on an already in-focus field (koreader#12449) @NiLuJe * Update el_popup.lua (koreader#12463) @ichnilatis-gr * [Dispatcher] add [missing] conditions to dispatcher items (koreader#12453) @Commodore64user * ReaderView: Fix a couple of edge-cases in onReaderFooterVisibilityChange (koreader#12466) @NiLuJe * misc: Move cleanupSelectedText to util (koreader#12477) @NiLuJe * ReaderHighlight: Fix an old typo in the OCR help string (koreader#12479) @NiLuJe * Highlights popup dialog: add "Color" and "Details" buttons (koreader#12461) @hius07 * ButtonDialogTitle: remove (koreader#12394) @hius07 * [Dispatcher] new events for selecting links (koreader#12474) @Commodore64user * Doc: Update quickstartguide on kindle non touch (koreader#12426) @Commodore64user * [NT] Call GotoViewRel for page up/down on useDPadAsActionKeys devices (koreader#12495) @cccslater * bump base (koreader#12503) @benoit-pierre * Defaults: Deprecate DKOPTREADER_CONFIG_DOC_LANGS_TEXT (koreader#12504) @NiLuJe * ReaderAnnotation: skip old highlights with invalid xpointers (koreader#12494) @hius07 * Cloud Sync: handles potentially incompatible backup and incoming dbs (koreader#12470) @weijiuqiao * VocabBuilder.koplugin: Better handling word context (koreader#12469) @weijiuqiao * Write highlights into PDF: revisited (koreader#12509) @hius07 * SyncService: Only require online connectivity when using Dropbox (koreader#12520) @NiLuJe * Chore: Don't cache static data tables (ui/elements, ui/data, kb layouts & co) (koreader#12519) @NiLuJe * Fix page gap inconsistencies between doc open & config change (koreader#12507) @NiLuJe * Kindle: Restore hall effect sensor state on startup (koreader#12521) @NiLuJe * VocabBuilder.koplugin: save dict headword as entry (koreader#12530) @weijiuqiao * Add PocketBook Verse Pro Color (PB634K3) (koreader#12476) @ElimGarak1 * Onyx Boox Go 10.3 support (koreader#12533) @hugleo * Statistics: fix null terminated strings, add toggle stats (koreader#12547) @hius07 * input: drop use of `ffi/input` (koreader#12486) @benoit-pierre * ci/circle: fix tests parallelism (koreader#12551) @benoit-pierre * input: fix kobo init (koreader#12553) @benoit-pierre * [ReaderFooter] update status bar in real time when inverting page turning (koreader#12424) @Commodore64user * input: fix pocketbook init (koreader#12555) @benoit-pierre * use new `ffi.loadlib` helper (koreader#12545) @benoit-pierre * input: fix loading of `fbink_input` library (koreader#12560) @benoit-pierre * Fix typos in quickstart (koreader#12566) @cheywood * File search: improve (koreader#12549) @hius07 * [DictQuickLookup] shortcuts for editing searches (koreader#12539) @Commodore64user * Profiles: auto-execute on events (koreader#12564) @hius07 * Exporter: export to book folder, register to Dispatcher (koreader#12569) @hius07 * sdl: use our compiled version for the AppImage (koreader#12575) @benoit-pierre * Add kobov4 & kobov5 targets (koreader#12568) @NiLuJe * android: switch to monolibtic mode (koreader#12576) @benoit-pierre * android: fix apkanalyzer location (koreader#12563) @benoit-pierre * Disabled rotation map for Android devices where "hw media keys follow gravity" (koreader#12557) @jkl16 * Quickstart guide kindle illustrations (koreader#12577) @Commodore64user * [plugin] Exporter: fix typo (koreader#12592) @Frenzie * Dispatcher: values' range consistent with bottom menu (koreader#12580) @hius07 * metadata translations for fdroid/flathub (koreader#12581) @pazos * ProgressWidget: fix misalignment (koreader#12526) @wbstx * Add wifi support to Cervantes Touch Light devices (koreader#12148) @jsanpe * ReaderTypeset: fix margins info (koreader#12609) @hius07 * DocSettingsTweak: apply to books with status (koreader#12606) @hius07 * Menu widget: rotation handler (koreader#12573) @hius07 * Profiles: add "on rotation" auto-execute event (koreader#12612) @hius07 * ReaderHighlight: adjustable "Edit highlight" dialog position (koreader#12599) @hius07 * KOSync/Kobo: Explicitly kill Wi-Fi on suspend (koreader#12616) @NiLuJe * Kobo: Minor simplification after koreader#12616 (koreader#12629) @NiLuJe * bump luajit-launcher (koreader#12630) @benoit-pierre * Terminal: Minor cleanups (koreader#12631) @NiLuJe * Menu widget: draw focus on focused item (koreader#12583) @hius07 * Cloud storage: fix sorting non-English filenames (koreader#12644) @hius07 * Text editor: show current line number (koreader#12632) @hius07 * InputText: fix Terminal plugin input (koreader#12652) @hius07 * ReaderFont test: Error correction -- new fonts were not rendering in own font (koreader#12646) @jonnyl2 * ReaderHighlight: fix color for extended highlight in pdf (koreader#12654) @hius07 * Kindle Scribe: fix elseif for orientation code (koreader#12663) @mergen3107 * Menu: Fix fallout from koreader#12583 on hasDPad devices (koreader#12657) @NiLuJe * Misc: Some more FocusManager fallout (koreader#12667) @NiLuJe * plugin messages (koreader#12636) @pazos * And deprecation to autofrontlight (koreader#12619) @pazos * Add deprecation for unmaintained targets (koreader#12620) @pazos * Profiles: more auto-exec triggers (koreader#12651) @hius07 * PatchManagement: prettify menu (koreader#12668) @hius07 * Fix KOReader spelling in the code (koreader#12670) @mergen3107 * Update Building.md - meson install from backports (koreader#12680) @mergen3107 * OTA: Make the final ConfirmBox undismissable (koreader#12676) @NiLuJe * Text editor: rotate (koreader#12658) @hius07 * Profiles: more auto-exec triggers 2 (koreader#12691) @hius07 * Add authentication request and cookies management for news (koreader#12496) @bateast * calibre: always use custom fast parser (koreader#12714, koreader#12715) @pazos * Add Nextcloud Notes to exporter plugin (koreader#12301) @kotnik * [plugin] Wallabag: remove empty file when download fails (koreader#12723) @fenuks * [plugin] Wallabag: separate option to delete ‘on hold’ articles (koreader#12722) @fenuks * kopt: fix OCR segmentation mode (koreader#12726) @benoit-pierre * bump base (koreader#12727) @benoit-pierre Most notably dropbear: update to 2024.85 (koreader/koreader-base#1964) * bump android-luajit-launcher (koreader#12734) @pazos * ci/circle: manually cap build jobs (koreader#12751) @benoit-pierre * Adjust Minimum Font Size in Terminal Emulator (koreader#12752) @pphszx * ReaderHighlight: Highlight menu, change 'Delete' to Trash can icon (koreader#12758) @jonnyl2 * @trash-pandy made their first contribution in koreader#12197 * @fbriere made their first contribution in koreader#12296 * @charrarr made their first contribution in koreader#12351 * @smasher816 made their first contribution in koreader#11044 * @bitterlox made their first contribution in koreader#12446 * @cccslater made their first contribution in koreader#12495 * @cheywood made their first contribution in koreader#12566 * @jkl16 made their first contribution in koreader#12557 * @wbstx made their first contribution in koreader#12526 * @jsanpe made their first contribution in koreader#12148 * @fenuks made their first contribution in koreader#12723 * @pphszx made their first contribution in koreader#12752 **Full Changelog**: koreader/koreader@v2024.07...v2024.11 — [closed milestone issues](https://github.com/koreader/koreader/milestone/74?closed=1) --- **Installation instructions**: [Android](https://github.com/koreader/koreader/wiki/Installation-on-Android-devices) • [Cervantes](https://github.com/koreader/koreader/wiki/Installation-on-BQ-devices) • [ChromeOS](https://github.com/koreader/koreader/wiki/Installation-on-Chromebook-devices) • [Kindle](https://github.com/koreader/koreader/wiki/Installation-on-Kindle-devices) • [Kobo](https://github.com/koreader/koreader/wiki/Installation-on-Kobo-devices) • [PocketBook](https://github.com/koreader/koreader/wiki/Installation-on-PocketBook-devices) • [ReMarkable](https://github.com/koreader/koreader/wiki/Installation-on-ReMarkable) • [Desktop Linux](https://github.com/koreader/koreader/wiki/Installation-on-desktop-linux) • [MacOS](https://github.com/koreader/koreader/wiki/Installation-on-MacOS)
|
@hius07 |
|
Looks like there is no option to disable statistics "per document" (except frozen statistics for finished books), directory defaults are not applicable. |
|
Thanks @hius07 ! |
|
I tried with two Profiles that toggle statistics: on book opening + path check for [manga] and the same for book closing. The toggle itself worked. However, Reading statistics still created an entry for this book with 00:00:00 time. I will follow up with a new ticket |
Confirm, the statistics cannot be disabled. koreader/plugins/statistics.koplugin/main.lua Lines 2824 to 2830 in 70d13be I'll look into it. |
|
Can you play with this statistics.koplugin main, various testing? It should toggle well. I agree that statistics on/off should be added to the action list. If the above fix works, I'll do that all together. |
|
@hius07 Notifications told me auto-exec worked, but it still made an entry with 00:00:00 |
…2547) (1) Followup to koreader/koreader-base#1921, koreader#12503: when getting a pdf book id in the stats db, check for old records with null terminated authors and title. Discussed in koreader#11318. There are some other reasons for disappearing old stats for pdf (md5 changes when converting/writing to pdf or something else), hence not closing the issue. (2) Add Dispatcher action for toggling statistics. Closes koreader#11422.
(1) Followup to koreader/koreader-base#1921, #12503: when getting a pdf book id in the stats db, check for old records with null terminated authors and title.
Discussed in #11318. There are some other reasons for disappearing old stats for pdf (md5 changes when converting/writing to pdf or something else), hence not closing the issue.
(2) Add Dispatcher action for toggling statistics. Closes #11422.
This change is