Skip to content

Export: solved my_clippings OOM#12107

Merged
Frenzie merged 1 commit into
koreader:masterfrom
Mochitto:fix/OOM-my_clippings
Jun 28, 2024
Merged

Export: solved my_clippings OOM#12107
Frenzie merged 1 commit into
koreader:masterfrom
Mochitto:fix/OOM-my_clippings

Conversation

@Mochitto

@Mochitto Mochitto commented Jun 27, 2024

Copy link
Copy Markdown
Contributor

This should take care of #10745 regarding my_clippings; I'm not sure if it's the last one remaining and the issue can be closed.
The result is the same and I've followed the instructions given by @pazos; hopefully the OOM risk is no more.

Thanks for tagging me in #12106 (probably needs to solve conflicts with this fix)


This change is Reviewable

@hius07

hius07 commented Jun 27, 2024

Copy link
Copy Markdown
Member

I'd propose building each line and then table.concat(tbl, "\n") to avoid extra concatenation.

@Frenzie Frenzie added this to the 2024.06 milestone Jun 27, 2024
Comment on lines 42 to 46
local content = table.concat(tbl)
return content

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reads a bit funny. ;-)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh gosh ahah thanks for pointing it out!

@Mochitto

Mochitto commented Jun 27, 2024

Copy link
Copy Markdown
Contributor Author

@hius07

I've tried using the concat "\n" but the result is inconsistent..?
I'm not sure why that would be so, since I'm matching the same structure as the one with the concat "\n"

Can you help me out? I'm not very familiar with lua

local function format(booknotes)
    local tbl = {}

    for ___, entry in ipairs(booknotes) do
        for ____, clipping in ipairs(entry) do
            if booknotes.title and clipping.text then
                table.insert(tbl, booknotes.title)

                local header = T(_("- Your highlight on page %1 | Added on %2"), clipping.page,
                    os.date("%A, %B %d, %Y %I:%M:%S %p", clipping.time))
                table.insert(tbl, header)

                table.insert(tbl, "")
                table.insert(tbl, clipping.text)
                table.insert(tbl, "==========")

                if clipping.note then
                    table.insert(tbl, booknotes.title)

                    header = T(_("- Your note on page %1 | Added on %2"), clipping.page,
                        os.date("%A, %B %d, %Y %I:%M:%S %p", clipping.time))
                    table.insert(tbl, header)

                    table.insert(tbl, "")
                    table.insert(tbl, clipping.note)
                    table.insert(tbl, "==========")
                end
            end
        end
    end

    local content = table.concat(tbl, "\n")
    return content
end

The outputs; notice how some titles are in the same line as the divider ==========

My experience with Anki
- Your highlight on page 1 | Added on Thursday, June 27, 2024 08:38:52 PM

I prepared some questions that are primarily for reflection, feel free
==========
My experience with Anki
- Your highlight on page 1 | Added on Thursday, June 27, 2024 08:38:54 PM

Do you have a special philosophy/methodology behind
==========
My experience with Anki
- Your highlight on page 6 | Added on Thursday, June 27, 2024 08:38:58 PM

ease hell](https://readbroca.com/anki/ease-hell/)
- Progress
==========KOReader Quickstart Guide
- Your highlight on page 1 | Added on Thursday, June 27, 2024 08:35:36 PM

You can access the complete user manual from
==========LinuxCommand.Find and replace
- Your highlight on page 1 | Added on Thursday, June 27, 2024 08:39:19 PM

-rli "1.0.5" . | xargs -i@ sed -i "s/1.0
==========
LinuxCommand.Find and replace
- Your highlight on page 1 | Added on Thursday, June 27, 2024 08:39:57 PM

the matching string with [[LinuxTool.grep]].
`r` recursively, `l` print name of files with matching strings, `i`
==========
LinuxCommand.Find and replace
- Your note on page 1 | Added on Thursday, June 27, 2024 08:39:57 PM

This is my note
==========

@Frenzie

Frenzie commented Jun 27, 2024

Copy link
Copy Markdown
Member

At the beginning insert an extra "\n" (so it'll turn into two). Or just an empty string if you only want one.

@Frenzie

Frenzie commented Jun 27, 2024

Copy link
Copy Markdown
Member

That is, if you do concat with \n you get line1\nline2\nline3. Attach that to another set and you're missing a newline.

@Mochitto Mochitto force-pushed the fix/OOM-my_clippings branch from 97c9d2e to ddbed8d Compare June 27, 2024 20:19
@Mochitto

Mochitto commented Jun 27, 2024

Copy link
Copy Markdown
Contributor Author

Oh, I've got it now! It actually needed to be added to the end part :)
To avoid having a newline at the start of the file and output the same content as before the fix.

I lost focus of the fact that the format function is called separately for each book, instead of the entry i'm iterating through being a continuous stream.

tysm, fixed ! :)

@Mochitto Mochitto force-pushed the fix/OOM-my_clippings branch 2 times, most recently from 1cfcf71 to 2d11f0d Compare June 27, 2024 20:35
Comment thread plugins/exporter.koplugin/target/my_clippings.lua Outdated
@Frenzie Frenzie added the Plugin label Jun 27, 2024
@Mochitto Mochitto force-pushed the fix/OOM-my_clippings branch from 2d11f0d to 6b81b51 Compare June 27, 2024 20:43

@pazos pazos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!, thank you very much :)

@arpanghosh8453

Copy link
Copy Markdown
Contributor

@Mochitto and @pazos In order to include the features in #12106 (now with conflicting changes) you need to change the following in line 19 (OOM fix version). I have added the author as "Unknown" if the author is not presenting preserving Kindle syntax as mentioned in @hius07 in #12106

- table.insert(tbl, booknotes.title)

+ if booknotes.author then
+     table.insert(tbl, booknotes.title .. " (" .. booknotes.author .. ")" )
+ else
+     table.insert(tbl, booknotes.title .. " (Unknown)" )
+ end

And the following in line 30 (OOM fix version)

- table.insert(tbl, booknotes.title)

+ if booknotes.author then
+     table.insert(tbl, booknotes.title .. " (" .. booknotes.author .. ")" )
+ else
+     table.insert(tbl, booknotes.title .. " (Unknown)" )
+ end

@arpanghosh8453

arpanghosh8453 commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

Also @Mochitto and @pazos, I have noticed something, that is worth fixing. The time in the added export are in local time. But I assume Kindle uses GMT/UTC timestamps so when importing to myclippings, it auto-converts them to local time based on browser timezone/system time. I have illustrated an example here -

image

image

As you can see, 05:30 was added to the timestamp given in the file ( I am from GMT + 5:30 time zone )

I am not sure what is the format of the time in clipping.time , but it's best if you can somehow convert that to the UTC time zone matching Kindle syntax for seamless importing by clippings.io. I think Koreader already has the time zone information, so you need to use that for the conversion. Maybe @Frenzie or @pazos can help with this issue better.

@hius07

hius07 commented Jun 28, 2024

Copy link
Copy Markdown
Member

I am not sure what is the format of the time in clipping.time , but it's best if you can somehow convert that to the UTC time zone matching Kindle syntax for seamless importing by clippings.io.

It is converted exactly to the Kindle format:

local header = T(_("- Your highlight on page %1 | Added on %2"), clipping.page,
os.date("%A, %B %d, %Y %I:%M:%S %p", clipping.time))

@arpanghosh8453

Copy link
Copy Markdown
Contributor

I am not sure what is the format of the time in clipping.time , but it's best if you can somehow convert that to the UTC time zone matching Kindle syntax for seamless importing by clippings.io.

It is converted exactly to the Kindle format:

local header = T(_("- Your highlight on page %1 | Added on %2"), clipping.page,
os.date("%A, %B %d, %Y %I:%M:%S %p", clipping.time))

The problem is outlined above. What I meant is I do not know what object clipping.time variable is ( is it a formatted string, datetime object, Epoch time integer ... ) . The format there is fine, but the time is in local time zone. Which is not how Kindle exports it. So when uploaded to myclippings.io (as intended for this plugin by @Mochitto), the timestamp gets modified to wrong value ( It always assumes the given time is in UTC and stores accordingly, shows the user in local time by calculating from browser time ). See the images I attached in my comment. That might help you understand the issue.

@hius07

hius07 commented Jun 28, 2024

Copy link
Copy Markdown
Member

What does myclippings.io import from the Kindle?
My Clippings.txt or something else?

@arpanghosh8453

Copy link
Copy Markdown
Contributor

What does myclippings.io import from the Kindle? My Clippings.txt or something else?

Yes, It imports the My Clippings.txt file (The exact format we are trying to replicate here)

@hius07

hius07 commented Jun 28, 2024

Copy link
Copy Markdown
Member

And our Export plugin provides the same timestamp as the Kindle does.
Maybe myclippings.io wants to connect to the device to get its timezone and to recalculate the time?

@arpanghosh8453

Copy link
Copy Markdown
Contributor

And our Export plugin provides the same timestamp as the Kindle does. Maybe myclippings.io wants to connect to the device to get its timezone and to recalculate the time?

I don't have a kindle right now with me, but can you check if the exported file has the timestamp converted to UTC?

@hius07

hius07 commented Jun 28, 2024

Copy link
Copy Markdown
Member

Okay, I got your point.
I am looking at the My Clippings.txt made just now.
It contains the timestamp exactly with the current time of my Kindle. It is UTC+2.

@arpanghosh8453

arpanghosh8453 commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

Okay, I got your point. I am looking at the My Clippings.txt made just now. It contains the timestamp exactly with the current time of my Kindle. It is UTC+2.

Ah, Thank you. So you mean it's the local time and not UTC. Do you mind uploading that to the service ( you just need a simple https://clippings.io account ) to see if the time appears right?

May I see a preview of that file please?

@Mochitto

Copy link
Copy Markdown
Contributor Author

I haven't used apps that integrate with my clippings, only obsidian plugins which worked fine; I just made the plugin since kindle has a huge market-share and thought it would make for a very compatibility with other projects.

I'll wait for the updates but my 2 cents are: I don't think kindle end-users would know what timestamps other than local timestamps would be.
I'm rather perplexed about the way this service handles time, as we are just exploring a timezone-less datetime string...
So any parsing should be pretty forward and not apply mutations; if kindle devices export in local timezones (which I'd expect, else I could imagine all the calls to Amazon's support services), could it be that the app applies the timezones erroneously?

Maybe you can shoot an email to the app's support services and ask about it :)

@Mochitto

Mochitto commented Jun 28, 2024

Copy link
Copy Markdown
Contributor Author

Maybe a new issue can be opened about this as it's out of scope with the out of memory fix and the authors fix and, maybe, a bug on the app's end instead of koreader's, so if we have to wait for updates this PR wouldn't be the best to store that information for the future.

@Frenzie Frenzie merged commit 138e26a into koreader:master Jun 28, 2024
@arpanghosh8453

arpanghosh8453 commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

I haven't used apps that integrate with my clippings, only obsidian plugins which worked fine; I just made the plugin since kindle has a huge market-share and thought it would make for a very compatibility with other projects.

I'll wait for the updates but my 2 cents are: I don't think kindle end-users would know what timestamps other than local timestamps would be. I'm rather perplexed about the way this service handles time, as we are just exploring a timezone-less datetime string... So any parsing should be pretty forward and not apply mutations; if kindle devices export in local timezones (which I'd expect, else I could imagine all the calls to Amazon's support services), could it be that the app applies the timezones erroneously?

Maybe you can shoot an email to the app's support services and ask about it :)

Then I assume that is the issue with the web service (clippings.io) itself. Don't bother about it. For now, I will be happy with merging #12106 and #12107 (both looks approved, but have conflicting changes). I am not sure how @pazos or @Frenzie will resolve the conflict, but in case of any issue, I am down to help.

Commodore64user pushed a commit to Commodore64user/KOReader_fork that referenced this pull request Aug 2, 2024
0xstillb pushed a commit to 0xstillb/koreader-thai that referenced this pull request May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants