Skip to content

[ReaderDictionary] clean interpuncts, pipes and up arrows from text selection#14031

Merged
Frenzie merged 4 commits into
koreader:masterfrom
Commodore64user:clean-up
Jul 14, 2025
Merged

[ReaderDictionary] clean interpuncts, pipes and up arrows from text selection#14031
Frenzie merged 4 commits into
koreader:masterfrom
Commodore64user:clean-up

Conversation

@Commodore64user

@Commodore64user Commodore64user commented Jul 6, 2025

Copy link
Copy Markdown
Member

what's new

  • frontend/apps/reader/modules/readerdictionary.lua: Added logic to remove interpuncts (·), pipes (|), and up arrows () from text selections, as these characters are used in some dictionaries for syllable delimitation or indicating related words.
Image

as seen is #14013


This change is Reviewable

Comment on lines +856 to +858
text = text:gsub("·", "") -- interpunct
text = text:gsub("|", "") -- pipe
text = text:gsub("↑", "") -- and up arrow, used in some dictionaries to indicate related words

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(inconsistant indentation of the comment on the last one)

@hius07

hius07 commented Jul 7, 2025

Copy link
Copy Markdown
Member

It can be one line using brackets.

@Frenzie

Frenzie commented Jul 7, 2025

Copy link
Copy Markdown
Member

Agreed, it's simple enough for that.

@mergen3107

Copy link
Copy Markdown
Contributor

Sorry, don't really how gsub works, but isn't this now looking for the combination of the three characters?

Also, comments of the first two didn't make it to the revised one-liner

@Frenzie

Frenzie commented Jul 9, 2025

Copy link
Copy Markdown
Member

isn't this now looking for the combination of the three characters?

No, [] and () are special. The [] means any of these characters, but (in this case) only once.

@Commodore64user

Copy link
Copy Markdown
Member Author

what do you mean by "only once"?

@Frenzie

Frenzie commented Jul 9, 2025

Copy link
Copy Markdown
Member

I phrased that badly. I was trying to explain that the pattern itself matches a single character. In this case the gsub function means that pattern (i.e., the single character) will be matched any time it occurs.

Whereas something like [.?][.?] would match all of .?, ?., ?? and ... No more, no less.

@mergen3107

Copy link
Copy Markdown
Contributor

No, [] and () are special.

What do () do?

@Frenzie

Frenzie commented Jul 9, 2025

Copy link
Copy Markdown
Member

Seeing an example probably explains best. It's mostly used for getting specific parts out of a larger string. The ? means the preceding is optional.

--- Returns normalized version of KOReader git-rev input string.
-- @string rev full KOReader git-rev such as `v2015.11-982-g704d4238`
-- @treturn int version in the form of a 12 digit number such as `201511000982`
-- @treturn string short git commit version hash such as `704d4238`
function Version:getNormalizedVersion(rev)
if not rev then return end
local year, month, point, revision = rev:match("v(%d%d%d%d)%.(%d%d)%.?(%d?%d?)-?(%d*)")
year = tonumber(year)
month = tonumber(month)
point = tonumber(point)
revision = tonumber(revision)
local commit = rev:match("-%d*-g(%x*)[%d_%-]*")
-- NOTE: * 10000 to handle at most 9999 commits since last tag ;).
return ((year or 0) * 100 + (month or 0)) * 1000000 + (point or 0) * 10000 + (revision or 0), commit
end

Edit: http://lua-users.org/wiki/PatternsTutorial

@Commodore64user

Copy link
Copy Markdown
Member Author

I thought you were suggesting there was a problem, unless you are ;)

@poire-z

poire-z commented Jul 9, 2025

Copy link
Copy Markdown
Contributor

Also, comments of the first two didn't make it to the revised one-liner

This one is still to be solved :) (and/or the current splitting on 2 lines is odd, first line with a full comment ends with a period. 2nd line tail of the code starts with a lowercase).

@Commodore64user Commodore64user requested a review from poire-z July 11, 2025 23:01

@poire-z poire-z left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with me, if you have checked it works with each of the 3 chars after your last rewrite.

@Commodore64user

Copy link
Copy Markdown
Member Author

Works fine with interpuncts and pipes, i don’t have any dicts with arrows though. But you also sound as if you're suggesting something is wrong, are you?

@poire-z

poire-z commented Jul 14, 2025

Copy link
Copy Markdown
Contributor

No no, but with regexps, best to triple check when making changes.
(The pipe in the middle of [·|↑] made it look like an alternative when in parens (·|↑), but if it's ok for @Frenzie , fine.)

@Frenzie Frenzie merged commit 105abfa into koreader:master Jul 14, 2025
4 checks passed
@Frenzie Frenzie added this to the 2025.06 milestone Jul 14, 2025
@Commodore64user Commodore64user deleted the clean-up branch July 24, 2025 12:24
@Commodore64user

Commodore64user commented Jul 24, 2025

Copy link
Copy Markdown
Member Author

oops, can I add this one as well?

text = text:gsub("%d+(%a+)", "%1"):gsub("(%a+)%d+", "%1")

@Frenzie

Frenzie commented Jul 24, 2025

Copy link
Copy Markdown
Member

What's it for though? :-D

@Commodore64user

Commodore64user commented Jul 24, 2025

Copy link
Copy Markdown
Member Author

these pesky little guys: 1word or word1 --> word

@mergen3107

Copy link
Copy Markdown
Contributor

Is this for number of syllables?
Or "Note 1"?

@Commodore64user

Commodore64user commented Jul 24, 2025

Copy link
Copy Markdown
Member Author

it removes numbers from either left or right of a selection. that happens when words have superscripts, for example to say meaning 2 of word X, or footnotes (although crengine is pretty good at not getting those, MuPDF is not)

Screenshot 2025-07-24 at 18 22 56 Screenshot 2025-07-24 at 18 23 52

@Frenzie

Frenzie commented Jul 24, 2025

Copy link
Copy Markdown
Member

I suppose exceptions like A1 Sauce are rare enough for it to be an improvement most of the time.

@poire-z

poire-z commented Jul 24, 2025

Copy link
Copy Markdown
Contributor

text = text:gsub("%d+(%a+)", "%1"):gsub("(%a+)%d+", "%1")

I guess %a may not match Chinese or Arabic letters.
May be %D (not a digit) could be used instead of %a ?
Is it possible to have numbers in dictionary entries ? Couldn't we just remove any digit ? (Just asking, I guess we could have a wikipedictionary with entries for years.
Is it possible also that some dictionary use superscript letters ? ◌ͣfall, ◌ͨfall ?

@Commodore64user

Copy link
Copy Markdown
Member Author

I guess %a may not match Chinese or Arabic letters. May be %D (not a digit) could be used instead of %a ?

if you think that is safe, ... I don't mind.

Is it possible to have numbers in dictionary entries ?

apparently, but that wouldn't be problematic here, what would be is as @Frenzie mentioned, things like A1 (the sauce and the paper size), 3D, MI6

Screenshot 2025-07-26 at 00 39 51 Screenshot 2025-07-26 at 01 39 45

Couldn't we just remove any digit ? (Just asking, I guess we could have a wikipedictionary with entries for years.

what do you mean by "remove any digit"?

Is it possible also that some dictionary use superscript letters ? ◌ͣfall, ◌ͨfall ?

that seems highly unlikely but I suppose not impossible, I think is safe to assume it does not happen.

@poire-z

poire-z commented Jul 26, 2025

Copy link
Copy Markdown
Contributor

I guess %a may not match Chinese or Arabic letters. May be %D (not a digit) could be used instead of %a ?

if you think that is safe, ... I don't mind.

You should mind testing :) (I just gave a thought, no idea if it is safe and working).

what do you mean by "remove any digit"?

text = text:gsub("%d", ""):
But indeed, it would screw up 1080 and search for MI.

I remember that recently, you proposed to lookup an additional transformed word in some case (dunno if this was merged or not). Again, just asking: what if a word contains digits, lookup first a word with the digits removed, then the original word ?

@Commodore64user

Copy link
Copy Markdown
Member Author

#13884, but i am not sure that is a better approach, suddenly we'd be searching for {MI6, mi6, MI, mi}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants