Skip to content

kopt: improve OCR user experience#12497

Draft
benoit-pierre wants to merge 1 commit into
koreader:masterfrom
benoit-pierre:pr/improve_ocr_user_experience
Draft

kopt: improve OCR user experience#12497
benoit-pierre wants to merge 1 commit into
koreader:masterfrom
benoit-pierre:pr/improve_ocr_user_experience

Conversation

@benoit-pierre

@benoit-pierre benoit-pierre commented Sep 9, 2024

Copy link
Copy Markdown
Member

Differentiate between a failure to initialize tesseract data (bad or missing), and OCR failed (no results).

Initialization failed:
2024-09-09-180221_540x720_scrot

No results:
2024-09-09-180158_540x720_scrot


This change is Reviewable

Differentiate between a failure to initialize tesseract data
(bad or missing), and OCR failed (no results).
@benoit-pierre benoit-pierre force-pushed the pr/improve_ocr_user_experience branch from affcd1a to 213d911 Compare September 9, 2024 16:16
@benoit-pierre

Copy link
Copy Markdown
Member Author

So what's the point of getOCRText? None of document implementations support it.

@Frenzie

Frenzie commented Sep 9, 2024

Copy link
Copy Markdown
Member

It does say "not implemented yet". :-) As you probably noticed it says this where it's designed to use it:

local text = self.ui.document:getOCRText(self.hold_pos.page, selected_text.sboxes)
if not text then
-- getOCRText is not implemented in some document backends, but
-- getOCRWord is implemented everywhere. As such, fall back to
-- getOCRWord.
text = ""
for _, sbox in ipairs(selected_text.sboxes) do
local word = self.ui.document:getOCRWord(self.hold_pos.page, { sbox = sbox })
logger.dbg("OCRed word:", word)
--- @fixme This might produce incorrect results on RTL text.
if word and word ~= "" then
text = text .. word
end
end
end
logger.dbg("OCRed text:", text)

My guess is it was intended as a follow-up to koreader/libk2pdfopt@537d0a5

Based on ocrtess_ocrwords_from_bmp8 rather than ocrtess_single_word_from_bmp8.
https://github.com/koreader/libk2pdfopt/blob/6e79f026b044beef0015f3ddedc818a6d4a01f44/willuslib/willus.h#L1692-L1694

I'll note that this has already changed in k2pdfopt_tocr_single_word in the meantime as part of your recent update, though of course it doesn't take sboxes. I suspect OCR output might be better if it had more context to work with? Or maybe it'd just be for performance reasons.

@benoit-pierre

Copy link
Copy Markdown
Member Author

So that comment is wrong and misleading ("is not implemented in some document backends").

@Frenzie

Frenzie commented Sep 10, 2024

Copy link
Copy Markdown
Member

It would seem so.

@benoit-pierre

Copy link
Copy Markdown
Member Author

OK, so another libk2pdfopt change and corresponding base update is going to be needed… In no particular order:

  • ocr_type, passed to ocrtess_ocrwords_from_bmp8 from k2pdfopt_tocr_single_word, was previously unused, but should be changed to -1 (the default: assume a single uniform block of text): this seem to make a big difference for Arabic.
  • for the getOCRText / getOCRWord situation: "implement" it in koptinterface (move the workaround from ReaderHighlight:lookup) (this will also fix ReaderHighlight:translate when OCR is involved).
  • there's an issue with text selection when the document language is Chinese: if the tesseract API as not been initialized, then k2pdfopt_get_reflowed_word_boxes & k2pdfopt_get_native_word_boxes will fail when k2pdfopt_get_word_boxes_from_tesseract is involved…

@Frenzie

Frenzie commented Sep 10, 2024

Copy link
Copy Markdown
Member

All sounds good. 👍

@NiLuJe

NiLuJe commented Sep 10, 2024

Copy link
Copy Markdown
Member
  • there's an issue with text selection when the document language is Chinese: if the tesseract API as not been initialized, then k2pdfopt_get_reflowed_word_boxes & k2pdfopt_get_native_word_boxes will fail when k2pdfopt_get_word_boxes_from_tesseract is involved…

Possibly related: I never looked into it, but there's potentially a fair number of old open issues about crashes on CJK when OCR and/or reflow are involved (issues that, IIRC, may have started since the previous mupdf bump ages ago ;)).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants