@MartinThoma
I'm interested in your position about phi / phi1 being crossed

Codecov Report

Patch coverage: 92.85% and project coverage change: +0.03% 🎉

Comparison is base (890c93a) 94.03% compared to head (9a06598) 94.07%.
Report is 1 commits behind head on main.

@@            Coverage Diff             @@
##             main    #2016      +/-   ##
==========================================
+ Coverage   94.03%   94.07%   +0.03%     
==========================================
  Files          33       33              
  Lines        7076     7104      +28     
  Branches     1413     1421       +8     
==========================================
+ Hits         6654     6683      +29     
  Misses        263      263              
+ Partials      159      158       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

except from My comment above, this PR is all yours

This is amazing 😲 😍 Thank you so much 🤗

I'm looking forward to the release on the weekend + an update of https://github.com/py-pdf/benchmarks/blob/main/benchmark.py 🎉

@pubpub-zz I've updated the benchmark:

The text extracting quality metric increased from 96% to 97%. I've also found a couple of places where the ground truth was wrong 🎉 We a now on-par with Tika / PyMuPDF. However, the felt quality is still slightly worse as Tika / PyMuPDF typically deal with whitespaces better.

I had a look at what would be necessary to lift the text extraction to the next step (from a users perspective):

Local optimizations

Ligature replacement:

• fi should be fi
• fl should be fl
• ff should be ff

I know that this actually goes away from "raw" text extraction, but I think this is what most users want. Maybe we need to re-define what we want to achieve and potentially add flags / methods for common post-processing 🤔

Composed characters:

• ¯x should be x̄

• ˆx should be x̂

• Chinese characters in arxiv 2201.00021: The name one the first page.

• Removal of hypens inserted solely to fit on the line: Here I'm uncertain. I think most people use the text extraction to do Natural Language Processing (NLP). For them, the hyphens are just noise. But some might need the layout mode to do post-processing on their own. Then hyphen-removal might actually harm.

• Superscript / subscripts: Especially squares (x²) and cubes (x³) as well as zero-subscripts (x₀) and one-subscripts (x₁)

Whitespace

• Most important are inner-word spaces that often occur after the first letter of a word. See Random whitespaces are inserted when using page.extract_text() #1507
• Newlines, especially for arXiv 2201.00029
• Spaces around math-mode stuff
• Spaces after dots: New line character missing and URLs adding periods and space #1974

Layout-mode

• Indentation of code blocks currently completely breaks.
• Multiple newlines to represent paragraph / section boundaries

Advanced text extraction normalization

This will likely never go into pypdf as it requires a level of document understanding that is likely only achievable with machine learning. Still interesting to think about it:

• Detection of tables + automatic application of layout mode for them, while not using layout mode for e.g. two-column pages.
• Removal of footers (page numbers)
• Removal of headers
• Removal of spaces used for thousands separation
• Detection of text that belongs to an image / diagram
• Re-structuring of text that is broken up by an image to ensure a smooth text flow