@MartinThoma
For review only, still some works before merging into 2.0

Codecov Report

Merging #881 (4baedb2) into main (42d4659) will increase coverage by 5.83%.
The diff coverage is 73.78%.

❗ Current head 4baedb2 differs from pull request most recent head 121420c. Consider uploading reports for the commit 121420c to get more accurate results

@@            Coverage Diff             @@
##             main     #881      +/-   ##
==========================================
+ Coverage   78.25%   84.09%   +5.83%     
==========================================
  Files          16       18       +2     
  Lines        4346     4068     -278     
  Branches      821      854      +33     
==========================================
+ Hits         3401     3421      +20     
+ Misses        758      460     -298     
  Partials      187      187              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42d4659...121420c. Read the comment docs.

deep changes in the text extractions. still some analysis with some fonts(truetype,type 3)

Looking for so beta testers.

Nice! I'll let it run with https://github.com/py-pdf/benchmarks this evening if I have some time :-)

I fixed the merge conflicts due to the recent PEP8 renamings + I adjusted the PR to the new syntax. There are still mypy issues though.

Running the benchmark, I get:

  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/_page.py", line 1141, in extract_text
    ft,cmap,cmap2 = buildCharMap(self.pdf,operands[0])
  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/_page.py", line 1073, in buildCharMap
    fontType = pdf.pages[0]["/Resources"]["/Font"][font_name]["/Subtype"]
  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/generic.py", line 519, in __getitem__
    return dict.__getitem__(self, key).get_object()
KeyError: '/R167'

@MartinThoma

Running the benchmark, I get:

  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/_page.py", line 1141, in extract_text
    ft,cmap,cmap2 = buildCharMap(self.pdf,operands[0])
  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/_page.py", line 1073, in buildCharMap
    fontType = pdf.pages[0]["/Resources"]["/Font"][font_name]["/Subtype"]
  File "/home/moose/Github/py-pdf/PyPDF2/PyPDF2/generic.py", line 519, in __getitem__
    return dict.__getitem__(self, key).get_object()
KeyError: '/R167'

Thanks for the test and the report. I've fixed it; also I've introduced a test using testbench dataset. using urllib.request to get them. It's increasing the test duration. I've set a way to bypass it.

I think that this pull now should be proposed for testing

PS : I merged with the new API and 2 functions are missing to prevent compatibility loss : getPage and getObject
PPS: I have some issues with mypy, can you fix the errors for my training ?

def read_from_stream(stream: StreamType, pdf: Any, forcedEncoding: Union[None,str,List[str],dict[int,str]] = None) -> "ArrayObject":  # PdfReader

E TypeError: 'type' object is not subscriptable

You need to change dict[int,str] to Dict[int,str] (capital D). The syntax you've used was only introduced in Python 3.9 or 3.10, I think

A general question: If I see such things, should I directly adjust it in your PR? (I know some people love it and others hate it)

Thanks, for the advice. I've tried to improve myself.
I successed to fix it 😊
During your review, I would like to have your opinion : the solution I've used to prevent mypy errors with cast makes the code not so easy to read for me: can you give me some advice ?

Glancing at the first result of the benchmark the results look amazing 😍

Left is old, right is new:

It will take a while until the benchmark completed. I'll share the results later :-)

@MartinThoma,
My latest status
a) about issues with asian characters : I've not been able to reproduce the issue.
my code in python to get the file:
import io,urllib.request,PyPDF2;p = PyPDF2.PdfReader(io.BytesIO(urllib.request.urlopen("https://arxiv.org/pdf/2201.00151.pdf").read()));e=p.pages[0].extract_text();print(e);f=open('e:/out.txt','w',encoding='utf-8');f.write(e);f.close()
the result of comparison between out.txt(left) and copy/paste from adobe reader DC:

b) spacing:
I've sligthtly modified the code to allow to pass the space width criteria. I' still do not understand some case where a lower width is sometimes interpreted as a space whereas is some cases not.

c) mypy.
Your proposal seems to have reintroduced some error reporting. Can you check on your side and reajust enventually?

d)benchmark
Can you tell me how are you obtaining the percentage result?

I've extracted the spacing improvement heuristic to #922 . That should be possible to get merged really soon

I've just fixed the merge conflict I've introduced

@pubpub-zz Here is the simplification for the casts: c488734 - essentially just assigning variable names to intermediate results

Hey man, just want to say thank you for you hard work on this one. I got some interest on do this improved myself and end up finding this PR. Really nice work, @pubpub-zz. Are you checking list your past and future updates somewhere? I would like to follow and see if I can help to add html features to it (while avoiding disturb your work)

@MartinThoma, your proposal is quite heavy to read. May I propose this solution?
font_type: str = self["/Resources"]["/Font"][font_name]["/Subtype"] # type: ignore

That is completely fine for me as well :-)

Hey man, just want to say thank you for you hard work on this one. I got some interest on do this improved myself and end up finding this PR. Really nice work, @pubpub-zz. Are you checking list your past and future updates somewhere? I would like to follow and see if I can help to add html features to it (while avoiding disturb your work)

@LucasWolfgang, thanks for your nice comments. There will be still some improvement to be done. Don't know i will be able to release new changes.
What do you intend to propose (font? style? position?)

@pubpub-zz , I was thinking of implemting the functionalities available in PyMuPDF's get_text. Of course it would be a LOT of work. So I would start fist by just creating the blocks (paragraphs) and continue from there to extract font, style and Images (as Base64 or maybe as references) a PR at a time. About positions, I am still unsure how I would use it besides ordering the blocks.

maybe what you could do would be to branch from my dev branch https://github.com/pubpub-zz/PyPDF2/tree/extractText
If you had an as_html boolean parameter to add so if branches to be able keep text and html at the same time. This solution may help you to merge my changes later

@pubpub-zz #924 got merged 🎉 I'd make a minor release (2.1.0) on Monday.

I've moved the buildCharMap function to its own module: 4baedb2 - I hope this simplifies future PRs / makes it easier for me to understand what they are actually doing.