Two hours ago I edited a larger pdf-file created by

qpdf --stream-data=uncompress

I remembered this old PR and used update_offsets.py to fix the offsets. I added

len_stream = None

to fix a bug occuring if there are several streams.

I think this would be a great addition to pdfly 👍

Are you still willing to work on this @srogmann? 🙂

@Lucas-C
The essential work on update_offsets.py was completed in August 2022. As of August 2022, I was able to automatically supplement offsets in simple PDFs. I don't know why the PR was not merged at the time; perhaps I should have made it clearer that, in my view, it was ready.

As mentioned above, in May 2024, I remembered pdfly and used update-offsets to correct a manually edited PDF file. In my view, the PR was also ready in May 2024.

An example is in the attached file-in.pdf , I used it to test the text-extraction of documents with Tm operators.

By update-offsets the XREF-section

xref
0 7
0000000000 65535 f 
0000000015 00000 n 
0000000015 00000 n 
0000000015 00000 n 
0000000015 00000 n 
0000000015 00000 n 
0000000015 00000 n 
trailer << /Size 7 /Info 2 0 R /Root 1 0 R >>
startxref
000000
%%EOF

will be converted into

xref
0 7
0000000000 65535 f 
0000000015 00000 n 
0000000081 00000 n 
0000000158 00000 n 
0000000217 00000 n 
0000000380 00000 n 
0000001005 00000 n 
trailer << /Size 7 /Info 2 0 R /Root 1 0 R >>
startxref
1086
%%EOF

The "target audience" for update_offsets are simple PDF documents that have been manually created using an editor. It is not suitable for complex or obfuscated PDFs.

Thank you for your detailed answer @srogmann 👍

I'll be happy to review & merge this PR, but could you rebase it and solve the minor merge conflict, please?

@Lucas-C
I updated this PR.

During testing, I noticed an issue (specifically with pytest on Unix). In the Makefile, the test directory is written as 'Tests' with a capital 'T', but in the filesystem, it is written in lowercase.

During testing, I noticed an issue (specifically with pytest on Unix). In the Makefile, the test directory is written as 'Tests' with a capital 'T', but in the filesystem, it is written in lowercase.

Thank you for notifying this problem 👍

Could you please fix this as part of this PR?

PS: I myself wrote a similar script some time ago: https://github.com/Lucas-C/dotfiles_and_notes/blob/master/languages/python/set_pdf_xref.py

I'm really happy to include this feature in pdfly, it's a very good idea and would come handy to many people 👍 🙂

The GitHub Actions pipeline is currently failing due to black not being applied on some files, to ensure a consistent code style:

$ black --extend-exclude sample-files .
would reformat /home/runner/work/pdfly/pdfly/pdfly/cli.py
would reformat /home/runner/work/pdfly/pdfly/tests/test_update_offsets.py
would reformat /home/runner/work/pdfly/pdfly/pdfly/update_offsets.py

@Lucas-C
I have slightly revised the implementation of update_offsets.py. The lines are now read in binary mode so that line breaks are not distorted. This is important when reading binary data in streams. As a result, a PDF file like 002-trivial-libre-office-writer.pdf now matches the original byte for byte after processing.

In the tests, I have commented out four PDF documents that cannot be correctly processed with the current implementation. The current implementation is quite simple and works with regular expressions; it was originally intended to revise hand-edited PDF documents via an editor. The more accurately the script should work, the more it would be appropriate to parse the tokens according to chapter 3 of the PDF specification. Technically, this is possible, but it would far exceed the original goal of my implementation.

I have slightly revised the implementation of update_offsets.py. The lines are now read in binary mode so that line breaks are not distorted. This is important when reading binary data in streams. As a result, a PDF file like 002-trivial-libre-office-writer.pdf now matches the original byte for byte after processing.

Awesome! Good job 🙂

In the tests, I have commented out four PDF documents that cannot be correctly processed with the current implementation. The current implementation is quite simple and works with regular expressions; it was originally intended to revise hand-edited PDF documents via an editor. The more accurately the script should work, the more it would be appropriate to parse the tokens according to chapter 3 of the PDF specification. Technically, this is possible, but it would far exceed the original goal of my implementation.

That's fine really.
You did an excellent job on this, thank you for your contribution! 👍

I added a commit on the branch to fix some minor typing related issues.

This has been released in version 0.4.0: https://pypi.org/project/pdfly/#history