BUG: Using compress_identical_objects on transformed content duplicates differing content by danio · Pull Request #3197 · py-pdf/pypdf

danio · 2025-03-19T16:01:48Z

compress_identical_objects() can result in lost content, if page.add_transformation is used.

Test that creates bad output:

from pypdf import PdfReader, PdfWriter, Transformation

def test_transform_compress_identical_objects():
    reader = PdfReader(RESOURCE_ROOT / "two-different-pages.pdf")
    writer = PdfWriter()

    for page in reader.pages:
        op = Transformation().scale(sx=0.8, sy=0.8)
        page.add_transformation(op)
        writer.add_page(page)
    writer.compress_identical_objects()
    with open("result.pdf", "wb") as fp:
        writer.write(fp)

Input file:
two-different-pages.pdf

This contains "1" on page 1 and "2" on page 2

Result before fix:
result-before.pdf
This contains "1" on page 1 and "1" on page 2

Result after fix:
result-withfix.pdf
This contains "1" on page 1 and "2" on page 2

The issue is around the EncodedStreamObject which gets converted into a ContentStream after the transformation. BothContentStream objects on each page have the same obj.hash_value(), even though they have the same contents. Printing the _data value out for the contentstreams showed it was empty after the transformation.

I tried creating a simple test to reproduce just the hashcode calculation for the content streams but couldn't get it to work, I'm not sure quite what is happening inside page.add_transformation to create the content stream with no data yet calculated. This is what I tried (inspired by test_contentstream_arrayobject_containing_nullobject):

stream_object = DecodedStreamObject()
stream_object.set_data(b"Hello World!")
content_stream = ContentStream(stream=stream_object, pdf=None)
writer = PdfWriter()
cs2 = content_stream.clone(writer) # I also tried .replicate(), with the same result
print(cs2.hash_value()) 
print(cs2._data) # this was set correctly, I was hoping it would also show the issue

…es differing content

codecov · 2025-03-19T16:10:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.54%. Comparing base (a548ca1) to head (7567944).
Report is 74 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3197   +/-   ##
=======================================
  Coverage   96.54%   96.54%           
=======================================
  Files          53       53           
  Lines        8935     8935           
  Branches     1642     1642           
=======================================
  Hits         8626     8626           
  Misses        186      186           
  Partials      123      123

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stefan6419846 · 2025-03-19T16:20:48Z

Thanks for the report and PR. Could you please add a corresponding test as well?

The function provided by you should be rather easy to transform into an integration test if you run page.extract_text().

As for why _data is not populated after the transformation: The content stream can either be an unparsed string or a parsed list of operations.

pypdf/pypdf/_page.py

Lines 1003 to 1018 in a548ca1

    
           @staticmethod 
        
           def _add_transformation_matrix( 
        
               contents: Any, 
        
               pdf: Optional[PdfCommonDocProtocol], 
        
               ctm: CompressedTransformationMatrix, 
        
           ) -> ContentStream: 
        
               """Add transformation matrix at the beginning of the given contents stream.""" 
        
               contents = ContentStream(contents, pdf) 
        
               contents.operations.insert( 
        
                   0, 
        
                   [ 
        
                       [FloatObject(x) for x in ctm], 
        
                       b"cm", 
        
                   ], 
        
               ) 
        
               return contents

is responsible for the actual transformation and operates on the operations, thus _data is not set unless you call get_data() explicitly. (contents is page.get_contents() in this case.)

danio · 2025-03-19T17:06:15Z

@stefan6419846 That test would be nice, but I can't see how I can extract the text from the PdfWriter.
When I try writer.pages[0].extract_text() I just get empty data.
Would the test have to write to file, then open it? (I'm thinking it should work with a BytesIO so no need to actually write to disk)

…rmed content

danio · 2025-03-19T18:01:19Z

@stefan6419846 I've added a test that uses BytesIO to write/read the resulting PDF, let me know if there's a better way and I'll change it

tests/test_workflows.py

stefan6419846 · 2025-03-20T10:34:14Z

The current approach looks fine for me. I have added a comment about a small code style change we should add; afterwards, I see no problem with getting this merged.

stefan6419846

Thanks.

@bryan-brancotte

## What's new ### New Features (ENH) - Add support for IndirectObject.__iter__ (#3228) by @bryan-brancotte - Allow filtering by font when removing text (#3216) by @samuelbradshaw ### Bug Fixes (BUG) - Add missing named destinations being ByteStringObjects (#3282) by @stefan6419846 - Get font information more reliably when removing text (#3252) by @samuelbradshaw - T* 2D Translation consistent with PDF 1.7 Spec (#3250) by @hackowitz-af - Add font stack to q/Q operations in layout mode (#3225) by @hackowitz-af - Avoid completely hiding image loading issues like exceeding image size limits (#3221) by @stefan6419846 - Using compress_identical_objects on transformed content duplicates differing content (#3197) by @danio - Consider BlackIs1 parameter for CCITTFaxDecode filter (#3196) by @stefan6419846 ### Robustness (ROB) - Deal with insufficient cm matrix during text extraction (#3283) by @stefan6419846 - Allow merging when annotations miss D entry (#3281) by @stefan6419846 - Fix merging documents if there are no Dests (#3280) by @stefan6419846 - Fix crash on malformed action in outline (#3278) by @larsga - Fix compression issues for removed images which might be None (#3246) by @stefan6419846 - Attempt to deal with non-rectangular FlateDecode streams (#3245) by @stefan6419846 - Handle some None values for broken PDF files (#3230) by @stefan6419846 ### Developer Experience (DEV) - Multiple style improvements by @j-t-1 - Update ruff to 0.11.0 by @stefan6419846 ### Maintenance (MAINT) - Conform ASCIIHexDecode implementation to specification (#3274) by @j-t-1 - Modify comments of filters that do not use decode_parms (#3260) by @j-t-1 ### Code Style (STY) - Simplify warnings & debugging in layout mode text extraction (#3271) by @hackowitz-af - Standardize mypy assert statements (#3276) by @j-t-1 [Full Changelog](5.4.0...5.5.0)

BUG: using compress_identical_objects on transformed content duplicat…

70bd00e

…es differing content

stefan6419846 added the needs-test A test should be added before this PR is merged. label Mar 19, 2025

danio added 2 commits March 19, 2025 17:58

BUG: integration test for using compress_identical_objects on transfo…

7b3a292

…rmed content

BUG: integration test for using compress_identical_objects on transfo…

f504cb7

…rmed content

stefan6419846 reviewed Mar 20, 2025

View reviewed changes

tests/test_workflows.py Outdated Show resolved Hide resolved

stefan6419846 removed the needs-test A test should be added before this PR is merged. label Mar 20, 2025

stefan6419846 changed the title ~~BUG: using compress_identical_objects on transformed content duplicates differing content~~ BUG: Using compress_identical_objects on transformed content duplicates differing content Mar 20, 2025

Remove unneeded brackets

7567944

stefan6419846 approved these changes Mar 20, 2025

View reviewed changes

stefan6419846 merged commit aed50ad into py-pdf:main Mar 20, 2025
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Using compress_identical_objects on transformed content duplicates differing content#3197

BUG: Using compress_identical_objects on transformed content duplicates differing content#3197
stefan6419846 merged 4 commits intopy-pdf:mainfrom
danio:cs-fix

danio commented Mar 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 19, 2025 •

edited

Loading

Uh oh!

stefan6419846 commented Mar 19, 2025

Uh oh!

danio commented Mar 19, 2025 •

edited

Loading

Uh oh!

danio commented Mar 19, 2025

Uh oh!

Uh oh!

stefan6419846 commented Mar 20, 2025

Uh oh!

stefan6419846 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

danio commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stefan6419846 commented Mar 19, 2025

Uh oh!

danio commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danio commented Mar 19, 2025

Uh oh!

Uh oh!

stefan6419846 commented Mar 20, 2025

Uh oh!

stefan6419846 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

danio commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 19, 2025 •

edited

Loading

danio commented Mar 19, 2025 •

edited

Loading