Skip to content

Close file pointer copy in the libtiff encoder if still open#6986

Merged
radarhere merged 3 commits intopython-pillow:mainfrom
fcarron:fix_libtiff_file_pointer
Mar 5, 2023
Merged

Close file pointer copy in the libtiff encoder if still open#6986
radarhere merged 3 commits intopython-pillow:mainfrom
fcarron:fix_libtiff_file_pointer

Conversation

@fcarron
Copy link
Contributor

@fcarron fcarron commented Mar 3, 2023 •
edited by radarhere
Loading

Resolves #6985

Problem:
When saving many tiff files with compression="group4" I get the error OSError: [Errno 24] Too many open files (after ~8000 images)

def test_pil():
    im_pil = Image.open("T:/test.jpg")
    im_pil = im_pil.reduce(5)
    im_pil = im_pil.convert("1")
    for i in range(10000)
        file = f"{i:03}"
        out_name = os.path.join(
            "T:/trash", os.path.splitext(os.path.basename(file))[0] + ".tif")
        )
        im_pil.save(out_name, format="TIFF", compression="group4")

Without compression="group4" it works fine.

This also happens when a new image is created on each iteration and with explicit close()

def test_pil():
    for i in range(10000):
        file = f"{i:03}"
        im_pil = Image.open("T:/test.jpg")
        im_pil = im_pil.reduce(5)
        im_pil = im_pil.convert("1")
        out_name = os.path.join(
            "T:/trash", os.path.splitext(os.path.basename(file))[0] + ".tif")
        )
        im_pil.save(out_name, format="TIFF", compression="group4")
        im_pil.close()

Changes proposed in this pull request:

The file pointer seems to be cloned by the libtiff library, but not freed at the end.
Closing to file pointer solve the problem for me.

@geoawd
Copy link

geoawd commented Mar 3, 2023

Hi @fcarron - I opened #6985 (comment) 2 minutes before you posted this merge request. I've made the same changes and this seems to have resolved the issue in #6985 - in the test files I had provided as well. I can test this properly next week but thanks for submitting this.

@geoawd geoawd mentioned this pull request Mar 3, 2023
Closed
@radarhere radarhere changed the title Close the file pointer copy (_fp) in the libtiff encoder if it is sti… Close the file pointer copy (_fp) in the libtiff encoder if it is still open Mar 3, 2023
@radarhere
Copy link
Member

radarhere commented Mar 4, 2023 •
edited
Loading

I presume, like the other reports we've received about this, that you're a Windows user?

@radarhere radarhere removed the Windows label Mar 4, 2023
@radarhere radarhere mentioned this pull request Mar 4, 2023
Merged
@radarhere radarhere changed the title Close the file pointer copy (_fp) in the libtiff encoder if it is still open Close file pointer copy in the libtiff encoder if still open Mar 4, 2023
@fcarron
Copy link
Contributor Author

fcarron commented Mar 4, 2023

@radarhere Yes, exactly (sorry I forgot to mention that),
My setup:
Microsoft Windows 11 Pro, 10.0.22621 Build 22621
Python 3.11.2
Pillow version 9.4.0
But I first had this issue on an old Windows 7 machine with Python 3.8.6 and Pillow 9.4.0.

@radarhere
Copy link
Member

radarhere commented Mar 4, 2023

Thanks.

I've created fcarron#1 as a suggestion for a test to add, to help ensure that this change doesn't get broken again in the future.

@geoawd
Copy link

geoawd commented Mar 4, 2023

Thanks @fcarron for this pull request which has also resolved #6985. I’ve now tested this with a couple of sets of ~40k files and they have completed without issue.

@fcarron
Copy link
Contributor Author

fcarron commented Mar 4, 2023

@radarhere Thank you for adding this test.

@radarhere
Copy link
Member

radarhere commented Mar 5, 2023

Just to link, this PR is similar to #5946

@radarhere radarhere merged commit eda9534 into python-pillow:main Mar 5, 2023
@radarhere radarhere mentioned this pull request Mar 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@radarhere radarhere radarhere approved these changes

Assignees

No one assigned

Labels

TIFF Windows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Too Many Open Files when working with Tiffs (issue with files being closed)

3 participants

@fcarron @geoawd @radarhere