[Python] Fix crash related to file-like objects and fsspec#7012
Merged
Mytherin merged 3 commits intoduckdb:masterfrom Apr 11, 2023
Merged
[Python] Fix crash related to file-like objects and fsspec#7012Mytherin merged 3 commits intoduckdb:masterfrom
fsspec#7012Mytherin merged 3 commits intoduckdb:masterfrom
Conversation
Collaborator
|
Thanks for the fix! LGTM. Perhaps we can turn the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When reading from file-like objects with the python client, we rely on
fsspecto create an in-memory filesystem to store them.The import cache is utilized for this, but a recent change that allows importing optional attributes/modules is causing problems here. Because the import silently fails, the object.ptr() is left as
nullptr.When the object's constructor is called, this causes a segfault.
I've fixed this here by checking if the object could not be imported before calling the constructor, but there are likely other places where this could become a problem.
In #7005 I've made changes to
py::isinstanceto support nullptr objects, maybe a better change would be to modifypy::isinstanceto take aPythonImportCacheItemobject instead, and throw from theoperator ()of the item if it's called on an item that wasn't imported.Also, I can't add a test for this, I've tried monkeypatching, but in the
pyduckdb/filesystem.pywe import fsspec, overriding the monkeypatch.