Skip to content

Let fromiter handle object arrays #4791

@fgregg

Description

@fgregg

In 2006, Tim Hochberg questioned the reason why fromiter does not handle object arrays:

            /* We would need to alter the memory RENEW code to decrement any
               reference counts before just throwing away the memory.
             */

This doesn't seem right. The array that we would be RENEWing is a bunch 
of PyObject*s. The reference counts don't reside there, but in the 
objects themselves. When we do the RENEW, we don't want the reference 
counts to change at all. The one tricky case is if we run out of memory, 
I'm not certain that the current setup correctly deals with reference 
counts in this case, although it appears likely that it should work 
since ret->data should still point to a valid chunk of memory and 
decreffing ret should result in the subsequent deallocation of all the 
stored objects.

http://mail.scipy.org/pipermail/numpy-discussion/2006-November/024747.html

@teoliphant responded

I'm not sure if you are still thinking about this.  I suspect I added 
the comment because I was concerned about two things:

1) a resize that deletes memory without first decrefing object counts 
--- I suspose this can't happen here, though, because RENEW is always 
"growing" memory, right?

2) a resize that grows memory without inserting NULLS.  For Object array 
manipulation to work, you need initialized (to NULL) "empty" memory. 

>The one tricky case is if we run out of memory, 
>I'm not certain that the current setup correctly deals with reference 
>counts in this case, although it appears likely that it should work 
>since ret->data should still point to a valid chunk of memory and 
>decreffing ret should result in the subsequent deallocation of all the 
>  
>
>stored objects.
>  
>
As long as you don't store the pointer to new memory until after the 
RENEW was successful then this should work.

I think my comment came from the possibility of #1 in general use of 
RENEW, but I don't think that is going to happen.  Therefore, the only 
tweak that seams necessary is to initialize any new memory to NULL for 
OBJECT arrays.

http://mail.scipy.org/pipermail/numpy-discussion/2006-November/024841.html

I couldn't find any further discussion of this issue since 2006. Is this analysis of what needs to be done to PyArray_FromIter still sound?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions