I don't understand why this changed - maybe @seberg or @mhvk might have some insight?

I'm even confused why it worked. I guess the idea is that once you take out ob you have a new reference to it, as well as the existing one in the buffer array. Then, the in-place addition means that the object in the buffer array is replaced with a new object, so you get one less.

I guess on 3.14, the reference count does not get increased by taking the object out? I fear I'm as confused as you... Maybe some inspection along the way can help?

This has something to do with finalizers and NpyIter_Close. If I move the HAS_REFCOUNT check inside the with block, things work consistently. See f6ce63b.

@mattip it looks like you added the iterator-as-context-manager feature back in 2018. Maybe you have context about whether this refcounting behavior change is something to worry about.

Since free-threading is still rapidly evolving, perhaps remove 3.13t already? It seems unlikely it would test anything not done by 3.14t-dev.

I'm even confused why it worked. I guess the idea is that once you take out ob you have a new reference to it, as well as the existing one in the buffer array. Then, the in-place addition means that the object in the buffer array is replaced with a new object, so you get one less.

I guess on 3.14, the reference count does not get increased by taking the object out? I fear I'm as confused as you... Maybe some inspection along the way can help?

Truthfully, I am not sure what the refcount part of this test is supposed to be testing. It seems to be checking some internal state that I am not sure is really part of the API of iterators.

It's supposed to check cleanup witihn the buffered casting. The buffer cleanup probably happens either way as the iterator is exhausted, but there should be no reason to move it inside and if anything it might be wrong.

Did this really fail? The test seems OK as it was on first sight? Unless Python re-uses the large integer more agressively now? Maybe try adding some large number instead of the += 1?

EDIT: To explain, a contains no objects, but the iterator is set up with O, so buffers for O are created and must be cleaned up again.

Did this really fail? The test seems OK as it was on first sight? Unless Python re-uses the large integer more agressively now? Maybe try adding some large number instead of the += 1?

EDIT: To explain, a contains no objects, but the iterator is set up with O, so buffers for O are created and must be cleaned up again.

I tried adding a large integer, it still seems to fail:

_______________________________________________ test_iter_object_arrays_conversions _______________________________________________

    def test_iter_object_arrays_conversions():
        # Conversions to/from objects
        a = np.arange(6, dtype='O')
        i = nditer(a, ['refs_ok', 'buffered'], ['readwrite'],
                        casting='unsafe', op_dtypes='i4')
        with i:
            for x in i:
                x[...] += 1
        assert_equal(a, np.arange(6) + 1)

        a = np.arange(6, dtype='i4')
        i = nditer(a, ['refs_ok', 'buffered'], ['readwrite'],
                        casting='unsafe', op_dtypes='O')
        with i:
            for x in i:
                x[...] += 1
        assert_equal(a, np.arange(6) + 1)

        # Non-contiguous object array
        a = np.zeros((6,), dtype=[('p', 'i1'), ('a', 'O')])
        a = a['a']
        a[:] = np.arange(6)
        i = nditer(a, ['refs_ok', 'buffered'], ['readwrite'],
                        casting='unsafe', op_dtypes='i4')
        with i:
            for x in i:
                x[...] += 1
        assert_equal(a, np.arange(6) + 1)

        #Non-contiguous value array
        a = np.zeros((6,), dtype=[('p', 'i1'), ('a', 'i4')])
        a = a['a']
        a[:] = np.arange(6) + 98172488
        i = nditer(a, ['refs_ok', 'buffered'], ['readwrite'],
                        casting='unsafe', op_dtypes='O')
        with i:
            ob = i[0][()]
            if HAS_REFCOUNT:
                rc = sys.getrefcount(ob)
            for x in i:
                x[...] += 123456789
        if HAS_REFCOUNT:
            newrc = sys.getrefcount(ob)
>           assert_(newrc == rc - 1)
E           AssertionError

a          = array([221629277, 221629278, 221629279, 221629280, 221629281, 221629282],
      dtype=int32)
i          = <numpy.nditer object at 0x106d01ed0>
newrc      = 2
ob         = 98172488
rc         = 2
x          = array(None, dtype=object)

Do you happen to know where the buffer cleanup is implemented? I'm not terribly familiar with the nditer internals and unfortunately I don't have time right now to bisect to figure out where the different behavior is triggered.

@ngoldbaum I know where it's implemented, but I suspect it is a red herring.

I.e. NumPy fills the buffer on construction here, meaning we should see an initial refcount of 2+1 and it should clean during the iteration (if not, then it should clean up on with statement exit).

Could it be that sys.getrefcount() gets a new reference (the +1) sometimes and sometimes it doesn't!? I.e. Python doing fancy optimization that makes it very hard to predict what sys.getrefcount() returns?!

In which case, I have to ask whether Python should do something about that, at least document clearly in sys.getrefounct() when you see a +1 and when you don't. right now that documentation still says:

Return the reference count of the object. The count returned is generally one higher than you might expect, because it includes the (temporary) reference as an argument to getrefcount().

which, I admit, says "generally", but before it was always right, while it seems now almost always wrong?

I wonder if trying to use sys.getrefcount(obj) is becoming too weak a proxy for what the test is actually checking. Is there a way to add something to the cleanup call to directly verify that the cleanup is happening?

This shows the behavior:

import sys
from contextlib import nullcontext

l = list(range(500000, 5000005))

def test(l):
    with nullcontext():
        obj = l[1]
        print(sys.getrefcount(obj))  # 2

    print(sys.getrefcount(obj))  # 3


test(l)

running with python3.14t will print 2 and 3.

(Actually, adding obj = None before the with statement makes it print 2 in both cases.)

Weird. Kinda feels like a reference counting bug somewhere in CPython.

This isn't a bug but a perf optimization to avoid incref where the reference of frame surely outlives the ref of the operand stack.

import sys
from contextlib import nullcontext

l = list(range(500000, 5000005))

def test(l):
    with nullcontext():
        obj = l[1] 
        print(sys.getrefcount(obj)) # uses LOAD_FAST_BORROW so it doesn't increase the refcount as obj cannot be unbound

    print(sys.getrefcount(obj)) # uses LOAD_FAST_CHECK as obj can be unbound so it increases the refcount 


test(l)

Thanks, well, NumPy runs into it once, so maybe it is rather niche. Not sure if sys.getrefcount can/should explain or even account for this.

You can use _PyEval_GetFrame to get _PyInterpreterFrame directly here of the current frame without having to materialize a Python level frame object, this would also be more performant.

It looks like _PyEval_GetFrame isn't exposed in the public ABI, at least as of CPython commit 70b322d:

dlopen(/Users/goldbaum/Documents/numpy/build-install/usr/lib/python3.14/site-packages/numpy/_core/_multiarray_umath.cpython-314-darwin.so, 0x0002): symbol not found in flat namespace '__PyEval_GetFrame'

(Actually, adding obj = None before the with statement makes it print 2 in both cases.)