Open question:

• Which namespace should hold the trace_data_allocations() function?

The callback mechanism is pretty fast, not noticeably slower to run numpy.test() with tracing callbacks set to None than before this functionality. This should be quantified/verified, though.

This looks like a pretty useful facility. I don't remember seeing it advertised on the numpy-discussion list, though (but I might have missed it).

The trace_data_allocations function should probably be either in numpy.core or in a separate numpy.tools name-space. But, this is another good question for the mailing list.

The brief discussion is here: http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062373.html

Sorry if it was a bit rushed, I thought having something that worked in place to see what it would involve would be helpful.

This is still a WIP, especially since some tests fail if I run np.test() while tracing that don't fail when run outside of tracing:

import numpy as np

total = 0
szs = {}

def inc(ptr, size):
    global total
    szs[ptr] = size
    total += size
    if size > 100:
        print total

def sub(ptr):
    global total
    total -= szs[ptr]

def res(newptr, oldptr, newsize):
    global total
    total -= szs[oldptr]
    szs[newptr] = newsize
    total += newsize

np.core.trace_data_allocations(inc, sub, res)
np.test()
print "END", total

My theory is that some tests are failing due to refcounts being invalid when the new functions are called, and the gc being triggered or objects otherwise being freed.

One test that regularly fails is test_regression.py:test_refcount_vectorize().

Some tests are still failing when run under tracing, though not consistently. Valgrind is not producing showing me anything within them in particular, but disabling gc completely seems to prevent them from failing.

scipy.test() also has some errors that only occur while tracing.

I'm going to set this aside for a few days, and perhaps write a simpler version that doesn't call back into Python. Instead, it can expose the malloc/free counts/bytes via a function, and track them in pure C code. Then the same approach as https://github.com/fabianp/memory_profiler can be used to track allocation line-by-line and most of the complexity goes away.

All of the failing tests seem to have been caused by the buffer copy bug, fixed in https://github.com/mwiebe/numpy/tree/nditer_buffer_flag (but not yet pulled into numpy).

I also have a version that implements tracing, with pure C in the allocation functions writing to a dynamically allocated buffer, which must then be fetched proactively by Python. However, I think this version is a little nicer to use from the Python perspective.

I'd think the PyDataMem_* memory allocations/deallocations should remain completely independent of Python, I suspect that is why they are there.

Thinking more about this, the lower-level version is probably just as easy to use, and avoids the potential issues with GC and python allocation, so I'll close this PR and open a new one when it's ready.