-
Notifications
You must be signed in to change notification settings - Fork 311
iterated objects are not released by islice_extended when start<0 and step>0 #994
Description
iterated objects are not released by islice_extended when start<0 and step>0
Description
An iterator created from islice_extended with a negative start index, does not release the elements that have been returned/iterated on.
StopIteration exception.
Illustrative example
# assuming there are 3 elements in anIterator.
sliced_iterator = islice_extended(anIterator, -3, None, 1)
next(sliced_iterator)
next(sliced_iterator)Expected
The first 2 elements of anIterator are released/deleted (if not referenced anymore)
Observed
The first 2 elements of anIterator are not released/deleted (if not referenced anymore) until we've iterated to the end of sliced_iterator
Repo steps/Example
To illustrate the issue, I'll be leveraging this helper class to track and report when an iterator elements are still alive or have been deleted:
class IteratorWithWeakReferences:
"""
A class that wraps an iterator and provides weak references to its elements.
This class allows iteration over a collection of objects while maintaining
weak references to the original objects. It can be used to track the validity
of the objects (i.e., whether they are still alive or has been deleted).
"""
class AnObject:
def __init__(self, index: int):
self.index = index
def __str__(self):
return f"AnObject({self.index})"
@classmethod
def FROM_SIZE(cls, size: int) -> IteratorWithWeakReferences:
return cls([IteratorWithWeakReferences.AnObject(index=index) for index in range(size)])
def __init__(self, iterable: Iterable):
self._data = deque(element for element in iterable)
self._weakReferences = [weakref.ref(a) for a in self._data]
def __iter__(self) -> Iterator:
return self
def __next__(self) -> AnObject:
if (len(self._data) == 0):
raise StopIteration
return self._data.popleft()
def weakReferencesAliveStatePerIndex(self) -> list[bool]:
"""
Checks the alive state of weak references of the initial iterator elements.
Returns:
list[bool]: A list of boolean values where each element corresponds to the alive state
of a weak reference.
- `True` indicates the reference is still alive
- `False`indicates the reference has been deleted.
"""
return [wr() is not None for wr in self._weakReferences]
def __str__(self):
overview = f"IteratorWithWeakReferences('initial size':{len(self._weakReferences)} 'remaining elements':{len(self._data)})\n"
overview += str([f'[{index}]:{"Alive" if wr() else "Deleted"} ' for (index, wr) in enumerate(self._weakReferences)])
return overviewSanity check
Let's first confirm it behaves as expected when using islice_extended with positive indexes:
- Create a slice iterator with
islice_extendedwith positive indexes and check status:
iteratorSize = 3
iteratorwwr = IteratorWithWeakReferences.FROM_SIZE(size=iteratorSize)
islice_extended_iterator = islice_extended(iteratorwwr, 0, None, 1)
print(f"iteratorwwr state: {iteratorwwr}")Output:
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':3)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']
We get the expected result:
- ✅Iterator
iteratorwwrhas not yet been iterated. - ✅all its elements are still alive.
- Iterate on the first element and check status
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")Output:
Returned element: AnObject(0)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':2)
['[0]:Deleted ', '[1]:Alive ', '[2]:Alive ']
We get the expected result:
- ✅
iteratorwwrhas been iterated on once and has now only 2 elements remaining. - ✅
AnObject(0)has been returned - ✅
AnObject(0)is now deleted since it's not referenced anymore.
- Let's do it again and again
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")Output:
Returned element: AnObject(1)
Returned element: AnObject(2)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Deleted ', '[1]:Deleted ', '[2]:Deleted ']
We get the expected result:
- ✅
iteratorwwrhas been iterated on all its elements. - ✅
AnObject(1)andAnObject(2)has been returned - ✅
AnObject(1)andAnObject(2)are now deleted since they are not referenced anymore.
The detailed repro steps
- Create a slice iterator with
islice_extendedwith a negative index and check status:
iteratorSize = 3
iteratorwwr = IteratorWithWeakReferences.FROM_SIZE(size=iteratorSize)
islice_extended_iterator = islice_extended(iteratorwwr, -3, None, 1)
print(f"iteratorwwr state: {iteratorwwr}")Output:
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':3)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']
We get the expected result:
- ✅Iterator
iteratorwwrhas not yet been iterated. - ✅all its elements are still alive.
- Iterate on the first element and check status
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")Output:
Returned element: AnObject(0)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']
We do not get the exact expected result:
- ✅
iteratorwwrhas been iterated on all its elements, and that's expected when dealing with negative indexes. - ✅
AnObject(0)has been returned. - ❌
AnObject(0)is still 'alive'. It should have been deleted.
- Let's do it again and again
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")Output:
Returned element: AnObject(1)
Returned element: AnObject(2)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']
We do not get the exact expected result:
- ✅
iteratorwwrhas been iterated on all its elements. - ✅
AnObject(1)andAnObject(2)has been returned. - ❌
AnObject(1)andAnObject(2)are still 'alive'. They should have been deleted.
- Let's do it one last time to hit
StopIteration
try:
next(islice_extended_iterator)
except StopIteration:
print(">> encountered StopIteration as expected")
print(f"iteratorwwr state: {iteratorwwr}")Output:
>> encountered StopIteration as expected
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Deleted ', '[1]:Deleted ', '[2]:Deleted ']
We do get the exact expected result:
- ✅we encounter a
StopIterationexception - ✅
AnObject(0),AnObject(1)andAnObject(2)are deleted.
Environment
- python: 3.11.3
- more-itertools: 10.7.0