Skip to content

iterated objects are not released by islice_extended when start<0 and step>0 #994

@ben42code

Description

@ben42code

iterated objects are not released by islice_extended when start<0 and step>0

Description

An iterator created from islice_extended with a negative start index, does not release the elements that have been returned/iterated on.
⚠️✅They are released once we've iterated to the end of the sliced_iterator/encountered a StopIteration exception.

Illustrative example

# assuming there are 3 elements in anIterator.
sliced_iterator = islice_extended(anIterator, -3, None, 1)
next(sliced_iterator)
next(sliced_iterator)

Expected

The first 2 elements of anIterator are released/deleted (if not referenced anymore)

Observed

The first 2 elements of anIterator are not released/deleted (if not referenced anymore) until we've iterated to the end of sliced_iterator

Repo steps/Example

To illustrate the issue, I'll be leveraging this helper class to track and report when an iterator elements are still alive or have been deleted:

class IteratorWithWeakReferences:
    """
    A class that wraps an iterator and provides weak references to its elements.
    This class allows iteration over a collection of objects while maintaining
    weak references to the original objects. It can be used to track the validity
    of the objects (i.e., whether they are still alive or has been deleted).
    """
    class AnObject:
        def __init__(self, index: int):
            self.index = index

        def __str__(self):
            return f"AnObject({self.index})"

    @classmethod
    def FROM_SIZE(cls, size: int) -> IteratorWithWeakReferences:
        return cls([IteratorWithWeakReferences.AnObject(index=index) for index in range(size)])

    def __init__(self, iterable: Iterable):
        self._data = deque(element for element in iterable)
        self._weakReferences = [weakref.ref(a) for a in self._data]

    def __iter__(self) -> Iterator:
        return self

    def __next__(self) -> AnObject:
        if (len(self._data) == 0):
            raise StopIteration

        return self._data.popleft()

    def weakReferencesAliveStatePerIndex(self) -> list[bool]:
        """
        Checks the alive state of weak references of the initial iterator elements.

        Returns:
            list[bool]: A list of boolean values where each element corresponds to the alive state
            of a weak reference.
            - `True` indicates the reference is still alive
            - `False`indicates the reference has been deleted.
        """
        return [wr() is not None for wr in self._weakReferences]

    def __str__(self):
        overview = f"IteratorWithWeakReferences('initial size':{len(self._weakReferences)} 'remaining elements':{len(self._data)})\n"
        overview += str([f'[{index}]:{"Alive" if wr() else "Deleted"} ' for (index, wr) in enumerate(self._weakReferences)])
        return overview

Sanity check

Let's first confirm it behaves as expected when using islice_extended with positive indexes:

  1. Create a slice iterator with islice_extended with positive indexes and check status:
iteratorSize = 3
iteratorwwr = IteratorWithWeakReferences.FROM_SIZE(size=iteratorSize)
islice_extended_iterator = islice_extended(iteratorwwr, 0, None, 1)
print(f"iteratorwwr state: {iteratorwwr}")

Output:

iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':3)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']

We get the expected result:

  • ✅Iterator iteratorwwr has not yet been iterated.
  • ✅all its elements are still alive.
  1. Iterate on the first element and check status
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")

Output:

Returned element: AnObject(0)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':2)
['[0]:Deleted ', '[1]:Alive ', '[2]:Alive ']

We get the expected result:

  • iteratorwwr has been iterated on once and has now only 2 elements remaining.
  • AnObject(0) has been returned
  • AnObject(0) is now deleted since it's not referenced anymore.
  1. Let's do it again and again
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")

Output:

Returned element: AnObject(1)
Returned element: AnObject(2)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Deleted ', '[1]:Deleted ', '[2]:Deleted ']

We get the expected result:

  • iteratorwwr has been iterated on all its elements.
  • AnObject(1) and AnObject(2) has been returned
  • AnObject(1) and AnObject(2) are now deleted since they are not referenced anymore.

The detailed repro steps

  1. Create a slice iterator with islice_extended with a negative index and check status:
iteratorSize = 3
iteratorwwr = IteratorWithWeakReferences.FROM_SIZE(size=iteratorSize)
islice_extended_iterator = islice_extended(iteratorwwr, -3, None, 1)
print(f"iteratorwwr state: {iteratorwwr}")

Output:

iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':3)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']

We get the expected result:

  • ✅Iterator iteratorwwr has not yet been iterated.
  • ✅all its elements are still alive.
  1. Iterate on the first element and check status
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")

Output:

Returned element: AnObject(0)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']

We do not get the exact expected result:

  • iteratorwwr has been iterated on all its elements, and that's expected when dealing with negative indexes.
  • AnObject(0) has been returned.
  • AnObject(0) is still 'alive'. It should have been deleted.
  1. Let's do it again and again
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"Returned element: {next(islice_extended_iterator)}")
print(f"iteratorwwr state: {iteratorwwr}")

Output:

Returned element: AnObject(1)
Returned element: AnObject(2)
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Alive ', '[1]:Alive ', '[2]:Alive ']

We do not get the exact expected result:

  • iteratorwwr has been iterated on all its elements.
  • AnObject(1) and AnObject(2) has been returned.
  • AnObject(1) and AnObject(2) are still 'alive'. They should have been deleted.
  1. Let's do it one last time to hit StopIteration
try:
    next(islice_extended_iterator)
except StopIteration:
    print(">> encountered StopIteration as expected")
print(f"iteratorwwr state: {iteratorwwr}")

Output:

>> encountered StopIteration as expected
iteratorwwr state: IteratorWithWeakReferences('initial size':3 'remaining elements':0)
['[0]:Deleted ', '[1]:Deleted ', '[2]:Deleted ']

We do get the exact expected result:

  • ✅we encounter a StopIteration exception
  • AnObject(0), AnObject(1) and AnObject(2) are deleted.

Environment

  • python: 3.11.3
  • more-itertools: 10.7.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions