DEV Community: Ashutosh Vaidya

Understanding The Recursion

Ashutosh Vaidya — Wed, 25 Jan 2023 14:11:51 +0000

In computer science, recursion is a method of solving a computational problem in which the solution is dependent on solutions to smaller instances of the same problem. The approach is suitable for a wide range of problems, and recursion is one of the core principles of computer science.

When a function is defined in such a way that it calls itself, it is regarded as a recursive function.

What is need for the recursion

Iterations such as the for loop and while loop can also be used to create a function that calls itself repeatedly. So you may be wondering why we require recursion in the first place.

A common algorithm design strategy is to break a problem down into sub-problems of the same type as the original, solve those sub-problems, and then combine the results. This is widely recognized as the divide-and-conquer method.

Recursion is one of the ways, how we can emulate the divide-and-conquer method. It is an effective tool for dividing larger problems into smaller subproblems. Recursive functions are also more pretty and readable than iterative functions.

Along with the divide-and-conquer method, recursion is widely utilized by many traditional programming algorithms such as,

Tower of Hanoi
Merge Sort
Quick Sort
Tree traversal (BFS and DFS)

How to implement recursion in python

Consider a classic example of calculating a factorial of a number. If one has to write the function using a loop it would look like the below:

def factorial_using_loop(num):
    factorial = 1
    for i in range(1, num+1):
        factorial *= i
    return factorial

num = 5
print(f"Factorial of {num} is => {factorial_using_loop(num)}")

# Output: Factorial of 5 is => 120

Now let us see how it will look like if we use a recursive function,

def factorial_using_recursion(num):
    if num == 0 or num == 1:
        return 1
    else:
        return num * factorial_using_recursion(num-1)

num = 5
print(f"Factorial of {num} is => {factorial_using_recursion(num)}")
# Output: Factorial of 5 is => 120

In this example, when num equals 0 or 1, functions returns 1. This is known as the base case, which is a condition whose output can be obtained without the need for recursion. The recursive case is when the num is greater than 1 then functions returns num multiplied by the factorial of num-1

Another popular use case for recursion is computing a Fibonacci series. This is how we can write the code to compute Fibonacci series for first 10 numbers,

def fibonacci_using_recursion(num):
    if num <= 0:
        return []
    elif num == 1:
        return [0]
    elif num == 2:
        return [0, 1]
    else:
        return fibonacci_using_recursion(num-1) + [fibonacci_using_recursion(num-1)[-1] + fibonacci_using_recursion(num-2)[-1]]
print(fibonacci_using_recursion(10))
# Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Comparing recursion with iterative functions

In Python, recursion, for loops, and while loops are all ways to control the flow of a programme, but they are used in slightly different ways.

Let us see how they compare in terms of memory consumption and execution speed.

Memory Consumption

Each time a recursive function is called, a new stack frame is created, using memory, making recursion potentially more memory-intensive than an iterative approach. This stack frame contains the function's variables as well as its call history.

After the function is called, a stack frame is produced, and it is deleted when the function returns. Recursively calling itself results in the creation of a stack of stack frames, with each new stack frame being added on top of the previous one. This stack of stack frames can eat a lot of memory, especially if the recursion is having complex calculation or the function is invoked frequently.

Execution Speed

Recursion typically outperforms its iterative counterpart because it divides the problem into smaller subproblems, resulting in smaller chunks or tasks. Furthermore, as a bounty point, it makes the function simple to grasp and read.

However, it is critical to have a clearly defined base case that will stop the recursion and produce the result; otherwise, there is a risk of running into an infinite loop due to recursion, which can cause all sorts of problems, including bringing down the entire system's network. When using recursions, use caution at all times.

To summarize,

Recursion is an effective technique for breaking down larger, more complex problems into smaller chunks.
A recursive function is one that repeatedly calls itself.
Always have a base case that stops the recursion and produces a result.
Recursion can be used to create shorter, more readable code fragments.
Recursion can be memory-intensive.
Always use recursion with caution.

I hope you found this post interesting and learned something new today. Please add anything I missed in this article, and as always, Happy Learning!

Mastering Higher-Order Functions

Ashutosh Vaidya — Thu, 19 Jan 2023 06:54:49 +0000

Introduction

A higher-order function in computer science is one that accomplishes at least one of the following:

Accepts one or more functions as arguments.
Returns a function as a result

These functions are quite beneficial for writing more abstract, reusable code.

In this post, we will learn about some of the most popular and often used higher-order functions in Python and how to write our own.

One of the most popular applications of higher-order functions in Python is data creation and manipulation. Higher-order functions such as map(), filter(), and reduce() are examples of built-in functions that may be used to transform and process data in several ways.

Let us go over them one by one.

Map

The map function accepts a function and an iterable as inputs and applies the function to each element of the iterable before producing a new iterable with the results. This is handy for performing a specific action on many data points at once without the need for a loop.

Take the problem of calculating the square of each number in the provided list, for instance. The simplest way to achieve this using a loop will be as shown below:

numbers = [1, 2, 3, 4, 5]
squares = []
for i in numbers:
    squares.append(i**2)
print(squares)  
# Output: [1, 4, 9, 16, 25]

Let's see how we can accomplish the same thing with map() instead:

numbers = [1, 2, 3, 4, 5]
squares = map(lambda x: x**2, numbers)
print(list(squares)) 
# Output: [1, 4, 9, 16, 25]

To clarify, the map takes iterable numbers and applies function lambda x: x**2 to it to produce a new iterable squares. Since it is iterable, we typecast the squares to a list before printing the result.

Filter

The filter function, on the other hand, takes a function and an iterable as inputs and produces a new iterable containing only the items of the original iterable for which the function returns True. This is useful for filtering big datasets and obtaining only the important information.

Consider the task where we want to return only even integers from the list. Using a loop, we can do this by doing the following:

numbers = [1,2,3,4,5,6,7,8,9,10]
even_numbers = []
for i in numbers:
    if i % 2 == 0:
        even_numbers.append(i)
print(even_numbers)
# Output: [2, 4, 6, 8, 10]

Now let us do the same using filter():

numbers = [1,2,3,4,5,6,7,8,9,10]
even_numbers = filter(lambda x: x % 2 == 0, numbers)
print(list(even_numbers))
# Output: [2, 4, 6, 8, 10]

We achieve the same result in one-liner clear code.

Reduce

The reduce function is a little different; for a starter, it is not a built-in function like the map and filter. We need to import it from the functools module. The reduce accepts a function and an iterable as parameters and, to reduce the iterable to a single value, cumulatively applies the function to each member of the iterable from left to right. This is efficient for performing complex calculations on huge datasets.

Assume you have a collection of family bills and your assignment is to determine the overall spending. The basic program will look like this:

bills = [100,200,300,400,500]
overall_spent = 0
for i in bills:
    overall_spent += i
print(overall_spent)
# Ouptut: 1500

This is how we can do it using reduce(),

from functools import reduce
bills = [100,200,300,400,500]
overall_spent = product = reduce(lambda x, y: x+y, bills)
print(overall_spent)
# Output: 1500

Custom higher-order function

Python also allows programmers to write their higher-order functions. This may be achieved by writing a function that accepts one or more other functions as arguments or by returning a function as a result.

These custom higher-order functions may be utilized to build more complex and efficient applications without the need for repeated or redundant coding.

Check out the following code snippet:

def make_greeting(greeting):
    def greet(name):
        return f"{greeting}, {name}!"
    return greet

hello = make_greeting("Hello")
print(hello("Jon")) # Output: "Hello, Jon!"

goodbye = make_greeting("Goodbye")
print(goodbye("Jane")) # Output: "Goodbye, Jane!"

Here, we wrote a custom higher-order function called make greeting that accepts a greeting as an argument and produces a new function called greet that accepts a name as an argument and returns a greeting message. We then built two welcome function instances, one for "Hello" and one for "Goodbye."

Conclusion

Higher-order functions are commonly used in software development to produce reusable, modular code. They are incredibly helpful and widely used in data science and machine learning. Consider working with enormous datasets and wanting to do preprocessing to clean data; here is where functions like map, reduce, and the filter comes into play.

Therefore, it is crucial to understand its fundamentals if one hopes to flourish in the fields of software engineering or data science. I hope this post is informative and gives you a good insight into higher-order functions. Happy Learning!

Python Functions: A Beginner’s Guide — Part 2

Ashutosh Vaidya — Tue, 17 Jan 2023 03:48:39 +0000

Introduction

In the previous article in the series Python Functions: A beginner’s Guide — Part 1, we learned about the fundamentals of functions, how to define them, and what terminology is used. This article will explore the local and global scope of variables, default and keyword paraments, and different types of functions.

Local and Global Scope

When a variable is defined within a function, it is considered a local variable and is accessible only within that function. For example, in the following code, the name variable is a local variable that can only be accessed within the greet function:

def greet(name):
    greeting = 'Hello, ' + name
    return greeting

print(greet('Jon'))  # Output: 'Hello, Jon'
print(name)  # This will cause an error, as the name variable is not defined outside of the greet function

A global variable, on the other hand, is defined outside of any function and can be accessed and modified from anywhere in the code. An example of a global variable is as follows:

name = 'Jon'  # This is a global variable
def greet():
    greeting = 'Hello, ' + name  #The name variable is accessed within the function
    return greeting

print(greet())  # Output: 'Hello, Jon'
print(name)  # Output: 'Jon'

Note: Global variables should be avoided wherever possible because they can make your code more difficult to understand and debug.

If you need to use a variable in multiple functions, it is often better to pass it as an argument to the functions that need it. This can make your code more modular and easier to maintain. We will explore different types of parameters in the following section.

Function Parameters

Function parameters in Python are variables used in the function definition to specify the input that the function can accept when called.

Take a look at the following function definition:

def greet(name, greeting='Hello'):
    return f'{greeting}, {name}'

The name and greeting variables in this example are function parameters. When calling the function, you can supply values for these parameters to specify the function’s arguments. As an example:

print(greet('Jon'))  # Output: 'Hello, Jon'
print(greet('Jon', 'Hi'))  # Output: 'Hi, Jon'

In Python, you can use a variety of function parameters, including:

Positional parameters: These are the most common kind of function parameters, and their position in the function declaration determines their value. In the preceding example, name is a positional parameter.
Default parameters: You can define default values for function parameters in the function declaration by including an equal sign and a value. In the example from earlier, greeting has a default value of ‘Hello,’ so if you don’t specify one when using the function, it will use the default value.
Keyword parameters: Function arguments can be specified using keyword notation, which enables you to specify the values of the arguments in any order as long as the names of the arguments are included. For example:

print(greet(greeting='Hi', name='Jon'))  # Output: 'Hi, Jon'

*args and **kwargs

The * and ** operators in Python can be used to pass a variable number of arguments to a function.

The *args operator allows you to pass a variable number of positional (i.e., non-keyword) arguments to a function. The arguments are passed as a tuple. For example:

def sum_numbers(*args):
    total = 0
    for number in args:
        total += number
    return total

print(sum_numbers(1, 2, 3))  # Output: 6
print(sum_numbers(1, 2, 3, 4, 5))  # Output: 15

The **kwargs operator allows you to pass a variable number of keyword arguments to a function. The arguments are passed as a dictionary. For example:

def print_keyword_arguments(**kwargs):
    for key, value in kwargs.items():
        print(f'{key} = {value}')

print_keyword_arguments(name='Jon', age=30)
    # Output:
    # name = Jon
    # age = 30

Note: Using *args and **kwargs can be useful when you want to write a flexible function that can handle a variable number of arguments.

Different types of Functions

In Python, there are several types of functions that you can use. These include:

Normal functions: These are the most common type of functions and are defined using the def keyword. They can take arguments and return a value.
Lambda functions: These are single-line, anonymous functions that are defined using the lambda keyword. They are often used when you need to define a simple function for a short period of time. For example:

add = lambda x, y: x + y
print(add(2, 3))  # Output: 5

Recursive functions: These are functions that call themselves in order to solve a problem. They can be used to solve problems that can be divided into smaller subproblems, such as calculating the factorial of a number. Here is an example of a recursive function that calculates the factorial of a number:

def factorial_recursive(num):
    if num == 0:
        return 1
    else:
        return num * factorial_recursive(num - 1)

fact = factorial_recursive(5)
print(f"Factorial of 5 is {fact}") # Output: Factorial of 5 is 120

Nested functions: In Python, you can define functions within other functions. These are called nested functions. The inner function has access to the variables of the outer function, but not vice versa. For example:

def outer(x):
    y = x + 1  # y is a local variable of the outer function
    def inner(z):
        return z + y  # The y variable is accessible within the inner function
    return inner

my_function = outer(2)
print(my_function(3))  # Output: 6
print(y)  # This will cause an error, as the y variable is not defined outside of the outer function

Note: Nested functions can be useful for organizing and structuring code, but it is important to be aware of the variable scopes and how they affect the accessibility of variables within the functions.

Generators: A generator is a special type of function that returns an iterator that produces a sequence of values on demand, instead of returning a single value. In Python, you can define a generator function using the yield keyword, like below:

def my_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

When you call a generator function, it does not execute the code within the function. Instead, it returns a generator object that you can iterate over to produce the values. For example, if you want to iterate over to generator 'my_range` we just created you can write something like below:

for i in my_range(3): print(i) #Output: 0, 1, 2

Note: Generators are useful for producing sequences of values that are too large to fit in memory all at once. They allow you to process the values one at a time, which can be more efficient and use less memory than storing all of the values in a list.

If you find generators interesting, you may like my article Behind the scenes of the For Loop where I discussed iterators, generators, and how the For Loop works internally.

Advantages of Functions

Code Reuse: Functions allow you to reuse code, which saves time and improves the efficiency of your code. Instead of repeating the same code, you can construct a function once and call it anytime the task requires it.
Modularization: Functions can assist you in breaking down your code into smaller, more manageable chunks. This can help you minimize repetition and make your code easier to read and understand.
Abstraction: Functions enable you to mask the specifics of how a task is carried out from the rest of your code. Because you don’t have to worry about the details of how the function works, your code will be easier to read and understand.
Testing and Debugging: Functions can let you test and debug your code more easily. Because functions are self-contained chunks of code, you can test and debug them independently of the rest of your code.
Code Maintenance: Functions can help you maintain your code more easily over time. If you need to update the code for a function, you only need to edit it once rather than searching through your entire program to identify all of the places where the code is utilized.

Final Summary

In a summary,

Functions are an essential aspect of Python programming and are used to organize and structure code. They let you reuse code, saving you time and making your code more efficient.
Functions have their own variable scope, and several sorts of functions, including conventional functions, lambda functions, and recursive functions to name a few.
Function parameters can also be used to establish default values and pass a variable number of arguments to a function.
Functions provide numerous advantages, including code reuse, modularization, automation, testing and debugging, and code maintenance.

I hope you find this article helpful, don’t forget to give a like, provide feedback and subscribe. Happy Reading!

Python Functions: A Beginner’s Guide - Part 1.

Ashutosh Vaidya — Sun, 15 Jan 2023 09:28:18 +0000

Introduction

As a programmer, I have always found functions very useful. They are a powerful tool that can improve both the clarity and expressiveness of the code. By the end of this article, you will get to know what is a function, how to write a function in the Python programming language, and get a brief overview of functions.

What is a Function?

A function is a block of organized and related statements that can take inputs and perform a specific task. A function allows us to use code repeatedly without worrying about duplication and maintenance if any changes are required in code. Functions are also called methods, procedures, subprograms or routines, etc.

There are two types of functions in Python: built-in and user-defined.

Built-in functions are provided by python and are immediately available. For example, print(), len() to name a few.
User-defined functions are created by programmers and are the main focus of this article.

Creating the first function

In Python, a function is defined using the def keyword. See the below example for reference,

    def hello_World():
        print("Hello World")

Here, the first line is a def statement, which defines a function named hello_world(), and the second line onwards is a body of a function. The code will be executed when the function is called not when it's defined which we have done right now.

Calling the function

The function is called as shown below and it will print Hello World

    hello_world()

    #Output:
    #Hello World

A function with a parameter and a return value

Now that we understand how to create a basic function, let’s create one function which takes the user’s name as input and greets the user.

    def greetings(name):
        print(f"Hello, {name}")

    greetings("Jon")

    #Output:
    #Hello, Jon

Here we have passed the user name as a parameter to the function and the function print’s the greeting. What will happen if we use the function calling enclosed in the print statement, let's try it out.

    print(greetings("Jon"))

    #Output:
    #Hello, Jon
    #None

What just happened? we got two outputs, the first Hello, Jon which was expected and the second line is None, where did it come from?

Well, the function can return the value to a Caller, in our example, we have not returned anything from the function hence it is returning the special type None. If we want to return something from the function then we will need to use the return statement.

Let’s modify our function to return from it.

    def greetings(name):
        print(f"Hello, {name}")
        return "It's nice to meet you!"

    print(greetings("Jon"))

    #Output:
    #Hello, Jon
    #It's nice to meet you!

In the above example, we have returned a string type of value. A function can return any python object. It can return integer, float, list, set, dictionary, etc. If there is no value to return then as we observed earlier function will return None

Terminologies

You must be thinking Define the function, call the function, pass the value, and return the value, so many things to remember. It can feel like a daunting task at first. So let’s explore them using a simple example by writing a function that accepts a number and returns its factorial.

To define a function means to create it. When the keyword def is used in the statement, it defines and names the function. In the above code example, line number 1 defines the function factorial().
num enclosed in parenthesis in line 1, is a parameter.
From Lines 2 onwards till line number 8 is a function body, here we write the logic we wanted to perform.
Line numbers 3 and 8 are a return statement, which returns the value of the factorial calculated in the function to the caller.
Line number 10 is a caller, who calls the function by passing a value 5 to parameter num. The value being passed is called an argument.
And finally, the variable fact holds the value returned by the function.

It’s easy to mix up these things, but hopefully, this will make them easier to understand and remember.

Conclusion

In this article, we understand the basics of functions, how to define them, and what are different terminologies used. In the next part, we will explore different types of arguments (Keyword, Positional, default, etc.), learn about variable scope, and also understand Lambda functions. So don’t forget to follow and stay tuned for the notification.

Infinity In Python

Ashutosh Vaidya — Thu, 12 Jan 2023 18:10:27 +0000

Infinity is defined as everything that is limitless, unending, or greater than any natural number. Infinity is a topic that philosophers and mathematicians frequently debate.

But what does infinite means in programming? Anyone with even a rudimentary understanding of a programming language knows that we first encounter infinity when we do some arithmetic operations and unintentionally divide a number by zero, aha, that famous Divide By Zero error.

In the field of computer science, infinity is commonly used to,

Evaluate and improve the performance of large-scale computing algorithms.
Set the minimum and maximum values.
To compare finite and infinite values

In Python, you cannot express infinite as an integer since it contradicts the notion of infinity, which is also true for other programming languages such as C, C++, and Java. However, because Python is a dynamic language, float values can be used to represent an infinite integer in a code.

Let us have a look at each of the numerous ways to express infinity in Python.

By Using float('inf') and float('-inf')

This is the easiest method because we don't need to import any libraries. To accomplish this, use the code snippet below.

    # Defining a positive infinite integer
    positive_infinity = float('inf')
    print('Positive Infinity: ', positive_infinity)

    # Defining a negative infinite integer
    negative_infinity = float('-inf')
    print('Negative Infinity: ', negative_infinity)

    # Expected Output:
    # Positive Infinity:  inf
    # Negative Infinity:  -inf

By Using Math module

Python's Math module is well-known and widely utilized for any mathematical operation we want to do. It also allows for infinity. For reference, below is a code excerpt.

    import math

    # Defining a positive infinite integer
    positive_infinity = math.inf
    print('Positive Infinity: ', positive_infinity)

    # Defining a negative infinite integer
    negative_infinity = -math.inf
    print('Negative Infinity: ', negative_infinity)

    # Expected Output:
    # Positive Infinity:  inf
    # Negative Infinity:  -inf

By Using Decimal module

Python's Decimal module also provides an infinite constant that we may make advantage of.

    from decimal import Decimal

    # Defining a positive infinite integer
    positive_infinity = Decimal('Infinity')
    print('Positive Infinity: ', positive_infinity)

    # Defining a negative infinite integer
    negative_infinity = Decimal('-Infinity')
    print('Negative Infinity: ', negative_infinity)

    # Expected Output:
    # Positive Infinity:  Infinity
    # Negative Infinity:  -Infinity

By Using Numpy module

Finally, we can utilise the ever-popular NumPy module to obtain the infinity constant for future use.

    import numpy as np

    # Defining a positive infinite integer
    positive_infinity = np.inf
    print('Positive Infinity: ', positive_infinity)

    # Defining a negative infinite integer
    negative_infinity = -np.inf
    print('Negative Infinity: ', negative_infinity)

    # Expected Output:
    # Positive Infinity:  inf
    # Negative Infinity:  -inf

Arithmetic operations on infinity

Now that we know how to define Infinity, let us go a little further into it and do some arithmetic operations on it. We are well aware that any mathematical operation is done on an infinite value will result in an endless value, whether it be addition, subtraction, multiplication, or division, for example.

The code sample below validates this idea.

    # Defining a positive infinite integer
    positive_infinity = float('inf')
    print('Positive Infinity: ', positive_infinity)

    # Defining a negative infinite integer
    negative_infinity = float('-inf')
    print('Negative Infinity: ', negative_infinity)

    print('Arithmetic operation on Positive infinity: ')
    # Multiply Positive infinity number by 5
    print('Multiplication : ',positive_infinity * 5)

    # Addition to Positive infinity Number
    print('Addition : ',positive_infinity + 5)

    # Subtraction to Positive infinity Number
    print('Subtraction : ',positive_infinity - 5)

    # Division to Positive infinity Number
    print('Division : ',positive_infinity / 5)

    print('Arithmetic operation on Negative infinity: ')
    # Multiply Negative infinity number by 5
    print('Multiplication : ',negative_infinity * 5)

    # Addition to Negative infinity Number
    print('Addition : ',negative_infinity + 5)

    # Subtraction to Negative infinity Number
    print('Subtraction : ',negative_infinity - 5)

    # Division to Negative infinity Number
    print('Division : ',negative_infinity / 5)

    # Expected output
    # Arithmetic operation on Positive infinity: 
    # Multiplication :  inf
    # Addition :  inf
    # Subtraction :  inf
    # Division :  inf
    # Arithmetic operation on Negative infinity: 
    # Multiplication :  -inf
    # Addition :  -inf
    # Subtraction :  -inf
    # Division :  -inf

Checking if a number is infinite

What if you are in need to check whether a number is infinite or not, we can use the math module’s isinf() function. It returns a Boolean result, which indicates that if the specified integer is infinite, it returns true; otherwise, it returns false. See the code excerpt below.

    import math

    # Defining a positive infinite integer
    positive_infinity = float('inf')
    print('Positive Infinity: ', positive_infinity)
    print('The variable is infinity = ',math.isinf(positive_infinity))

    # Defining a negative infinite integer
    negative_infinity = float('-inf')
    print('Negative Infinity: ', negative_infinity)
    print('The variable is infinity = ',math.isinf(negative_infinity))

    # Expected Output: 
    # Positive Infinity:  inf
    # The variable is infinity =  True
    # Negative Infinity:  -inf
    # The variable is infinity =  True

The positive infinity number is the greatest, and the negative infinity number is the smallest of all numbers.

Let us verify this with a program.

    if 99999999999999999 > positive_infinity:
        print('Numer is greater than Positive infinity')
    else:
        print('Positive infinity is greater')

    if -99999999999999999 < negative_infinity:
        print('Numer is smaller than Negative infinity')
    else:
        print('Negative infinity is smaller')

    #Expected Output:
    # Positive infinity is greater
    # Negative infinity is smaller

Conclusion
We looked at many approaches to express infinity in Python in this tutorial. We also look at the arithmetic operation on infinity and how it relates to other natural numbers. I hope you found this tutorial useful and learned something new today.

All the code snippets from the article are available on my GitHub repo if anyone wants to play around. You can get in touch with me on my LinkedIn as well.

Behind The Scenes Of For Loop

Ashutosh Vaidya — Wed, 11 Jan 2023 15:52:33 +0000

A For Loop is a control flow statement for specifying iteration in computer science. In simpler terms, a for loop is a section or block of code which runs repeatedly until a certain condition has been satisfied. It is the most basic concept, without which every programming language will be incomplete.

While exploring the ever-surprising python language, I uncovered that it is an Iterator that plays a huge part in the For Loop and many other higher-order functions such as the range function. This article is written to understand what is an iterator, what is a generator, and how it helps us to understand the crux behind the For Loop.

What is an Iterator in python?

An iterator is any python object which can be iterated upon and returns data one at a time. They are implemented in comprehension, loops, and generators but are not readily seen in plain sight. They are the pillars behind most of the beforementioned functionality of python.

An iterator object must support two methods namely iter() and next(). This is collectively called an Iterator protocol.

iter() is a function used to return an iterator by calling the iter() method on an Iterable object.
next() is used to iterate through each element. StopIteration is an exception raised whenever the end is reached.

Wait, iter() method is called on an Iterable object? What is the difference between an Iterator and an Iterable?

Well, an Iterable is an object, that one can iterate over. It generates an Iterator when passed to iter() method. Example of iterable objects includes string, list, tuple, dictionary, etc.

Remember, every iterator is also an iterable, but not every iterable is an iterator.

Now that we have some theoretical knowledge about an iterator let’s see them in action.

Let us first see how the iterable object can be converted into an iterator.

    # We know string can be iterated over.
    s = "Loreum Ipsum"
    for word in s:
        print(word)

Output:

What will happen if we call, next() on the string

    s = "Loreum Ipsum"
    next(s)

Output:

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    ~\AppData\Local\Temp/ipykernel_41560/3883344095.py in <module>
          1 # Calling next() on string
    ----> 2 next(s)

    TypeError: 'str' object is not an iterator

As we mentioned earlier we first need to call the iter method to create an iterator from the iterable object then only we can use this next method to access the element one by one. See the code snippet below which corrects that,

    s = "Loreum Ipsum"
    myIter = iter(s)
    print(next(myIter))
    print(next(myIter))
    print(next(myIter))
    print(next(myIter))

Output:

Now we will try to create an iterator that will print numbers from 1 to n.

    class PrintNumber:
        def __init__(self, max):
            self.max = max

        def __iter__(self):
            self.num = 0
            return self

        def __next__(self):
            if(self.num >= self.max):
                raise StopIteration
            self.num += 1
            return self.num

    print_num = PrintNumber(3)

    print_num_iter = iter(print_num)
    print(next(print_num_iter))  
    print(next(print_num_iter))  
    print(next(print_num_iter))

Output:

    1
    2
    3

Notice, we have called next only 3 times, since we explicitly provided max or n as 3. If we give another call to next it will throw StopIteration.

How does For loop work?

Equipped with our knowledge of iterator and how it needs to be implemented it is very easy now to understand how exactly the for loop works. For Loop calls an iter method to get an iterator and then calls method next over and over until a StopIteration exception is raised, which it handles gracefully to end the loop without breaking the code. That’s all, we have debunked the behind the scene of the For loop.

Still, something feels incomplete, inefficient. To create an iterator we have to implement an Iterator Protocol, a class with iter() and next() methods. We also need to keep track of the internal states and handle StopIteration exception so that the code is not broken. It seems like lots of work. This is where Generator comes in.

Generator in Python,

Simply speaking, a generator is a function that returns an iterator. It is very easy to create a generator in python. We simply need to define a normal function with one change being, using the keyword yield instead of return. If a function contains at least one yield statement then it is a generator.

The difference between return and yield is, while return terminates a function entirely, yield temporarily halts a function while retaining its local values and later continues from there on successive calls.

Below is the generator class which iterates the number from 0 to n and yields a number that is divisible by 7

    class divisible_by_7_generator:
        def __init__(self, num):
            self.num = num
        #Generator
        def get_nums_divisible_by_7(self):
            for i in range(0, self.num):
                if (i % 7 == 0):
                    yield i

    n = 100
    result = divisible_by_7_generator(n)
    print(f"Numbers which are divisible between 0 to {n} are:")
    for num in result.get_nums_divisible_by_7():
        print(num, end = ",")

Output:

    Numbers which are divisible between 0 to 100 are:
    0,7,14,21,28,35,42,49,56,63,70,77,84,91,98,

Python Generator Expression

A simple generator can be easily created on the fly using generator expressions. This is very similar to lambda functions with a syntax resembling to the list comprehension in python, with square brackets replaced by round parentheses. Unlike list comprehensions, generator expressions follow lazy execution, meaning producing items only when asked for and thus hugely beneficial for memory management.

Check out the below example, finding a cube of each element in a list.

    myList = [1,2,3,4,5,6]
    #list Comprehension
    cubeList = [x**3 for x in myList]
    # generator expressions are surrounded by parenthesis ()
    cubeGenerator = (x**3 for x in myList)

    print(cubeList)
    print(cubeGenerator)

Output:

    [1, 8, 27, 64, 125, 216]
    <generator object <genexpr> at 0x0000022F5D3EBD60>

We can use for loop to access the values from the generator as follow,

    for i in cubeGenerator: 
        print(i)

Output:

Notice, that list comprehension returns a list while the generator returns elements one at a time.

Conclusion,

In this article, we dug deep into the For Loop and in the process learned about very interesting and useful features such as Iterators, Generators. We also get to understand what is yield and how a generator can be implemented.

Finally, I would like to list a few frequently asked questions related to this topic in interviews:

What is the difference between an Iterator and an Iterable?
Can next() can be called on a string or on an iterable object?
What is an Iterator Protocol?
What is a Generator? What is the difference between an Iterator and a Generator?
What is the Difference between Generator and a Function?
What is Yield and what is the difference between Yield and Return?

If you like this article please follow me and don’t forget to like and give feedback. Every feedback will motivate me to explore more and come up with many more interesting topics in the future. All the code snippets from the article are available on my GitHub repo if anyone wants to play around. You can get in touch with me on my LinkedIn.