{"id":1127,"date":"2023-06-15T15:59:00","date_gmt":"2023-06-15T10:29:00","guid":{"rendered":"https:\/\/geekpython.in\/?p=1127"},"modified":"2024-03-05T13:42:38","modified_gmt":"2024-03-05T08:12:38","slug":"python-generators-with-yield-statement","status":"publish","type":"post","link":"https:\/\/geekpython.in\/python-generators-with-yield-statement","title":{"rendered":"Python Generators and the Yield Keyword &#8211; How They Work"},"content":{"rendered":"\n<p>Generators are defined by a function that generates values on the fly and can be iterated in the same way that we iterate strings, lists, and tuples in Python using the&nbsp;<code>\"for\"<\/code>&nbsp;loop.<\/p>\n\n\n\n<p>When the body of a normal function contains the&nbsp;<code>yield<\/code>&nbsp;keyword instead of the&nbsp;<code>return<\/code>&nbsp;keyword, it is said to be a&nbsp;<strong>generator function<\/strong>.<\/p>\n\n\n\n<p>In this article, we&#8217;ll look at:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What are generator and generator functions?<\/li>\n\n\n\n<li>Why do we need them?<\/li>\n\n\n\n<li>What does the yield statement do?<\/li>\n\n\n\n<li>Generator expression<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"heading-generator\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-generator\"><\/a>Generator<\/h2>\n\n\n\n<p><a target=\"_blank\" href=\"https:\/\/peps.python.org\/pep-0255\" rel=\"noreferrer noopener\">PEP 255<\/a>&nbsp;introduced the generator concept, as well as the&nbsp;<code>yield<\/code>&nbsp;statement, which is used in the generator function.<\/p>\n\n\n\n<p>When called, the&nbsp;<strong>generator function<\/strong>&nbsp;returns a generator object or generator-iterator object, which we can loop over in the same way that we do with lists.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \"># Generator function to generate odd numbers\ndef gen_odd(num):\n    n = 0\n    while n &lt;= num:\n        if n % 2 == 1:\n            yield n\n        n += 1\n\nodd_num = gen_odd(10)\nfor i in odd_num:\n    print(i)<\/pre><\/div>\n\n\n\n<p>The above code defines the&nbsp;<code>gen_odd<\/code>&nbsp;generator function, which accepts an arbitrary number and returns a generator object that can be iterated using either the&nbsp;<code>\"for\"<\/code>&nbsp;loop or the&nbsp;<code>next()<\/code>&nbsp;method.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">1\n3\n5\n7\n9<\/pre><\/div>\n\n\n\n<p>By iterating over the generator-iterator object, we obtained the odd numbers between 0 and 10.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"heading-why-generators\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-why-generators\"><\/a>Why Generators?<\/h3>\n\n\n\n<p>We now have a general understanding of generators, but why do we use them?&nbsp;<strong>The great thing about generator functions is that they return iterators, and iterators use a strategy known as<\/strong>&nbsp;<a target=\"_blank\" href=\"https:\/\/en.wikipedia.org\/wiki\/Lazy_evaluation\" rel=\"noreferrer noopener\"><strong>lazy evaluation<\/strong><\/a><strong>, which means that they return values only when requested<\/strong>.<\/p>\n\n\n\n<p>Consider a scenario in which we need to compute very large amounts of data. In that case, we can use generators to help us because generators compute the data on demand, eliminating the need to save data in memory.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"heading-yield-what-it-does\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-yield-what-it-does\"><\/a>yield &#8211; What It Does?<\/h3>\n\n\n\n<p>The&nbsp;<code>yield<\/code>&nbsp;statement is what gives generators their allure, but what does it do within the function body? Let&#8217;s look at an example to see how the process works.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">def gen_seq(num):\n    initial_val = 1\n    while initial_val &lt;= num:\n        yield initial_val\n        initial_val += 1\n\nsequence = gen_seq(3)<\/pre><\/div>\n\n\n\n<p>The generator function\u00a0<code>gen_seq<\/code>\u00a0generates a sequence of numbers up to the specified\u00a0<code>num<\/code>\u00a0argument. The generator function is called, and it will return a generator object.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">print(sequence)\n----------\n&lt;generator object gen_seq at 0x000001FFBDB55770&gt;<\/pre><\/div>\n\n\n\n<p>We can now use the generator object&#8217;s&nbsp;<code>next()<\/code>&nbsp;method. To get the values, use&nbsp;<code>sequence.__next__()<\/code>&nbsp;or&nbsp;<code>next(sequence)<\/code>.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">print(sequence.__next__())\nprint(sequence.__next__())\nprint(sequence.__next__())\n----------\n1\n2\n3<\/pre><\/div>\n\n\n\n<p>When we call the generator object&#8217;s\u00a0<code>__next__()<\/code>\u00a0method, the code inside the function executes until it reaches the\u00a0<code>yield<\/code>\u00a0statement.<\/p>\n\n\n\n<p><strong>What happens when the function code encounters the<\/strong>&nbsp;<code>yield<\/code>&nbsp;<strong>statement?<\/strong>&nbsp;The&nbsp;<code>yield<\/code>&nbsp;statement operates differently than the&nbsp;<code>return<\/code>&nbsp;statement.<\/p>\n\n\n\n<p><strong>The<\/strong>\u00a0<code>yield<\/code>\u00a0<strong>statement returns the value to the<\/strong>\u00a0<code>next()<\/code>\u00a0<strong>method&#8217;s caller and instead of exiting the program, it retains the function&#8217;s state<\/strong>. When we call the\u00a0<code><code>__next__()<\/code><\/code>\u00a0method again, the execution resumes where it was left.<\/p>\n\n\n\n<p>Check the code below to gain a better understanding.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">def gen_seq():\n    print(\"Start\")\n    val = 0\n    print(\"Level 1\")\n    while True:\n        print(\"Level 2\")\n        yield val\n        print(\"Level 3\")\n        val += 1\n\nsequence = gen_seq()\nprint(sequence.__next__())\nprint(sequence.__next__())\nprint(sequence.__next__())<\/pre><\/div>\n\n\n\n<p>The&nbsp;<code>print<\/code>&nbsp;statement is set at every level in the above code to determine whether or not the&nbsp;<code>yield<\/code>&nbsp;statement continues execution from where it was left.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">Start\nLevel 1\nLevel 2\n0\nLevel 3\nLevel 2\n1\nLevel 3\nLevel 2\n2<\/pre><\/div>\n\n\n\n<p>The code begins at the beginning and progresses through levels 1 and 2 before returning the yielded value to the\u00a0<code><code>__next__()<\/code><\/code>\u00a0method&#8217;s caller. When we call the\u00a0<code><code>__next__()<\/code><\/code>\u00a0method again, the previously yielded value increments by 1, and the execution cycle is resumed from where it was left.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"1000\" src=\"https:\/\/geekpython.in\/wp-content\/uploads\/2023\/08\/yield-working.png\" alt=\"Execution cycle of a generator\" class=\"wp-image-1159\" srcset=\"https:\/\/geekpython.in\/wp-content\/uploads\/2023\/08\/yield-working.png 1400w, https:\/\/geekpython.in\/wp-content\/uploads\/2023\/08\/yield-working-300x214.png 300w, https:\/\/geekpython.in\/wp-content\/uploads\/2023\/08\/yield-working-1024x731.png 1024w, https:\/\/geekpython.in\/wp-content\/uploads\/2023\/08\/yield-working-768x549.png 768w\" sizes=\"auto, (max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"heading-exception\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-exception\"><\/a>Exception<\/h3>\n\n\n\n<p>The generators, like all iterators, can become exhausted after all the iterable values are evaluated. Consider the generator function&nbsp;<code>gen_odd<\/code>&nbsp;from earlier.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \"># Generator function to generate odd numbers\ndef gen_odd(num):\n    n = 0\n    while n &lt;= num:\n        if n % 2 == 1:\n            yield n\n        n += 1\n\nodd_num = gen_odd(3)\nprint(odd_num.__next__())\nprint(odd_num.__next__())\nprint(odd_num.__next__())<\/pre><\/div>\n\n\n\n<p>The above code will generate odd numbers up to 3. As a result, the program will only generate 1 and 3, allowing us to call the\u00a0<code><code>__next__()<\/code><\/code>\u00a0method twice. When we run the above code, we will get the following result.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">1\n3\nTraceback (most recent call last):\n  ....\nStopIteration<\/pre><\/div>\n\n\n\n<p>When we called the first two\u00a0<code><code>__next__()<\/code><\/code>\u00a0methods on\u00a0<code>odd_num<\/code>, we got the yielded values, but when we called the last\u00a0<code><code>__next__()<\/code><\/code>\u00a0method, our code threw a\u00a0<code>StopIteration<\/code>\u00a0exception which indicates that the iterator has ended.<\/p>\n\n\n\n<p>Instead of raising the&nbsp;<code>StopIteration<\/code>&nbsp;exception, the program would have simply exited if we had used the&nbsp;<code>\"for\"<\/code>&nbsp;loop.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"heading-yield-in-tryfinally\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-yield-in-tryfinally\"><\/a>yield In try\/finally<\/h3>\n\n\n\n<p>Take a look at the example below in which we have a generator function and&nbsp;<strong><em>try\/except\/finally<\/em><\/strong>&nbsp;clause inside it.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">def func():\n    try:\n        yield 0\n        try:\n            yield 1\n        except:\n            raise ValueError\n    except: # program never get to this part\n        yield 2\n        yield 3\n    finally:\n        yield 4\n\nx = func()\nfor val in x:\n    print(val)<\/pre><\/div>\n\n\n\n<p>If we run the above code, we&#8217;ll get the following output:<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">0\n1\n4<\/pre><\/div>\n\n\n\n<p>We can see that we didn&#8217;t get the values 2 and 3 as our program didn&#8217;t reach that part because of the&nbsp;<code>ValueError<\/code>, and as is customary when an error occurs, the program proceeds to the&nbsp;<code>finally<\/code>&nbsp;clause, execute it, and exit the program.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"heading-generator-expression\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-generator-expression\"><\/a>Generator Expression<\/h2>\n\n\n\n<p>You must have used the list comprehension, The&nbsp;<strong>generator expression<\/strong>&nbsp;allows us to create a generator in a few lines of code. Unlike list comprehensions, the generator expressions are enclosed within parenthesis&nbsp;<code>()<\/code>.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">gen_odd_exp = (n for n in range(5) if n % 2 == 1)\nprint(gen_odd_exp)\n\n----------\n&lt;generator object &lt;genexpr&gt; at 0x000001D635E33C30&gt;<\/pre><\/div>\n\n\n\n<p>The above generator expression&nbsp;<code>gen_odd_exp<\/code>&nbsp;is somewhat equivalent to the generator function&nbsp;<code>gen_odd<\/code>&nbsp;which we saw at the beginning. We can iterate just like we would with a generator function.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">print(next(gen_odd_exp))\nprint(next(gen_odd_exp))\n\n----------\n1\n3<\/pre><\/div>\n\n\n\n<p>When we compare the memory requirements of the generator expression and list comprehension, we get the following result.<\/p>\n\n\n\n<div class=\"wp-block-urvanov-syntax-highlighter-code-block\"><pre class=\"lang:python decode:true \">import sys\n\ngen_odd_exp = (n for n in range(10000) if n % 2 == 1)\nprint(f'{sys.getsizeof(gen_odd_exp)} bytes')\n\nlist_odd_exp = [n for n in range(10000) if n % 2 == 1]\nprint(f'{sys.getsizeof(list_odd_exp)} bytes')\n\n----------\n104 bytes\n41880 bytes<\/pre><\/div>\n\n\n\n<p>The generator object in the case of generator expression took 104 bytes of memory, whereas the result in the case of list comprehension took 41880 bytes (almost 41 KB) of memory.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"heading-conclusion\"><a href=\"https:\/\/geekpython.in\/python-generators-with-yield-statement#heading-conclusion\"><\/a>Conclusion<\/h2>\n\n\n\n<p>A normal function with the&nbsp;<code>yield<\/code>&nbsp;keyword in its body defines the generator. This generator function returns a&nbsp;<strong>generator-iterator object<\/strong>&nbsp;that can be iterated over to generate a data sequence.<\/p>\n\n\n\n<p>It is said that generators are a Pythonic way to create iterators, and iterators use a strategy known as&nbsp;<strong>lazy evaluation<\/strong>, which means that they only return values when the caller requests them.<\/p>\n\n\n\n<p>Generators come in handy when we need to compute large amounts of data without storing them in memory.<\/p>\n\n\n\n<p>The quickest way to create a generator function in a few lines of code is to use a&nbsp;<strong>generator expression<\/strong>&nbsp;or generator comprehension (similar to list comprehension).<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>\ud83c\udfc6<strong>Other articles you might be interested in if you liked this one<\/strong><\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/context-managers-and-python-with-statement\" rel=\"noreferrer noopener\">What are context manager and the with statement in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/init-vs-new\" rel=\"noreferrer noopener\">What are __init__ and __new__ methods in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/init-and-call-method\" rel=\"noreferrer noopener\">What are __init__ and __call__ methods in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/seek-and-tell-in-python\" rel=\"noreferrer noopener\">What is the difference between seek() and tell() in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/tempfile-in-python\" rel=\"noreferrer noopener\">Generate temporary files and directories using tempfile module in Python<\/a>.<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/displaying-images-on-the-frontend-using-fastapi\" rel=\"noreferrer noopener\">How to display images on the frontend using FastAPI in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/match-case-in-python\" rel=\"noreferrer noopener\">How to use match case statements for pattern matching in Python<\/a>?<\/p>\n\n\n\n<p>\u2705<a target=\"_blank\" href=\"https:\/\/geekpython.in\/argparse-in-python\" rel=\"noreferrer noopener\">Build a command line interface using argparse in Python<\/a>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>That&#8217;s all for now<\/strong><\/p>\n\n\n\n<p><strong>Keep Coding\u270c\u270c<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Generators are defined by a function that generates values on the fly and can be iterated in the same way that we iterate strings, lists, and tuples in Python using the&nbsp;&#8220;for&#8221;&nbsp;loop. When the body of a normal function contains the&nbsp;yield&nbsp;keyword instead of the&nbsp;return&nbsp;keyword, it is said to be a&nbsp;generator function. In this article, we&#8217;ll look [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1129,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ocean_post_layout":"","ocean_both_sidebars_style":"","ocean_both_sidebars_content_width":0,"ocean_both_sidebars_sidebars_width":0,"ocean_sidebar":"0","ocean_second_sidebar":"0","ocean_disable_margins":"enable","ocean_add_body_class":"","ocean_shortcode_before_top_bar":"","ocean_shortcode_after_top_bar":"","ocean_shortcode_before_header":"","ocean_shortcode_after_header":"","ocean_has_shortcode":"","ocean_shortcode_after_title":"","ocean_shortcode_before_footer_widgets":"","ocean_shortcode_after_footer_widgets":"","ocean_shortcode_before_footer_bottom":"","ocean_shortcode_after_footer_bottom":"","ocean_display_top_bar":"default","ocean_display_header":"default","ocean_header_style":"","ocean_center_header_left_menu":"0","ocean_custom_header_template":"0","ocean_custom_logo":0,"ocean_custom_retina_logo":0,"ocean_custom_logo_max_width":0,"ocean_custom_logo_tablet_max_width":0,"ocean_custom_logo_mobile_max_width":0,"ocean_custom_logo_max_height":0,"ocean_custom_logo_tablet_max_height":0,"ocean_custom_logo_mobile_max_height":0,"ocean_header_custom_menu":"0","ocean_menu_typo_font_family":"0","ocean_menu_typo_font_subset":"","ocean_menu_typo_font_size":0,"ocean_menu_typo_font_size_tablet":0,"ocean_menu_typo_font_size_mobile":0,"ocean_menu_typo_font_size_unit":"px","ocean_menu_typo_font_weight":"","ocean_menu_typo_font_weight_tablet":"","ocean_menu_typo_font_weight_mobile":"","ocean_menu_typo_transform":"","ocean_menu_typo_transform_tablet":"","ocean_menu_typo_transform_mobile":"","ocean_menu_typo_line_height":0,"ocean_menu_typo_line_height_tablet":0,"ocean_menu_typo_line_height_mobile":0,"ocean_menu_typo_line_height_unit":"","ocean_menu_typo_spacing":0,"ocean_menu_typo_spacing_tablet":0,"ocean_menu_typo_spacing_mobile":0,"ocean_menu_typo_spacing_unit":"","ocean_menu_link_color":"","ocean_menu_link_color_hover":"","ocean_menu_link_color_active":"","ocean_menu_link_background":"","ocean_menu_link_hover_background":"","ocean_menu_link_active_background":"","ocean_menu_social_links_bg":"","ocean_menu_social_hover_links_bg":"","ocean_menu_social_links_color":"","ocean_menu_social_hover_links_color":"","ocean_disable_title":"default","ocean_disable_heading":"default","ocean_post_title":"","ocean_post_subheading":"","ocean_post_title_style":"","ocean_post_title_background_color":"","ocean_post_title_background":0,"ocean_post_title_bg_image_position":"","ocean_post_title_bg_image_attachment":"","ocean_post_title_bg_image_repeat":"","ocean_post_title_bg_image_size":"","ocean_post_title_height":0,"ocean_post_title_bg_overlay":0.5,"ocean_post_title_bg_overlay_color":"","ocean_disable_breadcrumbs":"default","ocean_breadcrumbs_color":"","ocean_breadcrumbs_separator_color":"","ocean_breadcrumbs_links_color":"","ocean_breadcrumbs_links_hover_color":"","ocean_display_footer_widgets":"default","ocean_display_footer_bottom":"default","ocean_custom_footer_template":"0","ocean_post_oembed":"","ocean_post_self_hosted_media":"","ocean_post_video_embed":"","ocean_link_format":"","ocean_link_format_target":"self","ocean_quote_format":"","ocean_quote_format_link":"post","ocean_gallery_link_images":"off","ocean_gallery_id":[],"footnotes":""},"categories":[2],"tags":[63,12,31],"class_list":["post-1127","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-generator","tag-python","tag-python3","entry","has-media"],"_links":{"self":[{"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/posts\/1127","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/comments?post=1127"}],"version-history":[{"count":5,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/posts\/1127\/revisions"}],"predecessor-version":[{"id":1666,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/posts\/1127\/revisions\/1666"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/media\/1129"}],"wp:attachment":[{"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/media?parent=1127"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/categories?post=1127"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/geekpython.in\/wp-json\/wp\/v2\/tags?post=1127"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}