{"id":319,"date":"2024-01-28T10:49:18","date_gmt":"2024-01-28T10:49:18","guid":{"rendered":"https:\/\/learnpython.elegantwallp.com\/?p=319"},"modified":"2024-01-28T10:49:19","modified_gmt":"2024-01-28T10:49:19","slug":"python-processpoolexecutor","status":"publish","type":"post","link":"https:\/\/learnpython.elegantwallp.com\/2024\/01\/28\/python-processpoolexecutor\/","title":{"rendered":"Python ProcessPoolExecutor"},"content":{"rendered":"\n

Summary<\/strong>: in this tutorial, you\u2019ll learn how to use the Python ProcessPoolExecutor<\/code> to create and manage a process pool effectively.<\/p>\n\n\n\n

Introduction to the Python ProcessPoolExecutor class<\/h2>\n\n\n\n

In the previous tutorial, you learned how to run code in parallel by creating processes manually using the\u00a0Process<\/code>\u00a0class from the\u00a0multiprocessing<\/code>\u00a0module. However, manually creating processes is not efficient.<\/p>\n\n\n\n

To manage the processes more efficiently, you can use a process pool. Like a\u00a0thread pool, a process pool is a pattern for managing processes automatically.<\/p>\n\n\n\n

The ProcessPoolExecutor<\/code> class from the concurrent.futures<\/code> module allows you to create and manage a process pool.<\/p>\n\n\n\n

For example, the ProcessPoolExecutor<\/code> class uses the number of CPU cores for creating an optimized number of processes to create.<\/p>\n\n\n\n

The ProcessPoolExecutor<\/code> extends the Executor<\/code> class that has three methods:<\/p>\n\n\n\n

    \n
  • submit()<\/code> \u2013 dispatch a function to be executed by the process and return a Future object.<\/li>\n\n\n\n
  • map()<\/code> \u2013 call a function to an iterable of elements.<\/li>\n\n\n\n
  • shutdown()<\/code> \u2013 shut down the executor.<\/li>\n<\/ul>\n\n\n\n

    To release the resources held by the executor, you need to call the\u00a0shutdown()<\/code>\u00a0method explicitly. To shut down the executor automatically, you can use a\u00a0context manager.<\/p>\n\n\n\n

    The Future<\/code> object represents an eventful result of an asynchronous operation. It has two main methods for getting the result:<\/p>\n\n\n\n

      \n
    • result()<\/code> \u2013 return the result from the asynchronous operation.<\/li>\n\n\n\n
    • exception()<\/code> \u2013 return an exception that occurred while running the asynchronous operation.<\/li>\n<\/ul>\n\n\n\n

      Python ProcessPoolExecutor example<\/h2>\n\n\n\n

      The following program uses a process pool to create thumbnails for pictures in the\u00a0images<\/code>\u00a0folder and save them to the\u00a0thumbs<\/code>\u00a0folder.<\/p>\n\n\n\n

      import time import os from PIL import Image, ImageFilter from concurrent.futures import ProcessPoolExecutor filenames = [ 'images\/1.jpg', 'images\/2.jpg', 'images\/3.jpg', 'images\/4.jpg', 'images\/5.jpg', ] def create_thumbnail(filename, size=(50,50), thumb_dir ='thumbs'): # open the image<\/em> img = Image.open(filename) # apply the gaussian blur filter<\/em> img = img.filter(ImageFilter.GaussianBlur()) # create a thumbnail<\/em> img.thumbnail(size) # save the image<\/em> img.save(f'{thumb_dir}\/{os.path.basename(filename)}') # display a message<\/em> print(f'{filename} was processed...') def main(): start = time.perf_counter() with ProcessPoolExecutor() as executor: executor.map(create_thumbnail, filenames) finish = time.perf_counter() print(f'It took {finish-start: .2f} second(s) to finish') if __name__ == '__main__': main()<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Output:<\/p>\n\n\n\n
      images\/5.jpg was processed... images\/4.jpg was processed... images\/3.jpg was processed... images\/2.jpg was processed... images\/1.jpg was processed... It took 0.79 second(s) to finish<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Note that to run the program, you need to install the Pillow<\/code> which is a popular library for image processing by running the pip command pip install Pillow<\/code>.<\/p>\n\n\n\n
      How it works.<\/p>\n\n\n\n
      First, declare a list of files for creating thumbnails:<\/p>\n\n\n\n
      filenames = [ 'images\/1.jpg', 'images\/2.jpg', 'images\/3.jpg', 'images\/4.jpg', 'images\/5.jpg', ]<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Second, define a function that creates a thumbnail from an image file and saves the output to the thumbnail folder:<\/p>\n\n\n\n
      def create_thumbnail(filename, size=(50,50), thumb_dir ='thumbs'): # open the image<\/em> img = Image.open(filename) # apply the gaussian blur filter<\/em> img = img.filter(ImageFilter.GaussianBlur()) # create a thumbnail<\/em> img.thumbnail(size) # save the image<\/em> img.save(f'{thumb_dir}\/{os.path.basename(filename)}') # display a message<\/em> print(f'{filename} was processed...')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Third, create a process pool and call the\u00a0create_thumbnail()<\/code>\u00a0function for each picture specified in the filenames:<\/p>\n\n\n\n
      with ProcessPoolExecutor() as executor: executor.map(create_thumbnail, filenames)<\/code><\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
      Summary: in this tutorial, you\u2019ll learn how to use the Python ProcessPoolExecutor to create and manage a process pool effectively. Introduction to the Python ProcessPoolExecutor class In the previous tutorial, you learned how to run code in parallel by creating processes manually using the\u00a0Process\u00a0class from the\u00a0multiprocessing\u00a0module. However, manually creating processes is not efficient. To manage the processes […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[],"class_list":["post-319","post","type-post","status-publish","format-standard","hentry","category-1-python-concurrency"],"_links":{"self":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/319","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/comments?post=319"}],"version-history":[{"count":1,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/319\/revisions"}],"predecessor-version":[{"id":320,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/319\/revisions\/320"}],"wp:attachment":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/media?parent=319"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/categories?post=319"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/tags?post=319"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}