{"id":303,"date":"2024-01-28T10:25:32","date_gmt":"2024-01-28T10:25:32","guid":{"rendered":"https:\/\/learnpython.elegantwallp.com\/?p=303"},"modified":"2024-01-28T10:25:33","modified_gmt":"2024-01-28T10:25:33","slug":"python-threadpoolexecutor","status":"publish","type":"post","link":"https:\/\/learnpython.elegantwallp.com\/2024\/01\/28\/python-threadpoolexecutor\/","title":{"rendered":"Python ThreadPoolExecutor"},"content":{"rendered":"\n

Summary<\/strong>: in this tutorial, you\u2019ll learn how to use the Python ThreadPoolExecutor<\/code> to develop multi-threaded programs.<\/p>\n\n\n\n

Introduction to the Python ThreadPoolExecutor class<\/h2>\n\n\n\n

In the multithreading tutorial, you learned how to manage multiple threads in a program using the\u00a0Thread<\/code>\u00a0class of the\u00a0threading<\/code>\u00a0module. The\u00a0Thread<\/code>\u00a0class is useful when you want to create threads manually.<\/p>\n\n\n\n

However, manually managing threads is not efficient because creating and destroying many threads frequently are very expensive in terms of computational costs.<\/p>\n\n\n\n

Instead of doing so, you may want to reuse the threads if you expect to run many ad-hoc tasks in the program. A thread pool allows you to achieve this.<\/p>\n\n\n\n

Thread pool<\/h3>\n\n\n\n

A thread pool is a pattern for achieving concurrency of execution in a program. A thread pool allows you to automatically manage a pool of threads efficiently:<\/p>\n\n\n\n

\"Python<\/figure>\n\n\n\n

Each thread in the pool is called a worker thread or a worker. A thread pool allows you to reuse the worker threads once the tasks are completed. It also protects against unexpected failures such as\u00a0exceptions.<\/p>\n\n\n\n

Typically, a thread pool allows you to configure the number of worker threads and provides a specific naming convention for each worker thread.<\/p>\n\n\n\n

To create a thread pool, you use the ThreadPoolExecutor<\/code> class from the concurrent.futures<\/code> module.<\/p>\n\n\n\n

ThreadPoolExecutor<\/h3>\n\n\n\n

The\u00a0ThreadPoolExecutor<\/code>\u00a0class\u00a0extends\u00a0the\u00a0Executor<\/code>\u00a0class and returns a\u00a0Future<\/code>\u00a0object.<\/p>\n\n\n\n

Executor<\/h3>\n\n\n\n

The Executor<\/code> class has three methods to control the thread pool:<\/p>\n\n\n\n

    \n
  • submit()<\/code> \u2013 dispatch a function to be executed and return a Future<\/code> object. The submit()<\/code> method takes a function and executes it asynchronously.<\/li>\n\n\n\n
  • map()<\/code> \u2013 execute a function asynchronously for each element in an iterable.<\/li>\n\n\n\n
  • shutdown()<\/code> \u2013 shut down the executor.<\/li>\n<\/ul>\n\n\n\n

    When you create a new instance of the ThreadPoolExecutor<\/code> class, Python starts the Executor<\/code>.<\/p>\n\n\n\n

    Once completing working with the executor, you must explicitly call the\u00a0shutdown()<\/code>\u00a0method to release the resource held by the executor. To avoid calling the\u00a0shutdown()<\/code>\u00a0method explicitly, you can use the\u00a0context manager.<\/p>\n\n\n\n

    Future object<\/h3>\n\n\n\n

    A Future<\/code> is an object that represents the eventual result of an asynchronous operation. The Future class has two useful methods:<\/p>\n\n\n\n

      \n
    • result()<\/code> \u2013 return the result of an asynchronous operation.<\/li>\n\n\n\n
    • exception()<\/code> \u2013 return the exception of an asynchronous operation in case an exception occurs.<\/li>\n<\/ul>\n\n\n\n

      Python ThreadPoolExecutor examples<\/h2>\n\n\n\n

      The following program uses a single thread:<\/p>\n\n\n\n

      from time import sleep, perf_counter def task(id): print(f'Starting the task {id}...') sleep(1) return f'Done with task {id}' start = perf_counter() print(task(1)) print(task(2)) finish = perf_counter() print(f\"It took {finish-start} second(s) to finish.\")<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Output:<\/p>\n\n\n\n
      Starting the task 1... Done with task 1 Starting the task 2... Done with task 2 It took 2.0144479 second(s) to finish.<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      How it works.<\/p>\n\n\n\n
      First, define the\u00a0task()<\/code>\u00a0function that takes about one second to finish. The\u00a0task()<\/code>\u00a0function calls the\u00a0sleep()<\/code>\u00a0function to simulate a delay:<\/p>\n\n\n\n
      def task(id): print(f'Starting the task {id}...') sleep(1) return f'Done with task {id}'<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Second, call the\u00a0task()<\/code>\u00a0function twice and print out the result. Before and after calling the\u00a0task()<\/code>\u00a0function, we use the\u00a0perf_counter()<\/code>\u00a0to measure the start and finish time:<\/p>\n\n\n\n
      start = perf_counter() print(task(1)) print(task(2)) finish = perf_counter()<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Third, print out the time the program took to run:<\/p>\n\n\n\n
      print(f\"It took {finish-start} second(s) to finish.\")<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Because the task()<\/code> function takes one second, calling it twice will take about 2 seconds.<\/p>\n\n\n\n
      Using the submit() method example<\/h3>\n\n\n\n
      To run the\u00a0task()<\/code>\u00a0function concurrently, you can use the\u00a0ThreadPoolExecutor<\/code>\u00a0class:<\/p>\n\n\n\n
      from time import sleep, perf_counter from concurrent.futures import ThreadPoolExecutor def task(id): print(f'Starting the task {id}...') sleep(1) return f'Done with task {id}' start = perf_counter() with ThreadPoolExecutor() as executor: f1 = executor.submit(task, 1) f2 = executor.submit(task, 2) print(f1.result()) print(f2.result()) finish = perf_counter() print(f\"It took {finish-start} second(s) to finish.\")<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Output:<\/p>\n\n\n\n
      Starting the task 1... Starting the task 2... Done with task 1 Done with task 2 It took 1.0177214 second(s) to finish.<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      The output shows that the program took about 1 second to finish.<\/p>\n\n\n\n
      How it works (we\u2019ll focus on the thread pool part):<\/p>\n\n\n\n
      First, import the\u00a0ThreadPoolExecutor<\/code>\u00a0class from the\u00a0concurrent.futures<\/code>\u00a0module:<\/p>\n\n\n\n
      from concurrent.futures import ThreadPoolExecutor<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Second, create a thread pool using the\u00a0ThreadPoolExecutor<\/code>\u00a0using a context manager:<\/p>\n\n\n\n
      with ThreadPoolExecutor() as executor:<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Third, calling the\u00a0task()<\/code>\u00a0function twice by passing it to the\u00a0submit()<\/code>\u00a0method of the executor:<\/p>\n\n\n\n
      with ThreadPoolExecutor() as executor: f1 = executor.submit(task, 1) f2 = executor.submit(task, 2) print(f1.result()) print(f2.result()) <\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      The submit()<\/code> method returns a Future object. In this example, we have two Future objects f1<\/code> and f2<\/code>. To get the result from the Future object, we called its result()<\/code> method.<\/p>\n\n\n\n
      Using the map() method example<\/h3>\n\n\n\n
      The following program uses a\u00a0ThreadPoolExecutor<\/code>\u00a0class. However, instead of using the\u00a0submit()<\/code>\u00a0method, it uses the\u00a0map()<\/code>\u00a0method to execute a function:<\/p>\n\n\n\n
      from time import sleep, perf_counter from concurrent.futures import ThreadPoolExecutor def task(id): print(f'Starting the task {id}...') sleep(1) return f'Done with task {id}' start = perf_counter() with ThreadPoolExecutor() as executor: results = executor.map(task, [1,2]) for result in results: print(result) finish = perf_counter() print(f\"It took {finish-start} second(s) to finish.\")<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      How it works.<\/p>\n\n\n\n
      First, call the\u00a0map()<\/code>\u00a0method of the executor object to run the task function for each id in the list [1,2]. The\u00a0map()<\/code>\u00a0method returns an iterator that contains the result of the function calls.<\/p>\n\n\n\n
      results = executor.map(task, [1,2])<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Second, iterate over the results and print them out:<\/p>\n\n\n\n
      for result in results: print(result)<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      Python ThreadPoolExecutor practical example<\/h2>\n\n\n\n
      The following program downloads multiple images from Wikipedia using a thread pool:<\/p>\n\n\n\n
      from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen import time import os def download_image(url): image_data = None with urlopen(url) as f: image_data = f.read() if not image_data: raise Exception(f\"Error: could not download the image from {url}\") filename = os.path.basename(url) with open(filename, 'wb') as image_file: image_file.write(image_data) print(f'{filename} was downloaded...') start = time.perf_counter() urls = ['https:\/\/upload.wikimedia.org\/wikipedia\/commons\/9\/9d\/Python_bivittatus_1701.jpg', 'https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/48\/Python_Regius.jpg', 'https:\/\/upload.wikimedia.org\/wikipedia\/commons\/d\/d3\/Baby_carpet_python_caudal_luring.jpg', 'https:\/\/upload.wikimedia.org\/wikipedia\/commons\/f\/f0\/Rock_python_pratik.JPG', 'https:\/\/upload.wikimedia.org\/wikipedia\/commons\/0\/07\/Dulip_Wilpattu_Python1.jpg'] with ThreadPoolExecutor() as executor: executor.map(download_image, urls) finish = time.perf_counter() print(f'It took {finish-start} second(s) to finish.')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      How it works.<\/p>\n\n\n\n
      First, define a function\u00a0download_image()<\/code>\u00a0that downloads an image from an URL and saves it into a file:<\/p>\n\n\n\n
      def download_image(url): image_data = None with urlopen(url) as f: image_data = f.read() if not image_data: raise Exception(f\"Error: could not download the image from {url}\") filename = os.path.basename(url) with open(filename, 'wb') as image_file: image_file.write(image_data) print(f'{filename} was downloaded...')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
      The download_image()<\/code> function the urlopen()<\/code> function from the urllib.request<\/code> module to download an image from an URL.<\/p>\n\n\n\n
      Second, execute the\u00a0download_image()<\/code>\u00a0function using a thread pool by calling the\u00a0map()<\/code>\u00a0method of the\u00a0ThreadPoolExecutor<\/code>\u00a0object:<\/p>\n\n\n\n
      with ThreadPoolExecutor() as executor: executor.map(download_image, urls)<\/code><\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
      Summary: in this tutorial, you\u2019ll learn how to use the Python ThreadPoolExecutor to develop multi-threaded programs. Introduction to the Python ThreadPoolExecutor class In the multithreading tutorial, you learned how to manage multiple threads in a program using the\u00a0Thread\u00a0class of the\u00a0threading\u00a0module. The\u00a0Thread\u00a0class is useful when you want to create threads manually. However, manually managing threads is not efficient […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[],"class_list":["post-303","post","type-post","status-publish","format-standard","hentry","category-1-python-concurrency"],"_links":{"self":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/303","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/comments?post=303"}],"version-history":[{"count":1,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/303\/revisions"}],"predecessor-version":[{"id":304,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/303\/revisions\/304"}],"wp:attachment":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/media?parent=303"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/categories?post=303"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/tags?post=303"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}