{"id":316,"date":"2024-01-28T10:48:08","date_gmt":"2024-01-28T10:48:08","guid":{"rendered":"https:\/\/learnpython.elegantwallp.com\/?p=316"},"modified":"2024-01-28T10:48:09","modified_gmt":"2024-01-28T10:48:09","slug":"python-multiprocessing","status":"publish","type":"post","link":"https:\/\/learnpython.elegantwallp.com\/2024\/01\/28\/python-multiprocessing\/","title":{"rendered":"Python Multiprocessing"},"content":{"rendered":"\n

Summary<\/strong>: in this tutorial, you\u2019ll learn how to run code in parallel using the Python multiprocessing module.<\/p>\n\n\n\n

Introduction to the Python multiprocessing<\/h2>\n\n\n\n

Generally, programs deal with two types of tasks:<\/p>\n\n\n\n

    \n
  1. I\/O-bound tasks: if a task does a lot of input\/output operations, it\u2019s called an I\/O-bound task. Typical examples of I\/O-bound tasks are reading from files, writing to files, connecting to databases, and making a network request. For I\/O-bound tasks, you can use\u00a0multithreading\u00a0to speed them up.<\/li>\n\n\n\n
  2. CPU-bound tasks: when a task does a lot of operations using a CPU, it\u2019s called a CPU-bound task. For example, number calculation, image resizing, and video streaming are CPU-bound tasks. To speed up the program with lots of CPU-bound tasks, you use multiprocessing.<\/li>\n<\/ol>\n\n\n\n

    Multiprocessing allows two or more processors to simultaneously process two or more different parts of a program.<\/p>\n\n\n\n

    In Python, you use the multiprocessing<\/code> module to implement multiprocessing.<\/p>\n\n\n\n

    Python multiprocessing example<\/h2>\n\n\n\n

    See the following program:<\/p>\n\n\n\n

    import time def task(): result = 0 for _ in range(10**8): result += 1 return result if __name__ == '__main__': start = time.perf_counter() task() task() finish = time.perf_counter() print(f'It took {finish-start:.2f} second(s) to finish')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Output:<\/p>\n\n\n\n
    It took 5.55 second(s) to finish<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    How it works.<\/p>\n\n\n\n
    First, define the\u00a0task()<\/code>\u00a0function is a CPU-bound task because it performs a heavy computation by executing a loop for 100 million iterations and incrementing a variable\u00a0result<\/code>:<\/p>\n\n\n\n
    def task(): result = 0 for _ in range(10**8): result += 1 return result<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Second, call the\u00a0task()<\/code>\u00a0functions twice and record the processing time:<\/p>\n\n\n\n
    if __name__ == '__main__': start = time.perf_counter() task() task() finish = time.perf_counter() print(f'It took {finish-start: .2f} second(s) to finish')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    On our computer, it took 5.55 seconds to complete.<\/p>\n\n\n\n
    Using multiprocessing module<\/h3>\n\n\n\n
    The following program uses the multiprocessing module but takes less time:<\/p>\n\n\n\n
    import time import multiprocessing def task() -> int: result = 0 for _ in range(10**8): result += 1 return result if __name__ == '__main__': start = time.perf_counter() p1 = multiprocessing.Process(target=task) p2 = multiprocessing.Process(target=task) p1.start() p2.start() p1.join() p2.join() finish = time.perf_counter() print(f'It took {finish-start:.2f} second(s) to finish')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Output:<\/p>\n\n\n\n
    It took 3.43 second(s) to finish<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    How it works.<\/p>\n\n\n\n
    First, import the multiprocessing module:<\/p>\n\n\n\n
    import multiprocessing<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Second, create two processes and pass the task function to each:<\/p>\n\n\n\n
    p1 = multiprocessing.Process(target=task) p2 = multiprocessing.Process(target=task)<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Note that the Process()<\/code> constructor returns a new Process<\/code> object.<\/p>\n\n\n\n
    Third, call the\u00a0start()<\/code>\u00a0method of the\u00a0Process<\/code>\u00a0objects to start the process:<\/p>\n\n\n\n
    p1.start() p2.start()<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Finally, wait for the processes to complete by calling the\u00a0join()<\/code>\u00a0method:<\/p>\n\n\n\n
    p1.join() p2.join()<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Python multiprocessing practical example<\/h2>\n\n\n\n
    We\u2019ll use the multiprocessing module to resize the high-resolution images.<\/p>\n\n\n\n
    First, install the\u00a0Pillow<\/code>\u00a0library for image processing:<\/p>\n\n\n\n
    pip install Pillow<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Second, develop a program that creates thumbnails of the pictures in the\u00a0images<\/code>\u00a0folder and save them to the\u00a0thumbs<\/code>\u00a0folder:<\/p>\n\n\n\n
    import time import os from PIL import Image, ImageFilter filenames = [ 'images\/1.jpg', 'images\/2.jpg', 'images\/3.jpg', 'images\/4.jpg', 'images\/5.jpg', ] def create_thumbnail(filename, size=(50,50), thumb_dir ='thumbs'): # open the image<\/em> img = Image.open(filename) # apply the gaussian blur filter<\/em> img = img.filter(ImageFilter.GaussianBlur()) # create a thumbnail<\/em> img.thumbnail(size) # save the image<\/em> img.save(f'{thumb_dir}\/{os.path.basename(filename)}') # display a message<\/em> print(f'{filename} was processed...') if __name__ == '__main__': start = time.perf_counter() for filename in filenames: create_thumbnail(filename) finish = time.perf_counter() print(f'It took {finish-start:.2f} second(s) to finish')<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    On our computer, it took about 4.06 seconds to complete:<\/p>\n\n\n\n
    images\/1.jpg was processed... images\/2.jpg was processed... images\/3.jpg was processed... images\/4.jpg was processed... images\/5.jpg was processed... It took 4.06 second(s) to finish<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Third, modify the program to use multiprocessing. Each process will create a thumbnail for a picture:<\/p>\n\n\n\n
    import time import os from PIL import Image, ImageFilter import multiprocessing filenames = [ 'images\/1.jpg', 'images\/2.jpg', 'images\/3.jpg', 'images\/4.jpg', 'images\/5.jpg', ] def create_thumbnail(filename, size=(50,50), thumb_dir ='thumbs'): # open the image<\/em> img = Image.open(filename) # apply the gaussian blur filter<\/em> img = img.filter(ImageFilter.GaussianBlur()) # create a thumbnail<\/em> img.thumbnail(size) # save the image<\/em> img.save(f'{thumb_dir}\/{os.path.basename(filename)}') # display a message<\/em> print(f'{filename} was processed...') def main(): start = time.perf_counter() # create processes<\/em> processes = [multiprocessing.Process(target=create_thumbnail, args=[filename]) for filename in filenames] # start the processes<\/em> for process in processes: process.start() # wait for completion<\/em> for process in processes: process.join() finish = time.perf_counter() print(f'It took {finish-start:.2f} second(s) to finish') if __name__ == '__main__': main()<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    Output:<\/p>\n\n\n\n
    images\/5.jpg was processed... images\/4.jpg was processed... images\/1.jpg was processed... images\/3.jpg was processed... images\/2.jpg was processed... It took 2.92 second(s) to finish<\/code>Code language: Python (python)<\/small><\/code><\/pre>\n\n\n\n
    In this case, the output shows that the program processed the pictures faster.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Summary: in this tutorial, you\u2019ll learn how to run code in parallel using the Python multiprocessing module. Introduction to the Python multiprocessing Generally, programs deal with two types of tasks: Multiprocessing allows two or more processors to simultaneously process two or more different parts of a program. In Python, you use the multiprocessing module to implement multiprocessing. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[],"class_list":["post-316","post","type-post","status-publish","format-standard","hentry","category-1-python-concurrency"],"_links":{"self":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/316","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/comments?post=316"}],"version-history":[{"count":1,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/316\/revisions"}],"predecessor-version":[{"id":317,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/posts\/316\/revisions\/317"}],"wp:attachment":[{"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/media?parent=316"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/categories?post=316"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learnpython.elegantwallp.com\/wp-json\/wp\/v2\/tags?post=316"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}