![\"Introduction](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/c_introduction_memory_profiling_python_1.png\")
\n Image by Author<\/font><\/center>
\n <\/p>\n
Profiling Python code is helpful to understand how the code works and identify opportunities for optimization. You\u2019ve probably profiled your Python scripts for time-related metrics<\/a>\u2014measuring execution times of specific sections of code.\u00a0<\/p> \t\t\t\t But profiling for memory\u2014to understand memory allocation and deallocation during execution\u2014is just as important. Because memory profiling can help identify memory leaks, resource utilization, and potential issues with scaling.\u00a0<\/p> In this tutorial, we\u2019ll explore profiling Python code for memory usage using the Python package memory-profiler<\/a>.<\/p>\n <\/p>\n <\/p>\n Let\u2019s start by installing the memory-profiler Python package using pip:<\/p>\n <\/p>\n \n Note<\/strong>: Install memory-profiler in a dedicated virtual environment<\/a> for the project instead of in your global environment. We'll also be using the plotting capabilities available in memory-profiler to plot the memory usage, which requires matplotlib<\/a>. So make sure you also have matplotlib installed in the project\u2019s virtual environment.<\/font> <\/p><\/blockquote>\n <\/p>\n <\/p>\n Let's create a Python script (say main.py) with a function <\/p>\n Running the script should give you a similar output:\u00a0<\/p>\n As seen in the output, we\u2019re able to see the memory used, the increment with each subsequent string creation and the string deletion step freeing up some of the used memory.<\/p>\n <\/p>\n <\/p>\n Instead of running the Python script as shown above, you can also run the <\/p>\n When you run this command, you should also be able to see a .dat file with the memory usage data. You\u2019ll have one .dat file every time you run the <\/p>\n <\/p>\n Sometimes it's easier to analyze memory usage from a plot instead of looking at numbers. Remember we discussed matplotlib being a required dependency to use the plotting capabilities.\u00a0<\/p>\n You can use the <\/p>\n By default, You can see the timestamps mentioned in the plot as well.<\/p>\n <\/p>\n <\/p>\n Alternatively, you can log the memory usage statistics to a preferred log file in the working directory. Here, we create a file handler <\/p>\n When you now run the script, you should be able to see the mem_profile.log file in your working directory with the following contents:<\/p>\n <\/p>\n <\/p>\n You can also use the The Here, we\u2019d like to find the memory usage of the <\/p>\n Here\u2019s the corresponding output:<\/p>\n <\/p>\n You can also adjust the sampling interval based on how often you want the memory usage to be captured. As an example, we set the interval to 0.01 s; meaning we\u2019ll now get a more granular view of the memory utilized.<\/p>\n <\/p>\n You should be able to see a similar output:<\/p>\n <\/p>\n <\/p>\n In this tutorial, we learned how to get started with profiling Python scripts for memory usage.<\/p>\n Specifically, we learned how to do this using the memory-profiler package. We used the ![\"BrightData](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s1-brightdata-2512.jpg\")
\nThe most reliable Web Scraping API<\/p>\n<\/a><\/div><\/div>\n![\"NWU](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s-nwu-2605.jpg\")
\nAnalytics, AI, and leadership skills.\t\t<\/p>\n<\/a><\/div><\/div>\nInstalling the memory-profiler Python Package<\/h1>\n
pip3 install memory-profiler<\/code><\/pre>\n<\/div>\nProfiling Memory Usage with the @profile Decorator<\/h1>\n
process_strs<\/code>:<\/p>\n\n
str1<\/code> and str2<\/code> and concatenates them.\u00a0\nreps<\/code> controls the number of times the hardcoded strings are to be repeated to create str1<\/code> and str2<\/code>. And we give it a default value of 10**6 which will be used if the function called does not specify the value of reps<\/code>.\nstr2<\/code>.\u00a0\nstr3<\/code>.\n<\/ul>\n# main.py\r\n\r\nfrom memory_profiler import profile\r\n\r\n@profile\r\ndef process_strs(reps=10**6):\r\n\tstr1 = 'python'*reps\r\n\tstr2 = 'programmer'*reps\r\n\tstr3 = str1 + str2\r\n\tdel str2\r\n\treturn str3\r\n\r\nprocess_strs(reps=10**7)<\/code><\/pre>\n<\/div>\n
\n![\"Introduction](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/c_introduction_memory_profiling_python_5.png\")
<\/center>
\n <\/p>\nRunning the mprof command\u00a0<\/h2>\n
mprof<\/code> command like so:<\/p>\nmprof run --python main.py<\/code><\/pre>\n<\/div>\nmprof<\/code> command\u2014identified by the timestamp.<\/p>\n
\n![\"Introduction](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/c_introduction_memory_profiling_python_3.png\")
<\/center><\/p>\nPlotting Memory Usage\u00a0<\/h2>\n
mprof plot<\/code> command to plot the data in the .dat file and save it to an image file (here output.png):<\/p>\nmprof plot -o output.png<\/code><\/pre>\n<\/div>\nmprof plot<\/code> used the data from the most recent run of the mprof<\/code> command.<\/p>\n
\n![\"Introduction](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/c_introduction_memory_profiling_python_2.png\")
<\/center>
\n <\/p>\nLogging Memory Usage Profile to a Log File<\/h2>\n
mem_logs<\/code> to the log file, and set the stream<\/code> argument in the @profile<\/code> decorator to the file handler:<\/p>\n# main.py\r\n\r\nfrom memory_profiler import profile\r\n\r\nmem_logs = open('mem_profile.log','a')\r\n\r\n@profile(stream=mem_logs)\r\ndef process_strs(reps=10**6):\r\n\tstr1 = 'python'*reps\r\n\tstr2 = 'programmer'*reps\r\n\tstr3 = str1 + str2\r\n\tdel str2\r\n\treturn str3\r\n\r\nprocess_strs(reps=10**7)<\/code><\/pre>\n<\/div>\n
\n![\"Introduction](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/c_introduction_memory_profiling_python_4.png\")
<\/center><\/p>\nProfiling Using the memory_usage Function\u00a0<\/h1>\n
memory_usage()<\/code> function to understand the resources required for a specific function to execute\u2014sampled at regular time intervals.<\/p>\nmemory_usage<\/code> function takes in the function to profile, positional and keyword arguments as a tuple.<\/p>\nprocess_strs<\/code> function with the keyword argument reps<\/code> set to 10**7. We also set the sampling interval to 0.1 s:<\/p>\n# main.py\r\n\r\nfrom memory_profiler import memory_usage\r\n\r\ndef process_strs(reps=10**6):\r\n\tstr1 = 'python'*reps\r\n\tstr2 = 'programmer'*reps\r\n\tstr3 = str1 + str2\r\n\tdel str2\r\n\treturn str3\r\n\r\nprocess_strs(reps=10**7)\r\n\r\nmem_used = memory_usage((process_strs,(),{'reps':10**7}),interval=0.1)\r\nprint(mem_used)<\/code><\/pre>\n<\/div>\nOutput >>>\r\n[21.21875, 21.71875, 147.34375, 277.84375, 173.93359375]<\/code><\/pre>\n<\/div>\n# main.py\r\n\r\nfrom memory_profiler import memory_usage\r\n\r\ndef process_strs(reps=10**6):\r\n\tstr1 = 'python'*reps\r\n\tstr2 = 'programmer'*reps\r\n\tstr3 = str1 + str2\r\n\tdel str2\r\n\treturn str3\r\n\r\nprocess_strs(reps=10**7)\r\n\r\nmem_used = memory_usage((process_strs,(),{'reps':10**7}),interval=0.01)\r\nprint(mem_used)<\/code><\/pre>\n<\/div>\nOutput >>>\r\n[21.40234375, 21.90234375, 33.90234375, 46.40234375, 59.77734375, 72.90234375, 85.65234375, 98.40234375, 112.65234375, 127.02734375, 141.27734375, 155.65234375, 169.77734375, 184.02734375, 198.27734375, 212.52734375, 226.65234375, 240.40234375, 253.77734375, 266.52734375, 279.90234375, 293.65234375, 307.40234375, 321.27734375, 227.71875, 174.1171875]<\/code><\/pre>\n<\/div>\nConclusion<\/h1>\n
@profile<\/code> decorator and the memory_usage()<\/code> function to get the memory usage of a sample Python script. We also learned how to use the capabilities such as plotting the memory usage and capturing the stats in a log file.<\/p>\n