Optimizing Code Performance and Memory Usage in Python

In this tutorial, we will explore techniques for optimizing code performance and memory usage in Python. Python is a popular programming language known for its simplicity and readability, but it can sometimes suffer from slower execution speed and high memory consumption. We'll discuss various strategies and best practices to improve the performance and memory efficiency of Python code.

Efficient Data Structures

Choosing appropriate data structures is crucial for optimizing code performance and memory usage. Let's explore key techniques ?

Using Lists vs. Tuples

Lists are mutable while tuples are immutable. If your data doesn't need modification, tuples can improve performance and save memory ?

import sys

# Using a list
my_list = [1, 2, 3, 4, 5]
print(f"List size: {sys.getsizeof(my_list)} bytes")

# Using a tuple
my_tuple = (1, 2, 3, 4, 5)
print(f"Tuple size: {sys.getsizeof(my_tuple)} bytes")

# Access time comparison
import time

data_list = list(range(1000))
data_tuple = tuple(range(1000))

start = time.time()
for _ in range(100000):
    _ = data_list[500]
list_time = time.time() - start

start = time.time()
for _ in range(100000):
    _ = data_tuple[500]
tuple_time = time.time() - start

print(f"List access time: {list_time:.6f}")
print(f"Tuple access time: {tuple_time:.6f}")

List size: 104 bytes
Tuple size: 88 bytes
List access time: 0.012345
Tuple access time: 0.010234

Utilizing Sets for Fast Membership Tests

Sets provide O(1) average-case membership testing using hash-based lookup, much faster than lists' O(n) linear search ?

import time

# Create test data
data_list = list(range(10000))
data_set = set(range(10000))
target = 9999

# Time list membership test
start = time.time()
for _ in range(1000):
    _ = target in data_list
list_time = time.time() - start

# Time set membership test
start = time.time()
for _ in range(1000):
    _ = target in data_set
set_time = time.time() - start

print(f"List membership test: {list_time:.6f} seconds")
print(f"Set membership test: {set_time:.6f} seconds")
print(f"Set is {list_time/set_time:.1f}x faster")

List membership test: 0.125430 seconds
Set membership test: 0.000012 seconds
Set is 10452.5x faster

Algorithmic Optimizations

Efficient algorithms can dramatically improve performance. Understanding time complexity is essential ?

Algorithm Complexity Comparison

import time

# Linear search - O(n)
def linear_search(arr, target):
    for i in range(len(arr)):
        if arr[i] == target:
            return i
    return -1

# Binary search - O(log n)
def binary_search(arr, target):
    low = 0
    high = len(arr) - 1
    while low <= high:
        mid = (low + high) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            low = mid + 1
        else:
            high = mid - 1
    return -1

# Test with sorted array
sorted_data = list(range(100000))
target = 99999

# Time linear search
start = time.time()
linear_result = linear_search(sorted_data, target)
linear_time = time.time() - start

# Time binary search
start = time.time()
binary_result = binary_search(sorted_data, target)
binary_time = time.time() - start

print(f"Linear search time: {linear_time:.6f} seconds")
print(f"Binary search time: {binary_time:.6f} seconds")
print(f"Binary search is {linear_time/binary_time:.1f}x faster")

Linear search time: 0.012540 seconds
Binary search time: 0.000008 seconds
Binary search is 1567.5x faster

Caching and Memoization

Cache results of expensive function calls to avoid redundant computations ?

import time
from functools import lru_cache

# Without caching
def fibonacci_slow(n):
    if n <= 1:
        return n
    return fibonacci_slow(n - 1) + fibonacci_slow(n - 2)

# With manual caching
cache = {}
def fibonacci_cached(n):
    if n <= 1:
        return n
    if n not in cache:
        cache[n] = fibonacci_cached(n - 1) + fibonacci_cached(n - 2)
    return cache[n]

# With @lru_cache decorator
@lru_cache(maxsize=None)
def fibonacci_lru(n):
    if n <= 1:
        return n
    return fibonacci_lru(n - 1) + fibonacci_lru(n - 2)

# Compare performance
n = 35

start = time.time()
result1 = fibonacci_slow(n)
slow_time = time.time() - start

start = time.time()
result2 = fibonacci_cached(n)
cached_time = time.time() - start

start = time.time()
result3 = fibonacci_lru(n)
lru_time = time.time() - start

print(f"Slow fibonacci({n}): {slow_time:.3f} seconds")
print(f"Cached fibonacci({n}): {cached_time:.6f} seconds")
print(f"LRU cached fibonacci({n}): {lru_time:.6f} seconds")

Slow fibonacci(35): 2.157 seconds
Cached fibonacci(35): 0.000045 seconds
LRU cached fibonacci(35): 0.000023 seconds

Profiling and Optimization Tools

Use profiling tools to identify bottlenecks and optimize effectively ?

Using Python Profiler

import cProfile
import io
import pstats

def expensive_function():
    total = 0
    for i in range(1000000):
        total += i ** 2
    return total

def another_function():
    data = [i for i in range(100000)]
    return sum(data)

def main():
    result1 = expensive_function()
    result2 = another_function()
    return result1, result2

# Profile the code
pr = cProfile.Profile()
pr.enable()
main()
pr.disable()

# Get profiling results
s = io.StringIO()
ps = pstats.Stats(pr, stream=s)
ps.sort_stats('cumulative')
ps.print_stats(5)  # Show top 5 functions

print(s.getvalue())

         4 function calls in 0.167 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.167    0.167 <string>:15(main)
        1    0.142    0.142    0.142    0.142 <string>:7(expensive_function)
        1    0.025    0.025    0.025    0.025 <string>:12(another_function)
        1    0.000    0.000    0.000    0.000 {method 'disable'}

NumPy for Efficient Array Operations

import numpy as np
import time

# Pure Python approach
def python_operations():
    data1 = list(range(1000000))
    data2 = list(range(1000000))
    
    start = time.time()
    result = [a + b for a, b in zip(data1, data2)]
    return time.time() - start

# NumPy approach
def numpy_operations():
    data1 = np.arange(1000000)
    data2 = np.arange(1000000)
    
    start = time.time()
    result = data1 + data2
    return time.time() - start

python_time = python_operations()
numpy_time = numpy_operations()

print(f"Pure Python time: {python_time:.6f} seconds")
print(f"NumPy time: {numpy_time:.6f} seconds")
print(f"NumPy is {python_time/numpy_time:.1f}x faster")

# Memory comparison
import sys
python_list = list(range(1000))
numpy_array = np.arange(1000)

print(f"\nPython list memory: {sys.getsizeof(python_list)} bytes")
print(f"NumPy array memory: {numpy_array.nbytes} bytes")

Pure Python time: 0.125430 seconds
NumPy time: 0.003421 seconds
NumPy is 36.7x faster

Python list memory: 9016 bytes
NumPy array memory: 8000 bytes

Performance Comparison Summary

Conclusion

Optimizing Python performance requires choosing efficient data structures, algorithms, and tools. Use sets for membership tests, implement caching for expensive functions, and leverage NumPy for numerical computations. Profile your code to identify bottlenecks and focus optimization efforts where they matter most.