python requests session

Python Requests Session: The Complete Guide

by

Python is widely regarded as one of the most beginner-friendly and versatile programming languages. Its ecosystem includes powerful libraries for almost every task, and one of the most popular for interacting with the web is Requests. While requests.get() and requests.post() are easy to use, they are stateless: each HTTP request is independent. A Python Requests Session allows developers to maintain persistent parameters across requests, such as cookies, headers, and authentication, while reusing TCP connections.

This not only makes code cleaner and easier to maintain but also improves performance.

Table of Contents

  1. Introduction to Python Requests Session
  2. Why Use a Requests Session?
  3. Creating a Requests Session
  4. Session Objects vs. Regular Requests
  5. Setting Headers and Authentication in Sessions
  6. Handling Cookies with Sessions
  7. Timeouts and Error Handling
  8. Advanced Session Features
  9. Performance Comparison: Sessions vs Stateless Requests
  10. Async Alternatives: httpx and aiohttp
  11. Real-World Examples Using Requests Sessions
  12. Common Troubleshooting Scenarios
  13. Best Practices for Python Requests Sessions
  14. Conclusion

Introduction to Python Requests Session

The Requests library is renowned for its simplicity and versatility. Whether you are retrieving web pages, consuming APIs, or scraping dynamic content, Requests simplifies HTTP communication.

However, using individual requests.get() calls for multiple requests introduces repetition, slows performance, and makes session management tedious. Python Requests Session solves these issues by:

  • Maintaining cookies automatically
  • Reusing headers, authentication, and query parameters
  • Reusing TCP connections for faster requests

This makes it the go-to solution for web scraping, API integrations, and any scenario that requires repeated HTTP requests.

Why Use a Requests Session?

Using a session has clear advantages:

  1. Persistent Cookies: Essential for login sessions, e-commerce sites, or dashboards.
  2. Shared Headers: Avoid repetitive code by storing headers for multiple requests.
  3. Connection Reuse: Sessions reuse TCP connections, reducing latency.
  4. Retries & Error Handling: Easily configure retry strategies.
  5. Proxy & SSL Management: Apply proxies or SSL settings globally.

Example Scenario: Logging into a website with multiple pages. Without a session, you’d need to manually handle cookies on each request. With a session, login cookies persist automatically.

Creating a Requests Session

Creating a session is simple:

import requests
# Create a session object
session = requests.Session()
# GET request
response = session.get('https://example.com')
print(response.status_code)

Basic Session Example

login_url = 'https://example.com/login'
payload = {'username': 'user', 'password': 'pass'}
with requests.Session() as session:
    # Login
    session.post(login_url, data=payload)
    
    # Access protected content
    dashboard = session.get('https://example.com/dashboard')
    print(dashboard.text[:200])

This code keeps cookies, headers, and authentication active for the session duration.

Session Objects vs. Regular Requests

Stateless requests.get() calls are fine for one-off requests but inefficient for repetitive tasks. Sessions are stateful, which is especially useful in the following cases:

  • Multiple API calls using the same token
  • Login-based websites
  • Repeated requests to the same domain

Table: Session vs Requests

Featurerequests.get()requests.Session()
Maintains cookies❌✅
Reuses TCP connections❌✅
Persistent headers❌✅
Authentication❌✅
Connection pooling❌✅
Ease of making multiple requestsMediumHigh

Setting Headers and Authentication in Sessions

Sessions make it easy to set default headers or authentication.

Custom Headers Example

headers = {
    'User-Agent': 'MyApp/1.0',
    'Accept-Language': 'en-US'
}
session = requests.Session()
session.headers.update(headers)
response = session.get('https://httpbin.org/headers')
print(response.json())

Authentication Example

from requests.auth import HTTPBasicAuth
session = requests.Session()
session.auth = HTTPBasicAuth('username', 'password')
response = session.get('https://httpbin.org/basic-auth/username/password')
print(response.status_code)

Handling Cookies with Sessions

Cookies persist automatically, allowing seamless login sessions.

Viewing and Modifying Cookies

session = requests.Session()
session.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
# Access cookies
print(session.cookies.get_dict())
# Add a new cookie manually
session.cookies.set('custom_cookie', 'abcdef')
print(session.cookies.get_dict())

Timeouts and Error Handling

Timeouts prevent requests from hanging indefinitely. Sessions simplify global error handling.

Timeout Example

try:
    response = session.get('https://httpbin.org/delay/5', timeout=2)
except requests.Timeout:
    print("Request timed out!")

Retry Strategies

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry = Retry(total=5, backoff_factor=1, status_forcelist=[502,503,504])
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

Advanced Session Features

Proxies

proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.10:1080",
}
session.proxies.update(proxies)

SSL Verification

response = session.get('https://expired.badssl.com/', verify=False)

Adapters and Connection Pooling

adapter = HTTPAdapter(pool_connections=100, pool_maxsize=100)
session.mount('https://', adapter)

Performance Comparison: Sessions vs Stateless Requests

  • Stateless Requests: Open a new TCP connection for every request → slower, high latency
  • Session Requests: Reuse TCP connections → faster, less overhead

Benchmark Example:

import time
import requests
urls = ['https://httpbin.org/get'] * 10
# Stateless requests
start = time.time()
for url in urls:
    requests.get(url)
print("Stateless:", time.time() - start)
# Session requests
session = requests.Session()
start = time.time()
for url in urls:
    session.get(url)
print("Session:", time.time() - start)

Sessions are significantly faster for repeated requests.

Async Alternatives: httpx and aiohttp

For large-scale applications or high-performance scraping:

  • httpx supports async requests, similar API to Requests
  • aiohttp is an async-first HTTP client

Example with httpx:

import httpx
import asyncio
async def fetch(url):
    async with httpx.AsyncClient() as client:
        r = await client.get(url)
        print(r.status_code)
asyncio.run(fetch('https://example.com'))

Real-World Examples Using Requests Sessions

Scraping Websites Efficiently

urls = ['https://example.com/page1', 'https://example.com/page2']
with requests.Session() as session:
    session.headers.update({'User-Agent': 'MyScraper/1.0'})
    for url in urls:
        r = session.get(url)
        print(len(r.text))

API Authentication with Tokens

token = 'your_api_token'
session.headers.update({'Authorization': f'Bearer {token}'})
response = session.get('https://api.example.com/data')

Geo-targeted API Requests

session.headers.update({'Accept-Language': 'en-US'})
response = session.get('https://api.weather.com/v3/wx/forecast/daily/5day?geocode=51.5074,-0.1278&format=json')
print(response.json())

This approach ensures geo-targeted SEO-friendly content for APIs or web services.

Common Troubleshooting Scenarios

  1. Session cookies not persisting: Ensure you are using the same session object.
  2. Slow requests: Use connection pooling or async requests.
  3. SSL errors: Use verify=False cautiously or install proper certificates.
  4. 403 Forbidden errors: Update headers like User-Agent or use proxies.

Best Practices for Python Requests Sessions

  1. Always use sessions for repeated requests.
  2. Use with statements to auto-close sessions.
  3. Manage headers, cookies, and authentication at session level.
  4. Implement retries and timeouts for robust applications.
  5. Use connection pooling to optimize performance.
  6. Consider async alternatives for high-performance or scraping tasks.

Conclusion

Python Requests Sessions are essential for developers needing persistent connections, cookies, and authentication. Sessions streamline web scraping, API integrations, and repetitive HTTP tasks. By leveraging sessions, retries, proxies, and connection pooling, developers can build high-performance, maintainable Python applications.

Leave a Comment