How to Scrape Dynamic Websites With Python

For websites built with static HTML and CSS, simple tools like Python's request library and Beautiful Soup can often do the job when web scraping. However, scraping dynamic content gets tricky when dealing with advanced websites built on dynamic JavaScript frameworks like React, Angular, and Vue. These frameworks streamline web development by providing pre-built components and architecture, but they also make dynamic websites difficult to scrape. 

In this blog post, we'll overview the differences between dynamic and static websites and give a step-by-step guide on dynamic web scraping, particularly those featuring infinite scroll.

What is a dynamic website, and how is it different from a static one?

A dynamic website is a site where HTML pages are not fixed but generated in real-time based on user actions, database queries, or other factors. Unlike static websites, which serve the same pre-built HTML file to every visitor, dynamic sites load content on the fly, meaning different users can see different versions of the same page depending on their behavior, preferences, or search inputs.

This difference comes down to how each type handles a request. With a static page, the server simply delivers a fixed file, fast, straightforward, and identical for everyone. With a dynamic web page, the server performs intermediate processing first, querying a database, applying logic, and personalizing content, before sending the final result to the browser. This makes dynamic sites more complex to serve, but far richer in interactivity and personalization.

E-commerce product pages, social media feeds, and account dashboards are all common examples – the underlying URL may be the same, but the content shifts based on who's viewing it and when. 

Challenge of web scraping dynamic content

Scraping dynamic content presents unique challenges that simple tools like Python's Requests library and Beautiful Soup aren't built to handle. Some of them can be:

• JavaScript rendering: basic HTTP requests won't capture HTML generated in real-time by frameworks like React, Angular, and Vue

• Pop-up windows and overlays: interstitial elements can block content access and interrupt the scraping flow

• Drop-down menus and live search suggestions: interactive elements that require user input to reveal their content

• Real-time updates: content that changes continuously without a page reload is harder to capture accurately

• Sortable tables: data that reorders dynamically based on user interaction requires additional handling

• Infinite scroll: content that only loads as the user scrolls down requires custom scraper logic to capture fully

• Anti-bot protection: dynamic sites often deploy CAPTCHAs and IP blocks that interrupt automated scraping

How to scrape dynamic web pages?

We've previously discussed dynamic scraping techniques like employing Selenium, utilizing a headless browser, and making AJAX request. So, we invite you to check these resources out as well. Additionally, check out the advanced web scraping with Python guide to learn more tactics, such as emulating AJAX calls and handling infinite scroll.

However, it's important to note that each target presents unique challenges. Indeed, there can be different elements that require special attention, such as pop-up windows, drop-down menus, live search suggestions, real-time updates, sortable tables, and more. 

One of the most encountered challenges is continuous or infinite scroll —  a technique that allows content to be dynamically loaded as the user scrolls down a webpage. To scrape target URL from websites with infinite scroll, you need to customize your scraper, which is exactly what we'll discuss below using Google Search as an example of a dynamic target.

How to scrape dynamic web pages using Selenium

This section will go through the numbered steps to scrape dynamic sites using Selenium in Python. For our target, we will use the Google Search page with some keywords. 

Step 1: Setting up the environment

First, make sure you have installed the latest version of Python on your system. You'll also need to install Selenium library using the following command:

pip install selenium

Then, download the Chrome driver and ensure that the version matches your Google Chrome version.

Step 2: Code in action

Start by creating a new Python file and import the required libraries:

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from bs4 import BeautifulSoup

Then set up Chrome Webdriver with Selenium by copying the path to your driver executable file and pasting in the following code:

# Set up the Chrome WebDriver
driver = webdriver.Chrome()

Following that, navigate to the Google Search Page and provide your search keyword:

# Navigate to Google Search
search_keyword = "adidas"
driver.get("https://www.google.com/search?q=" + search_keyword)

Now you can simulate a continuous scroll on Selenium. Using the script below, you’ll scroll multiple times to get more results and then scrape the results:

Define the number of times to scroll
scroll_count = 5


# Simulate continuous scrolling using JavaScript
for _ in range(scroll_count):
   driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
   time.sleep(2)  # Wait for the new results to load 

Finally, use BeautifulSoap to parse and extract the results:

page_source = driver.page_source


# Parse the page source with BeautifulSoup
soup = BeautifulSoup(page_source, 'html.parser')


search_results = soup.find_all('div', class_='tF2Cxc')
for result in search_results:
   title = result.h3.text
   link = result.a['href']
   print(f"Title: {title}")
   print(f"Link: {link}")
   print("\n")

Run the final code as specified below to see the results:

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from bs4 import BeautifulSoup

# Set up the Chrome WebDriver
driver = webdriver.Chrome()

# Navigate to Google Search
search_keyword = "adidas"
driver.get("https://www.google.com/search?q=" + search_keyword)

# Define the number of times to scroll
scroll_count = 5

# Simulate continuous scrolling using JavaScript
for _ in range(scroll_count):
   driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
   time.sleep(2)  # Wait for the new results to load (adjust as needed)

# Get the page source after scrolling
page_source = driver.page_source


# Parse the page source with BeautifulSoup
soup = BeautifulSoup(page_source, "html.parser")

# Extract and print search results
search_results = soup.find_all("div", class_="tF2Cxc")
for result in search_results:
   title = result.h3.text
   link = result.a["href"]
   print(f"Title: {title}")
   print(f"Link: {link}")
   print("\n")

# Close the WebDriver
driver.quit()

The output should look something like this:

One problem with this web scraping approach is that Google might mistake you for a malicious bot, which would frequently trigger CAPTCHA. This means you'll have to integrate reliable proxies into your script and rotate them continuously. 

So, if you're looking for a solution to extract data on a large scale, you might also want to consider a commercial solution that will deal with anti-bot systems for you, and we'll cover this in the subsequent section.

How to scrape dynamic web pages using Playwright

This section will go through the numbered steps to scrape dynamic sites using Playwright in Python. For our target, we will use the Oxylabs sandbox product listing page at sandbox.oxylabs.io/products. 

Step 1: Setting up the environment

First, make sure you have installed the latest version of Python on your system. You'll also need to install Playwright library using the following command:

python -m pip install playwright
python -m playwright install

Step 2: Code in action

Start by creating a new Python file and import the required libraries:

from playwright.sync_api import sync_playwright

TThen, launch a Chromium browser instance and open a new page:

# Launch Browser
with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

Navigate to the target product listing page and wait for the product cards to load. Since this is a dynamic site, we wait for the content to be visible before proceeding:

# Navigate to Target Page
page.goto("https://sandbox.oxylabs.io/products")
page.locator(".product-card").first.wait_for(state="visible")

Now check if pagination exists on the page, and print how many pages are available:

# Check for Pagination
pagination = page.locator(".pagination")
if pagination.is_visible():
    page_links = page.locator(".pagination a").all()
    print(f"Found {len(page_links)} pages")

If a "Next" button is found, click it and wait for the new page's products to load dynamically:

# Navigate to Next Page
    next_button = page.locator("text=Next").first
    if next_button.is_visible():
        print("clicking 'next' button")
        next_button.click()
        page.locator(".product-card").first.wait_for(state="visible")

Next, let’s scrape the products from the new page and close the browser:

# Scrape Products
        products = page.locator(".product-card").all()
        print(f"Found {len(products)} products on page 2")

    # Clean Up
    browser.close()

Now, let’s do parsing: 

# Parse Product Details
                for product in products:
                    title = product.locator("h4").inner_text()
                    price = product.locator(".price-wrapper").inner_text()
                    print(f"Title: {title} | Price: {price}")

Run the final code as specified below to see the results:

from playwright.sync_api import sync_playwright

def main():
    # Launch Browser
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        # Navigate to Target Page
        page.goto("https://sandbox.oxylabs.io/products")
        page.locator(".product-card").first.wait_for(state="visible")

        # Check for Pagination
        pagination = page.locator(".pagination")
        if pagination.is_visible():
            page_links = page.locator(".pagination a").all()
            print(f"Found {len(page_links)} pages")

            # Navigate to Next Page
            next_button = page.locator("text=Next").first
            if next_button.is_visible():
                print("clicking 'next' button")
                next_button.click()
                page.locator(".product-card").first.wait_for(state="visible")

                # Scrape Products
                products = page.locator(".product-card").all()
                print(f"Found {len(products)} products on page 2")

                # Parse Product Details
                for product in products:
                    title = product.locator("h4").inner_text()
                    price = product.locator(".price-wrapper").inner_text()
                    print(f"Title: {title} | Price: {price}")

        # Clean Up
        browser.close()

if __name__ == '__main__':
    main()

The parsing section loops through each product card and extracts data.

The output should look something like this in the Terminal:

Found 10 pages
clicking 'next' button
Found 32 products on page 2
Title: iPhone 15 Pro | Price: $999.99
Title: Samsung Galaxy S24 | Price: $849.99

If you're looking for a solution to extract data on a large scale, you might also want to consider a commercial solution that will deal with anti-bot systems for you, and we'll cover this in the subsequent section.