How to Scrape Multiple Pages Using Beautifulsoup

Understanding pagination patterns

Before writing code, look at how the site numbers its pages. Most sites fall into one of three patterns, and the right approach depends on which you are dealing with.

• Numbered URLs such as ?page=2 or /page/2/, where you can build the URL yourself.
• Next-link pagination, where each page contains a link to the following page and you follow it until the link disappears.
• Offset or cursor parameters, common on APIs and infinite-scroll pages, where a value increments each request.

Looping over numbered pages

When the page number is part of the URL, a simple loop is the cleanest solution. Build each URL, fetch it, parse it, and stop when a page returns no results.

import requests
from bs4 import BeautifulSoup

base = "https://example.com/products?page={}"
all_items = []

for page in range(1, 11):
    resp = requests.get(base.format(page), timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    items = soup.select(".product-title")
    if not items:
        break  # no more results, stop early
    all_items.extend(i.get_text(strip=True) for i in items)

print(len(all_items), "items collected")

Following a next-page link

Some sites do not expose a predictable page number, but every page links to the next one. Here you start at the first URL and follow the next link until it is gone.

from urllib.parse import urljoin

url = "https://example.com/blog"
results = []

while url:
    soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser")
    results += [h.get_text(strip=True) for h in soup.select("h2.post-title")]
    next_link = soup.select_one("a.next")
    url = urljoin(url, next_link["href"]) if next_link else None

Being polite and avoiding blocks

Sending requests as fast as possible is the quickest way to get rate limited or banned. A few habits keep a multi-page scrape healthy.

Add delays and identify yourself

import time

headers = {"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"}

for page in range(1, 11):
    resp = requests.get(base.format(page), headers=headers, timeout=10)
    # process page...
    time.sleep(2)  # pause between requests

Handle errors so one bad page does not kill the run

for page in range(1, 11):
    try:
        resp = requests.get(base.format(page), timeout=10)
        resp.raise_for_status()
    except requests.RequestException as e:
        print("Skipping page", page, "-", e)
        continue
    # parse normally...

Saving data as you go

For long runs, write results to disk page by page rather than holding everything in memory. Appending rows to a CSV after each page means a crash near the end does not lose all your work, and it makes the job restartable.

Where proxies become essential

Across many pages from one IP, sites often throttle or block you regardless of how polite you are. Rotating proxies distribute requests across different addresses so a long paginated crawl can complete. Cheapest Proxies is our featured value pick for this kind of steady, high-volume work, though it is always worth comparing providers on rotation frequency, country coverage and price for your specific target.

Reuse a session instead of bare requests.get

Calling requests.get repeatedly opens a new connection each time and discards any cookies the site sets. A requests.Session keeps the TCP connection alive across pages, carries cookies that some paginated listings depend on, and gives you one place to set default headers. For a multi-page run this is both a speed win and a reliability win, because many sites treat a cookie-less sequence of hits as suspicious.

import requests
session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"})
resp = session.get(url, timeout=10)

Make the run resumable

Long crawls fail in boring ways: a network blip, a process kill, a power cut. If you record the last completed page (or a set of done page numbers) to a small state file, the next run can skip what is finished and continue. Combined with appending results to disk after each page, this means an interruption costs you one page of work rather than the entire job.

A minimal resume pattern

• Read a state file listing completed page identifiers at startup.
• Skip any page already marked done before fetching it.
• After a page is parsed and saved, write its identifier to the state file.
• On restart, the loop naturally picks up where it left off.

Deduplicate across pages, not within them

Paginated sites are not static: items shift between pages as new entries are added, so the same record can appear on page two during one fetch and page three minutes later. Deduplicating by page number does nothing for this. Instead, derive a stable key from each record, such as a product id or a normalised URL, and keep a set of seen keys so repeats are dropped no matter which page surfaced them.

Detecting the end and backing off correctly

Stopping when a page yields no items is a good baseline, but pages can return an empty body during a transient block and trick your loop into quitting early. A safer end-of-pagination signal combines an empty result set with a successful status code, and treats a non-200 response as a retry rather than an end. When you do hit throttling, exponential backoff with a little random jitter spreads retries out and recovers gracefully where a fixed two-second sleep simply gets blocked again.

Keep comparing with these closely related pages.