How to Scrape a Table Using Beautifulsoup

How an HTML table is structured

A standard table nests several tags. The <table> wraps everything; <tr> defines each row; <th> holds header cells and <td> holds data cells. Many tables also separate <thead> and <tbody>. Knowing this hierarchy is the key to a reliable scraper, because you walk the same nested shape every time.

Locating the right table

Pages often contain several tables, so target the one you want by id, class, or position. Inspect the page in your browser to find a stable selector.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/stats", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")

# By id or class is most reliable:
table = soup.find("table", {"id": "results"})
# Or grab the first table on the page:
# table = soup.find("table")

Extracting the header row

Read the header cells first so you know your column names. Most tables put them in <th> elements inside the first row or the <thead>.

headers = [th.get_text(strip=True) for th in table.select("thead th")]
# Fallback if there is no thead:
if not headers:
    headers = [th.get_text(strip=True) for th in table.find("tr").find_all("th")]
print(headers)

Iterating over the data rows

Now loop through each row and pull the text from its cells. Using get_text(strip=True) trims whitespace and gives you clean values.

rows = []
for tr in table.select("tbody tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all("td")]
    if cells:  # skip empty or separator rows
        rows.append(cells)

for r in rows[:5]:
    print(r)

Pairing headers with values

To build dictionaries keyed by column name, zip the headers against each row. This makes downstream code much more readable.

records = [dict(zip(headers, row)) for row in rows]

Exporting the table

Once you have rows, saving them is straightforward. The standard csv module writes a flat file, while pandas gives you a DataFrame for filtering and analysis.

import csv

with open("table.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(headers)
    writer.writerows(rows)

import pandas as pd

df = pd.DataFrame(rows, columns=headers)
print(df.head())

The pandas shortcut

For simple, well-formed tables, pandas.read_html can parse every table on a page in one call. It is fast, but BeautifulSoup gives you finer control when tables have merged cells, nested markup, or messy structure that pandas misreads.

tables = pd.read_html(resp.text)
df = tables[0]  # first table on the page

Handling tricky tables

• Merged cells using colspan or rowspan break simple zipping; you may need to pad rows manually.
• Nested tables can cause your selector to grab cells from the wrong table, so scope selectors tightly.
• JavaScript-rendered tables will not appear in the raw HTML and require a browser automation tool first.

Scraping tables at scale

If you are collecting tables from many URLs, sending all requests from one IP often triggers blocks. Rotating proxies keep a large table-scraping job moving, and Cheapest Proxies is a strong value-focused option worth considering. Compare providers on rotation, reliability and price so you pay for what your job actually needs.

Cleaning cell values into real data types

BeautifulSoup hands you strings, and a column that looks numeric on screen is often "$1,299.00" or "12.4%" in the markup. Before you can sum, sort or chart it, you have to strip the currency symbol, remove thousands separators, drop the percent sign and cast to a number. Doing this in a small per-column cleaning function keeps your extraction loop readable and makes the cleaning rules easy to audit later.

def to_number(text):
    cleaned = text.replace(",", "").replace("$", "").replace("%", "").strip()
    try:
        return float(cleaned)
    except ValueError:
        return text  # leave genuinely non-numeric values alone

Expanding rowspan and colspan properly

The base guide notes that merged cells break simple zipping; the fix is to honour the span attributes as you build the grid. A cell with colspan="2" should occupy two columns, and a cell with rowspan="3" should reappear in the same column on the next two rows. Maintaining a small carry-over map of pending rowspans as you iterate lets you reconstruct a clean rectangular grid even from heavily merged tables.

The mental model

• Track each unfinished rowspan as a column index with a remaining count and value.
• At the start of every row, fill those columns first before placing new cells.
• Repeat a colspan cell's value across the number of columns it spans.
• Decrement rowspan counters until they reach zero and drop out.

Multi-row and grouped headers

Statistical and financial tables frequently use two header rows, where a top row groups several sub-columns beneath it. Reading only the first <tr> of the <thead> gives you half the labels. Either flatten the two rows into combined names like "2024 Revenue" by pairing the group label with each sub-label, or keep them as a hierarchical column index if you are loading into pandas for analysis.

Validating the grid before you trust it

A table can parse cleanly yet be quietly wrong: a stray separator row, a footer totals row, or one misread span can shift every value one column left. Cheap assertions catch this. Check that each data row has the same length as your header, watch for rows that are entirely empty or hold a single spanning cell, and spot-check a few known values against the live page. This verification step is what separates a reliable extraction from a dataset that looks fine until someone relies on it.

Keep comparing with these closely related pages.