How to Find All Href Attributes Using Beautifulsoup

What an href attribute actually is

In HTML, the href attribute lives on an <a> (anchor) tag and points to the destination of a link. It can hold an absolute URL, a relative path, an anchor fragment such as #section, or a special scheme like mailto: or tel:. When you scrape links, you usually want the real navigable URLs and not these special cases, so a little filtering goes a long way.

Setting up BeautifulSoup

You need the beautifulsoup4 and requests packages. Install them with pip, then load the page HTML into a soup object using a parser such as the built-in html.parser or the faster lxml.

pip install beautifulsoup4 requests lxml

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url, timeout=10)
soup = BeautifulSoup(response.text, "html.parser")

Finding every href on the page

The simplest approach is to find all anchor tags and read the href attribute from each one. Passing href=True to find_all tells BeautifulSoup to return only the anchors that actually have an href, which avoids errors from links that omit it.

links = []
for a in soup.find_all("a", href=True):
    links.append(a["href"])

print(links)

You can compress this into a single list comprehension, which is the most common idiom in real scraping scripts:

hrefs = [a["href"] for a in soup.find_all("a", href=True)]

Using a CSS selector instead

If you prefer CSS selectors, select does the same job and pairs nicely with attribute syntax. This is handy when you only want links inside a specific container.

hrefs = [a["href"] for a in soup.select("a[href]")]
# Only links inside the main content area:
main_links = [a["href"] for a in soup.select("main a[href]")]

Turning relative links into absolute URLs

Many sites use relative hrefs like /about or ../page.html. These are useless on their own, so resolve them against the page URL with urljoin from the standard library.

from urllib.parse import urljoin

absolute = [urljoin(url, a["href"]) for a in soup.find_all("a", href=True)]

Filtering out fragments, mailto and duplicates

Real pages contain anchor fragments, email links and repeated URLs. A short filter keeps only the links you care about and removes duplicates while preserving order.

from urllib.parse import urljoin

seen = set()
clean = []
for a in soup.find_all("a", href=True):
    href = a["href"].strip()
    if href.startswith(("#", "mailto:", "tel:", "javascript:")):
        continue
    full = urljoin(url, href)
    if full not in seen:
        seen.add(full)
        clean.append(full)

Common mistakes to avoid

• Reading a["href"] on a tag that has no href raises a KeyError; use href=True or a.get("href") instead.
• Forgetting to resolve relative paths means later requests fail.
• Scraping every link recursively without limits can hammer a server, so add delays and respect robots.txt.
• Some links are injected by JavaScript and never appear in the raw HTML; those need a browser-based tool, not plain requests.

Where proxies fit in

Crawling many pages to collect links from a single IP often leads to rate limits or blocks. Rotating proxies spread requests across multiple addresses so a link-discovery crawl can run reliably. For budget-conscious projects, Cheapest Proxies is a strong value-focused option worth considering, and it is sensible to compare providers on rotation, location coverage and price before committing.

Links live in more than just anchor tags

Most tutorials stop at <a> tags, but a thorough harvester should know that hrefs also appear on <area> elements inside image maps and on <link> tags in the document head, where canonical URLs, stylesheets and alternate language versions hide. If your goal is link discovery for a crawler, the <link rel="canonical"> href is often the single most valuable URL on the page because it tells you the site's preferred address for that content. A selector such as soup.select("a[href], area[href], link[href]") captures all three families in one pass, and you can branch on the tag name afterwards to treat them differently.

Resolving the base tag before urljoin

A subtle bug appears when a page includes a <base href> tag. That tag redefines the reference point for every relative link, so resolving against the request URL alone produces wrong results. Read the base tag first, fall back to the page URL when it is absent, and feed that into your resolution step so relative paths point where the browser would actually send them.

base_tag = soup.find("base", href=True)
ref = base_tag["href"] if base_tag else page_url
absolute = [urljoin(ref, a["href"]) for a in soup.find_all("a", href=True)]

Canonicalising to catch sneaky duplicates

A plain set deduplicates exact strings, but http://Site.com/page/, http://site.com/page and http://site.com/page?utm_source=x all point at the same content while looking different. Lowercasing the host, removing a trailing slash and stripping known tracking parameters collapses these into one entry, which can dramatically shrink a crawl frontier and cut wasted requests.

A light normalisation pass

• Lowercase the scheme and host but never the path, which can be case-sensitive.
• Drop common tracking parameters such as those beginning with utm_.
• Remove the fragment after # since it never changes the requested resource.
• Decide once whether a trailing slash matters for your target and apply it consistently.

Scope control: internal versus external

For a focused crawl you usually want only same-domain links, while a broken-link audit may want everything. Compare the resolved link's host against the page's host with urlparse and route each into an internal or external bucket. This single decision prevents a crawler from wandering across the entire web and keeps proxy usage predictable, since you only follow links that fall inside your declared boundary.

Keep comparing with these closely related pages.