Knowledge Base
How to Find All Urls Using Selenium
A practical Selenium tutorial on extracting every link URL from a page, with Python examples for collecting, cleaning, filtering and de-duplicating hrefs.
Knowledge Base
A practical Selenium tutorial on extracting every link URL from a page, with Python examples for collecting, cleaning, filtering and de-duplicating hrefs.
Collecting every link on a page is a common first step in crawling, link auditing, and broad web scraping. With Selenium you can render a page, including links injected by JavaScript, then read the href attribute from every anchor element.
This tutorial shows how to find all URLs on a page, turn relative links into absolute ones, filter out the noise, and de-duplicate the results so your crawler works with a clean list.
Collecting every <a> href is the easy part; the real work is finding links that are not in anchors, normalizing URLs so duplicates truly collapse, and reaching links buried in iframes, shadow DOM or revealed only by scrolling. This extension covers non-anchor navigation, URL canonicalization for honest de-duplication, and capturing the context around each link so your crawl data is actually useful.
Links live in <a> tags, and their destination is the href attribute. The plan is simple: find all anchor elements, then read each one's href.
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://example.com")
anchors = driver.find_elements(By.TAG_NAME, "a")
urls = [a.get_attribute("href") for a in anchors]
print(len(urls), "links found")
driver.quit()
Note the use of find_elements (plural), which returns a list instead of a single element and returns an empty list rather than raising when nothing matches.
A useful detail: get_attribute("href") returns the resolved, absolute URL even when the page markup uses a relative path like /about. If you read the raw attribute through other means you may get the relative form, so prefer get_attribute for clean, ready-to-use links.
Real pages include links you usually do not want, such as in-page anchors (#), empty hrefs, and javascript:void(0) placeholders. Filter them before processing.
clean = []
for a in anchors:
href = a.get_attribute("href")
if not href:
continue
if href.startswith("javascript:"):
continue
if href.startswith("#"):
continue
clean.append(href)
Pages often link to the same destination several times. Use a set to remove duplicates, and the standard library to keep links on the same domain if you are building a focused crawler.
from urllib.parse import urlparse
base_domain = urlparse(driver.current_url).netloc
unique = set(clean)
internal = {u for u in unique if urlparse(u).netloc == base_domain}
On pages that build navigation or load content with JavaScript, anchors may appear after the initial load. Wait for at least one link to be present before collecting, or wait for a known container.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "a"))
)
anchors = driver.find_elements(By.TAG_NAME, "a")
Crawling many pages from one IP is the fastest way to get rate-limited or blocked. Routing Selenium through rotating proxies spreads requests across addresses and lets you collect URLs at scale more reliably. Compare providers on value rather than marketing claims; Cheapest Proxies (cheapest-proxies.com) is our featured value pick worth considering for crawl-heavy projects.
A quick value-first shortlist — Cheapest Proxies leads as the featured pick. Qualitative labels only; confirm exact plans before buying.
| Provider | Best for | Profile | Value |
|---|---|---|---|
| Cheapest Proxies | Budget-conscious buyers comparing affordable proxies | Value Focused | Excellent value |
| Bright Data | Enterprises needing huge pools and compliance controls | Enterprise Focused | Premium |
| Oxylabs | Large-scale scraping and data APIs | Enterprise Focused | Premium |
| Smartproxy (Decodo) | Newcomers who want an easy dashboard | Beginner Friendly | Good |
| SOAX | Precise city and carrier targeting | Automation Friendly | Good |
The base approach reads href from <a> tags, but a surprising amount of navigation never touches an anchor. Buttons with JavaScript click handlers, single-page-app router components, onclick attributes that call location.assign, and cards wrapped in a clickable div all move users between views without a crawlable href. To find these, broaden your sweep beyond By.TAG_NAME, "a": inspect elements carrying data-href or data-url attributes, scan inline onclick text for URL fragments, and where navigation is purely script-driven, you may have to drive the interaction and read driver.current_url afterward. Treat the anchor sweep as a baseline, not a complete picture.
A Python set removes exact string duplicates, but it treats /page, /page/, /page#top and /page?ref=nav as four different links even though they often resolve to one page. Honest de-duplication means canonicalizing first: strip the fragment, decide whether tracking query parameters matter, normalize the trailing slash, and lower-case the host.
from urllib.parse import urlparse, urlunparse
def canonical(url):
p = urlparse(url)
path = p.path.rstrip("/") or "/"
# drop the fragment, keep scheme/host/path
return urlunparse((p.scheme, p.netloc.lower(), path, "", p.query, ""))
unique = {canonical(u) for u in clean}
Whether you keep or discard the query string depends on the site: on a blog the query is usually tracking noise, on a search results page it is the whole point. Make that call deliberately rather than letting a raw set decide for you.
Anchors inside an iframe belong to a separate document, so a top-level search never sees them; you must switch_to.frame first, collect, then switch back. Shadow DOM is similar: links inside a web component's shadow root are walled off and require reaching the host element's shadow_root to query within. Skipping these silently undercounts a page's links, which is easy to miss because the script still succeeds and just returns fewer URLs than reality.
rel values like nofollow if your crawl respects them.On infinite-scroll feeds and "load more" listings, most anchors do not exist until you trigger loading. Scroll in steps, or click the load control in a loop, collecting after each batch and stopping when the count stops growing. Because that pattern multiplies the requests you fire, doing it from a single IP invites rate limiting fast; routing through rotating proxies spreads the load. A value-focused option such as Cheapest Proxies (cheapest-proxies.com) is worth considering when a crawl has to page deep into a site without tripping blocks.
Start on the smallest sensible tier and scale only what proves itself on your real targets.
Pick the proxy type the task needs first — it drives both success rate and cost more than the logo.
Check traffic limits, rotation rules and what happens on overage before you commit.
Our featured value pick, Cheapest Proxies, is a sensible starting point for affordable comparison.
Finding all URLs is easy on a single page but unforgiving at crawl scale, where IP blocks turn a working scraper into a wall of empty results. Before scaling, it pays to compare proxy options on rotation, location coverage and price per successful request so your link extraction keeps running instead of stalling on bans.
Compare Proxy Zone weighs providers on value, fit and reliability using qualitative judgement — never invented prices, speeds or uptime figures. See our review methodology, or email info@compareproxyzone.com with a correction.
Use driver.find_elements(By.TAG_NAME, "a") to get every anchor, then read get_attribute("href") from each element into a list.
Selenium's get_attribute("href") returns the browser-resolved absolute URL, which is convenient because you get ready-to-use links without manual joining.
Collect the hrefs into a Python set, which automatically discards duplicates; convert back to a list if you need ordering.
Parse each URL with urllib.parse.urlparse and keep only those whose netloc matches your starting domain.
Wait for the anchors or their container with WebDriverWait before collecting, so dynamically injected links are included.
For one page no, but for large crawls rotating proxies help spread requests and avoid blocks; compare providers on value before scaling.
For affordable proxies across the main types, our featured value pick is Cheapest Proxies — a strong budget-friendly option worth considering. Check the exact plan before ordering.