How to Use Proxies with Python Requests

The basic proxy setup

Requests accepts a proxies argument on any call. You map each scheme (HTTP and HTTPS) to a proxy URL. In most modern setups the same endpoint serves both, so both keys point to the same address.

import requests

proxies = {
    "http": "http://proxy.example.com:8080",
    "https": "http://proxy.example.com:8080",
}

resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(resp.json())

Hitting an endpoint that echoes your IP, such as an IP-check service, is the fastest way to confirm the proxy is actually being used. If the returned address is your own, the request bypassed the proxy.

Adding authentication

Most commercial proxies require a username and password. The cleanest method is to embed credentials directly in the proxy URL using the user:pass@host:port format.

proxies = {
    "http": "http://USERNAME:PASSWORD@proxy.example.com:8080",
    "https": "http://USERNAME:PASSWORD@proxy.example.com:8080",
}

If your password contains special characters such as @ or :, URL-encode them with urllib.parse.quote so the URL parses correctly. A malformed credential string is a common cause of silent 407 Proxy Authentication Required errors.

Working with sessions

When you make many requests, create a Session and set the proxies once. This reuses connections, applies headers consistently, and keeps your code tidy.

session = requests.Session()
session.proxies.update(proxies)
session.headers.update({"User-Agent": "Mozilla/5.0 (research-bot)"})

for page in range(1, 5):
    r = session.get(f"https://example.com/list?page={page}", timeout=15)
    print(r.status_code)

Rotating proxies

For larger jobs you often rotate IPs to spread requests. Some providers handle rotation server-side through a single gateway endpoint, so your code never changes. Others give you a pool, and you rotate in Python.

import random

pool = [
    "http://USER:PASS@ip1:port",
    "http://USER:PASS@ip2:port",
    "http://USER:PASS@ip3:port",
]

def get_proxy():
    p = random.choice(pool)
    return {"http": p, "https": p}

r = requests.get("https://example.com", proxies=get_proxy(), timeout=15)

Rotating vs sticky sessions

• Rotating: a new IP per request, ideal for spreading load across many targets.
• Sticky: the same IP held for a short window, useful when a site tracks a session across several pages.

Check which model your provider supports before committing, because it shapes how you structure your code.

Handling errors gracefully

Proxies fail. Connections time out, IPs get blocked, and gateways occasionally drop. Wrap requests in try/except and retry sensibly rather than crashing.

from requests.exceptions import ProxyError, Timeout, ConnectionError

try:
    r = requests.get(url, proxies=proxies, timeout=10)
    r.raise_for_status()
except (ProxyError, Timeout, ConnectionError) as e:
    print(f"Proxy issue, retrying: {e}")

Add a short backoff between retries and a maximum attempt count so a dead proxy does not loop forever. A 407 points to credentials, a 403 often points to the target blocking you, and a timeout usually points to a slow or overloaded proxy.

Verifying TLS and avoiding leaks

Keep certificate verification on by default. Disabling it (verify=False) silences warnings but exposes you to interception, so only do it for deliberate, controlled testing. Also confirm your DNS resolution is going where you expect, since some setups leak DNS lookups outside the proxy.

Concurrency: where Requests quietly bottlenecks

A single proxy and a single thread will cap your throughput long before the proxy does, because Requests blocks on each call until the response returns. Most people discover this when a job that should take minutes takes hours. The library itself will not parallelise anything for you. The common patterns are a ThreadPoolExecutor for I/O-bound scraping, switching to an async client such as httpx or aiohttp when you need hundreds of concurrent connections, or spreading work across processes when CPU-bound parsing is the limit. Whichever you pick, give each worker its own Session; sharing one Session across threads is generally safe for reads but mixing proxy changes across threads on a shared Session invites surprises.

Connection pooling and keep-alive through a proxy

Under the hood Requests uses urllib3 connection pools. When you route through a proxy, the pool keeps a tunnel open to that proxy, which is exactly what you want for many requests to the same endpoint. You can tune the pool with an HTTPAdapter mounted on your Session, raising pool_maxsize so concurrent workers do not contend for connections. The subtlety: if you rotate the proxy URL on every request, you defeat keep-alive and pay a fresh handshake each time. Where latency matters, prefer a provider gateway that rotates the exit IP server-side while you keep one stable endpoint, so pooling still works.

Environment variables and NO_PROXY precedence

Requests reads proxy settings from the environment automatically. This is convenient until it is not: a stray HTTPS_PROXY in a shell profile or container image can route traffic you intended to send direct, and NO_PROXY can carve out hosts so they bypass your proxy entirely. When a request unexpectedly shows your real IP, or unexpectedly fails to reach an internal host, check the environment before the code. You can disable this behaviour per Session by setting session.trust_env = False, which makes the explicit dictionary the single source of truth.

A quick triage order when something looks wrong

• Confirm the exit IP with an echo endpoint before assuming the proxy works.
• Print session.trust_env and the relevant environment variables.
• Separate a 403 (target blocking) from a 407 (auth) from a timeout (proxy slow or dead).
• Retry the same URL direct to see whether the target or the proxy is at fault.

Instrumenting requests so failures are diagnosable

At any real volume you need to know not just that a request failed but which proxy served it, how long it took, and what the target returned. Wrap your call to capture the chosen proxy, status code, elapsed time, and final URL after redirects, then log them as structured fields. This lets you spot a single bad exit node dragging down a pool, distinguish target-side blocks from proxy-side faults, and decide whether to retry, rotate, or back off. The few extra lines pay for themselves the first time a job stalls.

Keep comparing with these closely related pages.