How to Use Curl with Python

Why Pair Curl with Python at All?

Many developers reach a point where a one-off curl command needs to run on a schedule, loop over many URLs, or feed its output into parsing logic. Rewriting everything in pure Python is one option, but sometimes you already have a working curl command and simply want to wrap it. Curl is also excellent for reproducing exact requests, since browser developer tools can export a request as a ready-to-run curl string.

The two most common goals are: reusing an existing curl command from inside a Python script, or getting curl-level control over the underlying transfer while still working in Python. Each goal points to a different technique.

Option One: Calling Curl with subprocess

The simplest path is to invoke the curl binary directly using Python's subprocess module. This is ideal when you have a known-good curl command and just want to run it and capture the result.

import subprocess

result = subprocess.run(
    ["curl", "-s", "https://httpbin.org/get"],
    capture_output=True,
    text=True,
)
print(result.stdout)

The -s flag keeps curl quiet about progress, capture_output=True grabs both streams, and text=True returns strings instead of bytes. Pass a list of arguments rather than a single string to avoid shell-quoting problems. This approach is quick, but you are parsing text output yourself and you depend on curl being installed on the host.

Adding a Proxy with subprocess

Routing through a proxy is just another curl flag:

subprocess.run([
    "curl", "-s",
    "-x", "http://user:pass@proxy-host:port",
    "https://httpbin.org/ip",
])

The -x flag sets the proxy, and credentials can be embedded in the URL or supplied separately. This makes it easy to test whether a proxy endpoint responds before wiring it into a larger job.

Option Two: PycURL for In-Process Control

If you want curl's transfer engine without spawning an external process, PycURL exposes libcurl directly to Python. It is faster for high-volume work and gives fine-grained control over timeouts, headers, and connection reuse.

import pycurl
from io import BytesIO

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, "https://httpbin.org/get")
c.setopt(c.PROXY, "http://proxy-host:port")
c.setopt(c.WRITEDATA, buffer)
c.perform()
c.close()
print(buffer.getvalue().decode("utf-8"))

PycURL has a steeper learning curve than the alternatives, but its performance and low-level options make it a developer-friendly choice for demanding pipelines.

Option Three: Translate Curl into requests

Often the cleanest long-term answer is to stop shelling out and rewrite the request natively. Python's requests library mirrors most common curl flags closely, which makes ongoing maintenance easier.

import requests

proxies = {"http": "http://proxy-host:port", "https": "http://proxy-host:port"}
r = requests.get("https://httpbin.org/get", proxies=proxies, timeout=10)
print(r.json())

Here a curl -x becomes the proxies dictionary, -H becomes the headers argument, and -d becomes data or json. Tools that convert a curl string into requests code can accelerate this migration considerably.

Where Proxies Make the Difference

Any of these methods can route traffic through a proxy, and for repeated or large-scale requests a proxy is usually essential. Datacenter proxies tend to be fast and economical for tolerant targets, while residential proxies look more like ordinary users and suit stricter sites. The right pick depends on your target, your volume, and your budget.

Because proxy quality and pricing vary widely, it pays to compare providers before committing. For value-focused buyers, Cheapest Proxies (our featured value pick) is a strong option worth considering when you want capable proxies without overspending.

Quick Tips for Reliable Runs

• Always set a timeout so a stalled request cannot hang your script.
• Test the proxy with a simple endpoint like an IP-echo service before scaling up.
• Prefer argument lists over shell strings to avoid injection and quoting bugs.
• Log the exact command or options used so failed runs are reproducible.

Reading Curl Exit Codes Instead of Guessing

When you call curl through subprocess, a failure does not raise a Python exception by default. Curl returns a numeric exit code and writes diagnostics to stderr, and your script keeps running as if nothing went wrong. The robust pattern is to check result.returncode and branch on it. Code 6 means the host could not be resolved, code 7 means the connection was refused, code 28 is a timeout, and codes in the higher ranges often point at TLS or proxy problems. Capturing stderr alongside stdout lets you log the human-readable reason. If you prefer exceptions, add check=True to subprocess.run so a non-zero exit raises CalledProcessError, then catch it where you handle retries.

Importing a Browser "Copy as cURL" Command Safely

Browser developer tools can export any request as a ready-to-run curl string, complete with headers, cookies, and body. This is gold for reproducing an authenticated or stubborn request, but pasting it straight into subprocess as one long shell string invites quoting and injection problems. Split it into an argument list instead, or feed it to a curl-to-requests converter and tidy the generated code. Watch for browser-specific headers like sec-ch-ua or a long cookie blob; those are often what makes the request work, and stripping them is a common reason a faithfully copied request suddenly starts failing.

Matching Curl's TLS and HTTP/2 Behaviour

Curl and Python's HTTP stack do not always negotiate the same connection. Curl may default to HTTP/2 where requests stays on HTTP/1.1, and the two can present different TLS cipher orders, which a fingerprinting endpoint can notice. If a request works in curl but not in requests, this mismatch is a frequent culprit. PycURL helps here because it shares libcurl's exact behaviour, so it is the closest in-process match to the command line. For mutual TLS, curl's --cert and --key map to the cert tuple in requests and to SSLCERT/SSLKEY options in PycURL.

Quick parity checklist

• Confirm both honour the same redirect policy before comparing output.
• Send each through an echo endpoint and diff the headers they actually emit.
• Check whether your proxy supports CONNECT for HTTPS in all three approaches.

Where a Proxy Belongs in Each Approach

The proxy endpoint is identical across methods, but its placement differs: a -x flag for subprocess curl, a PROXY setopt for PycURL, and a proxies dictionary for requests. For repeated automated runs, route through a rotating pool rather than one static address so a single IP is not rate-limited. If you are choosing a provider for that pool, a value-focused option such as Cheapest Proxies is worth comparing when you want capable endpoints without overpaying for capacity you will not use.

Keep comparing with these closely related pages.