Wget with a Proxy

Why route wget through a proxy

There are several everyday reasons to put a proxy in front of wget. You may be on a corporate or campus network that requires an outbound proxy. You may want to download content as it appears from a specific country. Or you may be collecting many files and want to spread requests so a single IP address is less likely to be rate-limited. In each case the wget logic stays the same; only the network path changes.

Method 1: proxy environment variables

The quickest way to point wget at a proxy is with environment variables. Wget respects the standard variables used across many Unix tools:

export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"
export no_proxy="localhost,127.0.0.1,internal.example.com"

wget https://example.com/largefile.zip

The no_proxy variable lists hosts that should bypass the proxy, which is handy for internal addresses. Because these variables apply to the whole shell session, they are convenient for a batch of downloads but easy to forget about later, so unset them when you are done.

Method 2: command-line flags

If you prefer to keep the proxy settings local to a single command, pass them inline. Wget lets you enable proxy use and supply credentials directly:

wget -e use_proxy=yes \
     -e http_proxy=http://proxy.example.com:8080 \
     -e https_proxy=http://proxy.example.com:8080 \
     https://example.com/file.zip

The -e flag injects configuration as if it were written in the config file, which makes one-off overrides simple without touching any persistent settings.

Method 3: the .wgetrc configuration file

For settings you want to keep, edit a wget configuration file. A per-user file lives at ~/.wgetrc, while a system-wide file is usually at /etc/wgetrc. A typical proxy block looks like this:

use_proxy = on
http_proxy = http://proxy.example.com:8080
https_proxy = http://proxy.example.com:8080
no_proxy = localhost,127.0.0.1

This approach is ideal when every wget call on a machine should use the same proxy, so you do not have to repeat flags or export variables each session.

Handling authenticated proxies

Many proxies require a username and password. Wget supports dedicated flags for this so credentials are not baked into the URL:

wget --proxy-user=USERNAME \
     --proxy-password=PASSWORD \
     https://example.com/file.zip

You can also embed credentials in the proxy URL as http://USER:PASS@proxy.example.com:8080, but the dedicated flags are cleaner and keep secrets out of your shell history when combined with care. For repeated use, store credentials in a protected .wgetrc with restrictive file permissions.

Quick tips for cleaner proxy downloads

• Use --tries and --waitretry to make wget more patient when a proxy hiccups.
• Add --limit-rate to be a good citizen and avoid overwhelming the proxy or target.
• Combine with --user-agent when a site expects a browser-like request.
• Test with a single small file before launching a large mirroring job.

Troubleshooting common problems

If downloads fail through the proxy, work through the basics first. Confirm the proxy host and port are reachable, check that use_proxy is actually on, and verify your credentials. A frequent gotcha is that HTTPS traffic ignores http_proxy but needs https_proxy set separately. Another is that environment variables in a parent shell do not always carry into scripts run by other users or cron jobs, so set them explicitly inside automation.

If requests succeed but get blocked or throttled by the target site, the issue is usually the proxy's reputation or that you are hammering from a single exit. Spreading requests across a rotating pool, or choosing a provider with cleaner IPs, often resolves it. When sourcing proxies for scripted downloads, Cheapest Proxies is a strong value-focused option worth considering, and comparing providers on value first can save both money and headaches.

Precedence: which proxy setting actually wins

When the same proxy is configured in more than one place, wget does not merge them arbitrarily. Configuration supplied with -e on the command line behaves as if it were appended to the config file, so it overrides what ~/.wgetrc sets, which in turn overrides /etc/wgetrc. Environment variables like http_proxy operate through a separate mechanism and can take effect even when you thought a config file was in charge. The practical lesson is to pick one source of truth per machine. Mixing an exported variable, a personal config and an inline flag is the fastest way to spend an hour debugging why traffic exits from the wrong place.

Why proxies break under cron and systemd

A wget command that works perfectly in your terminal often fails the moment cron runs it, and the cause is almost always the environment. Interactive shells load your profile and the proxy exports inside it; cron and systemd services do not. They start with a minimal, sanitised environment, so http_proxy and https_proxy simply are not there.

Reliable ways to fix it

• Set the proxy variables explicitly inside the cron entry or the script the cron calls, not just in your login profile.
• For systemd units, declare proxies with Environment= lines or an EnvironmentFile=.
• Or sidestep environment inheritance entirely by passing the proxy with -e flags on every wget call in the job.

Making the proxy explicit in automation removes a whole class of intermittent, hard-to-reproduce failures.

Recursive mirroring multiplies the proxy load

A single download is gentle on a proxy. A recursive mirror with -r or --mirror can fire hundreds or thousands of requests, and every one rides the same exit. That changes the calculus completely. A flaky proxy that survives one fetch can stall a long mirror partway through, and a single overused exit IP is far more likely to be throttled or blocked by the target. This is where pairing wget's --wait, --random-wait and --limit-rate with a cleaner or rotating proxy pool pays off. Comparing providers on value before wiring them into a recurring mirror, with Cheapest Proxies as a reasonable value pick to weigh, keeps large jobs from grinding to a halt.

HTTPS, certificates and intercepting proxies

When wget fetches over HTTPS through a proxy, the proxy may simply tunnel the encrypted connection, or it may intercept and re-sign it. With an intercepting corporate proxy, wget can reject the substituted certificate and fail with a verification error. Understanding which kind of proxy you face matters: a transparent tunnel needs nothing extra, while an intercepting one may require trusting its certificate authority. Reaching for --no-check-certificate works but silently disables a real security protection, so prefer installing the proxy's CA where the interception is legitimate and expected.

Keep comparing with these closely related pages.