Nine tactics, ranked: conditional requests, route interception, compression, dedupe and pool choice. With the arithmetic, and the ones not worth doing.
· 15 min read · Pricing and economics
Most advice on how to reduce proxy bandwidth costs starts with "block images." That is one tactic out of nine, it only applies if you are running a browser, and on a recurring crawl it is not the biggest one. The biggest one is conditional requests, and almost nobody implements them.
This article ranks nine tactics by the term each one attacks in the cost formula, gives the arithmetic for the saving, and names the ones that cost more in engineering time than they return. A vendor-neutral version, including the parts that make the vendor look worse.
You cannot cut what you have not attributed. Before changing anything, produce two breakdowns of your last run.
By response class. Successful 200s with valid content, 200s with invalid content (challenge pages, soft-404s), 3xx, 4xx, 5xx, timeouts. Bytes for each. If the non-success classes are more than a fifth of your bytes, stop reading this article and go read failed requests and retry spend — your problem is success rate, not payload size.
By content type. Document, script, stylesheet, image, font, XHR, other. Bytes for each. If you are running a plain HTTP client, this breakdown will be 100 percent document and half the advice on this topic does not apply to you.
The harness that produces both breakdowns is in measured bandwidth by page type. Run it once. It takes ten minutes and it will tell you which of the nine sections below is worth your afternoon.
| # | Tactic | Attacks | Applies to | Engineering cost |
|---|---|---|---|---|
| 1 | Conditional requests on re-crawls | Re-fetching unchanged pages | Any recurring crawl | Low |
| 2 | Route interception | Subresources you never parse | Browser clients only | Low |
| 3 | Drop the browser entirely | All subresources | Targets that render server-side | Medium |
| 4 | Frontier deduplication | Fetching the same document twice | Any crawl over ~10K URLs | Medium |
| 5 | Retry discipline | Buying the same failure repeatedly | Any job with blocking | Low |
| 6 | Early abort on non-200 | Challenge-page bodies | Any client that streams | Low |
| 7 | Verified compression | Uncompressed document transfer | Any client | Very low |
| 8 | Local cache and re-parse | Re-fetching to fix a parser bug | Any pipeline under development | Low |
| 9 | Cheapest pool that still succeeds | Cost per successful record | Any job | Low |
The ordering is by mechanism, not by a measured percentage. Which one is biggest for you depends entirely on your diagnosis above, which is why the diagnosis comes first.
The single largest saving available on any recurring crawl, and the least implemented.
When you first fetch a page, keep the ETag and Last-Modified response
headers. On the next crawl, send them back:
If-None-Match: "a1b2c3d4"
If-Modified-Since: Tue, 05 Aug 2026 11:04:22 GMT
If the document has not changed, the server replies 304 Not Modified with
headers and no body at all. You paid for a header exchange instead of a
document.
The arithmetic is exact. Let u be the fraction of your frontier unchanged between crawls, B the document size, and B₃₀₄ your measured 304 response size:
saving_per_cycle = frontier_size × u × (B − B_304)
A 250,000-page monthly re-crawl of 22 KB documents where 80 percent are unchanged: 250,000 × 0.8 × 22 KB = 4,400,000 KB = 4.4 GB saved per cycle, minus 200,000 × B₃₀₄. Since B₃₀₄ is header-only, the saving is close to the gross figure.
Measure B₃₀₄ rather than assuming it:
curl -sS -o /dev/null -D - \
-H 'If-None-Match: "a1b2c3d4"' \
-w '\nbody=%{size_download} headers=%{size_header}\n' \
https://example.com/some/page
In Scrapy, this is a settings change rather than code:
HTTPCACHE_ENABLED = True
HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy"
The RFC2616 policy is the one that issues conditional requests. The default dummy policy does not — it either serves from cache or fetches in full, so switching the policy is the whole change.
Two failure modes. Some servers return a 200 with a full body regardless of
your conditional headers, in which case you have lost nothing but gained
nothing; detect this by counting 304s and disable the header for those hosts.
And a weak ETag on a page with a rotating ad slot changes every cycle even
though your data did not — for those, compare a content hash of the parsed
fields instead and stop re-fetching pages whose data has been stable for n
cycles.
Applies only if you are driving a browser. If you are using an HTTP client, you already do not download subresources — skip to section 4.
Playwright:
BLOCK = {"image", "media", "font", "stylesheet"}
async def gate(route):
if route.request.resource_type in BLOCK:
await route.abort()
else:
await route.continue_()
await page.route("**/*", gate)
Puppeteer:
await page.setRequestInterception(true);
page.on('request', req => {
const t = req.resourceType();
if (['image', 'media', 'font', 'stylesheet'].includes(t)) req.abort();
else req.continue();
});
On the HTTP Archive July 2025 medians, images and fonts alone are 911 KB and 122 KB of the 2,559 KB median mobile home page. Blocking them is the difference between two very different rows of the per-100K table.
Route interception interacts with how you scope contexts and proxies, which is covered under aborting requests in Playwright.
Three cautions that the standard advice omits:
Blocking third parties by domain is usually a bigger win than blocking by resource type, because ad and analytics chains pull further resources of their own. Maintain a blocklist and match on the request URL host.
The largest single reduction available, when it is available. On median figures, a document-only fetch is a small fraction of a full page load — the comparison is laid out in measured bandwidth by page type.
Run the test before you decide: fetch the page with curl and search the raw
HTML for the field you extract. If it is there, the browser is buying you
detection resistance and nothing else, and that is a stealth decision priced
separately from a bandwidth decision. The full trade-off is in
the stealth-versus-gigabytes trade-off.
The middle option is usually the right one: use the browser once to discover the JSON endpoint the page is calling, then hit that endpoint directly with an HTTP client for the rest of the run.
Every URL you fetch twice is a gigabyte you bought twice. On a large crawl the duplicate rate is routinely double digits and almost always invisible.
Normalize before enqueueing:
utm_*, gclid, fbclid, ref, _ga.jsessionid, sid, PHPSESSID.?a=1&b=2 and ?b=2&a=1 collapse.rel="canonical". If page A declares B canonical, record A→B and
never fetch A again.Then measure it. Count distinct normalized URLs against total fetches for one run. That ratio is your duplicate factor, and it multiplies every other number in your budget.
Pagination overlap is the sibling problem. Offset-based pagination over a list that changes under you returns overlapping rows: you pay for page 3 twice and store it once. Where the target supports cursor pagination, use it. Where it does not, deduplicate by record ID after parsing and accept the overlap as a known cost rather than an invisible one.
Covered in full in
failed requests and retry spend,
so only the rule here: classify the error before retrying, and never retry a
403 challenge on the same exit IP. The arithmetic that matters is that
attempts per successful record equal 1 divided by your per-attempt success
rate, regardless of your retry cap. Raising the cap buys coverage. Only raising
the success rate lowers unit cost.
Three techniques, in the order you should reach for them.
Stream and abort on a disqualifying status or content type.
import httpx
MAX_BYTES = 512 * 1024
def fetch(client, url):
with client.stream("GET", url) as r:
ct = r.headers.get("content-type", "")
if r.status_code != 200 or "html" not in ct:
return None # body never transfers
buf = bytearray()
for chunk in r.iter_bytes():
buf += chunk
if len(buf) > MAX_BYTES:
break # connection closes here
return bytes(buf)
This is the highest-value change in this section, because it converts the most expensive failure class — a full challenge-page body — into a header exchange.
Range for partial reads. When you only need something near the top of a
document — a canonical tag, a meta refresh, a JSON-LD block in <head> — ask
for the first few kilobytes:
Range: bytes=0-8191
Servers that support it reply 206 Partial Content. Servers that do not reply
200 with the whole document, so check the status before assuming a saving.
HEAD for existence checks. Genuinely useful when you are validating a URL
list and will not fetch the body either way. See the caution in section 10 —
HEAD followed by GET on the same URL is usually a net loss.
Set a hard DOWNLOAD_MAXSIZE in Scrapy, or the equivalent ceiling in your
client, so an unexpected multi-megabyte response cannot run away with a plan.
Client-level configuration for a common case is in
configuring an Axios client.
Sending the header is not the same as receiving compressed bytes.
Accept-Encoding: gzip, br, zstd
Then verify from the response, not from your request:
curl -sS -o /dev/null -D - --compressed https://example.com/ \
| grep -i '^content-encoding'
No Content-Encoding line means the server ignored you and you are paying for
plain text.
Three reasons it silently fails:
Accept-Encoding by default, and some proxy
configurations rewrite it.Brotli beats gzip on text by a useful margin at equal effort. Zstandard is now widely supported and worth adding to the header. None of the three helps on content that is already compressed — images, video, PDFs — so do not expect a saving there.
Confirm the actual wire saving with the interface counter rather than with your client's reported download size. Depending on client and version, a "download size" may be the decoded size rather than the transferred size, which would make compression look like it saved nothing.
The cheapest gigabyte is the one you already bought.
During development, every parser bug that sends you back to the network is a second purchase of data you already had. Write raw responses to disk or object storage on first fetch, keyed by normalized URL and fetch timestamp, and point the parser at the store rather than at the target.
Two rules make this work in practice:
ETag and
Last-Modified for section 1, and the Content-Encoding for section 7.Storage is cheaper than proxy bandwidth by a wide margin at every provider. Compare your own object-storage rate against the per-GB rates and the decision makes itself.
The most common expensive mistake in this list: optimizing the rate card instead of the cost per successful record.
Cost per successful record is price_per_GB × bytes_per_page ÷ success_rate.
Which means a pool is only cheaper if its rate advantage survives its success
rate.
Worked example on a 100 KB page, per 1,000 collected pages, using the rates published on the per-GB rates page as of 2026-08-06 — datacenter $0.99/GB, residential $1.99/GB, mobile $3.99/GB. Rates change; the method does not.
| Pool | Rate | Success | Bytes billed per 1,000 collected | Cost |
|---|---|---|---|---|
| Datacenter | $0.99/GB | 25% | 0.400 GB | $0.396 |
| Residential | $1.99/GB | 85% | 0.118 GB | $0.234 |
The pool that costs half as much per gigabyte costs 69 percent more per record.
The general rule is a ratio of those two rates rather than either number:
Those two ratios are the whole pool decision, and they are stable even when the rates change — recompute the ratio, not the conclusion. Background on what each pool is for is in pool selection.
Run the test properly: same URL sample, same client, same hour, one pool per run. Success rate against a given target is not a property of a pool in the abstract, and a measurement taken last quarter is not evidence about today.
This is the section every vendor article omits, because it is the section where some tactics fail.
No percentage is published here, for the same reason none of the nine sections above carries one: a saving measured on someone else's targets is not evidence about yours. Run the comparison instead. It is four steps.
The fourth step is what makes the table reusable in six months. The third is what keeps it honest: a tactic that reduces bytes while reducing success rate has not saved anything. It has moved spend from the bandwidth line to the coverage line, and reporting only the first column hides the move.
Three candidates that plausibly do not pay off, to be confirmed or refuted by that run rather than asserted here:
HEAD before GET. Two round trips instead of one, two handshakes instead
of one, and you still fetch the body. Worth it only when you expect to skip a
high proportion of the URLs. On a list you will fetch anyway, it is a net loss.
Aggressive request-header trimming on a job dominated by large responses. Saving a kilobyte per request against a 2 MB response is a rounding error. The same change on a job of 400-byte JSON responses is material. It is the same tactic with opposite verdicts, decided by the ratio you measured in the diagnosis step.
Brotli over gzip on small documents. The compression-ratio advantage is real but small in absolute terms below a few kilobytes, and some origins fall back to identity encoding when asked for brotli. Ask for all three encodings and stop thinking about it.
All three are the same lesson from three directions: the verdict on a tactic is a property of your workload, not of the tactic. Measure, then decide, then write down what you measured so the next person does not repeat the run.
Diagnose first, then pick the tactic matching your diagnosis. If you run a browser, route interception blocking images, fonts and third-party domains is a one-hour change with immediate effect. If you run a recurring crawl, conditional requests are larger and take about the same effort. If your non-success bytes exceed a fifth of the total, neither matters until you fix the success rate.
On a recurring crawl, yes. A 304 Not Modified returns headers and no body, so
every unchanged page costs a header exchange instead of a document. Multiply
your frontier size by the fraction unchanged between crawls by the document
size and you have the saving. On a 250,000-page monthly crawl of 22 KB pages
with 80 percent unchanged, that is about 4.4 GB per cycle.
No. A plain HTTP client fetches the document you asked for and nothing else — no images, no scripts, no fonts. There is nothing to block. Advice about blocking images applies to Playwright, Puppeteer, Selenium and any other browser automation. If you are on an HTTP client, your levers are conditional requests, deduplication, compression and retry discipline.
No. Cost per collected record is the rate divided by the success rate, so a pool at half the price that succeeds a third as often is more expensive. Divide the two published per-GB rates to get the threshold: at current rates, datacenter beats residential only while its success rate stays above 49.7 percent of residential's. Measure both pools against the same URL sample in the same hour before deciding.
Check the response, not the request. Run curl -sS -o /dev/null -D - --compressed
against the URL and look for a Content-Encoding header. If it is absent, the
server ignored your Accept-Encoding and you are transferring plain text.
Verify the byte saving against a network-interface counter rather than your
client's reported download size, which may report decoded rather than
transferred bytes.
Yes, particularly during development. Storage costs a small fraction of proxy
bandwidth, and every parser fix that sends you back to the target is a second
purchase of data you already had. Store raw bytes plus response headers, keyed
by normalized URL and timestamp, so you keep the ETag values that make
conditional re-crawls work later.
Check the per-GB rates, then run the diagnosis before you change
a line of code. Route through gw.anonedge.com:823, break your last run down
by response class and content type, and pick the one section above that matches
what you find. Nine tactics is eight too many if you have not measured which
one applies.