A headless browser pays about 130 times the bytes of the HTML it renders. Here is the measured ratio, the break-even math, and when the cost is worth it.
· 13 min read · Pricing and economics
Should you use a headless browser or plain HTTP requests? Two SERPs give you opposite answers and neither acknowledges the other. The anti-blocking guides say browser. The bandwidth guides say HTTP client. Nobody prices the gap between them.
Here is the gap. In the HTTP Archive's July 2025 crawl the median home page transferred 2,862 KB on desktop, and the HTML inside that same median page was 22 KB (Web Almanac 2025, Page Weight, Figures 14.1 and 14.6). One page. Two clients. A factor of about 130 in what your proxy meters.
On a per-GB plan that factor is the whole decision.
Fetch with an HTTP client. Escalate to a browser only for the requests that an HTTP client cannot complete at all — not the ones it completes badly.
That sounds like a stealth argument. It is not. Work the break-even below and you find that bandwidth almost never justifies a browser, because the byte ratio is so lopsided that the HTTP client would have to fail more than 99% of the time before the browser catches up on cost per successful page. The browser is a feasibility purchase, not a bandwidth one. You buy it when the success rate of the cheap client is approximately zero, and you buy it for exactly those URLs.
An HTTP client sends one request and receives one response. No JavaScript executes. No subresource is fetched. No image, font, stylesheet, tracker, analytics beacon or third-party script is ever requested, because nothing parses the document and decides to request them.
What you get:
What you give away, immediately:
User-Agent spoofing changes the handshake
that happened before the header was sent.User-Agent and a
non-Chrome header sequence is a one-line detection rule.Rotating your exit address does nothing about any of this. A residential IP attached to a fingerprint that no residential user has ever produced is a residential IP that stands out more, not less. Choosing the exit pool matters, but it is a separate axis from the client — see choosing datacenter, residential or mobile exits.
A real browser engine, driven over a debugging protocol. It executes the page's JavaScript, builds a DOM, runs the event loop, sets and replays cookies and storage the way the page expects, and produces a handshake and header profile that belongs to the browser it actually is.
Concretely, it buys you four things an HTTP client cannot fake cheaply:
localStorage, sessionStorage and
IndexedDB behavior across a multi-step flow.And it buys all of that at the byte cost of every subresource on the page.
The HTTP Archive publishes exactly the comparison this decision needs: the median transfer bytes for a full page load, alongside the transfer bytes of the HTML document inside that same median page. July 2025 data, split by page role because home pages and inner pages are not the same purchase:
| Metric (median) | Desktop home | Mobile home | Desktop inner | Mobile inner |
|---|---|---|---|---|
| Total page weight | 2,862 KB | 2,559 KB | 1,963 KB | 1,769 KB |
| HTML | 22 KB | 22 KB | 21 KB | 20 KB |
| JavaScript | 697 KB | 632 KB | 719 KB | 660 KB |
| Images | 1,058 KB | 911 KB | 442 KB | 354 KB |
| Fonts | 139 KB | 122 KB | 138 KB | 119 KB |
| CSS | 82 KB | 77 KB | 85 KB | 80 KB |
Source: Web Almanac 2025, Page Weight.
Translate that into a proxy bill. 100,000 pages, cold, one pass. Decimal gigabytes, 1 GB = 1,000,000 KB:
| Client | Bytes per page | 100,000 pages |
|---|---|---|
| HTTP client, desktop home pages | 22 KB | 2.2 GB |
| Headless browser, desktop home pages | 2,862 KB | 286.2 GB |
| HTTP client, desktop inner pages | 21 KB | 2.1 GB |
| Headless browser, desktop inner pages | 1,963 KB | 196.3 GB |
Two decimal places of precision would be false. The shape is the point: two orders of magnitude between the two columns, on the same 100,000 URLs.
Three caveats, because publishing the ratio without them is how the "10x" folklore started:
For the HTTP side, count what goes over the wire rather than what your parser receives. Send the encoding header yourself so curl does not decompress and inflate the count:
curl -s -o /dev/null \
-H 'Accept-Encoding: gzip, br' \
-w 'body=%{size_download} headers=%{size_header}\n' \
https://example.com/product/123
For the browser side, ask the browser. encodedDataLength is the on-the-wire
figure, subresources included:
const client = await context.newCDPSession(page);
await client.send('Network.enable');
let bytes = 0;
client.on('Network.loadingFinished', e => { bytes += e.encodedDataLength; });
await page.goto(url, { waitUntil: 'networkidle' });
console.log(bytes);
Run both against 20 representative URLs from your actual target. The ratio you get is the only one that should drive your architecture.
| Target type | Where the data lives | Client | Bytes you pay for | GB per 100,000 |
|---|---|---|---|---|
| Static HTML | In the document | HTTP client | HTML only | ~2 GB at the 21-22 KB median |
| Server-rendered, light JS | In the document; scripts decorate | HTTP client | HTML only | ~2 GB at the median |
| SPA with a JSON API behind it | In one or more XHR responses | HTTP client, calling the API | HTML (once, to discover) + JSON | Below the HTML baseline per record; target-specific, measure it |
| SPA with JS-gated content | Written into the DOM at runtime | Browser, or replayed API call | Full page load | ~196-286 GB at the median, before context reuse |
| Behind an active challenge | Released after a script runs | Browser for the challenge only | Full page load plus challenge assets | Above the full-page figure; target-specific, measure it |
The rows that matter are rows three and four. Most pages people reach for a browser to scrape are row three wearing row four's clothes.
Two of the five cells give a direction rather than a number, deliberately. No public dataset measures the bytes per extracted record on a JSON-API-backed SPA, or the bytes of a page behind an active challenge including the challenge's own assets, and both are dominated by the specific target rather than by any web-wide median. The curl and CDP commands above produce both figures for your targets in an afternoon. Treat anyone quoting a general number for either as guessing.
Between "fetch the HTML" and "render the page" there are three techniques, and all three are cheaper than the browser.
Call the API the page calls. Open the target once with devtools, filter the network panel to XHR and fetch, and look at what populates the view. It is usually one JSON endpoint returning the records you were about to parse out of rendered HTML — often paginated, often with a larger page size available than the UI requests. You skip the document, the bundle, the fonts and the images, and you get structured data instead of markup that changes when a designer touches a class name.
Solve once, replay many. Where a challenge issues a cookie or token, run a browser for that one exchange, extract the credential, and hand it to the HTTP client for the following requests until it expires. You pay browser bytes once per token lifetime instead of once per page. Hold the exit address for the handoff — a token issued to one IP and replayed from another is the most common way this technique fails.
Hybrid crawls. Browser for entry points, category pages and anything that sets state. HTTP client for the long tail of detail pages. On a catalog crawl the long tail is the overwhelming majority of the URLs, and it is the majority you do not want to pay 2,862 KB for.
Match the concurrency of each half to your plan — a browser context and an HTTP connection both occupy a slot, and browser contexts hold several connections each. The limits are in thread limits and concurrent connections; the sizing formula is in sizing concurrency.
Cost per successful page is what you actually spend, because you pay for failed attempts too:
cost per success = price per GB x bytes per attempt / success rate
The browser is cheaper when:
bytes_browser / rate_browser < bytes_http / rate_http
Rearranged, the browser wins when:
rate_http / rate_browser < bytes_http / bytes_browser
Now solve it for two page weights, using the medians above.
Desktop home pages. bytes_http / bytes_browser = 22 / 2,862 = 0.0077.
If your browser succeeds 90% of the time, the HTTP client must succeed less
than 0.69% of the time before the browser is the cheaper purchase.
Desktop inner pages. 21 / 1,963 = 0.0107. Same 90% browser success rate, and the threshold moves to 0.96%.
The page got 31% lighter and the break-even barely moved. That is the finding.
There is no realistic block rate at which a browser is a bandwidth
optimization. The threshold only clears when rate_http is effectively zero —
which is precisely the case where the content does not exist without
execution.
So stop framing it as a cost-versus-stealth trade and frame it correctly:
Check the arithmetic against your own tier before committing — datacenter, residential and mobile are metered at different per-GB rates, and the same 286 GB costs very different amounts depending on which pool the job needs.
Most of a page's bytes are things a parser never reads. On the desktop medians above, images and fonts alone are 1,058 KB and 139 KB of 2,862 KB. Removing what you never parse is therefore a large reduction rather than a marginal one — but measure it before and after with the CDP counter above rather than trusting any estimate, including that one.
image, media, font, and the stylesheets you do not need for
layout-dependent extraction. Images alone are 1,058 KB of the 2,862 KB
desktop home page median.networkidle in production. Wait for the selector that
contains your data. networkidle waits for trackers you already blocked and
for polling that never stops.Point the browser at the gateway the same way you point any other client, using the host and port that match your session model — see configuring proxy settings, or the HTTP client setup if you are moving work onto the cheap path. Scoping a proxy to an individual browser context rather than the whole browser is covered under Playwright contexts and proxies, and the rest of the reduction tactics, ranked, are under cutting browser bandwidth.
Use an HTTP client unless the data does not exist without JavaScript execution. The break-even math says the browser only becomes cheaper per successful page once the HTTP client's success rate falls under about 1% of the browser's, which in practice means the content is gated rather than merely defended. Route the gated URLs to a browser and everything else to the cheap client.
No. That figure circulates widely and understates the difference by an order of magnitude. In the HTTP Archive's July 2025 crawl, the median desktop home page transferred 2,862 KB while its HTML document transferred 22 KB — a ratio near 130, not 10. Context reuse and resource blocking reduce it in practice, but a one-page cold load on a median page is nowhere near 10x.
Partly. Libraries that mimic a browser's TLS ClientHello and HTTP/2 settings close the fingerprint gap that JA3 and JA4 measure, and fixing header order is free. What they cannot do is execute a challenge script or produce content that only exists after JavaScript runs. Fingerprint parity is achievable; execution parity is not.
The client and the exit pool are independent choices. A browser does not require residential exits and an HTTP client is not restricted to datacenter ones. Pick the pool by how hard the target is, then pick the client by whether the data survives without JavaScript. Running a browser on an expensive pool multiplies two costs at once, so verify you need both.
For HTTP clients, use curl -w '%{size_download}' with an explicit
Accept-Encoding header so nothing is decompressed before it is counted. For
browsers, attach a CDP session, listen for Network.loadingFinished, and sum
encodedDataLength. Both report on-the-wire transfer, which is the same thing
a gateway meters, so the two are directly comparable.
It can. Some challenge flows fetch an image as part of their verification, and some targets score sessions on subresource completeness. Block by URL pattern rather than by resource type on those targets, and keep whatever the challenge requests. Measure the success rate before and after — if it does not move, keep the block.
Instrument both paths on 20 URLs, publish the ratio internally, and let the number pick the architecture. Most crawls that run in a browser today are one JSON endpoint away from running at 1% of the bytes.
Start routing on the client the measurement chose, and check the per-GB rates for the pool it needs.