HTML is 22 KB of a 2,559 KB median home page. Here is the per-100K-page arithmetic, the harness to measure your own targets, and what it costs.
· 15 min read · Pricing and economics
How much bandwidth does web scraping use depends almost entirely on one decision: whether your client downloads the page's subresources. On the public dataset with the most transparent methodology, the median mobile home page transfers 2,559 KB — and the HTML document inside it is 22 KB. That is a factor of 116 on the same URLs, from the same servers, in the same crawl.
Every range you will find quoted for this question — 5 to 15 GB per 100,000 pages, 50 to 200 GB per 100,000 — is published without a method. This article does it the other way around. The method comes first, the source is named and dated, and the harness is included so you can reproduce the numbers on your own targets and dispute them.
Two sources of numbers appear below, and they are not interchangeable.
Source 1 — a public dataset with a published method. The HTTP Archive 2025 Web Almanac, Page Weight chapter, from the crawl of July 2025. It reports transfer sizes from a real browser load of millions of pages, which makes it the closest thing to a neutral baseline that exists for this question. Its own note on compression matters here: "Thanks to 'on-the-fly' compression (like Gzip), the number of bytes sent over the wire is often much smaller than the original file size." Those are wire bytes, which is what a proxy meters.
Read its labels carefully, because most articles quoting it do not. The chapter publishes several different medians, and swapping one for another moves an estimate by a third:
| Figure | What it measures | Desktop | Mobile |
|---|---|---|---|
| 14.1 | Median home page weight | 2,862 KB | 2,559 KB |
| 14.2 | Median inner page weight | 1,963 KB | 1,769 KB |
| 14.3 | Median page weight, home and inner combined | 2,412 KB | 2,164 KB |
| 14.5 / 14.6 | HTML component of the median home page | 22 KB | 22 KB |
| 14.5 / 14.6 | HTML component of the median inner page | 21 KB | 20 KB |
The 2,559 / 2,862 KB pair is the median home page, not the median page and not an average. The median across home and inner pages together is lower — 2,164 KB on mobile. Which row you should budget against depends on which kind of page your crawler actually visits, and for most jobs the answer is inner pages.
Source 2 — your own harness, run against your own targets. Non-negotiable, because page weight is not a property of the web, it is a property of the sites you scrape. An e-commerce category listing and a government records page are not in the same order of magnitude.
Every figure below is labeled with which source it came from. Where a first-party measurement would go and does not yet exist, there is a method you can run rather than a number you have to trust.
#!/usr/bin/env bash
# bytes-http.sh — transferred bytes per URL, plain HTTP client, no subresources
# usage: ./bytes-http.sh urls.txt > http-bytes.tsv
printf 'url\treq\tresp_hdr\tresp_body\ttotal\tredirects\n'
while read -r url; do
curl -sS -o /dev/null -L --compressed \
-w "%{url_effective}\t%{size_request}\t%{size_header}\t%{size_download}\t%{num_redirects}\n" \
"$url" \
| awk -F'\t' '{print $1"\t"$2"\t"$3"\t"$4"\t"($2+$3+$4)"\t"$5}'
done < "$1"
Note what this does not see: the TLS handshake, TCP/IP framing and retransmits. Those are real bytes that may be on your bill depending on where the meter is read — see what counts as billable traffic for why your count will differ from the provider's and by roughly how much.
# bytes-browser.py — transferred bytes per page, headless Chromium, full load
# usage: python bytes-browser.py urls.txt > browser-bytes.tsv
import asyncio, sys
from playwright.async_api import async_playwright
async def measure(page, url):
totals = {}
async def on_response(resp):
try:
s = await resp.request.sizes()
except Exception:
return
rtype = resp.request.resource_type
n = s["responseBodySize"] + s["responseHeadersSize"] \
+ s["requestBodySize"] + s["requestHeadersSize"]
totals[rtype] = totals.get(rtype, 0) + n
page.on("response", on_response)
await page.goto(url, wait_until="networkidle", timeout=60000)
page.remove_listener("response", on_response)
return totals
async def main(path):
urls = [l.strip() for l in open(path) if l.strip()]
async with async_playwright() as p:
b = await p.chromium.launch()
ctx = await b.new_context()
print("url\ttotal\t" + "\t".join(
["document","script","stylesheet","image","font","xhr","fetch","other"]))
for u in urls:
page = await ctx.new_page()
t = await measure(page, u)
await page.close()
row = [t.get(k, 0) for k in
["document","script","stylesheet","image","font","xhr","fetch"]]
other = sum(t.values()) - sum(row)
print(f"{u}\t{sum(t.values())}\t" + "\t".join(map(str, row + [other])))
await b.close()
asyncio.run(main(sys.argv[1]))
Run both against the same URL list, on the same day, from the same host. The ratio between the two totals is the number that decides your budget.
Two rules make the output worth keeping. Fix the URL list — 200 URLs spanning
five page classes is enough, and changing it between runs invalidates the
comparison. And record the client versions, curl, Chromium and Playwright,
alongside the run date, because all three change transfer behavior between
releases.
Every per-page figure below is attributed to HTTP Archive. None of them is a first-party measurement, and none of them is presented as one.
From HTTP Archive, July 2025 crawl: the HTML component of the median home
page is 22 KB on both mobile and desktop, and 20 to 21 KB on the
median inner page. That is the whole cost of a page for a client that requests
the document and stops — curl, requests, httpx, Scrapy without a
browser middleware.
Two adjustments before you use 22 KB as your own number.
Medians are not your sites. The median is drawn from a broad crawl. A JavaScript-heavy single-page app ships a small HTML shell and would sit far below it; a server-rendered category listing with 200 products sits well above it. Run the harness.
Home pages are heavier than the pages you actually scrape. The same chapter reports that a home page "on average is close to 45.8% larger than inner pages," and that "the median home page used 239% the image bytes of similar inner pages." Most scraping targets inner pages, and the chapter publishes that row directly: the median inner page is 1,769 KB on mobile and 1,963 KB on desktop, against 2,559 and 2,862 KB for home pages.
Almost nobody quoting a bandwidth range for scraping makes this distinction. Making it moves the estimate by a third, and it does not require deriving anything — the figure is in the source.
From HTTP Archive, July 2025 crawl, the median home page by content type:
| Mobile | Desktop | |
|---|---|---|
| Total page weight | 2,559 KB | 2,862 KB |
| HTML | 22 KB | 22 KB |
| CSS | 77 KB | 82 KB |
| JavaScript | 632 KB | 697 KB |
| Images | 911 KB | 1,058 KB |
| Fonts | 122 KB | 139 KB |
One caveat that most write-ups omit: the component medians do not sum to the total median. Each row is the 50th percentile of a different distribution across a different set of pages. Use the rows to see proportion, use the total row to size a job, and do not subtract one from the other and call the result a measurement.
Set the fields you actually extract — a price, a title, a stock status — against that table. On the median mobile home page, images alone are 911 KB, or roughly 41 times the HTML document that contains the data. JavaScript is another 632 KB and most of it is framework code, analytics and tag managers. Fonts are 122 KB of typography for a page no human will look at.
Itemized, here is what a full browser load buys you that a parser never touches:
If the data is in the HTML, every one of those is pure loss. Whether it is worth paying anyway is a separate question — the answer depends on detection, not on bandwidth. See when a browser is worth the gigabytes.
Most pages that render dynamically are calling an endpoint that returns exactly the data you want, without markup, without assets, and usually with pagination built in. Find it and your per-record bandwidth drops by an order of magnitude.
How to find it, in four steps:
Two warnings. First, the minimum header set is often smaller than you expect, and every header you drop is bytes you stop paying on every request. Second, API endpoints are frequently rate-limited more aggressively than the HTML they back, so your bandwidth per record falls while your success rate may also fall. Measure both.
The endpoint's payload size is entirely target-specific, which is why there is no median worth quoting here. Run step 3 and read the number off the Network tab.
Arithmetic on the HTTP Archive July 2025 medians above. Decimal gigabytes, 1 GB = 1,000,000 KB. A provider billing in binary gigabytes (1 GB = 1,048,576 KB) would report the same traffic as roughly 5 percent fewer gigabytes, so check which basis your invoice uses.
| Page class | Bytes per page | Per 100,000 pages | Source |
|---|---|---|---|
| HTML document only | 22 KB | 2.2 GB | HTML component of the median home page |
| Inner page, full load, mobile | 1,769 KB | 176.9 GB | Median inner page, Figure 14.2 |
| Inner page, full load, desktop | 1,963 KB | 196.3 GB | Median inner page, Figure 14.2 |
| Home page, full load, mobile | 2,559 KB | 255.9 GB | Median home page, Figure 14.1 |
| Home page, full load, desktop | 2,862 KB | 286.2 GB | Median home page, Figure 14.1 |
| JSON endpoint | target-specific | measure it | Your harness |
| Browser with images and fonts blocked | target-specific | measure it | Your harness |
That last row has no published number, and it cannot be produced by subtracting
the image and font medians from the total median — those are percentiles of
different distributions, and the subtraction is not valid arithmetic. Get it by
running bytes-browser.py twice against the same URL list, once with route
interception blocking image and font requests. Any article that hands you a
number for this row without describing that run has invented it.
The headline comparison, using only figures the source publishes: 2.2 GB versus 255.9 GB for the same 100,000 URLs. Same targets. Same servers. The only variable is the client.
Rates as published on the per-GB rates for each pool page on 2026-08-06: datacenter $0.99/GB, residential $1.99/GB, mobile $3.99/GB. Rates change. Treat the table below as a method, not a quote.
| Page class (per 100,000 pages) | Datacenter $0.99 | Residential $1.99 | Mobile $3.99 |
|---|---|---|---|
| HTML document only — 2.2 GB | $2.18 | $4.38 | $8.78 |
| Inner page, full load, mobile — 176.9 GB | $175.13 | $352.03 | $705.83 |
| Inner page, full load, desktop — 196.3 GB | $194.34 | $390.64 | $783.24 |
| Home page, full load, mobile — 255.9 GB | $253.34 | $509.24 | $1,021.04 |
| Home page, full load, desktop — 286.2 GB | $283.34 | $569.54 | $1,141.94 |
Two readings of that table are worth taking away.
The client decision dwarfs the pool decision. Moving from a browser to a plain HTTP client on the same 100,000 URLs saves more than moving from mobile to datacenter on the same client — by two orders of magnitude. Optimize the client first.
The cheapest rate is not always the cheapest job. A datacenter pool that gets blocked half the time costs more per collected record than a residential pool at twice the rate that succeeds. Compare choosing between the three pools on cost per successful request, not on the rate card.
Every one of these is a multiplier on the table above, and they compose.
Retries. The dominant one. Billed volume is roughly the estimate divided by your success rate. A 70 percent success rate is a 1.43x multiplier before anything else.
Redirect chains. Each hop is a full request and response with headers in both directions. A chain of three means four exchanges per document. Resolve chains once during development and store the final URLs.
Compression not negotiated. Sending Accept-Encoding: gzip, br is not the
same as receiving compressed bytes. Some servers ignore it; some CDNs strip it;
some clients do not set it by default. Uncompressed HTML runs several times the
compressed size. Verify from the response, not from the request.
Duplicate crawls of the same canonical. Tracking parameters, session IDs in URLs, trailing slashes and sort-order variants all produce distinct URLs that return the same document. On a large frontier this routinely adds double-digit percentages.
Pagination overlap. Cursor-less pagination that shifts under you means pages 2 and 3 share rows. You pay for both and store one.
Assets you did not intend to load. A browser configured without route interception fetches everything, including the third-party chat widget. This is the difference between two rows of the table above.
Five inputs. Fill them in and do the arithmetic before you size a plan.
The formula:
billed_GB = (unique_pages × recrawls × dup_factor × bytes_per_page) / success_rate
+ (total_attempts × per_request_overhead)
total_attempts = (unique_pages × recrawls × dup_factor) / success_rate
A worked example. 250,000 unique product pages, HTML only at 22 KB, monthly re-crawl over six months, a 1.15 duplicate factor, a measured 75 percent success rate:
The value of h is the one input nobody publishes and everybody needs, and it is the reason the harness matters more than the table. Measure it. Then read cutting the number down to see which levers move which term in that formula.
It depends on the client, not the target. Using HTTP Archive's July 2025 medians, 100,000 HTML documents at 22 KB each is about 2.2 GB, while 100,000 full mobile home page loads at 2,559 KB each is about 255.9 GB. That is a 116x spread on identical URLs. Add your per-request protocol overhead, then divide by your success rate, to get a number you can budget against.
It is the HTML component of the median home page in a broad public crawl, not a guarantee for your targets. Server-rendered listing pages with hundreds of rows run far above it; single-page applications that ship a thin shell run below it, though they then require an API call or a browser to produce data. Measure ten representative pages from your own frontier before committing to any figure.
On median figures, roughly. A browser fetches the document plus scripts, stylesheets, images and fonts, and on the median mobile home page images alone are 911 KB against a 22 KB document. The multiplier drops sharply once you block images, fonts and third-party tags with route interception, but it never reaches parity with a plain HTTP client that requests one document.
Run two harnesses against the same URL list on the same day. For plain HTTP,
sum curl's size_request, size_header and size_download. For a browser, sum
Playwright's request.sizes() across every response event, grouped by resource
type. Both are in this article. Compare the totals against your provider's
reported usage and investigate any gap larger than a few percent.
Yes. A blocked request has already paid for the TLS handshake, the request headers and usually a full challenge-page body before your code sees the status. Budget by dividing your clean estimate by your measured success rate: collecting 10 TB of usable data at a 70 percent success rate means paying for about 14.29 TB.
The API, almost always, when one exists and is reachable. It returns the data without markup, without assets and usually with server-side pagination. Find it in DevTools under Fetch/XHR, sorted by response size. The trade-off is that API endpoints are often rate-limited more tightly than the HTML they back, so measure success rate as well as payload size before switching.
Check the per-GB rates for each pool, then run the harness above
against ten of your own URLs before you buy anything. Route it through
gw.anonedge.com:823 and you will have a real per-page number in under an hour
— which is one more real number than any bandwidth range on the internet.