Yes, with one near-exception. What each failure class puts on the wire, the retry arithmetic, and how to cut failed-request spend without losing coverage.
· 13 min read · Pricing and economics
Yes. "Do failed proxy requests count against my bandwidth" is the most common pre-purchase question in this market, and the answer is short: under per-GB billing, a failed request is billed like any other request, because the bytes crossed the wire before anyone knew it had failed. The one near-exception is a connection that was never established — a refused TCP connection or a DNS failure moves close to zero bytes.
Everything else costs money. A 403 costs money. A 429 costs money. A
timeout at 60 percent of the body costs you 60 percent of the body. The
question that actually matters is not whether failures are billed, but how much
each class of failure costs and how many attempts you are willing to buy.
Per-GB metering counts traffic in both directions. The failure is discovered after the transfer, not before it, so the ordering guarantees the charge:
Steps 1 through 5 happened. There is no mechanism by which a provider could un-send them.
The exception sits at step 1. If the connection is refused, or the exit cannot resolve the host, nothing after step 1 occurs. Whether that near-zero exchange appears on your bill at all depends on where the provider reads its meter — that is one of the five questions in what is inside a billed byte.
The practical consequence: fail fast at the connection layer is cheap, fail slow at the body layer is expensive. Every optimization in this article is a version of that sentence.
Take the components one at a time and note which are unavoidable.
DNS. Resolved at the exit, not by you. Normally not billed as your traffic. Confirm rather than assume.
The TLS handshake. Unavoidable and not under your control. The bulk of it is the server's certificate chain, which you did not choose. Connection reuse and TLS session resumption remove most of it on subsequent requests to the same host — which is a reason to keep connections alive even on a job that is mostly failing.
Your request headers. Fully under your control and fully wasted on a failure. This is where a bloated cookie jar hurts: it is sent on every attempt, including the ones that return a challenge page.
Response headers. A challenge response typically carries more headers than a
normal one — several Set-Cookie values, cache directives, security headers.
The challenge body. The expensive component. Bot-management interstitials from the major vendors are not empty responses. They are real HTML documents carrying inline JavaScript that the challenge needs in order to run. You paid for a program designed to prove you are not a program.
No figure is quoted here for how large a challenge page is, because challenge sizes are vendor-specific, versioned, and no one publishing a number for them publishes a method alongside it. Get your own, on your own targets:
# capture the transfer size of blocked responses on one target
for i in $(seq 1 20); do
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
'https://target.example/page'
done | sort -n
Run it until you are being challenged, take the median and the spread across three targets, and record the date. That number is an input to every budget in this article, and it is worth an afternoon.
| Failure class | What crossed the wire | Relative cost | Typical cause |
|---|---|---|---|
| Connection refused / DNS failure | Handshake never started | Near zero | Dead host, blocked port, bad target |
407 auth rejection |
Headers up, headers down | Low | Wrong credentials, exhausted thread limit |
403 / 429 with a challenge body |
Full handshake, headers both ways, complete HTML body | Highest | Bot management, rate limiting, fingerprint mismatch |
| Timeout mid-body | Handshake, headers, partial body | Proportional to how far it got | Slow origin, dropped exit, oversized asset |
Two of these four are worth engineering against.
The 403-with-a-body class is expensive per event and correlates: once a target
starts serving challenges, it usually keeps serving them to the same
fingerprint, so the cost repeats until you change something.
The timeout class is expensive because it is silent. A job that quietly times
out at 60 percent of every large asset can burn a substantial fraction of a
plan while your success counter reports a clean zero for those URLs. Cap
max_response_size and set an aggressive read timeout, and this class becomes
bounded rather than open-ended.
Two different situations get confused here, and they have opposite answers.
Case one: the target is blocking you deterministically. Your per-attempt success probability is effectively zero. Three attempts with exponential backoff cost exactly three times as much as one attempt and produce exactly the same amount of data: none. Backoff spaces the attempts out; it does not make them cheaper. This is the case the retry budget exists to stop.
Case two: failures are probabilistic. Let p be the per-attempt success probability and k the maximum attempts per URL. Then:
1 − (1 − p)^k.(1 − (1 − p)^k) / p.| p | k | Coverage | Attempts per URL | Attempts per success |
|---|---|---|---|---|
| 0.9 | 3 | 99.9% | 1.11 | 1.11 |
| 0.7 | 3 | 97.3% | 1.39 | 1.43 |
| 0.5 | 3 | 87.5% | 1.75 | 2.00 |
| 0.5 | 5 | 96.9% | 1.94 | 2.00 |
| 0.3 | 3 | 65.7% | 2.19 | 3.33 |
| 0.1 | 5 | 41.0% | 4.10 | 10.00 |
Read the last column carefully, because it is the counterintuitive result: attempts per success is always 1/p, no matter how many retries you allow. Raising the retry cap buys coverage. It does not change what a single collected record costs you.
Which means the retry cap is the wrong lever for cost control. The lever is p. Raising success probability from 0.3 to 0.7 cuts your cost per record by more than half. Raising the retry cap from 3 to 5 at p = 0.5 buys nine points of coverage at no change in unit cost — that is often a good trade, and it is a coverage decision, not a budget decision.
The caveat that keeps this honest: failed attempts and successful attempts do not weigh the same. A challenge page and a full product page are different sizes. Weight the arithmetic by your own measured bytes per attempt before you treat these as dollars.
Four rules. All of them are about refusing to buy the same failure twice.
Classify before you retry. A 429 is a timing problem and deserves a
retry. A 403 with a challenge body is a fingerprint or reputation problem and
does not — the same request from the same client will be challenged again. A
404 is a fact. Retrying all non-200s uniformly is the most expensive possible
policy.
Never retry a 403 on the same exit IP. If the exit is burned for that
target, every attempt from it is a purchase of the same challenge page. Rotate
first, then retry. On a rotating port that happens automatically; on a sticky
port it does not, which is exactly when this rule earns its keep.
Cap total attempts per job, not just per URL. A per-URL cap of 3 still allows a job that is failing globally to spend three times its budget. Add a job-level circuit breaker: if the rolling success rate over the last 500 requests drops below a threshold, stop and alert. A job that has started losing is not going to win by continuing.
Make backoff bound the spend, not just the rate. Exponential backoff with
jitter is correct for a 429, because the target is telling you the timing is
wrong. It is theater for a 403, because the target is telling you the client
is wrong.
Same arithmetic as the cost side of billing, framed as a budget line.
To end up with 10 TB of usable data at a 70 percent success rate you are billed for 10 / 0.70 = 14.29 TB. At 90 percent it is 11.11 TB. At 50 percent it is 20 TB. Priced against the residential rate published on the per-GB rates page as of 2026-08-06 — $1.99/GB — that is $28,437 at 70 percent versus $19,900 at 100 percent. The $8,537 gap is failure. Rates change; check the page, not this paragraph.
Put that next to the pool decision. A datacenter pool at a lower rate that succeeds 40 percent of the time against a hardened target is more expensive per record than a residential pool at twice the rate that succeeds 85 percent of the time. Compare datacenter, residential and mobile pools on cost per successful request, never on the rate card alone. The mechanism behind the mechanism is covered in what is inside a billed byte.
The definition does more work here than the measurement.
HTTP 200 is not success. A challenge page is very often served with a 200.
A soft-404 is served with a 200. An empty results grid rendered because your
session was rejected is served with a 200. If your success metric is
status == 200, your success rate is fiction and it is fiction in the
optimistic direction.
Define success as the response contains the field you came for. Then implement the check as a content assertion, not a status check:
def is_success(resp, selector, min_bytes=1024):
if resp.status_code != 200:
return False
if len(resp.content) < min_bytes: # challenge pages are often small
return False
if b"captcha" in resp.content[:4096].lower():
return False
return selector in resp.text # the field you actually parse
Record four counters per target, not one: attempts, HTTP-200s, content-valid responses, and bytes. The gap between counter two and counter three is your challenge rate, and it is the number that predicts your bill.
Track it per target and per hour. Success rate is not a property of your provider — it is a property of the pair (your client, that target) at a point in time, and it degrades over a run as the target learns.
Six changes, ordered by saving per hour of engineering.
Abort the body on a non-200. Stream the response, read the status line and headers, and close the connection before the body arrives if the status disqualifies it. This turns your most expensive failure class into your cheapest.
Cap the response size. Set a hard byte ceiling per request. Anything larger is either an asset you did not want or a failure mode you have not seen yet.
Fail fast on connect. A short connect timeout and a short first-byte timeout convert slow failures into cheap ones. Read timeouts on the body should be tighter than most defaults.
Stop crawling a target that has started blocking you. Circuit-break per target. Continuing produces challenge pages at full price.
Trim your request. Drop cookies you do not need, drop Accept-* headers
you do not use, and stop sending a 4 KB cookie jar to an endpoint that ignores
it. On a job dominated by small responses this is a real percentage.
Request compression and verify you got it. More on this, with the rest of the tactics ranked by measured saving, in nine ways to cut bandwidth spend.
Two documented gateway-side rejections are worth distinguishing from target failures, because they never reach your target at all:
400 NO_RAY — no exit IP matched the filter you requested. This is a
targeting problem. The country, state, city, ZIP or ASN combination has no
available exits at that moment. Widen the filter.407 THREADS_EXHAUSTED — you hit the concurrent connection ceiling on
your plan. See thread limits for the model, and
400 NO_RAY and 407 THREADS_EXHAUSTED for the
diagnostics.Both are refusals at the gateway, before any traffic reaches a target. Which
means they are also the cleanest test of a provider's metering rule: put in
writing whether a gateway-side rejection, and the CONNECT exchange that
preceded it, consume metered traffic. Ask before you build a retry policy that
generates them at volume.
Yes, in almost every case. Per-GB metering counts bytes in both directions, and a request only becomes a "failure" after the response has already arrived. The TLS handshake, your request headers and the response body were all transferred before your code saw the status code. The only near-exception is a connection that was refused or a hostname that failed to resolve, where almost nothing crossed the wire.
A 403 or 429 served with a full bot-management challenge page. Unlike a bare
rejection, a challenge is a complete HTML document containing the JavaScript the
challenge needs to execute, so you pay the handshake, both sets of headers and a
real body — and receive nothing parseable. Timeouts mid-body are second, because
you pay for however much of the response arrived before the connection died.
Against a target blocking you outright, yes: three attempts cost three times as much and return nothing. Where failures are probabilistic, the arithmetic is different. Attempts per successful record equal 1 divided by your per-attempt success rate, regardless of your retry cap. Raising the cap buys coverage, not cost. Raising the success rate is what lowers cost.
Do not use HTTP 200 as the definition. Challenge pages, soft-404s and session-rejected empty grids are all commonly served with a 200 status. Define success as the response containing the field you came to parse, then count four things per target: attempts, 200s, content-valid responses, and bytes. The gap between 200s and content-valid responses is your real challenge rate.
Partly. Stream the response instead of buffering it, inspect the status line and headers as they arrive, and close the connection before the body transfers when the status disqualifies the response. This does not help when the challenge is served with a 200 status, which is common — for those, a hard response-size cap and an early content-type check are the available defenses.
Because your bill includes everything the pages cost you, not just what you kept. Divide billed volume by collected volume: the ratio is roughly 1 divided by your success rate, adjusted for retries, redirect hops, and any subresources a headless browser fetched. A ratio near 1.4 is a 70 percent success rate. A ratio near 3 means something is failing that you have not attributed yet.
Start routing and measure the failure classes on your own
targets. Point a client at gw.anonedge.com:823, log the four counters, and
you will know your real cost per record inside an hour. Then check the
per-GB rates and size the plan against the number you measured,
not the number you hoped for.