How many proxy threads do you actually need?

How many proxy threads do I need? Every vendor help desk answers that question with a ceiling — 100 per proxy, 500 per plan, 10,000 on this tier — and none of them answer the question that was asked. A ceiling tells you what you are not allowed to exceed. It does not tell you what your job requires.

The number you need is arithmetic, and it takes one line:

threads = target requests per second x average response time in seconds

That is Little's Law, applied to a request pool. Everything else in this article is either how to get honest inputs for it, or what breaks when you get the answer wrong in either direction.

Threads, concurrent connections and workers are one limit under three names

Kill this ambiguity first, because the SERP for this question is full of pages that never do.

A thread, in every proxy provider's usage, is one connection open through the gateway at one instant. It has nothing to do with operating system threads, and nothing to do with how many CPU cores you have. Your crawler can run 200 green threads, 200 coroutines or 200 OS threads and still hold exactly 8 connections open, because 192 of them are parsing, sleeping or waiting on a queue.

The three words you will see mean the same thing:

• Thread — the provider's billing and limit term.
• Concurrent connection — the same quantity, named accurately.
• Worker — your side's name for whatever holds one.

What counts against the limit is a socket in flight. A sticky session holds one for as long as you hold the session, whether or not it is transferring bytes, which is why sticky work hits a ceiling that rotating work of the same volume does not. That mechanism is in rotating and sticky session ports.

How many proxy threads do I need? Work the formula twice

Little's Law states that the average number of items in a system equals the arrival rate multiplied by the average time each item spends in it. For a request pool, items are in-flight requests, arrival rate is your throughput target, and time in system is your response time.

Job one. You need 200,000 pages in an 8-hour window.

200,000 / (8 x 3600) = 6.94 requests per second

If your average response takes 1.8 seconds:

threads = 6.94 x 1.8 = 12.5  ->  13

Thirteen. Not 100, not 500.

Job two. Same 200,000 pages, same window, but the target is slower and the average response takes 6 seconds:

threads = 6.94 x 6 = 41.7  ->  42

The workload did not change. The response time tripled and the thread requirement tripled with it. This is the sensitivity nobody warns you about: your thread requirement is a property of the target, not of your crawler. A target that slows down under load will silently raise your requirement while you are running, which is exactly when you cannot afford to hit a ceiling.

Run the formula in the other direction to find your ceiling in useful units:

requests per hour = 3600 x threads / average response seconds

At 20 threads and a 3-second average, that is 24,000 requests per hour. If your plan tops out at 20 threads and you need 40,000 per hour, no amount of tuning gets you there. You need a higher limit or a faster target.

Measure your average response time before you guess

Two numbers, not one.

The mean sizes the pool. The p95 tells you whether the pool stays sized. If your mean is 1.8 seconds and your p95 is 6 seconds, then during the tail every slot is occupied for more than three times as long as the formula assumed, and throughput collapses to a third while the tail lasts. Size on the mean, then check that you can tolerate 3600 x threads / p95 as a floor.

Measure through the proxy, not against the origin directly. The path you are sizing includes the gateway, the exit and the return leg. A timing taken from your laptop to the target measures a route your crawler never takes.

Instrument three timestamps per request: dispatch, first byte, last byte. Mean and p95 of last-byte-minus-dispatch is what goes in the formula. Anything shorter understates your requirement.

Which binds first, threads or gigabytes?

This is the question a per-GB plan actually turns on, and it appears nowhere in the vendor help-desk pages that own this SERP. Threads cap your rate. Gigabytes cap your volume. Whichever runs out first is your real limit, and it is usually not the one you have been tuning.

Requests per hour is 3600 x threads / seconds. Multiply by 24 for a day, then by average response size for bandwidth. Two response sizes below: 22 KB, the HTML component of the median home page in the HTTP Archive's July 2025 crawl (Web Almanac 2025, Page Weight), and 250 KB for a heavier document or a small API payload. Gigabytes are decimal here, 1 GB = 1,000,000 KB.

Threads	Avg response	Requests/hour	Requests/day	GB/day at 22 KB	GB/day at 250 KB
5	8.0 s	2,250	54,000	1.2	13.5
10	1.0 s	36,000	864,000	19.0	216.0
20	3.0 s	24,000	576,000	12.7	144.0
50	3.0 s	60,000	1,440,000	31.7	360.0
100	5.0 s	72,000	1,728,000	38.0	432.0

Read the last two columns as a monthly bill and the table stops being abstract. At 20 threads, a 3-second average and 250 KB responses, a job that runs around the clock consumes roughly 4,320 GB a month. At 22 KB responses the identical concurrency consumes roughly 380 GB. Same threads. An 11x difference in spend, driven entirely by what you chose to download. What exactly lands inside those gigabytes — headers, TLS, redirects and failures included — is in how bandwidth is metered.

Which means the useful conclusion is the opposite of the one the SERP gives you. If your responses are heavy, gigabytes bind first and adding threads just spends your budget sooner. If your responses are light, threads bind first and your bandwidth budget sits idle. Work out which side you are on before you tune anything, and check the number against per-GB rates for the pool you are running on.

Symptoms of too many threads

More concurrency past the binding constraint does not add throughput. It moves your failures around.

• 407 THREADS_EXHAUSTED from the gateway. The connection ceiling on your plan was reached. Requests are rejected at the gateway, not at the target, and they are rejected instantly — which makes retry storms cheap to trigger and expensive to recover from. The error and the plan model are in thread limits on your plan, and the wider list is under 407 THREADS_EXHAUSTED and other gateway errors.
• Rising 429s from the target. The origin's rate limiter, not yours. This one costs you bytes, because you paid for the request and the response.
• Latency collapse. Mean response time climbs as you add workers. The target is queuing you. Every additional thread now occupies a slot for longer, so effective throughput falls while your thread count rises.
• Success rate that falls as you add workers. The clearest signal there is. Plot success rate against concurrency. If the curve turns down, you passed the optimum some workers ago.

The retry logic makes all four worse. A naive "retry three times immediately" policy converts one rejection into four connection attempts, each of which occupies a slot. Use exponential backoff with jitter, cap total attempts, and treat 407 differently from 429 — a 407 means back off globally, because every worker is hitting the same ceiling at the same moment. A 429 means back off for that host only.

Symptoms of too few

Less obvious, and more expensive over a quarter.

• A queue that grows monotonically. Your producer outpaces your consumers and the backlog never drains. Plot queue depth; a straight upward line is under-provisioned concurrency, not a slow target.
• A job that never finishes inside its window. Overnight crawls that are still running at 09:00.
• An idle bandwidth budget. You bought gigabytes you are not spending because you cannot generate requests fast enough to spend them.

The fix is not always more threads. If your average response is 6 seconds and your p95 is 40, you are sized for a target that is timing out. Cut the timeout before you raise the limit — a 30-second timeout on a target with a 2-second mean is a slot held hostage by a request that was never going to complete.

Threads change your IP footprint

Concurrency is not only a throughput knob. On a rotating port, every simultaneous connection is a separate exit address, so 50 threads means the target sees up to 50 distinct addresses hitting it at once. That looks different from 50 sequential requests from 5 addresses, and on some targets it looks worse.

Two consequences worth planning around:

• Rotating plus high concurrency spreads volume per address thin, which is what you want against per-IP rate limits and not what you want against targets that score bursts of first-time visitors.
• Sticky plus high concurrency is a different animal. Each held session occupies a slot for its whole window. Ten workers on ten sticky ports for 30 minutes is ten threads consumed for 30 minutes regardless of how few requests they send. Budget slots for held sessions separately from slots for active transfers.

Which of the two you should be running, and what each does to how the target sees you, is the subject of session model and IP footprint.

The setting that actually controls concurrency, per tool

Raising a provider's thread limit does nothing if your client is capped lower. These are the knobs that decide how many sockets you actually open.

Tool	Setting	Notes
Scrapy	CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN	The per-domain value is the effective cap on a single-target crawl. Raising the global one alone changes nothing.
aiohttp	TCPConnector(limit=..., limit_per_host=...)	Defaults to 100 total and unlimited per host. Set both explicitly.
httpx	Limits(max_connections=..., max_keepalive_connections=...)	Keepalive connections still occupy a slot at the gateway.
undici (Node)	new Agent({ connections })	Per-origin. Left unset it is unbounded, so your real concurrency is whatever your task loop allows. Set it explicitly.
Playwright	Browser contexts, and pages within a context	Each page holds its own connections. A context with 10 open pages is not 1 thread.
Selenium	Driver instances	One driver, one browser, many sockets. Size by driver count, not by request count.

Two rules that follow from the table. First, a browser-driven worker consumes several connections per page, not one, so the formula's output is a floor for browser work rather than a target. Second, connection pooling and keepalive mean an idle-but-open connection still counts — set your pool sizes to the number you sized for, not to the library default.

Raising the limit

If the formula says you need more than your plan allows, and you have already checked that gigabytes are not the binding constraint, ask for an increase. Bring three things:

1. The formula's output and the inputs you measured — throughput target and mean response time.
2. What you are collecting and from which target categories.
3. Your session model, because sticky work holds slots differently from rotating work.

Increases are handled per plan and may require KYC. Start at request a higher thread limit.

Frequently asked questions

How many proxy threads do I need for web scraping?

Multiply the requests per second you need by your average response time in seconds. A job needing 7 requests per second against a target averaging 1.8 seconds needs 13 concurrent connections. The same job against a 6-second target needs 42. There is no universal number, because the requirement is set by the target's response time, not by your crawler.

What does 407 THREADS_EXHAUSTED mean?

Your account reached its concurrent connection ceiling and the gateway rejected the request before it reached an exit. It is not a target block and not an authentication failure despite the 407 status. Reduce concurrency, add backoff that pauses all workers rather than one, and confirm that held sticky sessions are not consuming slots you assumed were free.

Do more threads make my proxies faster?

No. Threads control how many requests are in flight, not how fast any one of them completes. Past the point where the target starts queuing you, adding workers increases mean response time, which reduces throughput and raises your thread requirement at the same time. Plot success rate against concurrency and stop at the peak.

Do sticky sessions use more threads than rotating ones?

They use the same number of slots but hold them longer. A sticky session occupies a connection for the duration of its window even during idle periods between requests, while a rotating request releases its slot on completion. Ten long-lived sticky sessions can therefore exhaust a limit that would comfortably carry a far higher rotating request rate.

Should I size threads or bandwidth first?

Bandwidth, if your responses are heavy. Calculate 3600 x threads / average response seconds x 24 x average response size and compare it against your monthly gigabytes. If that number exceeds your budget, gigabytes are your real limit and extra threads only spend the budget faster. If it is well under, threads are the constraint and it is worth asking for more.

Size it, then ask

Measure your mean and p95 through the gateway. Run the formula. Compare the resulting daily gigabytes against what you are willing to spend. Only then decide whether the limit is the problem.

Talk to support about a limit increase at /contact, and check the per-GB rates before you raise the number.