Why Your Bandwidth Counter Disagrees With The Vendor Invoice

Every operator who’s run proxies at any real scale has had this argument with a vendor at some point. Your scraper logs 40GB for the month. The invoice says 61GB. Support tells you their meter is correct, points at a dashboard you can’t independently verify, and the conversation stalls. Before you assume anyone’s lying, it’s worth understanding that these two numbers are rarely measuring the same thing in the first place.

The two meters sit in different places

Your application counts bytes at the point where your code touches them: the response body your HTTP client hands back after it’s been decompressed, decoded, and parsed. The vendor’s meter sits upstream of that, usually right at the proxy server or gateway, counting whatever crossed the wire between their infrastructure and the target site, or between their infrastructure and you.

Those are two different vantage points on the same connection, and a lot happens between them that never shows up in your application-level log: TLS handshakes, failed connection attempts, redirect hops, retransmissions, and in some cases the vendor’s own per-request billing floor. None of that is fraud. It’s just that “bandwidth used” isn’t a single well-defined number until you specify where you’re measuring it and what you’re counting.

TLS handshakes aren’t free, and rotation means paying for them repeatedly

Every new TLS connection involves a ClientHello, a ServerHello, a certificate chain from the target site, and a key exchange, all before a single byte of the actual page comes back. That overhead is small per connection, but it’s not zero, and it’s billed as transferred bytes because it genuinely was transferred.

The part that catches people off guard is what rotation does to this. If you’re running a rotating residential or mobile pool and requesting a fresh exit IP on every request, you’re also usually forcing a fresh TLS connection on every request, because the old connection was tied to the old IP. No keep-alive, no session resumption. If your workflow does that thousands of times a day, the accumulated handshake overhead is a real line item, even though your app only ever sees “page 1, page 2, page 3” and never sees the handshakes at all.

A failed request still crossed the wire

This is the one people miss most often. If a request times out, gets refused by the target, or your library aborts it after a partial response, your application often doesn’t log any bytes for it, because as far as your code is concerned, nothing usable came back. But the DNS lookup happened. The TCP handshake happened. In many cases the TLS handshake happened too, and sometimes response headers or a partial body already crossed the proxy before the connection died.

The vendor’s meter, sitting closer to the wire, counted all of that. A pipeline with a high retry rate, whether from flaky mobile carrier links, aggressive target-side blocking, or an unstable modem, generates a meaningful amount of billed traffic that never produces a single row in your scraped-data table. If your retry rate is high, this is often the single biggest source of the gap.

Redirect chains inflate the real total

A lot of client libraries follow redirects transparently and only report the byte count of the final response, or they report the total but attribute it to one logical “request” in your logs. If a target site bounces you through two or three redirects before landing on the real page, each hop is a separate connection with its own headers and handshake, and the vendor’s meter counts every one of them individually. Your dashboard might show “1 request, 8KB,” when the wire actually carried three connections and several kilobytes more.

Minimum billable units turn small responses into fixed costs

Some proxy vendors, particularly on mobile and residential pools, bill with a minimum unit per request rather than metering the exact byte count. If a target returns a 400-byte JSON error and the vendor’s floor is a few kilobytes per request, you’ll be billed the floor. This exists partly because the carrier-side or gateway-side accounting the vendor is reselling often works the same way, and partly because handling a request at all has fixed cost regardless of payload size. It’s worth checking your vendor’s documentation for a stated minimum, because if one exists, a workload with a lot of small responses (status checks, availability pings, lightweight API calls) will look disproportionately expensive relative to the bytes your app actually sees.

Compression direction can go either way

This one can cut against the vendor too, so it’s worth being precise about it. If a vendor meters wire bytes, they’re counting the compressed payload as it actually traveled, gzip or brotli-encoded. Your HTTP client, on the other hand, typically reports the decompressed size after it inflates the response for you to parse. For text-heavy pages, decompressed HTML is commonly several times larger than what actually crossed the wire, which means your app-level number can legitimately be higher than the vendor’s, not lower. If your dashboard number and the invoice disagree, check which direction the gap runs before assuming either side is wrong. Compression is one of the few places where the app-level count can plausibly exceed the metered count.

Buffering proxies fetch before they let you cancel

Some proxy setups, particularly ones that do content inspection or caching at the edge, pull the entire response from the target before streaming it back to you. If your client sets a byte cap or cancels a download early because it only needs the first part of a page, the proxy may have already fetched the full body upstream before your cancel signal reached it. You get billed for what the proxy pulled, not for what your client accepted. This is architecture-dependent and not universal, but it’s worth checking with a vendor whose product does any kind of inline processing, rather than a plain pass-through tunnel.

What actually closes the gap

Chasing this down by comparing two aggregate numbers rarely resolves anything, because aggregates hide exactly the detail you need. The more useful approach is request-level correlation:

Run a small sample of traffic through a local proxy you control, like mitmproxy, sitting between your application and the vendor’s proxy, so you can see wire-level bytes on your own side of the connection, including handshakes and failed connections. Compare that against your vendor’s per-request logs if they expose them, matched by timestamp or request ID rather than by day-level totals. Separately track your retry and timeout rate, since that’s usually the largest hidden contributor and the easiest one to fix by reducing unnecessary reconnects.

If you do that and the gap holds steady in proportion to your traffic, and lines up with your handshake and retry overhead, that’s a metering difference, not a billing error. If the gap is erratic, or shows up on days with no significant traffic at all, that’s worth raising with the vendor directly, with the request-level log in hand rather than a monthly total. A vendor that can’t produce per-request logs to reconcile against is giving you a weaker guarantee than one that can, and that’s a reasonable thing to weigh when you’re evaluating who to route traffic through.

None of this is about proving a vendor is cheating you. Most of the gap between what your code sees and what you’re billed for is explainable by where the two counters sit relative to the wire. Understanding that split is what turns “the invoice looks wrong” into a specific, checkable claim.

If you’re building out scraping infrastructure and want more of this kind of ground-level detail on proxies, pipelines, and the systems you’re actually running, take a look at what else we’ve written on Data Research Tools.

Get new guides and videos first — join the Telegram channel.