HTTP/2 and TLS session reuse for scrapers: what connection-level behavior actually costs you

Why connection setup matters more than people think

Most scraping post-mortems focus on the request layer: headers, user agents, rate limits. Fewer people look one layer down, at what happens before a single byte of HTTP is exchanged. Every HTTPS request starts with a TCP handshake and a TLS handshake, and at scale, that setup cost is often bigger than the cost of the actual HTTP exchange. If you’re running a pipeline that fires thousands of requests an hour through a proxy pool, the difference between negotiating a fresh TLS session on every request and reusing one is a real, measurable chunk of your latency and CPU budget.

This isn’t a bypass technique. It’s plumbing. But plumbing has consequences for both performance and for how a target site’s defenses read your traffic, so it’s worth understanding the mechanics rather than treating requests.get() as a black box.

What a TLS handshake actually costs

A full TLS 1.2 handshake takes two round trips before any application data moves: client hello, server hello plus certificate plus key exchange, then the client’s key exchange and finished message. TLS 1.3 collapsed that to one round trip in the common case, and added a 0-RTT mode for resumed connections where the client can send application data in its very first flight, though 0-RTT has its own replay-risk caveats that most server operators disable or restrict.

On top of the round trips there’s real CPU work: asymmetric key exchange (typically ECDHE), certificate parsing and chain validation, and key derivation. None of this is expensive for a single request. It becomes expensive when you’re doing it tens of thousands of times a minute across a fleet of proxy exit nodes, especially if each exit node is a resource-constrained box like a mobile modem gateway rather than a beefy cloud VM.

Session resumption: session IDs, session tickets, and PSK

TLS has had a resumption mechanism since TLS 1.0 in the form of session IDs: the server caches session state keyed by an ID it hands the client, and the client can present that ID on a later connection to skip the full handshake. Session tickets (RFC 5077) moved that state out of server memory and into an encrypted blob the client stores and replays, which scales better across a load-balanced server fleet since no shared session cache is needed.

TLS 1.3 folded both mechanisms into a single PSK (pre-shared key) resumption model. After a full handshake, the server can issue one or more session tickets. On a later connection, the client offers a ticket, and if the server accepts it, the handshake becomes an abbreviated one-round-trip (or 0-RTT) exchange instead of the full negotiation.

For a scraper, the practical effect is: if your HTTP client reuses a TLS session across requests to the same host, you pay the full handshake cost once and then get cheap resumed handshakes for a while, until the ticket lifetime expires. If your client (or your proxy layer) tears down and rebuilds the TLS connection on every request, you’re paying full handshake cost every single time, which shows up as higher latency and higher CPU load on both ends of the connection.

HTTP/2 multiplexing and why connection reuse compounds the win

HTTP/1.1 scraping code often opens one connection per request, or relies on a small keep-alive pool. HTTP/2, negotiated via ALPN during the TLS handshake, changes the shape of the problem: a single TCP+TLS connection carries multiple concurrent streams, each an independent request/response pair, multiplexed over one wire with HPACK header compression to cut repeated header overhead.

That means the payoff from TLS session reuse and HTTP/2 connection reuse stack. Instead of N connections each paying handshake cost, you get one connection amortizing its handshake cost across N logical requests. For a pipeline scraping paginated listings or repeated API calls against the same host, this is often the single biggest lever for reducing wall-clock time and reducing the load you place on the target’s server, which matters both for your own throughput and for not being the reason a site’s ops team starts rate-limiting a subnet.

The tradeoff is operational complexity. A connection pool keyed by host, with sane idle-timeout and max-requests-per-connection settings, is more code than “open a socket per request,” and it requires your scraping framework and your proxy layer to actually agree on connection lifetime. If your proxy silently rotates the upstream IP mid-session, or your HTTP client and your proxy pool have different ideas about when a connection is stale, you can end up with confusing failures that look like server-side blocking but are actually your own connection-reuse logic falling over.

The fingerprinting side, explained defensively

Connection-level behavior is also one of the harder things to control, which is exactly why it’s used defensively by anti-bot systems. A TLS ClientHello carries the cipher suite list, extension list and order, supported groups, and signature algorithms in a specific sequence. That sequence is fingerprintable (JA3 is the best-known hash of this), and most HTTP client libraries (a stock requests/urllib3 stack, a given version of Go’s net/http, curl-impersonate, browser TLS stacks) produce a distinctive, stable fingerprint. Bot-detection vendors compare an incoming ClientHello’s fingerprint against a known-bad list of automation libraries, and separately watch for a mismatch between what the TLS fingerprint claims (a browser) and what the HTTP layer reveals (missing browser-only headers, a User-Agent string that doesn’t match the negotiated TLS profile).

HTTP/2 adds its own fingerprint surface on top of TLS: the order and values of the SETTINGS frame, the WINDOW_UPDATE behavior, and the order of HTTP/2 pseudo-headers (:method, :authority, :scheme, :path) in a request. Real browsers have consistent, well-documented orderings here; many HTTP client libraries either don’t support HTTP/2 pseudo-header ordering configuration at all or use an order that doesn’t match any real browser, which is itself a usable signal for detection systems, independent of anything at the TLS layer.

It’s worth being precise about what this means and doesn’t mean. Matching a browser’s TLS and HTTP/2 fingerprint is not a way to make traffic “undetectable.” Detection systems that care about this layer generally combine it with other signals: request timing, IP reputation, behavioral patterns, and JavaScript-side checks that a plain HTTP client can’t answer at all. A consistent, resumed TLS session that behaves like a real browser at the connection layer removes one signal from the pile; it doesn’t remove the pile. Sites that invest in this kind of defense are explicitly trying to make automated traffic expensive to disguise, and no proxy or client configuration changes that fact.

What this means for pipeline design

The practical takeaway for anyone running a production scraping pipeline is to treat connection reuse as a performance and reliability problem first. Configure your HTTP client to negotiate ALPN for HTTP/2 where the target supports it, keep TLS sessions and HTTP/2 connections alive for a sensible pool of requests per host, and make sure your proxy layer’s session affinity (how long a given proxy IP stays assigned to a given session) is consistent with your HTTP client’s connection lifetime. Tearing down a TLS session on every request while paying for premium residential or mobile proxy bandwidth is a straightforward way to waste both money and time.

Do this with realistic expectations. Session reuse and protocol-correct behavior reduce operational cost and remove one class of easy tells. They are not a guarantee against blocking, and no vendor or configuration can promise that in good faith. Any site that has invested in bot detection is looking at more than your connection fingerprint, and treating this as a checklist item to “beat” detection rather than an infrastructure efficiency question is the wrong frame.

If you’re building or auditing a scraping stack and want to see how these pieces fit together with proxy infrastructure and orchestration, head back to the Data Research Tools home page for the rest of our tooling breakdowns and infrastructure guides.

Get new guides and videos first — join the Telegram channel.