Web Scraping Legal Guide 2026: GDPR, CFAA, hiQ vs LinkedIn, and More

scraping publicly available data is generally legal in the United States and most of Europe in 2026. courts have consistently ruled that public web pages do not enjoy CFAA-style “without authorization” protection, and the hiQ Labs v LinkedIn line of cases makes that explicit for scraping. but personal data falls under GDPR even if it is public, ToS violations create separate contract risk, and copyright protects scraped content even when scraping is allowed.

this guide is informational and not legal advice. for any production scraping operation, consult a qualified attorney in the relevant jurisdictions.

the high-level rule

three legal frameworks apply to most scraping:

1. computer access laws (CFAA in the US, Computer Misuse Act in the UK, similar in EU member states): regulate unauthorized access to computer systems
2. data protection laws (GDPR in the EU/UK, CCPA in California, PIPL in China): regulate processing of personal data
3. contract and tort law: ToS breach, trespass to chattels, copyright

each can apply independently. you can be compliant with one and violate another. the safest scrapers map each project against all three.

CFAA and hiQ Labs v LinkedIn

the Computer Fraud and Abuse Act (CFAA, 18 USC § 1030) is the main US law sites have tried to use against scrapers. the key question: does scraping public data constitute access “without authorization”?

the hiQ Labs Inc v LinkedIn Corp case answered no, with caveats. the timeline matters:

• 2017: hiQ sued LinkedIn after LinkedIn sent a cease-and-desist for scraping public profiles. hiQ won a preliminary injunction.
• 2019: 9th Circuit affirmed: scraping public data is not “without authorization” under CFAA.
• 2021: Supreme Court (in Van Buren v. United States) narrowed CFAA generally, supporting hiQ’s reading.
• 2022: 9th Circuit reaffirmed on remand. The hiQ-LinkedIn dispute eventually settled, with the November 2022 final judgment and permanent injunction (final judgment text) finding hiQ liable for breach of contract under LinkedIn’s User Agreement. hiQ was permanently enjoined from scraping LinkedIn data and ordered to delete data already collected.

so the hiQ ruling has two threads:

• CFAA: hiQ won. Scraping public web data is not “unauthorized access.” This precedent stands and is followed in most US courts.
• Contract: hiQ lost. Even though scraping was not a CFAA violation, hiQ had agreed to LinkedIn’s User Agreement (which prohibits scraping), and that agreement was enforceable.

the practical takeaway: scraping public data is not a federal crime in the US, but it can still be a breach of contract if you have a binding agreement with the site (you logged in, accepted ToS, etc.) that prohibits it. the LinkedIn case is the clearest authority on this distinction.

current CFAA precedent post-Van Buren

Van Buren v United States (Supreme Court, 2021) tightened CFAA broadly. the Court ruled CFAA’s “exceeds authorized access” clause applies only to information you have no right to access at all, not to information you have access to but use for an improper purpose.

for scrapers this means:

• accessing public pages: not CFAA-covered, you have authorization
• scraping behind a login that you legitimately have: probably not CFAA, but may be ToS breach
• scraping using stolen credentials, or accessing pages you have been specifically blocked from: CFAA risk

hard cases live in the third category. if a site sends you a cease-and-desist or actively blocks your IP and you continue, courts have sometimes treated that as crossing into “without authorization.” the law is unsettled here. the conservative read: stop when explicitly told to.

GDPR and personal data scraping

if you scrape personal data of EU/UK residents, GDPR applies regardless of where you operate. “personal data” includes anything that identifies a natural person: name, email, photo, online identifier, profile URL.

GDPR requires you have a legal basis to process. the relevant ones for scraping:

• consent: rarely practical (you cannot ask each scraped person)
• legitimate interest: most common basis for B2B scraping. requires balancing test against the data subject’s rights
• legal obligation, vital interests, public task: rarely apply to commercial scraping

even with legitimate interest, you must:

• inform data subjects within 30 days (Article 14) unless that is “impossible or disproportionate”
• honor right to erasure, access, and objection requests
• implement appropriate security
• consider whether the data is “special category” (health, political opinion, sexual orientation, etc.) which has higher protection

some EU data protection authorities (notably France’s CNIL and the Italian Garante) have fined companies that scraped personal data without proper basis. Clearview AI received €20M fines from multiple EU regulators in 2022-2023 for scraping faces from social media.

practical compliance steps:

• exclude EU/UK data subjects where possible
• if you must include them, document your legitimate interest assessment (LIA)
• publish a privacy notice covering scraped data
• maintain a data deletion process that responds within 30 days

CCPA and US state privacy laws

California’s CCPA (and successor CPRA), Virginia’s VCDPA, Colorado’s CPA, and a growing list of other state laws apply to companies that process California/state residents’ data above thresholds. for scrapers:

• if you sell scraped data, that is a “sale” under CCPA and triggers opt-out requirements
• consumers can request deletion, access, and opt out
• “publicly available” data has a narrower exception under CCPA than people often assume; just because it is public on LinkedIn does not exempt it

federal privacy law in the US remains stalled in Congress as of mid-2026. expect continued state-by-state expansion.

copyright and database rights

even when scraping is legal, what you do with the scraped data is a separate question.

United States: data and facts are not copyrightable (Feist Publications v Rural Telephone established this). but creative arrangements, written content, photos, and original prose are copyrighted. scraping articles and republishing them is infringement. scraping prices and aggregating them into a database is generally fine.

European Union: the Database Directive gives database makers a separate “sui generis” right protecting substantial investment in database creation, even when the contents are not copyrighted. scraping a “substantial part” of an EU-protected database can violate this even when the underlying data is factual.

United Kingdom: post-Brexit, the UK retained the EU database rights regime. similar rules apply.

if you scrape and re-publish content (not just facts), you need either a license, fair use/fair dealing defense, or transformative use that does not substitute for the original. AI training has been a hot litigation area here in 2024-2026, with multiple cases pending in US and EU courts.

ToS breach and contract law

most major sites have ToS prohibiting scraping. legally, this matters when:

• you have a binding agreement (clicked “I agree”, created an account, logged in)
• the ToS clearly prohibits scraping
• the site can prove damages

US courts have enforced anti-scraping ToS in some cases (the LinkedIn Final Judgment 2022 against hiQ being the clearest), but ToS-only claims usually result in injunctions (stop scraping) rather than large damages. unless you ignored a cease-and-desist, you usually have time to comply once a dispute escalates.

mere browsewrap (ToS link in the footer that you never clicked) is harder for sites to enforce. clickwrap (you actively agreed) is much stronger.

trespass to chattels and “computer trespass”

a tort theory some sites have used: by sending too many requests, you interfere with the site’s servers (trespass to chattels). courts require actual server impairment to apply this, not just a ToS violation.

the bar is high. a scraper running a few thousand requests a day rarely meets it. but DDoS-style scraping at very high volume has triggered successful claims. rate limiting protects you legally, not just technically.

country-specific notes

United States: most permissive. CFAA narrow per Van Buren and hiQ. ToS enforceable but injunctive relief is the typical remedy. state privacy laws growing.

European Union: GDPR is the binding constraint. database rights add a copyright layer. various national WAFs interpret “necessary processing” differently.

United Kingdom: post-Brexit, mostly aligned with EU but on its own track. UK GDPR and Computer Misuse Act 1990 apply.

Australia: privacy law similar to GDPR-lite. anti-spam regulation strict. Copyright Act 1968 protects content.

Singapore: PDPA covers personal data with a “publicly available” exemption broader than GDPR. Computer Misuse Act applies to unauthorized access.

China: PIPL and the Cybersecurity Law are strict. data export controls add friction. scraping Chinese sites from outside is technically possible but legally fraught.

Canada: PIPEDA and CASL apply. Tucows v ICANN and similar cases have been scraping-permissive but cautious.

the safest pattern for production scraping

1. only scrape public pages without bypassing auth or paywalls
2. respect robots.txt as a courtesy, even though violation is not itself illegal
3. rate-limit to avoid trespass-to-chattels exposure (a few requests per second per domain max is a reasonable default)
4. identify your bot in user-agent if appropriate, or use realistic browser UAs without forging origin
5. honor cease-and-desist and IP blocks; do not work around them once explicit
6. exclude personal data unless you have a documented legal basis
7. respect copyright: scrape facts and structured data freely, but do not republish creative content without a license
8. document your decisions: keep a written record of what you scrape, why, and your legal basis

following all eight puts you in the safest position legally. our comprehensive scraping legal guide goes deeper on each point.

scraping specific big sites: what we know

Amazon: scraping public product data has been litigated and Amazon has lost CFAA claims when the data was public. ToS prohibits scraping, but enforcement is mostly IP blocks. our Amazon scraping legal guide covers this in detail.

LinkedIn: hiQ-style scraping of public profiles is permitted under CFAA but enjoined under contract law for users with accounts. scraping while not logged in is the safer pattern.

Google: SERP scraping violates ToS but is universally done. Google enforces with IP blocks and CAPTCHAs, not lawsuits, for normal-volume use.

Twitter/X: post-2023 API changes, scraping is more legally fraught. X has been aggressive with cease-and-desists. proceed cautiously.

Reddit: post-2023 API pricing change, scraping public threads is technically allowed but ToS prohibits it. Reddit has not been litigious historically.

Meta (Facebook, Instagram): very aggressive. multiple lawsuits won under CFAA-adjacent state laws and ToS. high risk for commercial scraping.

AI training data: a special case

scraping data to train AI models is a 2024-2026 hot legal topic with no settled answer. major lawsuits include:

• New York Times v OpenAI/Microsoft (filed Dec 2023): copyright infringement claim over training data
• Getty Images v Stability AI: image scraping for diffusion model training
• Authors Guild class actions: training on copyrighted books
• various artist class actions: training on artwork

courts have not yet given clear guidance. fair use arguments are central to defenses. the EU AI Act (effective 2024-2026) requires training data transparency for general-purpose models. expect more rulings and regulation in 2026-2027.

if you scrape for AI training, document sources, exclude opted-out content (most major sites now have AI-specific opt-outs in robots.txt and ai.txt), and consult counsel before commercial deployment.

faq

is scraping illegal?
in most cases, no. scraping public data without bypassing auth is legal in most jurisdictions. specific data types (personal data, copyrighted content) and specific sources (sites where you have a binding ToS prohibiting it) carry separate legal risk.

did hiQ Labs really beat LinkedIn?
on CFAA, yes. on contract (ToS), no. the November 2022 final judgment found hiQ in breach of LinkedIn’s User Agreement and permanently enjoined hiQ from scraping LinkedIn. so the hiQ case actually establishes both that scraping public data is not CFAA-illegal and that ToS breaches can still be enforced separately.

what about the EU AI Act?
applies from 2024 with full force in 2026-2027. most relevant for scrapers training general-purpose AI: training-data transparency requirements, copyright opt-out respect, and risk classification of AI systems. does not directly regulate scraping itself.

can I scrape data and sell it?
depends what data and which jurisdictions. selling scraped factual data (prices, business listings, public records) is generally legal in the US and often in the EU subject to GDPR if personal data is involved. selling scraped copyrighted content (articles, photos) is infringement.

do I need to respect robots.txt?
not legally required in most jurisdictions, but courts have cited robots.txt non-compliance as evidence of bad faith. respect it where reasonable; document why if you do not.

should I use a real browser to avoid legal liability?
no. browser vs HTTP client does not change the legal analysis. what matters is whether you bypass auth, what data you collect, and your purpose.

am I liable if my scraper accidentally hits a private endpoint?
possibly. unauthorized access claims focus on what you knew or should have known. discovering an exposed private endpoint by accident, then continuing to scrape it after realizing, is risky. stop and notify if you see something that looks like a leak.

conclusion

web scraping in 2026 is mostly legal mostly of the time, but “mostly” carries real risk. the CFAA does not generally apply to public-data scraping in the US after hiQ and Van Buren. GDPR creates a bigger constraint when personal data is involved. ToS breach is an ever-present contract risk if you have an account on the target site. copyright applies to what you do with what you scraped, separate from the scraping itself.

the safe path is: scrape public data, respect explicit blocks and cease-and-desists, exclude or carefully justify personal data, do not republish copyrighted content, and document your decisions. for any commercial operation, talk to a lawyer in your jurisdiction before scaling. this guide is informational and is not a substitute for legal advice on your specific case.

Two clients open a connection to the same site. One of them gets the page. The other gets a block, and it gets that block before it ever sends a request you would recognize as a request. No url, no headers, nothing a scraper author usually thinks of as “the traffic.” The decision was already made during the encrypted handshake that every https connection has to do first. That is tls fingerprinting, and it is one of the earliest and cheapest checks a site runs.

I run proxy infrastructure and production scrapers, so I spend a lot of time on the defending side of this, reading how detection works from the site’s point of view. This is a defensive explainer, not a guide to faking anything. The useful part is understanding why the handshake is such a strong signal, and why the durable answer to it is a client that is honestly what it claims to be rather than one dressed up to look like something else.

The handshake happens before anything else

Before any encrypted data can move, your client and the server have to agree on how to encrypt it. That negotiation is the tls handshake, and it starts with the client speaking first, in the clear, with a message called the client hello. Nothing is secret yet, because the secure channel does not exist until this exchange finishes building it.

The important thing is that the client hello is not the same for every piece of software. It is a detailed proposal, and the details vary depending on what created it. A browser assembles one shape of client hello. A scripting library assembles a different one. All of it travels on the very first packet, in plain view, before a single byte of real content is exchanged.

What the client hello gives away

Several fields in that opening message describe the client without ever naming it. The list of cipher suites, which are the encryption methods the client is willing to use. The set of extensions, the optional features it asks for, including the server name it wants in an extension called sni. The supported groups and signature methods. And, more revealing than any single value, the exact order all of these appear in.

None of that is meant to identify you, but together it does. Chrome offers a particular set of cipher suites in a particular order. Firefox offers a slightly different set arranged a different way. A plain http library that was written just to open a connection tends to offer something thinner or unusual, because nobody tuned it to resemble a browser. So the lineup and the ordering of these fields already sort clients into rough buckets before the page loads.

ja3 and ja4, turning a handshake into an id

Reading a full handshake by eye does not scale, so people built ways to compress it. ja3 is a method that takes several of those client hello fields, the tls version, the cipher list, the extensions, and the supported groups, joins them into one string, and hashes that down to a short fingerprint. Two clients that negotiate the same way land on the same ja3 value. A given browser version tends to produce a known fingerprint, and so does a given scripting library.

ja3 had a weak spot. Browsers began randomizing the order of some extensions on purpose, which made the simple hash jump around and lose its meaning. ja4 was designed to be sturdier. Instead of one brittle hash it builds a more structured fingerprint that stays stable when parts of the handshake shuffle, and it captures a bit more of the picture. The direction matters more than the mechanics: the measuring got more robust, not less, and it still reads only fields the client volunteers at the very start.

The contradiction that gets scrapers sorted

Here is where it meets scraping. A common automated client sets a user agent string that says it is a current browser. That field is just text, so you can put anything in it. But the same client still negotiates tls through whatever library sits underneath it, and that library was never trying to imitate a browser. So the handshake produces a fingerprint that reads as generic scripting tool, while the header a moment later claims modern Chrome.

Those two things do not agree, and detection lives in exactly that gap. The site is holding a client that connects one way and describes itself another way. It does not need to prove intent or run an expensive test. It just noticed that the tls signature and the stated identity are telling two different stories, and that disagreement is enough to drop the connection into a stricter lane.

Why the tls stack, not the user agent, decides

The reason this is hard to paper over is that the two signals come from different places in the stack. The user agent lives in the request, a line of text your code writes. The client hello is produced far below that, by the encryption library your language and tooling are built on, something like OpenSSL or an equivalent. Setting a browser user agent changes the label. It does not change the handshake, because the handshake was generated by the layer underneath, and the site is reading that layer.

This is why the fingerprint is so reliable. A real browser makes the two agree effortlessly, because one real piece of software produced both. Anything assembling an identity from separate parts has to keep every layer consistent, and the tls layer is the one most people forget is even speaking.

What consistency looks like from the honest side

If you have read how browser fingerprinting works, this is the same principle one layer down. Detection is not hunting for a single forbidden handshake. It is checking whether the handshake agrees with everything else the client claims: the headers, the user agent, the behavior that follows. A request from a residential or mobile network with a browser handshake and browser headers that all match is boring, and boring is what gets through.

So the honest path sidesteps the whole problem instead of fighting it. If a job genuinely needs a browser, drive a real one, because a real browser produces a real browser’s handshake with no imitation required. If a job does not need a browser, let the client be honestly a scripting tool and identify it where the site expects that. And when a site offers an official api or a data feed, prefer it, because that front door does not care about your handshake at all. A client that is what it says it is never creates the contradiction, so there is nothing for the fingerprint to catch.

The honest limits

I want to be straight about the boundaries. Nothing here makes a client undetectable, and I would not believe anyone who told you it did. tls fingerprinting keeps improving. ja4 exists because the measuring got sharper, and what slips by today can be flagged tomorrow. Trying to match a browser’s handshake exactly is a treadmill, not a fix. Browsers change their tls behavior as they update, imitations drift out of date, and an almost right handshake that matches no real browser version becomes its own tell.

The durable version of this work is not a better disguise. It is not needing one. The moment collection leans on pretending to be a browser it is not, or reaches for private data, or ignores what a site clearly asked, it has left the compliant lane, whatever the tooling looks like. Public data, honest identification, a gentle pace, and the rules respected, that is the version that is still running a year from now.

I run this infrastructure in production, so the handshake, the fingerprinting, and the compliant patterns I write about are the ones I actually deal with. If you want the full written guides and the way this works in practice, read them at dataresearchtools.com. No undetectable promises, no bypass how to, just how the handshake is actually read.

Get new guides and videos first — join the Telegram channel.

what is IP reputation? how trust scores affect proxy blocking

IP reputation is a numerical score (usually 0-100) that anti-fraud and anti-bot systems assign to every IP address based on its history of abuse, hosting type, and traffic patterns. low-reputation IPs get blocked, captcha’d, or shadowbanned. high-reputation IPs pass through cleanly. if you scrape, run multi-account workflows, or rely on proxies, IP reputation is the single biggest factor that decides whether your traffic survives.

what makes an IP have “good” or “bad” reputation

every major fraud database (MaxMind, IPQualityScore, Spamhaus, IP2Proxy, Sift, Forter) tracks IPs across thousands of signals. the headline factors are simple.

was the IP recently flagged for spam, brute-force logins, credential stuffing, or scraping? did it appear on a public blocklist (Spamhaus DROP, AbuseIPDB)? is it a datacenter IP from AWS, Hetzner, OVH, DigitalOcean, or known proxy AS networks? has it been used by Tor or VPN exit nodes?

the cleanest IPs are residential addresses assigned by ISPs to home internet customers, plus mobile IPs from carriers (Verizon, AT&T, Singtel, Vodafone). these get the highest trust scores because real humans use them, share them via CGNAT, and don’t generate bot traffic.

datacenter IPs from cloud providers sit at the bottom. they make up most cheap proxy pools, and anti-bot systems block them on sight for any high-value target like Amazon, Google, or LinkedIn.

who scores IPs and how

four major commercial databases dominate IP reputation in 2026. each anti-bot vendor (Cloudflare, DataDome, Akamai, PerimeterX) blends them with proprietary signals.

provider	focus	typical use
MaxMind minFraud	fraud detection	e-commerce checkout, account creation
IPQualityScore	bot + proxy detection	ad networks, registration forms
Spamhaus	email spam, malware	mail servers, blocklists
IP2Proxy	proxy/VPN detection	content geolocation, fraud
AbuseIPDB	community reports	open source, security teams

scoring is dynamic. an IP can have a 95 score today and a 20 score tomorrow if a bot operator burns it on a credential-stuffing campaign. residential proxy networks deal with this constantly because customers share the same pool.

scores feed into anti-bot pipelines like Cloudflare’s bot management or Akamai Bot Manager. those systems combine IP reputation with TLS fingerprint, browser fingerprint, and behavioral signals to decide whether to allow, challenge, or block your request.

for a deeper look at the proxy types most affected, see our residential proxy guide and the full proxy type breakdown.

how IP reputation shows up in practice

three things happen when your IP has poor reputation.

outright block (HTTP 403, Cloudflare 1020). the request never reaches the application. you see a generic “access denied” page. this is the loudest signal but also the easiest to detect and route around with a fresh IP.

captcha challenge (hCaptcha, reCAPTCHA, Turnstile). the page loads but every form submission triggers a puzzle. you can solve them with services like 2Captcha or CapSolver, but at scale this gets expensive fast.

silent shadow ban. the request returns 200 OK with fake or stale data. Google does this. so does LinkedIn. you only notice when your scraped data looks wrong, which can take days. shadow bans are the worst because nothing in your logs tells you what happened.

how to check your IP reputation

before you trust a proxy provider’s claims, test their IPs yourself. several free tools give a quick read.

ipqualityscore.com/free-ip-lookup-proxy-vpn-test gives a 0-100 fraud score plus proxy/VPN flags. spur.us shows whether the IP appears in known proxy networks. scamalytics.com gives a clean visual breakdown.

for bulk testing, IPQualityScore’s API costs about $0.001 per lookup. run 100 IPs from your provider’s pool and compare scores. if more than 10% score above 75 (high risk), the pool is dirty.

residential and mobile proxies should score under 25 on most tools. if they score in the 50s, the network has been abused and you’ll see blocks soon.

why mobile proxies usually win

mobile carriers assign IPs via carrier-grade NAT, which means hundreds or thousands of real users share the same IP at the same time. blocking that IP would cut off legitimate paying customers, so anti-bot systems rarely do it.

this is why mobile proxies cost 5-10x more per GB than residential ones. the IP reputation is structurally cleaner because of how mobile networks work, not because the proxy provider does anything special.

for sites like Instagram, TikTok, OnlyFans, or Telegram, mobile IPs are the practical floor. residential works for medium-difficulty targets. datacenter only works for low-difficulty pages like static blogs or public APIs.

if you’re choosing between provider tiers, our proxy provider comparison ranks the major networks by IP quality, plus the rotating proxy guide covers when rotation helps and when it hurts.

how to keep your proxy IPs clean

three habits separate scrapers who survive from scrapers who get burned constantly.

rotate at the right pace. rotating per request defeats most session-based detection but burns through clean IPs fast. rotating per session (10-30 minutes) builds session trust. match your rotation policy to the target site’s session model.

respect rate limits. a clean residential IP that fires 100 requests per second to amazon.com becomes a dirty IP within minutes. throttle to human-realistic rates (1-3 requests per second per IP) and your pool stays healthy.

don’t reuse IPs across targets. an IP that scraped LinkedIn yesterday already has reduced trust at LinkedIn today. some providers offer “sticky session per domain” where you get a fresh IP per target site. that’s the gold standard for multi-site operations.

faq

is IP reputation public or private data?
mostly private. companies like MaxMind sell access to their databases. some lists (Spamhaus DROP, AbuseIPDB) are public. anti-bot vendors blend several sources plus proprietary signals, so even checking your score on free tools won’t tell you exactly what Cloudflare or DataDome will see.

can I improve a bad IP’s reputation?
not directly, and not quickly. reputation databases update over weeks or months based on observed behavior. if an IP gets flagged, the cleanest fix is to swap to a new one. proxy providers that “burn” an IP usually quarantine it for 30-90 days before recycling.

why do my datacenter proxies work for some sites but not others?
sites have different sensitivity. a static blog cares only about volume. Cloudflare-protected sites filter on IP type before anything else. e-commerce checkout flows weight reputation heavily because fraud is expensive. match proxy type to site difficulty, not the other way around.

is a 0 score better than a 100 score?
depends on the tool. on most fraud databases, 0 means low risk and 100 means high risk. on quality-of-IP tools it can be the opposite. always read the legend. IPQualityScore uses 0-100 where higher is worse. spur.us flags risk categories rather than a numeric score.

do residential proxies always have clean IPs?
no. residential proxy pools share IPs across thousands of customers, so a single bad actor can burn an IP for everyone using that pool. premium providers actively monitor and quarantine flagged IPs. budget providers don’t, which is why their pools degrade fast.

conclusion

IP reputation decides whether your proxy traffic looks human or fraudulent to the systems guarding modern websites. it blends abuse history, network type, and live behavior into a score that determines whether you get blocked, challenged, or allowed.

the practical takeaway is simple. for tough targets, use mobile or premium residential IPs from a provider that monitors pool health. test reputation before you commit. throttle requests to human pace. rotate sessions, not requests, where session trust matters. these four habits beat 90% of cheap proxy buyers who treat all IPs as interchangeable.

reputation isn’t something you can fake. you either bring clean IPs to the fight, or you spend your day debugging why your scraper sees captchas on every page.