What scraping is actually allowed

I have paid for legal advice about scraping exactly once. One hour, one jurisdiction, one specific job. What I remember is that she would not answer until I had broken my question into pieces, because what I walked in with was four questions wearing one coat.

I am not a lawyer and this is not advice

I have never studied law. I run proxy farms and production scraping pipelines out of Singapore, and everything below came from reading terms of service, getting things wrong, and that one paid hour.

This varies enormously by country. It varies by what the data is, and by what you do with it after you have it. Two operators running an identical scraper in two jurisdictions can land in completely different places.

So if there is real money on it, or a client, go and pay somebody who knows your jurisdiction and your use case. An hour of that costs less than one letter from the other side.

I am also not going to quote court cases. I have read the same summaries as everyone else, I have never been in a courtroom, and secondhand case law from a scraper on the internet is worth nothing to you.

The four questions

Can you reach the page without an account.

Do the terms permit automated collection.

What may you do with the data once it is on your disk.

Who owns the content itself.

Those are separate questions. Answering one does not answer the others, and almost everybody who asks me about this has answered the first and assumed the rest came with it.

Public does not mean permitted

“Publicly reachable” means something narrow. No login, no paywall, no token, nothing you had to get past. The server handed the page to an anonymous request that turned up and asked.

That matters, because getting around a control somebody installed on purpose is where the serious exposure sits. It is the difference between walking through an open door and picking a lock.

But it settles one of four. It says nothing about the terms, nothing about redistribution, nothing about who wrote the paragraphs on the page. I treat it as a floor. No means stop. Yes means keep reading.

Robots.txt is a request, and I follow it

Nothing enforces that file. No gate, no token, no consequence at the protocol level. Your fetcher honours it only if you wrote the code that honours it, and plenty of people treat that as licence to skip it.

My reason for reading it is not the one you usually hear. That file is the only channel a site owner has to tell me anything before I arrive. They do not know who I am and they cannot email me. The conversation runs one way and it is the only one on offer.

If I read it and go anyway, I have surrendered any standing to complain about what happens next. Blocks, bans, a letter. I was told, in the only way available to them, and I went anyway.

One exception I will defend and some people will not like. I will fetch a disallowed path once, by hand, before deciding whether the pipeline goes near it. I do not point a crawler down it.

An account is a promise you made

The moment you register, you clicked something, and it usually said accept. A vague question about public data became a specific commitment in writing.

The same applies to an api key, which people find much easier to accept. Nobody argues that api terms are optional. A login is that shape with a worse interface.

So one page fetched two ways sits in two categories. Anonymous, you are a stranger the server chose to serve. Signed in, you are a party who agreed to something and then did the opposite. That second one is far easier to lose, because nobody has to prove intent or damage. They show the clause and they show the traffic.

There is also a person attached to an account. An email, often a phone, sometimes a card. Anonymous scraping is anonymous. Account scraping is you, by name, in somebody’s logs.

Collecting and using are two different jobs

You can be completely fine gathering something and completely wrong publishing it. Plenty of terms permit access and forbid redistribution a few lines apart in the same document.

Ask it as three questions. Can I fetch this. Can I keep this. Can I republish, resell, or train on this.

Internal analysis is the safest by a distance. A price history you use to set your own prices is a different animal from a comparison site you charge people to read. Resale is where I would want real advice every time. The moment somebody is paying for the data rather than for something you built on it, the conversation changes.

Facts and writing are not the same material

Nothing about reachability or terms answers the copyright question.

The rough shape, read by a layman. Facts are hard to own. A price is a fact. A stock count is a fact. A street address is a fact.

Expression is owned. A nine hundred word review somebody sat down and wrote is theirs. A product photograph belongs to whoever shot it.

So I store facts and a reference back to the source, and I leave the prose alone. Copyright is the easiest of the four to get right, which is why scraped article farms irritate me more than they should.

Personal data is where the weight actually is

Names, emails, phone numbers, photographs, profile pages, anything that points at a specific living person. That category is regulated in most jurisdictions with real teeth, and how visible it was does not undo that.

Singapore has the Personal Data Protection Act. Europe has gdpr. California has its own. They differ in the detail and agree on the principle: the fact that anybody could see it does not mean you may collect it and build on it.

The underrated half is holding it. Collecting takes an afternoon. Holding is forever. You have to secure it, know where every copy lives, delete it when somebody asks, and answer for it if a backup ever walks out of the building.

The risk is also lopsided in a way nothing else in this work is. Get a product price wrong and a dashboard is wrong for a week. Get personal data wrong and there is a regulator in the room.

So the rule is blunt. I do not collect personal fields unless those fields are the entire point of the job. Otherwise they get dropped at the parser, before anything reaches storage.

Rate is an ethics question with its own answer

This one sits outside the four and day to day it matters more than any of them. A scraper that costs somebody real money in serving costs is doing harm even where it is entirely permitted.

I have an unusual angle on this because I sell bandwidth. Proxy traffic on my own lines is metered by the gigabyte, so I know what a gigabyte costs to move. When a headless browser loads a full page it pulls the html, the css, every javascript bundle, the fonts and the images, and the site on the other end is paying its own version of that number to send it. Run two hundred of those in parallel and you have moved a line item onto somebody’s invoice.

On a large platform that disappears into the noise. On a small site on shared hosting it is the whole box, and I have watched one of my own go down exactly that way.

So I run slower than I need to. One request at a time per host, a delay that turns an hour into a night, scheduled for their quiet hours rather than mine.

The five rules I operate by

Prefer an official interface. If there is an api I use it, and if it is metered I pay. A paid channel is the site stating in plain numbers how much automated traffic it wants.

Identify yourself. I put a real name and a contact address in the user agent for anything I am not embarrassed by. Most scraping advice says the opposite, blend in and look like a browser, and that was written for a different kind of job. If you would not sign your name to the traffic, you have your answer.

Go slowly, per above.

Take only what you need. Every extra field is something to justify later and something that can be quietly wrong for months.

Be willing to stop. If a site asks me to stop, I stop that day, and I do not argue about whether they were entitled to ask.

The thing I collected and should not have

I was tracking listing prices across sellers on a marketplace. Three fields mattered: price, currency, timestamp.

But I stored the raw json response whole, because that is exactly what I tell everybody to do. Keep the payload, parse from it later, never lose what you fetched.

That response carried the seller profile with it. Display names, rough locations, contact handles, join dates, review histories. I never used one of them. I never really looked. I just had them, on a box in my house and inside every encrypted backup taken since, for months.

The fix took an afternoon. An allowlist at the fetch step so the raw store only keeps declared fields, then a pass over the old objects to strip and rewrite.

What bothers me is that there was never a decision. I did not weigh it up and choose badly. My default was keep everything and sort it out later, and for personal data that default is the mistake.

The one I still cannot answer

When a page fetches its own data from a json endpoint and no account is required to call it, I treat that endpoint as publicly reachable, same as the page. That is my working position.

The argument against it is that the endpoint was never published for anyone but their own front end, so calling it directly is a different act from loading the page. I do not think that argument is stupid. I have just never seen anything that settles it.

So I behave identically either side of that line. Slow, identified, terms read, no personal fields kept. The full set of guides and the tooling I actually run is at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.