Running a scraper inside a client’s own cloud account

Why the account boundary matters more than the code

When you build a scraping pipeline for a client, one of the first decisions is where it actually runs. Most vendors default to running it on their own infrastructure and handing over a dashboard or a data feed. But a growing number of engagements, especially with larger companies or anyone in a regulated industry, ask for the opposite: the scraper runs inside the client’s own cloud account. Same code, same proxies, same schedule, different owner of the compute and the bill.

This isn’t a cosmetic change. It shifts who holds the IAM keys, who sees the logs, who pays the invoice, and who is accountable if something in the account gets misused. If you run pipelines for other people, it’s worth understanding exactly what moves and what doesn’t when you switch from “our servers” to “their account.”

What actually changes

The core reason clients ask for this is control and continuity, not detection. Three things genuinely change:

Billing and ownership. Compute, storage, and networking costs land on the client’s cloud bill directly. There’s no markup buried in a vendor invoice, and the client can see exactly what the scraper costs to run on any given day.

Data residency. If the client operates under GDPR or a similar regime, having scraped data land and stay inside their own VPC and their own region satisfies auditors in a way that “trust us, it’s on our servers in a different country” does not.

Exit strategy. If the vendor relationship ends, the client still has a working pipeline. This matters a lot in practice. I’ve seen agencies build a scraper, run it for two years, and then the client discovers that shutting off the contract means the data feed stops too, because everything, code, cron jobs, credentials, lived only in the vendor’s account. Deploying inside the client’s account with infrastructure as code (Terraform or CloudFormation, checked into a repo the client owns) means the client can keep it running, hire someone else to maintain it, or hand it to an internal team without a migration project.

What doesn’t change

This is the part that gets misunderstood most often: moving the compute into the client’s AWS or GCP account does nothing on its own to reduce how visible the scraper is to the target site.

Bot detection systems mostly key on the network path the request actually takes, not on who owns the account that launched the container. If the container makes outbound requests through a NAT gateway using the cloud provider’s own IP ranges, the target site sees an AWS, GCP, or Azure ASN. Datacenter ASNs are heavily flagged by IP reputation databases (used by services like Cloudflare, Akamai, and PerimeterX/HUMAN) precisely because real browser traffic almost never originates from a hyperscaler’s IP block. A request coming from an EC2 subnet gets treated with more suspicion than one coming from a residential ISP, regardless of whose AWS account is paying for that EC2 instance.

So if the pipeline needs to look like normal traffic to the target, the egress path still has to go through separate proxy infrastructure, residential or mobile IPs sourced from a proxy provider, or a proxy network you operate yourself, routed out from inside the client’s VPC. The client’s cloud account changes who owns the box making the request. It does not change what IP address the target site sees unless you specifically route egress through something other than the cloud’s own NAT gateway.

This is also where the honesty matters with the client. No proxy pool and no cloud account arrangement makes a scraper undetectable or immune to being rate limited or blocked. Target sites that care about scraping traffic use a layered set of signals: IP reputation, TLS and HTTP/2 fingerprinting, header consistency, request timing, and behavioral patterns on JS-heavy pages. Moving the account of record doesn’t touch most of those signals. It’s an operational and legal improvement, not a stealth upgrade.

The architecture that actually gets deployed

A typical setup looks like a scoped-down version of what you’d run yourself:

Compute. Containerized workers on ECS Fargate, GKE Autopilot, or Azure Container Instances, so there’s no server for either side to patch. The scraper image is built and pushed by the vendor but runs under the client’s account.

Scheduling. Cloud-native schedulers (EventBridge + Lambda, Cloud Scheduler, or a self-hosted Airflow/Prefect instance running as a container) trigger runs. This keeps orchestration inside the same account boundary instead of calling back out to a vendor-controlled server, which is often the actual thing a security team objects to.

Networking. A dedicated VPC and subnet, isolated from the client’s other workloads, with its own NAT gateway or a route out through whatever proxy layer the pipeline uses. Security teams generally want the scraper’s network blast radius to be as small as possible: it should not be able to reach the client’s internal databases or other services.

IAM. The vendor gets a role scoped to exactly what the pipeline needs, deploy permissions on its own ECS service or Cloud Run revision, write access to its own S3 bucket or BigQuery dataset, and nothing else. No admin access, no access to unrelated resources. This is also the part clients’ security reviews scrutinize hardest, and rightly so.

Secrets. API keys for proxy providers, target-site credentials if the scrape involves authenticated access, and any third-party service keys live in the client’s own secrets manager (AWS Secrets Manager, GCP Secret Manager), not in a vendor’s password store. That way access can be revoked instantly if the relationship ends.

Observability. CloudWatch, Cloud Logging, or whatever the client already uses gets wired up so the client can see run history, error rates, and block/retry counts without needing to ask the vendor for a status update. This is often the single biggest trust builder in these engagements, because the client stops having to take the vendor’s word for whether the pipeline is healthy.

Legal and policy boundaries don’t move either

Running inside the client’s account doesn’t change what’s legal or permitted to scrape, and it doesn’t shift that responsibility onto the vendor either, arguably it makes it more clearly the client’s own decision, since the traffic is now originating from infrastructure they control. Whatever target site’s terms of service and robots.txt say still apply the same way. Scraping personal data, paywalled content, or anything behind an authentication wall you don’t have rights to still carries the same legal exposure it would running from anywhere else. A vendor who builds pipelines for clients should push the client to get their own legal sign-off on what’s in scope before a single request goes out, cloud account or not.

When this setup actually makes sense

It’s worth this complexity when the client has a compliance team that needs residency guarantees, when the engagement is long enough that an exit strategy matters, or when the client simply doesn’t want to depend on a vendor’s uptime and account hygiene for a pipeline feeding their own product. It’s usually overkill for a short one-off scrape or a small dataset that’s easier to just deliver as a file. The added IAM, VPC, and handoff documentation work is real overhead, and it’s worth being upfront with clients about that tradeoff rather than selling it as a strictly better default.

If you’re evaluating whether to bring a scraping pipeline in-house, or run it through your own account instead of a vendor’s, that decision usually comes down to exactly these questions: who owns the exit, who owns the IP reputation, and who’s accountable for what gets requested. Get those three answered clearly before worrying about anything else.

For more breakdowns of how scraping infrastructure, proxy routing, and bot detection actually work under the hood, visit Data Research Tools.

Get new guides and videos first — join the Telegram channel.