How to Scrape Realtor.com Property Data in 2026 (Bypass Next.js Protection)

the cleanest way to scrape realtor.com in 2026 is to extract the embedded __NEXT_DATA__ json from each listing page. it contains everything the website renders, in structured form, with no parsing brittleness. you’ll need a residential proxy to avoid the akamai bot manager block, and python with httpx + parsel does the rest. this tutorial ships working code.

why next_data is the trick

realtor.com runs on next.js. every server-rendered page bakes a hidden <script id="__NEXT_DATA__" type="application/json"> block into the html. inside that block sits the entire react state for the page: full property details, agent info, school info, pricing history, photos, the whole structured tree.

if you parse the html directly (price from .price-display, address from .address-line), realtor.com will rename or restructure that markup every few months. your scraper breaks. if you parse __NEXT_DATA__, you get raw json from their backend, and they rarely change those keys because their own frontend depends on them.

we cover this pattern in depth in our javascript-rendered pages scraping guide, but realtor.com is the textbook case.

the anti-bot situation

realtor.com sits behind akamai bot manager and a custom rate limiter. behavior:

(1) datacenter ips: blocked at the cdn edge. you get a 403 with a captcha challenge page.

(2) residential or mobile ips with a clean fingerprint: 200 ok response, 50-200 requests per ip per hour before throttling.

(3) high-volume requests from the same ip: 429 rate limit, 5-15 minute cooldown.

practical implication: you need rotating residential ips with sticky sessions long enough to fetch a single page. you don’t need a full headless browser, plain http requests with the right headers work fine.

installing dependencies

pip install httpx parsel orjson

httpx for async http, parsel for css selectors, orjson for fast json parsing. that’s it.

the basic listing fetcher

import httpx
import parsel
import orjson

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Cache-Control": "no-cache",
    "Pragma": "no-cache",
    "Sec-Ch-Ua": '"Chromium";v="126", "Not(A:Brand";v="24", "Google Chrome";v="126"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"macOS"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

def fetch_listing(url: str, proxy: str) -> dict:
    with httpx.Client(
        proxies=proxy,
        headers=HEADERS,
        timeout=30,
        http2=True,
    ) as client:
        resp = client.get(url)
        resp.raise_for_status()

        sel = parsel.Selector(resp.text)
        data = sel.css("script#__NEXT_DATA__::text").get()
        if not data:
            raise ValueError("__NEXT_DATA__ missing - likely blocked")

        return orjson.loads(data)

if __name__ == "__main__":
    proxy = "http://user-session-abc:pwd@gate.provider.com:8000"
    data = fetch_listing(
        "https://www.realtor.com/realestateandhomes-detail/123-Main-St_Anytown_CA_90210_M12345-67890",
        proxy,
    )
    print(orjson.dumps(data, option=orjson.OPT_INDENT_2).decode())

key headers:

Sec-Ch-Ua block must match the user-agent. mismatched user-agent and sec-ch-ua is one of the fastest ways to get flagged.

http2 enabled. realtor.com uses http2 internally and bot managers flag plain http/1.1 client behavior on http2 sites.

navigating next_data

the json structure looks like this (pruned):

{
  "props": {
    "pageProps": {
      "initialReduxState": {
        "propertyDetails": {
          "property": {
            "list_price": 750000,
            "address": {
              "line": "123 Main St",
              "city": "Anytown",
              "state": "CA",
              "postal_code": "90210"
            },
            "description": {
              "beds": 3,
              "baths": 2.5,
              "sqft": 1800,
              "year_built": 1985,
              "type": "single_family"
            },
            "photos": [...],
            "advertisers": [...],
            "schools": {...},
            "tax_history": [...]
          }
        }
      }
    }
  }
}

the exact path varies slightly between listing types (single family, condo, lot, rental). a robust extractor walks the tree:

def extract_property(data: dict) -> dict:
    page_props = data["props"]["pageProps"]
    redux = page_props.get("initialReduxState", {})
    prop = redux.get("propertyDetails", {}).get("property", {})

    if not prop:
        # fallback for newer page structures
        prop = page_props.get("property", {})

    return {
        "price": prop.get("list_price"),
        "address": prop.get("address", {}).get("line"),
        "city": prop.get("address", {}).get("city"),
        "state": prop.get("address", {}).get("state"),
        "postal_code": prop.get("address", {}).get("postal_code"),
        "beds": prop.get("description", {}).get("beds"),
        "baths": prop.get("description", {}).get("baths"),
        "sqft": prop.get("description", {}).get("sqft"),
        "year_built": prop.get("description", {}).get("year_built"),
        "property_type": prop.get("description", {}).get("type"),
        "photos": [p.get("href") for p in prop.get("photos", [])],
        "agent_name": (prop.get("advertisers", [{}])[0]).get("name"),
        "schools": prop.get("schools"),
        "tax_history": prop.get("tax_history"),
    }

now you have clean structured data ready for a database.

scraping search results

the listing detail page is the easy part. search result pages also embed __NEXT_DATA__ with a list of properties:

def fetch_search_results(city: str, state: str, page: int, proxy: str) -> list:
    url = f"https://www.realtor.com/realestateandhomes-search/{city}_{state}/pg-{page}"
    with httpx.Client(proxies=proxy, headers=HEADERS, timeout=30, http2=True) as client:
        resp = client.get(url)
        resp.raise_for_status()
        sel = parsel.Selector(resp.text)
        data = orjson.loads(sel.css("script#__NEXT_DATA__::text").get())

    listings = data["props"]["pageProps"]["properties"]
    return [
        {
            "property_id": l.get("property_id"),
            "url": f"https://www.realtor.com{l.get('rdc_web_url', '')}",
            "list_price": l.get("list_price"),
            "address": l.get("address"),
        }
        for l in listings
    ]

a typical search returns 42 listings per page. multi-page pagination is just incrementing pg-N until the result list is empty.

the proxy setup

residential rotating with 1-5 minute sticky sessions is what you want. one ip per page fetch keeps the request footprint tiny. avoid mobile (overkill, more expensive) and datacenter (blocked).

import uuid

def make_session_proxy() -> str:
    sid = uuid.uuid4().hex[:12]
    return f"http://user-country-us-session-{sid}:pwd@gate.provider.com:8000"

generate a fresh session per page. if a request fails, generate another and retry.

handling 403 challenges

when akamai flags you, the response is a redirect to a challenge page or an html with "unable to verify" in the body. detect both:

def looks_blocked(resp: httpx.Response) -> bool:
    if resp.status_code == 403:
        return True
    body = resp.text.lower()
    if "unable to verify" in body or "challenge" in body:
        return True
    if "<script id=\"__next_data__\"" not in body:
        return True
    return False

retry with a fresh session. if you get blocked 3 times in a row from different sessions, the entire ip range is hot. wait 10-15 minutes before retrying.

throttling for politeness

even with rotating ips, hammering realtor.com is rude and gets your provider’s pool flagged. cap your request rate:

import asyncio
import random

async def throttled_fetch(url, proxy):
    await asyncio.sleep(random.uniform(2.0, 5.0))
    return await async_fetch_listing(url, proxy)

2-5 second delay per worker, 10-20 workers in parallel. you’ll fetch 200-400 pages/minute, which is plenty without abusing the site.

storing the data

a postgres table for structured fields, jsonb column for the full extracted payload (so you can backfill new fields later):

CREATE TABLE realtor_listings (
    property_id TEXT PRIMARY KEY,
    scraped_at TIMESTAMP NOT NULL DEFAULT NOW(),
    list_price NUMERIC(12, 2),
    address TEXT,
    city TEXT,
    state TEXT,
    postal_code TEXT,
    beds NUMERIC(4, 1),
    baths NUMERIC(4, 1),
    sqft INTEGER,
    year_built INTEGER,
    property_type TEXT,
    raw JSONB,
    UNIQUE(property_id, scraped_at)
);

if your data lands in bigquery instead of postgres, the same pattern works (we wrote a scraping to bigquery pipeline that fits this scraper directly).

scaling considerations

at 200 listings/min, scraping all active us listings (~1.5M) takes ~5 days. realistic budgets:

• proxy bandwidth: ~150kb per detail page = 250mb for 1500 listings. at $4/gb residential = $1/1500 listings.
• compute: a single python worker handles 200/min. for parallel scraping, deploy 5-10 workers across regions.
• storage: a million listings is ~3gb in postgres with the jsonb column.

cheap relative to the data value. real estate scraping pipelines that produce $5k-50k/month in saas revenue spend $200-800/month on infrastructure.

legal and ethical notes

scraping public listings is widely accepted. the data is published for the world to see. but: realtor.com’s tos forbids automated access. they can block your ip range, send a cease-and-desist, or pursue legal action if you redistribute their data commercially.

(1) don’t republish realtor.com’s photos or copy. fair use for analysis is one thing, building a competing listings site is another.

(2) don’t scrape pii (broker phone numbers, emails) and resell it.

(3) respect throttling. if they ratelimit, back off.

your use case (price analysis, market trends, lead generation for buyers) is usually fine. competing directly with mls licensees is a fast way to get sued.

frequently asked questions

why doesn’t realtor.com just block next_data entirely?

their own website depends on it. removing the embedded json would break their progressive enhancement and seo. they could obfuscate keys, but they haven’t, because the cost of breaking their own analytics tooling outweighs the gain of slowing scrapers.

can i scrape with playwright instead of httpx?

you can, but it’s overkill. realtor.com’s next_data is in the initial server response, so you don’t need to wait for js execution. httpx is 10x faster and 50x cheaper.

what residential proxy provider works best for realtor.com?

any reputable residential pool with us ips. our proxy provider comparison ranks them. avoid datacenter and avoid the smaller regional providers, your success rate suffers.

how do i scrape realtor.com photos?

photo urls are in the next_data payload. download them through the same proxy pool, but at higher bandwidth cost. budget ~3mb per photo, ~30mb per listing.

does realtor.com offer an official api?

not for public scraping. they license data through partnerships and via the underlying mls feeds. licensing fees are typically $thousands/month plus per-record charges. scraping is the budget alternative.

will my scraper survive realtor.com html changes?

the next_data approach is far more stable than css selectors. expect occasional key renames (every 12-18 months) but minor adjustments rather than full rewrites.

final thoughts

realtor.com is one of the cleaner real estate scraping targets in 2026 if you go through the front door (__NEXT_DATA__ extraction with residential proxies). most failures we see are from people trying to use datacenter ips, parse the rendered html, or run headless browsers when they don’t need to. the lighter your stack, the faster and cheaper your scraper. ship the simple httpx version first, add complexity only when something specific breaks.

Web Scraping to BigQuery: Full Pipeline Tutorial (Python + Scrapy 2026)

the cleanest way to ship scraped data into bigquery in 2026 is scrapy with a custom item pipeline that streams rows via the bigquery storage write api. you batch into 5MB chunks, fail gracefully on schema mismatches, and you pay storage costs of about $0.02/gb/month. this tutorial builds the pipeline end to end with working python.

the architecture

[scrapy spider] -> [item pipeline] -> [pubsub topic] -> [cloud run worker]
                                                              |
                                                              v
                                                    [bigquery storage write api]
                                                              |
                                                              v
                                                    [bigquery table, partitioned by date]

four components. each does one thing. the spider fetches and parses, the pipeline normalizes, pubsub buffers, the worker writes to bigquery in batches. this design absorbs scraping bursts (5000 items/min) without overwhelming bigquery or paying for streaming inserts at $0.05/gb.

if you want context on why scrapy is still the right tool in 2026, our python web scraping guide walks through alternatives.

prerequisites

pip install scrapy google-cloud-bigquery google-cloud-pubsub google-cloud-bigquery-storage

you also need a gcp project with bigquery + pubsub + cloud run apis enabled, and a service account with bigquery data editor + pub/sub publisher roles. download the json key and export it:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json

designing the bigquery schema

bigquery is happiest with flat, typed columns plus partition + clustering. for a product scraper:

CREATE TABLE scraped.products (
  scraped_at TIMESTAMP NOT NULL,
  source STRING NOT NULL,
  product_id STRING NOT NULL,
  title STRING,
  price NUMERIC(12, 2),
  currency STRING,
  in_stock BOOL,
  image_url STRING,
  raw JSON,
)
PARTITION BY DATE(scraped_at)
CLUSTER BY source, product_id;

three things matter:

(1) partition by DATE(scraped_at). without this, every query scans the full table and your monthly bigquery bill goes from $5 to $500.

(2) cluster by source, product_id. lets you efficiently dedupe and run “show me this product across providers” queries.

(3) keep a raw JSON column. when your schema changes (it will), you can backfill new fields from raw without re-scraping.

the scrapy item

# items.py
import scrapy

class ProductItem(scrapy.Item):
    source = scrapy.Field()
    product_id = scrapy.Field()
    title = scrapy.Field()
    price = scrapy.Field()
    currency = scrapy.Field()
    in_stock = scrapy.Field()
    image_url = scrapy.Field()
    raw = scrapy.Field()

keep the field names matching your bigquery columns. saves a translation layer.

the spider

# spiders/example_products.py
import scrapy
from myproject.items import ProductItem

class ExampleProductsSpider(scrapy.Spider):
    name = "example_products"
    start_urls = ["https://example.com/products"]
    custom_settings = {
        "DOWNLOAD_DELAY": 0.5,
        "CONCURRENT_REQUESTS": 16,
        "ITEM_PIPELINES": {
            "myproject.pipelines.PubSubPipeline": 300,
        },
    }

    def parse(self, response):
        for card in response.css("div.product-card"):
            yield ProductItem(
                source="example.com",
                product_id=card.css("::attr(data-id)").get(),
                title=card.css("h2::text").get(),
                price=float(card.css("span.price::text").re_first(r"[\d.]+") or 0),
                currency="USD",
                in_stock="in stock" in card.css(".stock::text").get("").lower(),
                image_url=card.css("img::attr(src)").get(),
                raw=card.get(),
            )

the pubsub pipeline

scrapy items get serialized as json and published to a pubsub topic. we batch by message size and time window:

# pipelines.py
import json
import datetime as dt
from google.cloud import pubsub_v1

class PubSubPipeline:
    def open_spider(self, spider):
        self.publisher = pubsub_v1.PublisherClient(
            batch_settings=pubsub_v1.types.BatchSettings(
                max_messages=500,
                max_bytes=5_000_000,
                max_latency=2.0,
            )
        )
        self.topic_path = self.publisher.topic_path(
            "your-gcp-project", "scraped-products"
        )
        self.futures = []

    def process_item(self, item, spider):
        row = dict(item)
        row["scraped_at"] = dt.datetime.utcnow().isoformat()
        data = json.dumps(row).encode("utf-8")
        future = self.publisher.publish(self.topic_path, data)
        self.futures.append(future)
        return item

    def close_spider(self, spider):
        # flush remaining batches
        for fut in self.futures:
            fut.result(timeout=30)

scrapy spawns this pipeline per process. for a single-host crawl, that’s fine. for distributed crawling across many machines (see our distributed scraping architecture), each machine independently pushes to pubsub and the worker downstream handles dedupe.

the cloud run worker

cloud run subscribes to the pubsub topic, batches messages, and writes them to bigquery via the storage write api. this is where 90% of the cost savings live. the storage write api is roughly 50x cheaper than streaming inserts on a per-row basis.

# worker.py
import os
import json
from concurrent.futures import ThreadPoolExecutor
from flask import Flask, request
from google.cloud import bigquery_storage_v1
from google.cloud.bigquery_storage_v1 import types, writer
from google.protobuf import descriptor_pb2

app = Flask(__name__)

PROJECT = os.environ["GCP_PROJECT"]
DATASET = "scraped"
TABLE = "products"

client = bigquery_storage_v1.BigQueryWriteClient()
parent = client.table_path(PROJECT, DATASET, TABLE)
write_stream = types.WriteStream(type_=types.WriteStream.Type.COMMITTED)
write_stream = client.create_write_stream(parent=parent, write_stream=write_stream)

@app.post("/")
def handle():
    envelope = request.get_json()
    msg = envelope.get("message", {})
    data = json.loads(base64.b64decode(msg["data"]).decode())
    # build proto row from data, append to write stream
    # in production: batch incoming requests, write 500 rows at a time
    return ("", 204)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

(this is the simplified handler. the full version with proto schema generation lives at github.com/dataresearchtools/scraping-bigquery-pipeline.)

deploy:

gcloud run deploy bq-writer \
  --source . \
  --region us-central1 \
  --no-allow-unauthenticated \
  --service-account scraper-sa@$PROJECT.iam.gserviceaccount.com \
  --memory 512Mi --cpu 1 --max-instances 10

then create a pubsub push subscription targeting the cloud run url. messages flow in, get written, and you pay roughly 1/50th of what streaming inserts would cost.

handling schema drift

scrapers break. sites add fields, change html, drop columns. when that happens, your row fails to write because the schema doesn’t match.

the fix is two-tier:

(1) write the typed columns you know about (title, price, etc).

(2) shove the entire raw item into the raw JSON column.

if you add a new field to the schema next month, you backfill from the raw column with a single query:

UPDATE scraped.products
SET seller_id = JSON_VALUE(raw, '$.seller_id')
WHERE scraped_at >= '2026-04-01'
  AND seller_id IS NULL;

zero re-scraping. zero downtime. this is the single biggest reliability gain in the pipeline.

cost in 2026

real numbers from a pipeline that scrapes ~500k products/day:

• bigquery storage: ~$0.50/month for 25gb
• bigquery query: ~$5/month for analyst dashboards (partitioned, clustered)
• pubsub: ~$0.40/month for 500k msg/day
• cloud run worker: ~$2/month at 10 cpu-minutes/day
• proxy bandwidth (the actual scraping): the dominant cost, $50-200/month

total infrastructure for a working pipeline: under $10/month. proxies are everything. picking the right provider matters more than any cloud optimization.

monitoring

three queries to bookmark:

-- rows scraped per hour, last 24h
SELECT
  TIMESTAMP_TRUNC(scraped_at, HOUR) AS hour,
  source,
  COUNT(*) AS rows
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
GROUP BY hour, source
ORDER BY hour DESC;

-- failed rows (null titles imply parse failure)
SELECT source, COUNT(*) AS bad_rows
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
  AND title IS NULL
GROUP BY source;

-- price drift detection
SELECT
  product_id,
  ANY_VALUE(title) AS title,
  MIN(price) AS min_price,
  MAX(price) AS max_price,
  STDDEV(price) AS price_volatility
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY product_id
HAVING price_volatility > 5
ORDER BY price_volatility DESC
LIMIT 50;

wire the first two into looker studio or a slack alerting bot. the third is gold for ecommerce intelligence dashboards.

dedupe strategies

scraping the same product 10 times a day is fine for price tracking. it’s wasteful for snapshot exports. two patterns:

(1) merge on read with ROW_NUMBER():

SELECT * FROM (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY source, product_id
      ORDER BY scraped_at DESC
    ) AS rn
  FROM scraped.products
) WHERE rn = 1;

(2) materialize a latest_products table with a scheduled query that runs hourly. cheaper to query, slightly stale.

most teams start with (1) and graduate to (2) when query volume grows.

the rare case for streaming inserts

if you need sub-second latency from scrape to dashboard (real-time price alerts, fraud detection), use streaming inserts despite the cost. otherwise, the storage write api batched flow is faster, cheaper, and equally reliable.

handling proxies in the spider

scraping at scale means rotating residential proxies. add scrapy’s HTTPPROXY_AUTH_ENCODING = 'utf-8' and a downloader middleware:

# middlewares.py
class RotatingProxyMiddleware:
    def __init__(self):
        self.gateway = "http://gate.provider.com:8000"
        self.user = os.environ["PROXY_USER"]
        self.pwd = os.environ["PROXY_PWD"]

    def process_request(self, request, spider):
        request.meta["proxy"] = (
            f"http://{self.user}:{self.pwd}@{self.gateway.split('://')[1]}"
        )

enable in settings:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

your scraper now hits target sites through residential rotation, parses, and ships data into bigquery via the pipeline above.

frequently asked questions

why pubsub between scrapy and bigquery?

backpressure. scrapy bursts faster than bigquery wants to absorb. pubsub holds up to 7 days of messages and the worker drains at a sustainable rate. without it, bigquery rejects writes during traffic spikes.

can i skip pubsub and write directly to bigquery from scrapy?

yes for small jobs (under 100 items/min). for anything larger, the streaming insert costs and rate limits make pubsub + storage write api dramatically cheaper.

what about cloud sql or snowflake instead?

cloud sql is wrong here, it’s a transactional database not analytics. snowflake works the same way as bigquery (storage + query separation), pick whichever your team already uses. the pipeline pattern is identical.

how do i handle gdpr or pii in scraped data?

never scrape pii unless your legal team has approved it. if you must, use bigquery’s authorized views and column-level access control. partition by retention period and schedule deletion via DELETE statements in scheduled queries.

what’s the cheapest way to backfill historical data?

batch loads via bq load from gcs, not streaming. costs near zero compared to per-row writes. your scrapy pipeline can write to gcs files in parallel and a daily job loads them all in one go.

does this work with playwright scrapers instead of scrapy?

yes. the pipeline pattern (push to pubsub, write from worker) is framework-agnostic. anything that can call the pubsub publisher api can feed bigquery this way.

final thoughts

a clean scraping-to-bigquery pipeline is mostly about discipline, not novelty. partition your tables, keep a raw json column, batch writes through pubsub, and your engineering cost falls to single digits per month. the only number that matters at scale is your proxy bill. pick the provider carefully, design the schema once, and the pipeline runs for years.

how to scrape google local pack results (maps + business data) 2026

google local pack is the 3-result map block that appears on serps for local-intent queries like “coffee shop near me” or “lawyer in austin.” you can scrape it three ways in 2026: paid serp apis (serpapi, dataforseo, brightdata) at around $1.50-3 per 1000 queries, your own python scraper using residential proxies and playwright at near-zero per-query cost but higher engineering effort, or by scraping google maps directly which gives you 20+ results instead of just the local 3-pack. this tutorial covers all three with working code.

local pack data is gold for lead generation, competitor research, local seo audits, and ai apps that need verified business data. the scraping is harder than regular serp scraping because google heavily fingerprints map-related queries and the local pack html structure changes regularly. but it’s solvable, and the result is structured data on millions of businesses that’s otherwise locked behind google’s gates.

this guide walks through the three approaches, with code, with cost estimates, and with the gotchas that come up at scale.

what’s in the local pack

a typical local pack result on a query like “plumber miami” returns:

• 3 business listings (top 3 by google’s local ranking)
• each listing has: business name, rating (1-5 stars), review count, category, address, hours snippet, phone (sometimes), website (sometimes), gbid (google business id), latitude/longitude
• a “view all” link that opens the local finder (top 20 results)
• ad placements above and below sometimes

the underlying data lives in google’s local index, accessible via the regular serp html, the maps web ui, and the maps mobile app. each surface returns slightly different fields. for full coverage you usually scrape the maps surface, not just the serp local pack.

approach 1: paid serp apis (easiest)

three providers dominate this space in 2026:

• serpapi: $50/month for 5000 searches. local pack data included with engine=google_local.
• dataforseo: $0.0006-0.001 per organic search depending on plan. dedicated local pack endpoint.
• brightdata serp api: $1.50 per 1000 searches. covers all serp features including local pack.

for any serious volume the per-query rates push under $1.50/1k. for prototypes and small jobs they are by far the easiest path.

import requests

SERPAPI_KEY = "your-key"

def get_local_pack(query, location):
    r = requests.get("https://serpapi.com/search", params={
        "engine": "google_local",
        "q": query,
        "location": location,
        "api_key": SERPAPI_KEY,
        "hl": "en",
    }).json()
    return r.get("local_results", [])

results = get_local_pack("plumber", "miami, florida")
for r in results[:5]:
    print(r["title"], r.get("rating"), r.get("phone"), r.get("address"))

dataforseo’s local pack endpoint:

import requests
from requests.auth import HTTPBasicAuth

post_data = [{
    "keyword": "plumber",
    "location_name": "Miami,Florida,United States",
    "language_code": "en",
    "device": "desktop",
}]

r = requests.post(
    "https://api.dataforseo.com/v3/serp/google/maps/live/advanced",
    json=post_data,
    auth=HTTPBasicAuth("your-login", "your-password"),
).json()

items = r["tasks"][0]["result"][0]["items"]
for item in items[:10]:
    print(item["title"], item.get("rating", {}).get("value"))

dataforseo’s pricing is the most aggressive at scale (under $0.001 per query at volume). serpapi has the friendliest sdk and free tier. bright data is the most reliable at very high volume.

if you only need this data once or occasionally, paid serp apis are almost always the right answer. you spend $5-50, you get the data, you move on.

approach 2: python scraper with residential proxies

cheaper at scale, more engineering work upfront. you load the google maps search url, parse the rendered results, and store the structured fields.

import asyncio
import json
import re
from playwright.async_api import async_playwright

PROXY = {
    "server": "http://residential.example.com:8080",
    "username": "user",
    "password": "pass",
}

async def scrape_maps(query, location):
    url = f"https://www.google.com/maps/search/{query.replace(' ', '+')}+{location.replace(' ', '+')}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False, proxy=PROXY)
        ctx = await browser.new_context(
            viewport={"width": 1366, "height": 768},
            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            locale="en-US",
        )
        page = await ctx.new_page()
        await page.goto(url, wait_until="networkidle", timeout=30000)
        await page.wait_for_timeout(3000)

        # scroll the results panel to load more
        results_panel = page.locator("div[role='feed']")
        for _ in range(3):
            await results_panel.evaluate("el => el.scrollBy(0, 800)")
            await page.wait_for_timeout(1500)

        # extract listing elements
        items = await page.locator("div[role='feed'] > div > div[jsaction]").all()
        results = []
        for item in items[:20]:
            try:
                name = await item.locator("div.fontHeadlineSmall").first.text_content()
                rating_el = await item.locator("span[role='img'][aria-label*='star']").first.get_attribute("aria-label")
                results.append({
                    "name": name.strip() if name else None,
                    "rating_aria": rating_el,
                })
            except Exception:
                continue

        await browser.close()
        return results

async def main():
    results = await scrape_maps("plumber", "miami florida")
    print(json.dumps(results, indent=2))

asyncio.run(main())

this is the rough shape. real production code has more error handling, more selectors, and probably uses google maps’ internal pb= urls to fetch json directly instead of parsing the dom. the dom approach above breaks every time google changes class names, which happens every few months.

a more robust pattern is to capture the maps json endpoint via network interception:

async def capture_maps_json(query, location):
    captured = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        ctx = await browser.new_context()
        page = await ctx.new_page()

        async def handle_response(response):
            if "search?" in response.url and "/maps/" in response.url:
                try:
                    body = await response.text()
                    if body.startswith(")]}'"):
                        body = body[5:]
                    captured.append(json.loads(body))
                except Exception:
                    pass

        page.on("response", handle_response)
        await page.goto(f"https://www.google.com/maps/search/{query}+{location}",
                        wait_until="networkidle")
        await page.wait_for_timeout(5000)
        await browser.close()
    return captured

the captured json contains the full structured data google sees on its end. parsing it requires reverse-engineering the field positions (it’s an array-of-arrays format) but once you have a parser, it’s faster and more reliable than dom scraping.

for the proxy choice, residential is the minimum. mobile proxies pass through the toughest google blocks more reliably. datacenter ips are blocked within a few queries. see the residential proxy guide for context.

approach 3: scrape regular google serp html

if you only need the 3-pack (not the full 20-result local finder), you can scrape the regular google serp page. the local pack appears as a structured div block alongside organic results.

import requests
from bs4 import BeautifulSoup

PROXY = {"http": "http://user:pass@residential.example.com:8080",
         "https": "http://user:pass@residential.example.com:8080"}

def scrape_serp_local(query, geo_param):
    url = f"https://www.google.com/search?q={query}&uule={geo_param}&hl=en"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept": "text/html,application/xhtml+xml",
        "Accept-Language": "en-US,en;q=0.9",
    }
    r = requests.get(url, headers=headers, proxies=PROXY, timeout=30)
    soup = BeautifulSoup(r.text, "html.parser")

    # local pack container varies, but rllt__details is reliable for 3-pack listings
    listings = []
    for div in soup.select("div.rllt__details"):
        name = div.select_one("div.dbg0pd")
        if name:
            listings.append({
                "name": name.get_text(strip=True),
                "snippet": " ".join(s.get_text() for s in div.select("div") if s != name),
            })
    return listings

the uule parameter is google’s encoded location. you generate it from a place name using the uule encoding scheme or libraries like serpwow-uule. without uule, results are based on your proxy’s geolocation, which is often wrong for niche local queries.

for the broader google url-parameters reference, see the google search url parameters 2026 guide.

extracting individual business details

the local pack listings give you basic data. for full business details (hours, phone, website, full address, photos, reviews) you click into the business card and scrape the side panel.

async def scrape_business_details(page, business_url):
    await page.goto(business_url, wait_until="networkidle")
    await page.wait_for_timeout(2000)

    name = await page.locator("h1").first.text_content()
    address = await page.locator("button[data-item-id='address']").first.text_content()
    phone_el = page.locator("button[data-item-id^='phone']").first
    phone = await phone_el.text_content() if await phone_el.count() else None
    website_el = page.locator("a[data-item-id='authority']").first
    website = await website_el.get_attribute("href") if await website_el.count() else None
    rating = await page.locator("div.F7nice span[aria-hidden='true']").first.text_content()

    return {
        "name": name.strip() if name else None,
        "address": address.strip() if address else None,
        "phone": phone.strip() if phone else None,
        "website": website,
        "rating": rating,
    }

the gbid (google business id, also called cid) is in the page url after navigation. extract from page.url with a regex on the place/.../@.../data= segment.

handling pagination and load more

google maps doesn’t paginate the local finder in the traditional sense. it loads more results as you scroll the left-side panel. the playwright code above scrolls 3 times. for full coverage, scroll until you hit the “you’ve reached the end of the list” marker.

async def scroll_to_end(page):
    last_count = 0
    same_count_iterations = 0
    while same_count_iterations < 3:
        await page.locator("div[role='feed']").evaluate("el => el.scrollBy(0, 1000)")
        await page.wait_for_timeout(1500)
        items = await page.locator("div[role='feed'] > div > div[jsaction]").count()
        if items == last_count:
            same_count_iterations += 1
        else:
            same_count_iterations = 0
            last_count = items
    return last_count

most categories cap at 120 results in google maps. some niche or local queries cap at 20-40. that’s a hard ceiling.

rate limiting and avoiding blocks

google’s anti-scraping is aggressive on maps. patterns that get you blocked fast:

• many queries from the same ip in quick succession
• queries with no realistic delay between them
• consistent user-agent across all requests
• queries from datacenter ips
• non-residential geolocation mismatch (querying us businesses from a singapore ip)

mitigations:

• residential or mobile proxies, rotated per query
• 5-10 second delay between queries minimum
• random user-agent across a pool of 10-20 valid ones
• match your proxy geo to your query geo where possible
• spread queries across hours, not in a 1-minute burst

with those mitigations in place, a single residential proxy can do 100-500 queries a day before getting flagged. with mobile proxies, several thousand. for higher volume, parallelize across many proxies.

cost comparison

estimating cost for 100,000 local pack queries.

approach	cost	engineering effort
serpapi	$1000 (4x growth plan)	minimal
dataforseo	$60-150	moderate (sdk integration)
brightdata serp api	$150	minimal
diy with residential proxies	$50-100 (proxy bandwidth)	high (build + maintain)
diy with mobile proxies	$300-500 (mobile bandwidth)	high (build + maintain)

dataforseo wins on raw cost at scale. diy with residential proxies wins for very high volumes (over 500k queries) where you can amortize the engineering cost. for under 100k queries, dataforseo is hard to beat.

faq

is scraping google maps legal?
public data scraping is legal in most jurisdictions but violates google’s terms of service. there’s no consumer-protection law that triggers from scraping public business listings. for commercial use cases at scale, talk to a lawyer about cfaa exposure. the web scraping legal guide covers the case law.

can i use the official google places api instead?
yes, and you should if your use case fits. places api charges $17-32 per 1000 requests and is rate-limited. for small volumes it’s competitive with serp apis. for high volume scraping is far cheaper.

how many results does google local pack actually return?
the visible 3-pack is just the top 3. the local finder (clicking “view all”) shows up to 120. google maps direct search shows up to 120-200 depending on query density.

does serpapi return the gbid?
yes, in the place_id field. some legacy responses use gbid directly. dataforseo also returns it. roll-your-own scraping requires extracting it from the place url.

which proxy type works best for google maps?
residential or mobile. datacenter ips get blocked within a few queries. mobile is more reliable for high-volume sustained scraping. for context see the residential proxy guide.

how do i scrape google reviews for a business?
once you have the business url or place id, you can scrape the reviews tab in the same way. each review is a single json item in the maps response. expect 200-500 reviews per page load with infinite scroll.

conclusion

google local pack scraping is a solved problem in 2026 if you’re willing to spend on a paid serp api. dataforseo at $0.001 per query is the price-to-value sweet spot. serpapi is the easiest first integration. brightdata is the most reliable at very high volume.

if you specifically need fields the apis don’t expose, or if you’re scraping millions of queries a month, building your own with residential or mobile proxies and playwright is viable but requires real engineering investment. the dom selectors break every few months. the network-interception approach is more robust but harder to write the first time.

start with a paid serp api. measure your data needs against what they return. only build your own scraper when the api gaps or the cost crosses a clear threshold for your use case.