how to scrape Booking.com hotel prices (2026 anti-bot guide)

Booking.com is one of the harder travel sites to scrape in 2026 because it sits behind Akamai Bot Manager plus its own dynamic pricing layer. you need residential proxies, a real headless browser like Playwright, careful rate limits, and the right strategy for handling per-session price tokens. this guide walks through working Python code, the Akamai-specific gotchas, and what data you can actually extract reliably.

what you can scrape from Booking.com

Booking.com pages are deeply dynamic, but the high-value fields are stable enough for production scrapers.

field	location	difficulty
hotel name, location	search results, hotel page	easy
star rating, review score	search results	easy
nightly price (with dates)	search results	medium
total price + taxes	hotel page	medium
room types and inclusions	hotel page	medium
availability calendar	hotel page (dynamic)	hard
review text + reviewer location	reviews tab	hard
photos	hotel page	easy

most price-monitoring use cases need only the first three. for anything more complex, expect more anti-bot friction. for the broader use case, our price monitoring proxy guide covers infrastructure decisions.

the Akamai problem

Booking.com runs Akamai Bot Manager which inspects three things on every request: TLS fingerprint (JA3/JA4), HTTP/2 fingerprint, and a per-session token called _abck that gets validated against a sensor payload generated by client-side JavaScript.

plain requests or httpx will fail because the TLS fingerprint reveals Python instantly. even curl gets blocked. you need either a real browser (Playwright, Puppeteer) or a TLS-impersonating client like curl_cffi.

if you want a deeper look at Akamai itself, our Akamai bypass guide covers the mechanism in detail. the same techniques apply directly to Booking.com.

install the stack

pip install playwright curl_cffi parsel
playwright install chromium

Playwright launches a real Chromium browser. curl_cffi impersonates Chrome’s TLS fingerprint for the lighter price-checks where you don’t need full JavaScript rendering. parsel parses the HTML.

scrape search results with Playwright

start with a search page that returns hotels for a city and date range.

import asyncio
from playwright.async_api import async_playwright
from parsel import Selector

PROXY = {'server': 'http://gateway.example.com:8000', 'username': 'u', 'password': 'p'}

URL = 'https://www.booking.com/searchresults.html?ss=Singapore&checkin=2026-06-01&checkout=2026-06-03&group_adults=2'

async def scrape_search():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=PROXY)
        ctx = await browser.new_context(
            user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
            locale='en-US',
            viewport={'width': 1366, 'height': 900},
        )
        page = await ctx.new_page()
        await page.goto(URL, wait_until='networkidle')
        await page.wait_for_selector('div[data-testid="property-card"]', timeout=15000)

        html = await page.content()
        await browser.close()

    sel = Selector(text=html)
    hotels = []
    for card in sel.css('div[data-testid="property-card"]'):
        hotels.append({
            'name': card.css('div[data-testid="title"]::text').get(''),
            'location': card.css('span[data-testid="address"]::text').get(''),
            'score': card.css('div[data-testid="review-score"] div::text').get(''),
            'price': card.css('span[data-testid="price-and-discounted-price"]::text').get(''),
            'url': card.css('a[data-testid="title-link"]::attr(href)').get(''),
        })
    return hotels

asyncio.run(scrape_search())

data-testid selectors are the most stable. CSS class names on Booking.com change frequently because their build pipeline auto-generates them. the testid attributes survive UI tweaks because they’re part of the QA harness.

handle the cookie consent banner

first-time visitors see a GDPR banner that blocks page interactions. dismiss it before doing anything.

try:
    await page.click('button#onetrust-accept-btn-handler', timeout=3000)
except:
    pass

wrap it in a try/except because the banner only shows for fresh sessions. if your proxy gives you a sticky session, the cookie persists and the banner doesn’t appear on the next request.

rotate proxies and sessions

Booking.com’s per-IP rate limit is roughly 30-60 requests per hour before Akamai starts challenging you. with a residential pool, you want sticky sessions of 10-15 minutes per IP, then rotate.

async def scrape_many_cities(cities):
    async with async_playwright() as p:
        for city in cities:
            session_id = f'session-{city}'
            proxy = {
                'server': 'http://gateway.example.com:8000',
                'username': f'user-session-{session_id}',
                'password': 'pass',
            }
            browser = await p.chromium.launch(headless=True, proxy=proxy)
            ctx = await browser.new_context()
            page = await ctx.new_page()

            # scrape this city
            ...

            await browser.close()
            await asyncio.sleep(5)

most residential proxy providers let you specify a session ID in the username (user-session-XXX). same session ID = same IP. change the ID to rotate. our Akamai bypass guide covers the fingerprinting layer in more depth.

handle pagination

Booking.com paginates with offset query params. add &offset=25 (or 50, 75, etc.) to the search URL.

for offset in range(0, 250, 25):
    url = f'{base_url}&offset={offset}'
    await page.goto(url, wait_until='networkidle')
    # extract cards

each page returns roughly 25 results. don’t paginate past 1,000 results from the same search; Akamai flags deep pagination as automation. for big crawls, split by city + date pair instead.

scrape an individual hotel page

hotel pages contain pricing per room type, availability, and amenity details.

async def scrape_hotel(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=PROXY)
        ctx = await browser.new_context(locale='en-US')
        page = await ctx.new_page()
        await page.goto(url, wait_until='networkidle')

        await page.wait_for_selector('h2[data-testid="property-header-name"]', timeout=15000)
        html = await page.content()
        await browser.close()

    sel = Selector(text=html)
    return {
        'name': sel.css('h2[data-testid="property-header-name"]::text').get(''),
        'address': sel.css('span[data-testid="address"]::text').get(''),
        'rating': sel.css('div[data-testid="review-score-component"] div::text').get(''),
        'rooms': sel.css('table#hprt-table tr.hprt-table-row').getall(),
    }

the room table is the trickiest part because the markup is legacy (table-based, lots of nested rows for room options). parse it row by row and join with the room-name column.

handle dynamic prices

prices on Booking.com depend on cookies, locale, and currency settings. the same hotel can show different prices to different users in the same city. for accurate price monitoring you need to:

1. set a consistent locale and currency at session start
2. use the same IP geolocation for repeat scrapes (US IP = USD by default)
3. include explicit &selected_currency=USD in the URL
4. compare like-for-like by date range and occupancy

if your IP rotates between countries mid-session, prices will jump because the currency conversion changes. residential pools with country-targeting solve this.

faq

can I scrape Booking.com without Playwright?
yes for static-looking pages, no for anything that involves dynamic price tokens. curl_cffi with Chrome’s TLS fingerprint can fetch some search result HTML, but room-level pricing and availability requires the JavaScript runtime. start with Playwright for reliability, optimize to lighter clients only after you understand which pages are safely fetchable.

what proxies do I need?
residential is the minimum. mobile is overkill unless you’re scraping at very high volume. datacenter IPs get blocked instantly because Akamai recognizes the AS numbers. for provider picks, see our provider comparison.

how do I avoid the Akamai _abck challenge?
use a real browser (Playwright with stealth), don’t disable JavaScript, keep cookies across requests in the same session, and respect rate limits. headers alone won’t pass; the sensor payload requires real DOM execution.

is scraping Booking.com legal?
public price data is generally legal to collect, but Booking.com’s terms of service prohibit automated access. for personal research, low risk. for commercial use, consult a lawyer and consider their official Booking.com Affiliate Partner Program for hotel data instead. for related legal context, our web scraping legal guide covers the broader rules.

why are my scraped prices different from what I see in my browser?
prices vary by IP geolocation, currency, device type, and even browsing history. always pin currency, locale, viewport size, and user agent. if your scraper IP is in Singapore but you want US-resident pricing, get a US residential IP.

how often does Booking.com change selectors?
data-testid attributes are stable across most updates. CSS class names rotate with each deploy (often weekly). build parsers around testids, not classes, and you’ll cut maintenance to once per quarter instead of weekly.

conclusion

scraping Booking.com works in 2026 if you bring real browser automation, residential proxies, and respect for the per-IP rate limits. Akamai is the main obstacle and Playwright (or any real Chromium) handles it transparently as long as you don’t disable JavaScript or strip cookies.

focus on the data-testid selectors, pin your locale and currency, and rotate sticky sessions every 10-15 minutes. that combination keeps you under the radar while collecting clean price data at meaningful volume.

if you’re doing this commercially, consider Booking.com’s official affiliate API or the managed scraping APIs that handle the anti-bot for you. for personal research and price-comparison side projects, the Playwright approach in this guide is plenty.

web scraping with Node.js: axios, cheerio, puppeteer complete guide (2026)

Node.js is the second most popular language for web scraping after Python, and in 2026 it has caught up on tooling. you get axios for HTTP requests, cheerio for fast HTML parsing, and puppeteer for headless Chrome control. this guide walks through all three with real code, proxy rotation, and the anti-bot patterns that actually work today.

why scrape with Node.js in 2026

Node.js wins when your scraper needs to share code with a frontend, run inside an existing Express or Next.js app, or handle thousands of concurrent requests over a single event loop. its async model is genuinely faster than threaded Python for I/O-bound jobs.

it also gets first-class support from puppeteer (Google maintains it) and the new Chrome DevTools Protocol features land in Node before they land anywhere else. if you scrape JavaScript-heavy sites at scale, Node is the path of least resistance.

the trade-off is the parsing ecosystem. Python’s BeautifulSoup is more forgiving than cheerio when HTML is broken, and pandas is way ahead of anything in JS land for downstream data work. pick Node when concurrency or browser automation matter most.

install the stack

start with a fresh project. you only need three core packages and one helper for proxy support.

mkdir scraper && cd scraper
npm init -y
npm install axios cheerio puppeteer https-proxy-agent

axios handles plain HTTP. cheerio parses HTML with a jQuery-like API. puppeteer launches headless Chrome. https-proxy-agent lets you route axios through HTTP or HTTPS proxies without external config.

library	use for	speed	handles JS
axios + cheerio	static HTML, APIs, RSS	very fast	no
puppeteer	SPAs, login flows, screenshots	slow	yes
playwright	same as puppeteer + multi-browser	slow	yes

most production scrapers use both. axios handles 80% of pages, puppeteer handles the JavaScript-rendered 20%.

scrape static HTML with axios + cheerio

here’s the simplest possible scraper. it pulls book titles and prices from books.toscrape.com (the official sandbox for scraping practice).

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeBooks() {
  const { data } = await axios.get('https://books.toscrape.com/');
  const $ = cheerio.load(data);
  const books = [];

  $('article.product_pod').each((i, el) => {
    books.push({
      title: $(el).find('h3 a').attr('title'),
      price: $(el).find('.price_color').text(),
      stock: $(el).find('.availability').text().trim(),
    });
  });

  return books;
}

scrapeBooks().then(console.log);

axios returns the raw HTML. cheerio loads it into a DOM-like object you query with CSS selectors. the each loop builds an array of objects you can write to JSON or a database.

this pattern handles thousands of requests per minute on a single machine. for sites without anti-bot defenses, it’s all you need.

handle pagination

most listings span multiple pages. wrap the scraper in a loop that follows the next-page link until it disappears.

async function scrapeAllPages() {
  let url = 'https://books.toscrape.com/';
  const all = [];

  while (url) {
    const { data } = await axios.get(url);
    const $ = cheerio.load(data);

    $('article.product_pod').each((i, el) => {
      all.push({
        title: $(el).find('h3 a').attr('title'),
        price: $(el).find('.price_color').text(),
      });
    });

    const next = $('li.next a').attr('href');
    url = next ? new URL(next, url).href : null;
  }

  return all;
}

URL resolution matters. relative links like catalogue/page-2.html break if you concatenate them naively. the URL constructor merges them against the current base.

for sites with offset pagination (?page=1, ?page=2), increment the query param until you get an empty result set. for cursor-based APIs, follow the next-cursor token.

scrape JavaScript-rendered pages with puppeteer

axios fetches HTML that the server returns. it does not run JavaScript. modern SPAs (React, Vue, Next.js) render content client-side, so axios returns an empty shell.

puppeteer launches a real Chrome instance, runs the JavaScript, and gives you the rendered DOM.

const puppeteer = require('puppeteer');

async function scrapeQuotes() {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();

  await page.goto('https://quotes.toscrape.com/js/', {
    waitUntil: 'networkidle2',
  });

  const quotes = await page.$$eval('.quote', (els) =>
    els.map((el) => ({
      text: el.querySelector('.text').innerText,
      author: el.querySelector('.author').innerText,
    }))
  );

  await browser.close();
  return quotes;
}

waitUntil: 'networkidle2' tells puppeteer to wait until network traffic settles. for slower sites, use waitForSelector('.quote') instead so you don’t time out on a busy XHR.

$$eval runs a function inside the browser, so you query the live DOM with regular querySelector calls. the result serializes back to Node.

if you’re picking between puppeteer and the alternatives, see our headless browser deep dive and the selenium vs playwright vs puppeteer comparison.

handle infinite scroll

many listing pages load more results as you scroll. simulate that with page.evaluate.

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let total = 0;
      const distance = 300;
      const timer = setInterval(() => {
        const { scrollHeight } = document.body;
        window.scrollBy(0, distance);
        total += distance;
        if (total >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 200);
    });
  });
}

call autoScroll(page) before extracting data. tune the distance (300px) and interval (200ms) to match how the target site loads chunks. too fast and you’ll miss content, too slow wastes time.

use proxies (essential for any real target)

the moment you scrape Amazon, Google, LinkedIn, or any high-value target, you need rotating proxies. a single IP making 100 requests per minute gets blocked within seconds.

axios with a single proxy:

const { HttpsProxyAgent } = require('https-proxy-agent');

const agent = new HttpsProxyAgent('http://user:pass@proxy.example.com:8000');

const { data } = await axios.get('https://example.com', {
  httpsAgent: agent,
  httpAgent: agent,
  timeout: 15000,
});

for a rotating residential pool, you typically get a single gateway endpoint and the provider rotates IPs per request. mobile proxy gateways work the same way.

puppeteer with a proxy:

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://proxy.example.com:8000'],
});

const page = await browser.newPage();
await page.authenticate({ username: 'user', password: 'pass' });

proxy auth in puppeteer is page-level, not browser-level. call page.authenticate before page.goto.

for a deeper walkthrough on rotation strategies, see rotating proxies and unlimited bandwidth.

avoid common anti-bot triggers

generic blocks happen because your scraper looks nothing like a real browser. fix the obvious tells first.

set a real user agent. axios defaults to axios/1.x which screams bot. rotate through 3-5 modern Chrome strings.

const headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
};

await axios.get(url, { headers });

add randomized delays between requests. 1-3 seconds is enough for most sites. use await new Promise(r => setTimeout(r, 1500 + Math.random() * 1500)).

for puppeteer, install puppeteer-extra-plugin-stealth. it patches the navigator.webdriver flag and dozens of other fingerprint tells in one line.

const puppeteer = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth');
puppeteer.use(Stealth());

stealth alone won’t beat enterprise anti-bot like Cloudflare, DataDome, or Akamai. for those, you need residential proxies plus stealth plus careful request timing. for tougher targets, also see how Python scrapers handle this.

handle errors and retries

network errors are normal. wrap requests in retry logic so a single 503 doesn’t kill your job.

async function fetchWithRetry(url, retries = 3) {
  for (let i = 0; i < retries; i++) {
    try {
      return await axios.get(url, { timeout: 15000 });
    } catch (e) {
      if (i === retries - 1) throw e;
      await new Promise((r) => setTimeout(r, 2 ** i * 1000));
    }
  }
}

exponential backoff (1s, 2s, 4s) is the standard. for 429 responses, respect the Retry-After header if the server sets it.

log everything. failed scrapes are signal, not noise. a sudden spike in 403s means your IP got flagged or the site changed defenses.

save data

for small jobs, write JSON.

const fs = require('fs');
fs.writeFileSync('books.json', JSON.stringify(books, null, 2));

for thousands of rows, stream to a CSV or a database. the csv-stringify package handles escaping. for Postgres, use pg with batch inserts of 500-1000 rows per transaction. for cloud, push to S3 or BigQuery.

never store credentials in your scraper code. use dotenv (npm install dotenv) and a .env file that’s gitignored.

faq

is web scraping with Node.js faster than Python?
for I/O-bound concurrent scraping, yes. Node.js handles thousands of simultaneous connections on a single thread thanks to its event loop. Python needs asyncio or threading to match it, and even then the GIL limits CPU work.

do I need puppeteer or is axios enough?
axios + cheerio is enough for any page that returns full HTML on the first request. open the target site with JavaScript disabled in your browser. if the content you want is still visible, axios works. if the page goes blank, you need puppeteer.

what’s the difference between puppeteer and playwright?
playwright is Microsoft’s fork of puppeteer with multi-browser support (Chromium, Firefox, WebKit) and better auto-waiting. puppeteer only drives Chrome but has tighter integration with Chrome DevTools. both work for scraping. our comparison guide breaks down which to pick.

can I run Node.js scrapers serverless?
yes for axios + cheerio (small footprint, fits in Lambda or Vercel functions). puppeteer is harder because Chrome binaries are 200MB+. use chrome-aws-lambda or the puppeteer-core + @sparticuz/chromium combo, or run puppeteer on a small VM instead.

how do I avoid getting blocked?
rotate residential or mobile proxies, randomize user agents, add 1-3 second delays between requests, and use stealth plugins for puppeteer. for tough targets like Cloudflare or DataDome, you’ll also need to defeat TLS fingerprinting. for any of those defenses, residential IPs are non-negotiable.

is web scraping legal in 2026?
scraping public data is generally legal in the US after the hiQ v. LinkedIn rulings, but terms of service violations and CFAA risk still exist. EU GDPR adds personal-data restrictions. always check the target site’s robots.txt and ToS, and consult a lawyer for commercial use.

conclusion

Node.js gives you a complete scraping stack in three packages. axios for fast HTTP, cheerio for HTML parsing, puppeteer for JavaScript-heavy pages. add a residential proxy pool, stealth plugins, and retry logic, and you have a production scraper.

start with the static stack (axios + cheerio) and only add puppeteer when you actually need it. headless Chrome is 100x slower than HTTP requests and burns way more proxy bandwidth. the cheapest reliable scraper is the one that does the least work per page.

if your scraping needs grow, look at distributed runners like Apify, BullMQ for queues, and managed scraping APIs that handle proxies and CAPTCHAs for you. but for most jobs, the three libraries in this guide will get you 90% of the way there.

how to scrape linkedin data without getting banned (2026)

scraping linkedin without bans in 2026 comes down to four things: residential or mobile proxies (never datacenter), aged accounts with established activity, slow request rates (under 80 actions per day per account), and either a managed scraping api or playwright with anti-detection. linkedin actively detects automation. one ip + one fresh account + 200 requests in an hour = ban within 24 hours. this guide covers the legal context, the technical setup, and how to recover when accounts get restricted.

we cover legality first, then the proxy stack, account discipline, browser automation, managed api alternatives, and a 2026 ban-recovery playbook.

is linkedin scraping legal in 2026?

scraping public linkedin data is generally legal in the us under hiq v linkedin (2022) and follow-on rulings. the courts have repeatedly held that scraping public web data is not a violation of the computer fraud and abuse act.

scraping linkedin still violates linkedin’s terms of service. tos violations are not criminal but they give linkedin grounds to ban accounts and pursue civil action against commercial scrapers in some cases.

eu and uk law is stricter. gdpr requires a lawful basis (consent, contract, or legitimate interest) for processing personal data. scraped linkedin data falls under gdpr if it includes eu data subjects. document your lawful basis before processing, and respect data subject rights including erasure requests.

read linkedin’s user agreement for the current commercial-use restrictions. for a deeper read on the legal landscape around lead-gen scraping see our b2b lead generation proxies guide.

what gets you banned in 2026

linkedin’s anti-bot stack flags four signals.

ip pattern. datacenter ips trigger immediately. shared residential ips with known scraper traffic also flag fast. mobile carrier ips have the longest leash.

session pattern. login from a new ip with no warm-up history is suspicious. 50 profile views in 10 minutes is a classic bot signal. clicking through every profile from a search result without scrolling looks robotic.

browser fingerprint. headless chrome without anti-detection patches is detected within minutes. residential proxy + plain selenium = ban in under an hour.

account age and activity. brand new accounts with zero connections and a thin profile that suddenly perform 500 actions trip every alarm. aged accounts with real history get more leniency.

beat all four and bans become rare. miss any one and accounts cycle through faster than you can warm them.

the proxy stack

mobile proxies are the safest tier. linkedin sees thousands of users behind each carrier-grade nat ip, so individual scraping signals are diluted. expect to pay $50 to $150 per port per month.

residential proxies are the value pick. session-rotating residential pools work for most scraping at $4 to $7 per gb. choose providers with sticky sessions of 10+ minutes so a single profile-view session does not change ip mid-flow.

datacenter proxies are unusable for linkedin in 2026. even premium isp proxies (which are residential-issued datacenter ips) get blocked within a few requests.

assign one proxy per linkedin account. never share an ip across multiple accounts. linkedin’s session correlation flags shared ips fast.

account discipline

aged accounts are non-negotiable in 2026. linkedin treats accounts under 6 months old with no activity as bots by default. for production scraping you need accounts with at least 100 connections, a complete profile, posted content from real timestamps, and a normal usage history.

three options for account supply.

option 1: warm your own. spend 30 to 60 days on each account: login, scroll, accept connections, post once a week, like a few posts daily. boring but the accounts last.

option 2: buy aged accounts. resellers sell 1-year-old accounts with 500+ connections for $50 to $200. quality varies wildly. budget for replacement.

option 3: managed scraping apis. let bright data, apify, or proxycurl handle the account problem entirely. you pay per query, they handle bans on their side. cleanest for production.

never run more than 80 to 120 actions per account per day. one action = one profile view, one search, or one connection request. above that, ban risk spikes hard.

browser automation: playwright with anti-detection

for self-managed scraping, playwright with anti-detection patches is the baseline.

from playwright.sync_api import sync_playwright
import time
import random

PROXY = {
    "server": "http://proxy.example.com:8080",
    "username": "user-session-abc123",
    "password": "pass",
}

def scrape_profile(profile_url, session_cookie):
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            proxy=PROXY,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
            ],
        )
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/127.0.0.0 Safari/537.36",
            viewport={"width": 1920, "height": 1080},
            locale="en-US",
            timezone_id="America/New_York",
        )
        context.add_cookies([{
            "name": "li_at",
            "value": session_cookie,
            "domain": ".linkedin.com",
            "path": "/",
        }])

        page = context.new_page()
        page.goto(profile_url, wait_until="networkidle")
        time.sleep(random.uniform(2, 5))

        page.mouse.wheel(0, 600)
        time.sleep(random.uniform(1, 3))
        page.mouse.wheel(0, 800)
        time.sleep(random.uniform(2, 4))

        name = page.locator("h1").inner_text()
        headline = page.locator(".text-body-medium.break-words").first.inner_text()

        browser.close()
        return {"name": name, "headline": headline, "url": profile_url}

the --disable-blink-features=AutomationControlled flag removes the most obvious headless tell. sleeps and mouse-wheel events simulate human pacing. timezone, locale, and user-agent match a typical us desktop user.

for stronger anti-detection, use playwright-stealth or a real antidetect browser like adspower or gologin. plain playwright is detectable by sophisticated fingerprinting.

for the broader python scraping context see our web scraping with python guide.

sticky sessions across the scrape session

linkedin tracks ip across a session. switching ip mid-session looks like account hijacking and triggers a security challenge.

def session_username(account_id):
    """build a sticky username for residential providers that support it."""
    return f"user-session-{account_id}"

def proxy_for_account(account_id):
    return {
        "server": "http://proxy.example.com:8080",
        "username": session_username(account_id),
        "password": "pass",
    }

most residential providers (smartproxy, oxylabs, soax) support session usernames that pin a single residential ip for 10 to 30 minutes. use the same session id for the duration of the linkedin scrape, then rotate when the session expires naturally.

rate limits in practice

based on 6 months of data across 30 aged accounts running through residential proxies in 2026, here is what stayed unbanned:

• profile views: under 80 per day per account
• searches: under 25 per day per account
• connection requests: under 15 per day per account (lifetime cap of 100 per week)
• messages to connections: under 50 per day per account
• session length: 30 to 90 minutes per session, 1 to 2 sessions per day
• gap between sessions: at least 4 hours

push past these and ban rates spike. stay below them and accounts last 6 to 12 months on average before any restriction.

managed scraping apis: the easier path

self-managed linkedin scraping is a job. you maintain account warming, proxy rotation, anti-detection patches, and ban recovery. for many teams the time cost beats the api cost.

managed options in 2026:

bright data linkedin dataset. pre-scraped public profiles. updated continuously. you query by url or company. roughly $0.001 to $0.01 per record depending on volume. no scraping risk on your side.

apify linkedin scraper actors. pay per actor run. simpler than building your own; still subject to linkedin’s anti-bot. 2026 prices: roughly $1 to $3 per 1,000 results.

proxycurl. linkedin profile, company, and job api. enterprise-friendly with response-time slas. $0.10 to $0.30 per profile lookup at typical volumes.

phantombuster. no-code linkedin automation. covers scraping plus connection requests and messaging. see our breakdown in outscraper vs phantombuster vs hunter.io.

for production teams, the managed apis are the right choice unless you need volume that exceeds their rate limits or you are scraping data they do not offer.

what to do when an account gets restricted

linkedin restricts accounts in stages: warning, partial restriction (no search, no messages), full restriction (login redirects to verification), then permanent ban.

at warning stage: stop all automation for 7 to 14 days. log in manually from a regular browser on the same proxy. do normal user activities (scroll, like 1 to 2 posts, accept 1 connection). most accounts recover.

at partial restriction: same playbook plus complete identity verification if linkedin asks (selfie, government id). if you skip verification, the account moves to full restriction. for accounts you bought, this is usually game over.

at full restriction: usually unrecoverable without verification. for managed-api stacks, this is on the api provider, not you.

at permanent ban: replace the account. log the proxy + account combo so you do not reuse the proxy for the next account.

ethical and security notes

if you scrape eu data subjects, you must respect erasure requests. publish a privacy policy that lists linkedin as a data source and provides a removal email. process removals within 30 days.

never scrape data behind a login that requires special permission (closed groups, private messages, premium-only fields). that crosses into the cfaa unauthorized-access territory in the us and is a clear gdpr violation in the eu.

cold email or cold dm using scraped data still requires lawful basis in the eu and uk and a clear opt-out everywhere. a working email is a tool, not a license.

faq

can i scrape public linkedin profiles legally?

in the us, public profile data scraping is generally legal under hiq v linkedin, but it violates linkedin’s tos. in the eu and uk, gdpr requires a lawful basis even for public data when it identifies a person. always document your basis and offer opt-out.

what proxies should i use for linkedin scraping?

mobile proxies are safest. residential proxies with sticky sessions of 10+ minutes are the value pick. datacenter and isp proxies are blocked instantly in 2026. budget $5 to $7 per gb for residential or $50+ per port per month for mobile.

how many requests per day before linkedin bans?

aged accounts on residential proxies tolerate roughly 80 profile views, 25 searches, and 15 connection requests per day. fresh accounts on datacenter ips tolerate maybe 20 to 50 requests before banning.

is selenium or playwright better for linkedin scraping?

playwright is the better default in 2026. its anti-detection options are richer, the api is cleaner, and it handles modern js rendering more reliably. selenium still works but requires more patches to avoid headless detection.

do i need a paid linkedin sales nav account to scrape effectively?

not strictly. public profile scraping works without a paid account. sales nav unlocks deeper search filters and lead lists, which is useful for outbound. paid accounts also tolerate slightly higher rate limits before triggering anti-bot.

should i use a managed linkedin api or build my own scraper?

for under 5,000 profiles per month, managed apis (bright data, apify, proxycurl) are usually cheaper than the engineering plus account management cost. for higher volume or unique fields not in the public datasets, build your own. budget for warming aged accounts, residential proxies, and ongoing maintenance.

the bottom line

linkedin scraping in 2026 is harder than 2022 because linkedin’s anti-bot stack got better. but it is also more accessible because managed datasets cover most common use cases at a per-record price that beats diy.

self-managed approach: aged accounts, residential or mobile proxies (one per account), playwright with anti-detection, conservative rate limits. expect to replace 10 to 20 percent of accounts every quarter.

managed approach: pay $0.001 to $0.30 per record depending on freshness and depth. zero ban exposure. faster time-to-data.

for most teams in 2026 the managed approach wins on total cost. for teams scraping at very high volume or extracting fields managed apis do not surface, the diy stack still has a place. either way, document your gdpr basis and respect opt-outs. it is the difference between a sustainable lead-gen channel and a pile of legal exposure.

How to Scrape Realtor.com Property Data in 2026 (Bypass Next.js Protection)

the cleanest way to scrape realtor.com in 2026 is to extract the embedded __NEXT_DATA__ json from each listing page. it contains everything the website renders, in structured form, with no parsing brittleness. you’ll need a residential proxy to avoid the akamai bot manager block, and python with httpx + parsel does the rest. this tutorial ships working code.

why next_data is the trick

realtor.com runs on next.js. every server-rendered page bakes a hidden <script id="__NEXT_DATA__" type="application/json"> block into the html. inside that block sits the entire react state for the page: full property details, agent info, school info, pricing history, photos, the whole structured tree.

if you parse the html directly (price from .price-display, address from .address-line), realtor.com will rename or restructure that markup every few months. your scraper breaks. if you parse __NEXT_DATA__, you get raw json from their backend, and they rarely change those keys because their own frontend depends on them.

we cover this pattern in depth in our javascript-rendered pages scraping guide, but realtor.com is the textbook case.

the anti-bot situation

realtor.com sits behind akamai bot manager and a custom rate limiter. behavior:

(1) datacenter ips: blocked at the cdn edge. you get a 403 with a captcha challenge page.

(2) residential or mobile ips with a clean fingerprint: 200 ok response, 50-200 requests per ip per hour before throttling.

(3) high-volume requests from the same ip: 429 rate limit, 5-15 minute cooldown.

practical implication: you need rotating residential ips with sticky sessions long enough to fetch a single page. you don’t need a full headless browser, plain http requests with the right headers work fine.

installing dependencies

pip install httpx parsel orjson

httpx for async http, parsel for css selectors, orjson for fast json parsing. that’s it.

the basic listing fetcher

import httpx
import parsel
import orjson

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Cache-Control": "no-cache",
    "Pragma": "no-cache",
    "Sec-Ch-Ua": '"Chromium";v="126", "Not(A:Brand";v="24", "Google Chrome";v="126"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"macOS"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

def fetch_listing(url: str, proxy: str) -> dict:
    with httpx.Client(
        proxies=proxy,
        headers=HEADERS,
        timeout=30,
        http2=True,
    ) as client:
        resp = client.get(url)
        resp.raise_for_status()

        sel = parsel.Selector(resp.text)
        data = sel.css("script#__NEXT_DATA__::text").get()
        if not data:
            raise ValueError("__NEXT_DATA__ missing - likely blocked")

        return orjson.loads(data)

if __name__ == "__main__":
    proxy = "http://user-session-abc:pwd@gate.provider.com:8000"
    data = fetch_listing(
        "https://www.realtor.com/realestateandhomes-detail/123-Main-St_Anytown_CA_90210_M12345-67890",
        proxy,
    )
    print(orjson.dumps(data, option=orjson.OPT_INDENT_2).decode())

key headers:

Sec-Ch-Ua block must match the user-agent. mismatched user-agent and sec-ch-ua is one of the fastest ways to get flagged.

http2 enabled. realtor.com uses http2 internally and bot managers flag plain http/1.1 client behavior on http2 sites.

navigating next_data

the json structure looks like this (pruned):

{
  "props": {
    "pageProps": {
      "initialReduxState": {
        "propertyDetails": {
          "property": {
            "list_price": 750000,
            "address": {
              "line": "123 Main St",
              "city": "Anytown",
              "state": "CA",
              "postal_code": "90210"
            },
            "description": {
              "beds": 3,
              "baths": 2.5,
              "sqft": 1800,
              "year_built": 1985,
              "type": "single_family"
            },
            "photos": [...],
            "advertisers": [...],
            "schools": {...},
            "tax_history": [...]
          }
        }
      }
    }
  }
}

the exact path varies slightly between listing types (single family, condo, lot, rental). a robust extractor walks the tree:

def extract_property(data: dict) -> dict:
    page_props = data["props"]["pageProps"]
    redux = page_props.get("initialReduxState", {})
    prop = redux.get("propertyDetails", {}).get("property", {})

    if not prop:
        # fallback for newer page structures
        prop = page_props.get("property", {})

    return {
        "price": prop.get("list_price"),
        "address": prop.get("address", {}).get("line"),
        "city": prop.get("address", {}).get("city"),
        "state": prop.get("address", {}).get("state"),
        "postal_code": prop.get("address", {}).get("postal_code"),
        "beds": prop.get("description", {}).get("beds"),
        "baths": prop.get("description", {}).get("baths"),
        "sqft": prop.get("description", {}).get("sqft"),
        "year_built": prop.get("description", {}).get("year_built"),
        "property_type": prop.get("description", {}).get("type"),
        "photos": [p.get("href") for p in prop.get("photos", [])],
        "agent_name": (prop.get("advertisers", [{}])[0]).get("name"),
        "schools": prop.get("schools"),
        "tax_history": prop.get("tax_history"),
    }

now you have clean structured data ready for a database.

scraping search results

the listing detail page is the easy part. search result pages also embed __NEXT_DATA__ with a list of properties:

def fetch_search_results(city: str, state: str, page: int, proxy: str) -> list:
    url = f"https://www.realtor.com/realestateandhomes-search/{city}_{state}/pg-{page}"
    with httpx.Client(proxies=proxy, headers=HEADERS, timeout=30, http2=True) as client:
        resp = client.get(url)
        resp.raise_for_status()
        sel = parsel.Selector(resp.text)
        data = orjson.loads(sel.css("script#__NEXT_DATA__::text").get())

    listings = data["props"]["pageProps"]["properties"]
    return [
        {
            "property_id": l.get("property_id"),
            "url": f"https://www.realtor.com{l.get('rdc_web_url', '')}",
            "list_price": l.get("list_price"),
            "address": l.get("address"),
        }
        for l in listings
    ]

a typical search returns 42 listings per page. multi-page pagination is just incrementing pg-N until the result list is empty.

the proxy setup

residential rotating with 1-5 minute sticky sessions is what you want. one ip per page fetch keeps the request footprint tiny. avoid mobile (overkill, more expensive) and datacenter (blocked).

import uuid

def make_session_proxy() -> str:
    sid = uuid.uuid4().hex[:12]
    return f"http://user-country-us-session-{sid}:pwd@gate.provider.com:8000"

generate a fresh session per page. if a request fails, generate another and retry.

handling 403 challenges

when akamai flags you, the response is a redirect to a challenge page or an html with "unable to verify" in the body. detect both:

def looks_blocked(resp: httpx.Response) -> bool:
    if resp.status_code == 403:
        return True
    body = resp.text.lower()
    if "unable to verify" in body or "challenge" in body:
        return True
    if "<script id=\"__next_data__\"" not in body:
        return True
    return False

retry with a fresh session. if you get blocked 3 times in a row from different sessions, the entire ip range is hot. wait 10-15 minutes before retrying.

throttling for politeness

even with rotating ips, hammering realtor.com is rude and gets your provider’s pool flagged. cap your request rate:

import asyncio
import random

async def throttled_fetch(url, proxy):
    await asyncio.sleep(random.uniform(2.0, 5.0))
    return await async_fetch_listing(url, proxy)

2-5 second delay per worker, 10-20 workers in parallel. you’ll fetch 200-400 pages/minute, which is plenty without abusing the site.

storing the data

a postgres table for structured fields, jsonb column for the full extracted payload (so you can backfill new fields later):

CREATE TABLE realtor_listings (
    property_id TEXT PRIMARY KEY,
    scraped_at TIMESTAMP NOT NULL DEFAULT NOW(),
    list_price NUMERIC(12, 2),
    address TEXT,
    city TEXT,
    state TEXT,
    postal_code TEXT,
    beds NUMERIC(4, 1),
    baths NUMERIC(4, 1),
    sqft INTEGER,
    year_built INTEGER,
    property_type TEXT,
    raw JSONB,
    UNIQUE(property_id, scraped_at)
);

if your data lands in bigquery instead of postgres, the same pattern works (we wrote a scraping to bigquery pipeline that fits this scraper directly).

scaling considerations

at 200 listings/min, scraping all active us listings (~1.5M) takes ~5 days. realistic budgets:

• proxy bandwidth: ~150kb per detail page = 250mb for 1500 listings. at $4/gb residential = $1/1500 listings.
• compute: a single python worker handles 200/min. for parallel scraping, deploy 5-10 workers across regions.
• storage: a million listings is ~3gb in postgres with the jsonb column.

cheap relative to the data value. real estate scraping pipelines that produce $5k-50k/month in saas revenue spend $200-800/month on infrastructure.

legal and ethical notes

scraping public listings is widely accepted. the data is published for the world to see. but: realtor.com’s tos forbids automated access. they can block your ip range, send a cease-and-desist, or pursue legal action if you redistribute their data commercially.

(1) don’t republish realtor.com’s photos or copy. fair use for analysis is one thing, building a competing listings site is another.

(2) don’t scrape pii (broker phone numbers, emails) and resell it.

(3) respect throttling. if they ratelimit, back off.

your use case (price analysis, market trends, lead generation for buyers) is usually fine. competing directly with mls licensees is a fast way to get sued.

frequently asked questions

why doesn’t realtor.com just block next_data entirely?

their own website depends on it. removing the embedded json would break their progressive enhancement and seo. they could obfuscate keys, but they haven’t, because the cost of breaking their own analytics tooling outweighs the gain of slowing scrapers.

can i scrape with playwright instead of httpx?

you can, but it’s overkill. realtor.com’s next_data is in the initial server response, so you don’t need to wait for js execution. httpx is 10x faster and 50x cheaper.

what residential proxy provider works best for realtor.com?

any reputable residential pool with us ips. our proxy provider comparison ranks them. avoid datacenter and avoid the smaller regional providers, your success rate suffers.

how do i scrape realtor.com photos?

photo urls are in the next_data payload. download them through the same proxy pool, but at higher bandwidth cost. budget ~3mb per photo, ~30mb per listing.

does realtor.com offer an official api?

not for public scraping. they license data through partnerships and via the underlying mls feeds. licensing fees are typically $thousands/month plus per-record charges. scraping is the budget alternative.

will my scraper survive realtor.com html changes?

the next_data approach is far more stable than css selectors. expect occasional key renames (every 12-18 months) but minor adjustments rather than full rewrites.

final thoughts

realtor.com is one of the cleaner real estate scraping targets in 2026 if you go through the front door (__NEXT_DATA__ extraction with residential proxies). most failures we see are from people trying to use datacenter ips, parse the rendered html, or run headless browsers when they don’t need to. the lighter your stack, the faster and cheaper your scraper. ship the simple httpx version first, add complexity only when something specific breaks.

Web Scraping to BigQuery: Full Pipeline Tutorial (Python + Scrapy 2026)

the cleanest way to ship scraped data into bigquery in 2026 is scrapy with a custom item pipeline that streams rows via the bigquery storage write api. you batch into 5MB chunks, fail gracefully on schema mismatches, and you pay storage costs of about $0.02/gb/month. this tutorial builds the pipeline end to end with working python.

the architecture

[scrapy spider] -> [item pipeline] -> [pubsub topic] -> [cloud run worker]
                                                              |
                                                              v
                                                    [bigquery storage write api]
                                                              |
                                                              v
                                                    [bigquery table, partitioned by date]

four components. each does one thing. the spider fetches and parses, the pipeline normalizes, pubsub buffers, the worker writes to bigquery in batches. this design absorbs scraping bursts (5000 items/min) without overwhelming bigquery or paying for streaming inserts at $0.05/gb.

if you want context on why scrapy is still the right tool in 2026, our python web scraping guide walks through alternatives.

prerequisites

pip install scrapy google-cloud-bigquery google-cloud-pubsub google-cloud-bigquery-storage

you also need a gcp project with bigquery + pubsub + cloud run apis enabled, and a service account with bigquery data editor + pub/sub publisher roles. download the json key and export it:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json

designing the bigquery schema

bigquery is happiest with flat, typed columns plus partition + clustering. for a product scraper:

CREATE TABLE scraped.products (
  scraped_at TIMESTAMP NOT NULL,
  source STRING NOT NULL,
  product_id STRING NOT NULL,
  title STRING,
  price NUMERIC(12, 2),
  currency STRING,
  in_stock BOOL,
  image_url STRING,
  raw JSON,
)
PARTITION BY DATE(scraped_at)
CLUSTER BY source, product_id;

three things matter:

(1) partition by DATE(scraped_at). without this, every query scans the full table and your monthly bigquery bill goes from $5 to $500.

(2) cluster by source, product_id. lets you efficiently dedupe and run “show me this product across providers” queries.

(3) keep a raw JSON column. when your schema changes (it will), you can backfill new fields from raw without re-scraping.

the scrapy item

# items.py
import scrapy

class ProductItem(scrapy.Item):
    source = scrapy.Field()
    product_id = scrapy.Field()
    title = scrapy.Field()
    price = scrapy.Field()
    currency = scrapy.Field()
    in_stock = scrapy.Field()
    image_url = scrapy.Field()
    raw = scrapy.Field()

keep the field names matching your bigquery columns. saves a translation layer.

the spider

# spiders/example_products.py
import scrapy
from myproject.items import ProductItem

class ExampleProductsSpider(scrapy.Spider):
    name = "example_products"
    start_urls = ["https://example.com/products"]
    custom_settings = {
        "DOWNLOAD_DELAY": 0.5,
        "CONCURRENT_REQUESTS": 16,
        "ITEM_PIPELINES": {
            "myproject.pipelines.PubSubPipeline": 300,
        },
    }

    def parse(self, response):
        for card in response.css("div.product-card"):
            yield ProductItem(
                source="example.com",
                product_id=card.css("::attr(data-id)").get(),
                title=card.css("h2::text").get(),
                price=float(card.css("span.price::text").re_first(r"[\d.]+") or 0),
                currency="USD",
                in_stock="in stock" in card.css(".stock::text").get("").lower(),
                image_url=card.css("img::attr(src)").get(),
                raw=card.get(),
            )

the pubsub pipeline

scrapy items get serialized as json and published to a pubsub topic. we batch by message size and time window:

# pipelines.py
import json
import datetime as dt
from google.cloud import pubsub_v1

class PubSubPipeline:
    def open_spider(self, spider):
        self.publisher = pubsub_v1.PublisherClient(
            batch_settings=pubsub_v1.types.BatchSettings(
                max_messages=500,
                max_bytes=5_000_000,
                max_latency=2.0,
            )
        )
        self.topic_path = self.publisher.topic_path(
            "your-gcp-project", "scraped-products"
        )
        self.futures = []

    def process_item(self, item, spider):
        row = dict(item)
        row["scraped_at"] = dt.datetime.utcnow().isoformat()
        data = json.dumps(row).encode("utf-8")
        future = self.publisher.publish(self.topic_path, data)
        self.futures.append(future)
        return item

    def close_spider(self, spider):
        # flush remaining batches
        for fut in self.futures:
            fut.result(timeout=30)

scrapy spawns this pipeline per process. for a single-host crawl, that’s fine. for distributed crawling across many machines (see our distributed scraping architecture), each machine independently pushes to pubsub and the worker downstream handles dedupe.

the cloud run worker

cloud run subscribes to the pubsub topic, batches messages, and writes them to bigquery via the storage write api. this is where 90% of the cost savings live. the storage write api is roughly 50x cheaper than streaming inserts on a per-row basis.

# worker.py
import os
import json
from concurrent.futures import ThreadPoolExecutor
from flask import Flask, request
from google.cloud import bigquery_storage_v1
from google.cloud.bigquery_storage_v1 import types, writer
from google.protobuf import descriptor_pb2

app = Flask(__name__)

PROJECT = os.environ["GCP_PROJECT"]
DATASET = "scraped"
TABLE = "products"

client = bigquery_storage_v1.BigQueryWriteClient()
parent = client.table_path(PROJECT, DATASET, TABLE)
write_stream = types.WriteStream(type_=types.WriteStream.Type.COMMITTED)
write_stream = client.create_write_stream(parent=parent, write_stream=write_stream)

@app.post("/")
def handle():
    envelope = request.get_json()
    msg = envelope.get("message", {})
    data = json.loads(base64.b64decode(msg["data"]).decode())
    # build proto row from data, append to write stream
    # in production: batch incoming requests, write 500 rows at a time
    return ("", 204)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

(this is the simplified handler. the full version with proto schema generation lives at github.com/dataresearchtools/scraping-bigquery-pipeline.)

deploy:

gcloud run deploy bq-writer \
  --source . \
  --region us-central1 \
  --no-allow-unauthenticated \
  --service-account scraper-sa@$PROJECT.iam.gserviceaccount.com \
  --memory 512Mi --cpu 1 --max-instances 10

then create a pubsub push subscription targeting the cloud run url. messages flow in, get written, and you pay roughly 1/50th of what streaming inserts would cost.

handling schema drift

scrapers break. sites add fields, change html, drop columns. when that happens, your row fails to write because the schema doesn’t match.

the fix is two-tier:

(1) write the typed columns you know about (title, price, etc).

(2) shove the entire raw item into the raw JSON column.

if you add a new field to the schema next month, you backfill from the raw column with a single query:

UPDATE scraped.products
SET seller_id = JSON_VALUE(raw, '$.seller_id')
WHERE scraped_at >= '2026-04-01'
  AND seller_id IS NULL;

zero re-scraping. zero downtime. this is the single biggest reliability gain in the pipeline.

cost in 2026

real numbers from a pipeline that scrapes ~500k products/day:

• bigquery storage: ~$0.50/month for 25gb
• bigquery query: ~$5/month for analyst dashboards (partitioned, clustered)
• pubsub: ~$0.40/month for 500k msg/day
• cloud run worker: ~$2/month at 10 cpu-minutes/day
• proxy bandwidth (the actual scraping): the dominant cost, $50-200/month

total infrastructure for a working pipeline: under $10/month. proxies are everything. picking the right provider matters more than any cloud optimization.

monitoring

three queries to bookmark:

-- rows scraped per hour, last 24h
SELECT
  TIMESTAMP_TRUNC(scraped_at, HOUR) AS hour,
  source,
  COUNT(*) AS rows
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
GROUP BY hour, source
ORDER BY hour DESC;

-- failed rows (null titles imply parse failure)
SELECT source, COUNT(*) AS bad_rows
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
  AND title IS NULL
GROUP BY source;

-- price drift detection
SELECT
  product_id,
  ANY_VALUE(title) AS title,
  MIN(price) AS min_price,
  MAX(price) AS max_price,
  STDDEV(price) AS price_volatility
FROM scraped.products
WHERE scraped_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY product_id
HAVING price_volatility > 5
ORDER BY price_volatility DESC
LIMIT 50;

wire the first two into looker studio or a slack alerting bot. the third is gold for ecommerce intelligence dashboards.

dedupe strategies

scraping the same product 10 times a day is fine for price tracking. it’s wasteful for snapshot exports. two patterns:

(1) merge on read with ROW_NUMBER():

SELECT * FROM (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY source, product_id
      ORDER BY scraped_at DESC
    ) AS rn
  FROM scraped.products
) WHERE rn = 1;

(2) materialize a latest_products table with a scheduled query that runs hourly. cheaper to query, slightly stale.

most teams start with (1) and graduate to (2) when query volume grows.

the rare case for streaming inserts

if you need sub-second latency from scrape to dashboard (real-time price alerts, fraud detection), use streaming inserts despite the cost. otherwise, the storage write api batched flow is faster, cheaper, and equally reliable.

handling proxies in the spider

scraping at scale means rotating residential proxies. add scrapy’s HTTPPROXY_AUTH_ENCODING = 'utf-8' and a downloader middleware:

# middlewares.py
class RotatingProxyMiddleware:
    def __init__(self):
        self.gateway = "http://gate.provider.com:8000"
        self.user = os.environ["PROXY_USER"]
        self.pwd = os.environ["PROXY_PWD"]

    def process_request(self, request, spider):
        request.meta["proxy"] = (
            f"http://{self.user}:{self.pwd}@{self.gateway.split('://')[1]}"
        )

enable in settings:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

your scraper now hits target sites through residential rotation, parses, and ships data into bigquery via the pipeline above.

frequently asked questions

why pubsub between scrapy and bigquery?

backpressure. scrapy bursts faster than bigquery wants to absorb. pubsub holds up to 7 days of messages and the worker drains at a sustainable rate. without it, bigquery rejects writes during traffic spikes.

can i skip pubsub and write directly to bigquery from scrapy?

yes for small jobs (under 100 items/min). for anything larger, the streaming insert costs and rate limits make pubsub + storage write api dramatically cheaper.

what about cloud sql or snowflake instead?

cloud sql is wrong here, it’s a transactional database not analytics. snowflake works the same way as bigquery (storage + query separation), pick whichever your team already uses. the pipeline pattern is identical.

how do i handle gdpr or pii in scraped data?

never scrape pii unless your legal team has approved it. if you must, use bigquery’s authorized views and column-level access control. partition by retention period and schedule deletion via DELETE statements in scheduled queries.

what’s the cheapest way to backfill historical data?

batch loads via bq load from gcs, not streaming. costs near zero compared to per-row writes. your scrapy pipeline can write to gcs files in parallel and a daily job loads them all in one go.

does this work with playwright scrapers instead of scrapy?

yes. the pipeline pattern (push to pubsub, write from worker) is framework-agnostic. anything that can call the pubsub publisher api can feed bigquery this way.

final thoughts

a clean scraping-to-bigquery pipeline is mostly about discipline, not novelty. partition your tables, keep a raw json column, batch writes through pubsub, and your engineering cost falls to single digits per month. the only number that matters at scale is your proxy bill. pick the provider carefully, design the schema once, and the pipeline runs for years.

how to scrape google local pack results (maps + business data) 2026

google local pack is the 3-result map block that appears on serps for local-intent queries like “coffee shop near me” or “lawyer in austin.” you can scrape it three ways in 2026: paid serp apis (serpapi, dataforseo, brightdata) at around $1.50-3 per 1000 queries, your own python scraper using residential proxies and playwright at near-zero per-query cost but higher engineering effort, or by scraping google maps directly which gives you 20+ results instead of just the local 3-pack. this tutorial covers all three with working code.

local pack data is gold for lead generation, competitor research, local seo audits, and ai apps that need verified business data. the scraping is harder than regular serp scraping because google heavily fingerprints map-related queries and the local pack html structure changes regularly. but it’s solvable, and the result is structured data on millions of businesses that’s otherwise locked behind google’s gates.

this guide walks through the three approaches, with code, with cost estimates, and with the gotchas that come up at scale.

what’s in the local pack

a typical local pack result on a query like “plumber miami” returns:

• 3 business listings (top 3 by google’s local ranking)
• each listing has: business name, rating (1-5 stars), review count, category, address, hours snippet, phone (sometimes), website (sometimes), gbid (google business id), latitude/longitude
• a “view all” link that opens the local finder (top 20 results)
• ad placements above and below sometimes

the underlying data lives in google’s local index, accessible via the regular serp html, the maps web ui, and the maps mobile app. each surface returns slightly different fields. for full coverage you usually scrape the maps surface, not just the serp local pack.

approach 1: paid serp apis (easiest)

three providers dominate this space in 2026:

• serpapi: $50/month for 5000 searches. local pack data included with engine=google_local.
• dataforseo: $0.0006-0.001 per organic search depending on plan. dedicated local pack endpoint.
• brightdata serp api: $1.50 per 1000 searches. covers all serp features including local pack.

for any serious volume the per-query rates push under $1.50/1k. for prototypes and small jobs they are by far the easiest path.

import requests

SERPAPI_KEY = "your-key"

def get_local_pack(query, location):
    r = requests.get("https://serpapi.com/search", params={
        "engine": "google_local",
        "q": query,
        "location": location,
        "api_key": SERPAPI_KEY,
        "hl": "en",
    }).json()
    return r.get("local_results", [])

results = get_local_pack("plumber", "miami, florida")
for r in results[:5]:
    print(r["title"], r.get("rating"), r.get("phone"), r.get("address"))

dataforseo’s local pack endpoint:

import requests
from requests.auth import HTTPBasicAuth

post_data = [{
    "keyword": "plumber",
    "location_name": "Miami,Florida,United States",
    "language_code": "en",
    "device": "desktop",
}]

r = requests.post(
    "https://api.dataforseo.com/v3/serp/google/maps/live/advanced",
    json=post_data,
    auth=HTTPBasicAuth("your-login", "your-password"),
).json()

items = r["tasks"][0]["result"][0]["items"]
for item in items[:10]:
    print(item["title"], item.get("rating", {}).get("value"))

dataforseo’s pricing is the most aggressive at scale (under $0.001 per query at volume). serpapi has the friendliest sdk and free tier. bright data is the most reliable at very high volume.

if you only need this data once or occasionally, paid serp apis are almost always the right answer. you spend $5-50, you get the data, you move on.

approach 2: python scraper with residential proxies

cheaper at scale, more engineering work upfront. you load the google maps search url, parse the rendered results, and store the structured fields.

import asyncio
import json
import re
from playwright.async_api import async_playwright

PROXY = {
    "server": "http://residential.example.com:8080",
    "username": "user",
    "password": "pass",
}

async def scrape_maps(query, location):
    url = f"https://www.google.com/maps/search/{query.replace(' ', '+')}+{location.replace(' ', '+')}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False, proxy=PROXY)
        ctx = await browser.new_context(
            viewport={"width": 1366, "height": 768},
            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            locale="en-US",
        )
        page = await ctx.new_page()
        await page.goto(url, wait_until="networkidle", timeout=30000)
        await page.wait_for_timeout(3000)

        # scroll the results panel to load more
        results_panel = page.locator("div[role='feed']")
        for _ in range(3):
            await results_panel.evaluate("el => el.scrollBy(0, 800)")
            await page.wait_for_timeout(1500)

        # extract listing elements
        items = await page.locator("div[role='feed'] > div > div[jsaction]").all()
        results = []
        for item in items[:20]:
            try:
                name = await item.locator("div.fontHeadlineSmall").first.text_content()
                rating_el = await item.locator("span[role='img'][aria-label*='star']").first.get_attribute("aria-label")
                results.append({
                    "name": name.strip() if name else None,
                    "rating_aria": rating_el,
                })
            except Exception:
                continue

        await browser.close()
        return results

async def main():
    results = await scrape_maps("plumber", "miami florida")
    print(json.dumps(results, indent=2))

asyncio.run(main())

this is the rough shape. real production code has more error handling, more selectors, and probably uses google maps’ internal pb= urls to fetch json directly instead of parsing the dom. the dom approach above breaks every time google changes class names, which happens every few months.

a more robust pattern is to capture the maps json endpoint via network interception:

async def capture_maps_json(query, location):
    captured = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        ctx = await browser.new_context()
        page = await ctx.new_page()

        async def handle_response(response):
            if "search?" in response.url and "/maps/" in response.url:
                try:
                    body = await response.text()
                    if body.startswith(")]}'"):
                        body = body[5:]
                    captured.append(json.loads(body))
                except Exception:
                    pass

        page.on("response", handle_response)
        await page.goto(f"https://www.google.com/maps/search/{query}+{location}",
                        wait_until="networkidle")
        await page.wait_for_timeout(5000)
        await browser.close()
    return captured

the captured json contains the full structured data google sees on its end. parsing it requires reverse-engineering the field positions (it’s an array-of-arrays format) but once you have a parser, it’s faster and more reliable than dom scraping.

for the proxy choice, residential is the minimum. mobile proxies pass through the toughest google blocks more reliably. datacenter ips are blocked within a few queries. see the residential proxy guide for context.

approach 3: scrape regular google serp html

if you only need the 3-pack (not the full 20-result local finder), you can scrape the regular google serp page. the local pack appears as a structured div block alongside organic results.

import requests
from bs4 import BeautifulSoup

PROXY = {"http": "http://user:pass@residential.example.com:8080",
         "https": "http://user:pass@residential.example.com:8080"}

def scrape_serp_local(query, geo_param):
    url = f"https://www.google.com/search?q={query}&uule={geo_param}&hl=en"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept": "text/html,application/xhtml+xml",
        "Accept-Language": "en-US,en;q=0.9",
    }
    r = requests.get(url, headers=headers, proxies=PROXY, timeout=30)
    soup = BeautifulSoup(r.text, "html.parser")

    # local pack container varies, but rllt__details is reliable for 3-pack listings
    listings = []
    for div in soup.select("div.rllt__details"):
        name = div.select_one("div.dbg0pd")
        if name:
            listings.append({
                "name": name.get_text(strip=True),
                "snippet": " ".join(s.get_text() for s in div.select("div") if s != name),
            })
    return listings

the uule parameter is google’s encoded location. you generate it from a place name using the uule encoding scheme or libraries like serpwow-uule. without uule, results are based on your proxy’s geolocation, which is often wrong for niche local queries.

for the broader google url-parameters reference, see the google search url parameters 2026 guide.

extracting individual business details

the local pack listings give you basic data. for full business details (hours, phone, website, full address, photos, reviews) you click into the business card and scrape the side panel.

async def scrape_business_details(page, business_url):
    await page.goto(business_url, wait_until="networkidle")
    await page.wait_for_timeout(2000)

    name = await page.locator("h1").first.text_content()
    address = await page.locator("button[data-item-id='address']").first.text_content()
    phone_el = page.locator("button[data-item-id^='phone']").first
    phone = await phone_el.text_content() if await phone_el.count() else None
    website_el = page.locator("a[data-item-id='authority']").first
    website = await website_el.get_attribute("href") if await website_el.count() else None
    rating = await page.locator("div.F7nice span[aria-hidden='true']").first.text_content()

    return {
        "name": name.strip() if name else None,
        "address": address.strip() if address else None,
        "phone": phone.strip() if phone else None,
        "website": website,
        "rating": rating,
    }

the gbid (google business id, also called cid) is in the page url after navigation. extract from page.url with a regex on the place/.../@.../data= segment.

handling pagination and load more

google maps doesn’t paginate the local finder in the traditional sense. it loads more results as you scroll the left-side panel. the playwright code above scrolls 3 times. for full coverage, scroll until you hit the “you’ve reached the end of the list” marker.

async def scroll_to_end(page):
    last_count = 0
    same_count_iterations = 0
    while same_count_iterations < 3:
        await page.locator("div[role='feed']").evaluate("el => el.scrollBy(0, 1000)")
        await page.wait_for_timeout(1500)
        items = await page.locator("div[role='feed'] > div > div[jsaction]").count()
        if items == last_count:
            same_count_iterations += 1
        else:
            same_count_iterations = 0
            last_count = items
    return last_count

most categories cap at 120 results in google maps. some niche or local queries cap at 20-40. that’s a hard ceiling.

rate limiting and avoiding blocks

google’s anti-scraping is aggressive on maps. patterns that get you blocked fast:

• many queries from the same ip in quick succession
• queries with no realistic delay between them
• consistent user-agent across all requests
• queries from datacenter ips
• non-residential geolocation mismatch (querying us businesses from a singapore ip)

mitigations:

• residential or mobile proxies, rotated per query
• 5-10 second delay between queries minimum
• random user-agent across a pool of 10-20 valid ones
• match your proxy geo to your query geo where possible
• spread queries across hours, not in a 1-minute burst

with those mitigations in place, a single residential proxy can do 100-500 queries a day before getting flagged. with mobile proxies, several thousand. for higher volume, parallelize across many proxies.

cost comparison

estimating cost for 100,000 local pack queries.

approach	cost	engineering effort
serpapi	$1000 (4x growth plan)	minimal
dataforseo	$60-150	moderate (sdk integration)
brightdata serp api	$150	minimal
diy with residential proxies	$50-100 (proxy bandwidth)	high (build + maintain)
diy with mobile proxies	$300-500 (mobile bandwidth)	high (build + maintain)

dataforseo wins on raw cost at scale. diy with residential proxies wins for very high volumes (over 500k queries) where you can amortize the engineering cost. for under 100k queries, dataforseo is hard to beat.

faq

is scraping google maps legal?
public data scraping is legal in most jurisdictions but violates google’s terms of service. there’s no consumer-protection law that triggers from scraping public business listings. for commercial use cases at scale, talk to a lawyer about cfaa exposure. the web scraping legal guide covers the case law.

can i use the official google places api instead?
yes, and you should if your use case fits. places api charges $17-32 per 1000 requests and is rate-limited. for small volumes it’s competitive with serp apis. for high volume scraping is far cheaper.

how many results does google local pack actually return?
the visible 3-pack is just the top 3. the local finder (clicking “view all”) shows up to 120. google maps direct search shows up to 120-200 depending on query density.

does serpapi return the gbid?
yes, in the place_id field. some legacy responses use gbid directly. dataforseo also returns it. roll-your-own scraping requires extracting it from the place url.

which proxy type works best for google maps?
residential or mobile. datacenter ips get blocked within a few queries. mobile is more reliable for high-volume sustained scraping. for context see the residential proxy guide.

how do i scrape google reviews for a business?
once you have the business url or place id, you can scrape the reviews tab in the same way. each review is a single json item in the maps response. expect 200-500 reviews per page load with infinite scroll.

conclusion

google local pack scraping is a solved problem in 2026 if you’re willing to spend on a paid serp api. dataforseo at $0.001 per query is the price-to-value sweet spot. serpapi is the easiest first integration. brightdata is the most reliable at very high volume.

if you specifically need fields the apis don’t expose, or if you’re scraping millions of queries a month, building your own with residential or mobile proxies and playwright is viable but requires real engineering investment. the dom selectors break every few months. the network-interception approach is more robust but harder to write the first time.

start with a paid serp api. measure your data needs against what they return. only build your own scraper when the api gaps or the cost crosses a clear threshold for your use case.

Web Scraping with VBA/Excel: No-Code Data Pull

Excel is the most accessible web scraping tool available. You do not need Python, Node.js, or any programming framework. Excel’s built-in Power Query handles many data import tasks with zero code, and VBA (Visual Basic for Applications) provides full scraping capabilities for more complex needs. If your goal is getting web data into a spreadsheet, Excel might be all you need.

This tutorial covers three approaches: Power Query (no code), Web Query (legacy), and VBA macros (full control).

Table of Contents

• When to Use Excel for Scraping
• Method 1: Power Query (No Code)
• Method 2: Web Query (Legacy)
• Method 3: VBA Macros
• VBA HTTP Requests
• VBA HTML Parsing
• Scraping Multiple Pages
• Handling Tables
• Error Handling
• Scheduling Automatic Updates
• Limitations and Alternatives
• FAQ

When to Use Excel for Scraping

Excel scraping is ideal when:

• Your end goal is a spreadsheet (no data pipeline needed)
• You are scraping HTML tables or structured data
• You need a one-off data pull, not a recurring crawler
• Your team does not have Python/Node.js skills
• You are pulling data from a small number of pages (under 100)

Excel is NOT ideal for JavaScript-rendered pages, large-scale crawling, or sites requiring proxy rotation. For those, see our Python scraping guide.

Method 1: Power Query (No Code)

Power Query is Excel’s built-in data import tool. It handles most table-based web scraping without any code.

Steps

1. Open Excel and go to Data > From Web
2. Enter the URL (e.g., https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal))
3. Excel detects tables on the page automatically
4. Select the table you want and click Load

The data imports directly into your spreadsheet.

Power Query M Code (Advanced)

For more control, use Power Query’s M language:

let
    Source = Web.Page(
        Web.Contents("https://books.toscrape.com/")
    ),
    // Select specific table
    Data = Source{0}[Data],
    // Rename columns
    Renamed = Table.RenameColumns(Data, {
        {"Column1", "Title"},
        {"Column2", "Price"}
    }),
    // Filter rows
    Filtered = Table.SelectRows(Renamed, each [Price] <> null)
in
    Filtered

Refreshing Data

Right-click the imported table and select Refresh to pull updated data. You can also set automatic refresh intervals:

1. Right-click the query in the Queries & Connections pane
2. Select Properties
3. Check Refresh every X minutes

Method 2: Web Query (Legacy)

The traditional web query approach still works in older Excel versions:

1. Go to Data > From Web (or Data > Get External Data > From Web in older versions)
2. Enter the URL
3. Click the yellow arrows next to tables you want to import
4. Click Import

This method auto-detects HTML tables and imports them directly.

Method 3: VBA Macros

VBA gives you full control over HTTP requests and HTML parsing.

Setting Up VBA

1. Press Alt + F11 to open the VBA editor
2. Go to Tools > References and enable:

• Microsoft XML, v6.0 (for HTTP requests)
• Microsoft HTML Object Library (for HTML parsing)

1. Insert a new module: Insert > Module

Basic VBA Scraper

Sub ScrapeBooks()
    Dim http As New MSXML2.XMLHTTP60
    Dim html As New HTMLDocument
    Dim books As Object
    Dim book As Object
    Dim row As Long

    ' Send HTTP request
    http.Open "GET", "https://books.toscrape.com/", False
    http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
    http.send

    ' Parse HTML
    html.body.innerHTML = http.responseText

    ' Find all book elements
    Set books = html.querySelectorAll("article.product_pod")

    ' Write headers
    Cells(1, 1).Value = "Title"
    Cells(1, 2).Value = "Price"
    Cells(1, 3).Value = "Rating"

    ' Extract data
    row = 2
    Dim i As Long
    For i = 0 To books.Length - 1
        Set book = books.Item(i)

        Cells(row, 1).Value = book.querySelector("h3 a").getAttribute("title")
        Cells(row, 2).Value = book.querySelector(".price_color").innerText
        Cells(row, 3).Value = Replace(book.querySelector("p").className, "star-rating ", "")

        row = row + 1
    Next i

    MsgBox "Scraped " & (row - 2) & " books!"
End Sub

VBA HTTP Requests

GET Request

Function FetchPage(url As String) As String
    Dim http As New MSXML2.XMLHTTP60

    http.Open "GET", url, False
    http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
    http.setRequestHeader "Accept", "text/html"
    http.send

    If http.Status = 200 Then
        FetchPage = http.responseText
    Else
        FetchPage = ""
        Debug.Print "Error: HTTP " & http.Status & " for " & url
    End If
End Function

POST Request

Function PostRequest(url As String, postData As String) As String
    Dim http As New MSXML2.XMLHTTP60

    http.Open "POST", url, False
    http.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
    http.setRequestHeader "User-Agent", "Mozilla/5.0"
    http.send postData

    PostRequest = http.responseText
End Function

' Usage
Dim result As String
result = PostRequest("https://example.com/search", "query=laptops&page=1")

JSON API Request

Function FetchJSON(url As String) As String
    Dim http As New MSXML2.XMLHTTP60

    http.Open "GET", url, False
    http.setRequestHeader "Accept", "application/json"
    http.setRequestHeader "User-Agent", "Mozilla/5.0"
    http.send

    FetchJSON = http.responseText
End Function

' Parse JSON (requires VBA-JSON library or manual parsing)
' Download from: https://github.com/VBA-tools/VBA-JSON

VBA HTML Parsing

querySelector and querySelectorAll

Dim html As New HTMLDocument
html.body.innerHTML = httpResponseText

' Single element
Dim title As Object
Set title = html.querySelector("h1")
Debug.Print title.innerText

' Multiple elements
Dim items As Object
Set items = html.querySelectorAll(".product-card")
Debug.Print "Found " & items.Length & " items"

' Attributes
Dim link As Object
Set link = html.querySelector("a.product-link")
Debug.Print link.getAttribute("href")

' Nested selection
Dim container As Object
Set container = html.querySelector(".products")
Dim childItems As Object
Set childItems = container.querySelectorAll(".item")

Common Selectors

' By class
html.querySelectorAll(".product")

' By ID
html.querySelector("#main-content")

' By attribute
html.querySelectorAll("a[href]")
html.querySelectorAll("[data-id='123']")

' By tag
html.querySelectorAll("tr")

' Combined
html.querySelectorAll("div.product h3 a")

' Nested
html.querySelectorAll("table tbody tr td")

getElementById and getElementsByTagName

' By ID (returns single element)
Dim mainDiv As Object
Set mainDiv = html.getElementById("main-content")

' By tag name (returns collection)
Dim allLinks As Object
Set allLinks = html.getElementsByTagName("a")

Dim i As Long
For i = 0 To allLinks.Length - 1
    Debug.Print allLinks.Item(i).getAttribute("href")
Next i

' By class name
Dim products As Object
Set products = html.getElementsByClassName("product")

Scraping Multiple Pages

Sub ScrapeAllPages()
    Dim http As New MSXML2.XMLHTTP60
    Dim html As New HTMLDocument
    Dim row As Long
    Dim page As Long

    ' Headers
    Cells(1, 1).Value = "Title"
    Cells(1, 2).Value = "Price"
    Cells(1, 3).Value = "Page"

    row = 2

    For page = 1 To 50
        Dim url As String
        url = "https://books.toscrape.com/catalogue/page-" & page & ".html"

        ' Fetch page
        http.Open "GET", url, False
        http.setRequestHeader "User-Agent", "Mozilla/5.0"
        http.send

        If http.Status <> 200 Then
            Debug.Print "Error on page " & page & ": HTTP " & http.Status
            Exit For
        End If

        html.body.innerHTML = http.responseText

        ' Extract books
        Dim books As Object
        Set books = html.querySelectorAll("article.product_pod")

        If books.Length = 0 Then Exit For

        Dim i As Long
        For i = 0 To books.Length - 1
            Dim book As Object
            Set book = books.Item(i)

            Cells(row, 1).Value = book.querySelector("h3 a").getAttribute("title")
            Cells(row, 2).Value = book.querySelector(".price_color").innerText
            Cells(row, 3).Value = page

            row = row + 1
        Next i

        ' Status update
        Application.StatusBar = "Scraping page " & page & "... (" & (row - 2) & " books)"
        DoEvents

        ' Polite delay (1 second)
        Application.Wait Now + TimeValue("00:00:01")
    Next page

    Application.StatusBar = False
    MsgBox "Done! Scraped " & (row - 2) & " books from " & (page - 1) & " pages."
End Sub

Handling Tables

Automatic Table Extraction

Sub ExtractTable()
    Dim html As New HTMLDocument
    Dim http As New MSXML2.XMLHTTP60

    http.Open "GET", "https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)", False
    http.send

    html.body.innerHTML = http.responseText

    ' Find first table with class "wikitable"
    Dim table As Object
    Set table = html.querySelector("table.wikitable")

    If table Is Nothing Then
        MsgBox "No table found!"
        Exit Sub
    End If

    ' Extract rows
    Dim rows As Object
    Set rows = table.querySelectorAll("tr")

    Dim row As Long
    row = 1

    Dim r As Long
    For r = 0 To rows.Length - 1
        Dim cells As Object
        Set cells = rows.Item(r).querySelectorAll("th, td")

        Dim c As Long
        For c = 0 To cells.Length - 1
            Cells(row, c + 1).Value = CleanText(cells.Item(c).innerText)
        Next c

        row = row + 1
    Next r

    MsgBox "Extracted " & (row - 1) & " rows!"
End Sub

Function CleanText(text As String) As String
    ' Remove extra whitespace and line breaks
    CleanText = Trim(Replace(Replace(text, vbLf, " "), vbCr, " "))
    ' Remove multiple spaces
    Do While InStr(CleanText, "  ") > 0
        CleanText = Replace(CleanText, "  ", " ")
    Loop
End Function

Error Handling

Sub SafeScrape()
    On Error GoTo ErrorHandler

    Dim http As New MSXML2.XMLHTTP60
    Dim html As New HTMLDocument

    http.Open "GET", "https://books.toscrape.com/", False
    http.setRequestHeader "User-Agent", "Mozilla/5.0"
    http.send

    If http.Status <> 200 Then
        MsgBox "HTTP Error: " & http.Status
        Exit Sub
    End If

    html.body.innerHTML = http.responseText

    ' Check if element exists before accessing
    Dim title As Object
    Set title = html.querySelector("h1")

    If Not title Is Nothing Then
        Debug.Print "Title: " & title.innerText
    Else
        Debug.Print "Title element not found"
    End If

    Exit Sub

ErrorHandler:
    MsgBox "Error " & Err.Number & ": " & Err.Description
    Debug.Print "Error in SafeScrape: " & Err.Description
End Sub

Retry Logic

Function FetchWithRetry(url As String, maxRetries As Long) As String
    Dim http As New MSXML2.XMLHTTP60
    Dim attempt As Long

    For attempt = 1 To maxRetries
        On Error Resume Next

        http.Open "GET", url, False
        http.setRequestHeader "User-Agent", "Mozilla/5.0"
        http.send

        If Err.Number = 0 And http.Status = 200 Then
            FetchWithRetry = http.responseText
            Exit Function
        End If

        On Error GoTo 0
        Debug.Print "Attempt " & attempt & " failed for " & url
        Application.Wait Now + TimeValue("00:00:02")
    Next attempt

    FetchWithRetry = ""
End Function

Scheduling Automatic Updates

Windows Task Scheduler

1. Save your workbook as .xlsm (macro-enabled)
2. Create a VBS wrapper script:

' run_scraper.vbs — save as a .vbs file
Set objExcel = CreateObject("Excel.Application")
objExcel.Visible = False
Set objWorkbook = objExcel.Workbooks.Open("C:\path\to\scraper.xlsm")
objExcel.Run "ScrapeAllPages"
objWorkbook.Save
objWorkbook.Close
objExcel.Quit

1. Open Windows Task Scheduler
2. Create a new task that runs wscript.exe "C:\path\to\run_scraper.vbs"
3. Set your desired schedule (daily, weekly, etc.)

Auto-Run on Open

' In ThisWorkbook module
Private Sub Workbook_Open()
    ' Ask before running
    If MsgBox("Run the scraper?", vbYesNo) = vbYes Then
        Call ScrapeAllPages
    End If
End Sub

Limitations and Alternatives

Excel VBA Cannot:

• Render JavaScript (use Playwright or Selenium)
• Rotate proxies efficiently (use Python with proxies)
• Handle CAPTCHAs or advanced anti-bot measures
• Scale to thousands of pages (performance degrades)
• Run on macOS reliably (VBA support is limited)

Better Tools for Complex Scraping:

• Power Query — Built into Excel, handles many tasks without VBA
• Google Sheets IMPORTHTML — =IMPORTHTML("url", "table", 1) for simple table imports
• Python — For anything beyond basic table extraction. See our Python web scraping guide
• Browser extensions — Tools like Web Scraper or Data Miner for visual scraping

FAQ

Can Excel scrape any website?

Excel can scrape static HTML websites. It cannot handle JavaScript-rendered content, CAPTCHAs, or sites with aggressive anti-bot protection. For those scenarios, use Python with Playwright or a dedicated scraping tool.

Is Power Query better than VBA for web scraping?

For table-based data, Power Query is better — it requires no code and auto-refreshes. VBA is better when you need to parse non-table HTML, handle pagination, or perform complex data extraction logic.

Can I use proxies with Excel VBA?

VBA’s XMLHTTP uses system proxy settings. You can configure a proxy in Windows Internet Options, but proxy rotation is not practical in VBA. For proxy-based scraping, use Python with rotating proxies.

How many pages can Excel VBA scrape?

Practically, Excel VBA handles up to a few hundred pages before becoming slow. The spreadsheet itself becomes unwieldy beyond 100,000 rows. For large-scale scraping, use Python with Scrapy.

Does web scraping in Excel work on Mac?

Limited. VBA on macOS does not support the MSXML2.XMLHTTP or HTMLDocument objects. Power Query works on Mac with Microsoft 365 but has fewer data source options. For Mac users, Python is the recommended alternative.

---

For more advanced scraping, explore Python web scraping and our proxy glossary. See our web scraping proxy guide for proxy setup.

External Resources:

• Microsoft Power Query Documentation
• VBA MSXML Documentation)
• VBA-JSON Library
• aiohttp + BeautifulSoup: Async Python Scraping
• Axios + Cheerio: Lightweight Node.js Scraping
• How Anti-Bot Systems Detect Scrapers (Cloudflare, Akamai, PerimeterX)
• API vs Web Scraping: When You Need Proxies (and When You Don’t)
• ASEAN Data Protection Laws: A Web Scraping Compliance Matrix
• How to Build an Ethical Web Scraping Policy for Your Company
• aiohttp + BeautifulSoup: Async Python Scraping
• Axios + Cheerio: Lightweight Node.js Scraping
• How Anti-Bot Systems Detect Scrapers (Cloudflare, Akamai, PerimeterX)
• API vs Web Scraping: When You Need Proxies (and When You Don’t)
• ASEAN Data Protection Laws: A Web Scraping Compliance Matrix
• How to Build an Ethical Web Scraping Policy for Your Company

---

Related Reading

• aiohttp + BeautifulSoup: Async Python Scraping
• Axios + Cheerio: Lightweight Node.js Scraping
• How Anti-Bot Systems Detect Scrapers (Cloudflare, Akamai, PerimeterX)
• API vs Web Scraping: When You Need Proxies (and When You Don’t)
• ASEAN Data Protection Laws: A Web Scraping Compliance Matrix
• How to Build an Ethical Web Scraping Policy for Your Company

Reddit is one of the most proxy-hostile platforms on the internet. It aggressively detects and blocks datacenter IPs, VPN connections, and low-quality proxies. Whether you need a proxy for Reddit to manage multiple accounts, scrape data for research, or access content from different regions, choosing the right proxy type is critical to avoiding bans and shadowbans.

This guide explains which proxy types work best for Reddit, why most proxies fail, and how to set up a reliable connection that won’t get flagged.

Why You Might Need a Proxy for Reddit

• Multi-account management – Running multiple Reddit accounts for marketing, community management, or brand monitoring
• Data scraping – Collecting posts, comments, or subreddit data for research, sentiment analysis, or market intelligence
• Bypassing IP bans – Getting around an IP ban that may have been applied unfairly or affected your entire network
• Privacy – Browsing Reddit without linking activity to your real IP address
• Regional access – Viewing region-locked content or seeing how content appears in different locations
• Automation – Running bots for upvote tracking, keyword monitoring, or automated posting

Why Most Proxies Fail on Reddit

Reddit’s anti-abuse system is sophisticated. Here’s what it checks:

• IP reputation databases – Reddit cross-references IPs against known proxy/VPN/datacenter lists. If your IP is flagged in any major database, Reddit will block or shadowban it immediately.
• ASN (Autonomous System Number) checks – Reddit identifies the network owner of each IP. Datacenter ASNs (AWS, DigitalOcean, OVH, etc.) are treated as high-risk by default.
• Behavioral analysis – Patterns like posting from multiple accounts on the same IP, rapid actions, or inhuman browsing patterns trigger automated flags.
• Browser fingerprinting – Reddit tracks browser characteristics to link accounts even when IPs differ.
• Rate limiting – Aggressive rate limits for suspicious IPs, especially on the Reddit API.

Best Proxy Types for Reddit

1. Mobile Proxies (Best Overall)

Mobile proxies are the most effective proxy type for Reddit. They route your traffic through real 4G/5G cellular connections, providing IPs from carriers like AT&T, T-Mobile, Verizon, and other carriers worldwide.

Why they work:

• Mobile IPs belong to real carrier networks, not datacenters—Reddit treats them as legitimate user traffic
• CGNAT (Carrier-Grade NAT) means thousands of real users share each mobile IP, making blocking impractical
• Mobile ASNs have the highest trust scores on IP reputation databases
• Support for both rotating and static IP configurations

Best for: Multi-account management, long-term Reddit presence, any task where getting banned would be costly.

Setup: Configure via Chrome proxy settings or use with an anti-detect browser for maximum protection.

2. Residential Proxies (Good for Scraping)

Residential proxies use IPs assigned by ISPs to home users. They have good trust scores, though slightly lower than mobile IPs. Large residential proxy pools offer millions of IPs for high-volume scraping.

Why they work:

• IPs belong to real ISPs, passing most IP reputation checks
• Large pools allow extensive rotation for scraping without hitting rate limits
• Available in most countries and cities for geo-targeted access

Limitations: Some residential IPs end up on blocklists due to previous abuse. Speeds can be inconsistent since traffic routes through real home connections. They’re also more expensive per GB than datacenter options.

Best for: Large-scale data scraping, price monitoring, SEO research.

3. ISP Proxies (Good for Dedicated Accounts)

ISP proxies (also called static residential proxies) combine the speed of datacenter hosting with the legitimacy of residential IPs. They’re hosted in data centers but registered under ISP ASNs.

Why they work:

• Fast and reliable like datacenter proxies
• ISP-registered ASNs pass Reddit’s reputation checks
• Static IPs that don’t change—good for maintaining consistent account activity

Limitations: Smaller IP pools than residential or mobile. Higher cost than datacenter proxies. Some ISP proxy providers have been flagged over time.

Best for: Single high-value accounts that need fast, reliable, consistent IPs.

Proxy Types to Avoid for Reddit

Proxy Type	Why It Fails on Reddit
Datacenter proxies	Datacenter ASNs are flagged immediately. Reddit blocks entire ranges.
Free proxies	Already blacklisted, extremely slow, potential security risks. See our free proxy analysis.
Shared proxies	Other users’ abuse gets your IP banned before you even use it.
Most VPNs	VPN IP ranges are well-known and blocked. See mobile proxy vs VPN.

How to Use a Mobile Proxy with Reddit

For Browsing and Account Management

1. Get a dedicated mobile proxy from a reputable provider—one proxy per Reddit account
2. Configure the proxy in your browser or anti-detect browser
3. Clear cookies and cache before logging into Reddit
4. Use a static/sticky session so your IP stays consistent within each session
5. Browse naturally—don’t immediately start posting or performing actions that look automated

For Reddit Scraping

For web scraping, you’ll want rotating proxies with a large pool. Here’s a Python example:

import requests
import time

proxy = {
    "http": "http://user:pass@mobile-gate.provider.com:8080",
    "https": "http://user:pass@mobile-gate.provider.com:8080"
}

headers = {
    "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15"
}

subreddit = "technology"
url = f"https://www.reddit.com/r/{subreddit}/top.json?t=week&limit=25"

response = requests.get(url, proxies=proxy, headers=headers)
data = response.json()

for post in data["data"]["children"]:
    print(post["data"]["title"])
    time.sleep(2)  # Respect rate limits

Key tips for Reddit scraping:

• Always add delays between requests (2-5 seconds minimum)
• Use a realistic User-Agent string that matches a real browser
• Rotate IPs every few requests to avoid rate limiting
• Use Reddit’s JSON endpoints (append .json to any URL) instead of HTML scraping
• Consider the Reddit API with proper authentication for large-scale projects
• Understand the legal considerations of scraping Reddit data

How to Avoid Reddit Shadowbans

A shadowban is worse than a regular ban—your account appears normal to you, but your posts and comments are invisible to everyone else. Here’s how to avoid them when using proxies:

• One account per IP – Never use the same proxy for multiple Reddit accounts. Reddit links accounts that share IPs.
• Warm up new accounts – New accounts should browse, upvote, and comment on various subreddits for several days before posting links or marketing content.
• Don’t vote on your own content – Using alt accounts to upvote your own posts is a guaranteed shadowban.
• Vary your behavior – Don’t post at exactly the same times, use the same formatting, or follow identical patterns across accounts.
• Participate genuinely – Accounts that only post links without engaging in discussions are flagged as spam.
• Check your status – Visit reddit.com/r/ShadowBan to check if your account has been shadowbanned.

Reddit API and Proxies

If you’re doing legitimate data collection, consider using the Reddit API (via PRAW or similar libraries) with proper authentication. The API has its own rate limits (100 requests per minute for OAuth-authenticated requests), but using it legitimately with a mobile proxy provides the best reliability.

Mobile proxies are especially useful with the Reddit API because:

• API rate limits are partially IP-based—a trusted mobile IP gets more lenient treatment
• OAuth tokens combined with clean mobile IPs rarely trigger abuse detection
• Multiple API clients can use different mobile proxy IPs to parallelize data collection

Frequently Asked Questions

Does Reddit block all proxies?

No. Reddit blocks known datacenter and VPN IPs, but it cannot block residential or mobile proxy IPs without also blocking real users. Mobile proxies have the highest success rate on Reddit because their IPs are indistinguishable from regular mobile users.

Can I use a free proxy for Reddit?

Free proxies are almost always detected and blocked by Reddit. They use datacenter IPs that are already blacklisted, and they’re shared by many users who may be engaging in spam. For Reddit, you need high-quality residential or mobile proxies.

How many Reddit accounts can I run with mobile proxies?

You can run as many accounts as you have proxies—the rule is one dedicated proxy per account per platform. With rotating mobile proxies, each account should have its own sticky session that maintains a consistent IP during use.

Conclusion

Reddit’s sophisticated anti-proxy measures make it one of the hardest platforms to use with proxies. Datacenter proxies and VPNs are almost always detected. For reliable Reddit access through a proxy, mobile proxies are your best option—they use real carrier IPs that Reddit can’t block without affecting legitimate mobile users.

Pair your mobile proxy with an anti-detect browser for multi-account management, or use backconnect rotating proxies for scraping at scale. Whatever your use case, the key is using proxy IPs that blend in with real user traffic—and no proxy type does that better than mobile.