how to scrape Booking.com hotel prices (2026 anti-bot guide)

Booking.com is one of the harder travel sites to scrape in 2026 because it sits behind Akamai Bot Manager plus its own dynamic pricing layer. you need residential proxies, a real headless browser like Playwright, careful rate limits, and the right strategy for handling per-session price tokens. this guide walks through working Python code, the Akamai-specific gotchas, and what data you can actually extract reliably.

what you can scrape from Booking.com

Booking.com pages are deeply dynamic, but the high-value fields are stable enough for production scrapers.

most price-monitoring use cases need only the first three. for anything more complex, expect more anti-bot friction. for the broader use case, our price monitoring proxy guide covers infrastructure decisions.

the Akamai problem

Booking.com runs Akamai Bot Manager which inspects three things on every request: TLS fingerprint (JA3/JA4), HTTP/2 fingerprint, and a per-session token called _abck that gets validated against a sensor payload generated by client-side JavaScript.

plain requests or httpx will fail because the TLS fingerprint reveals Python instantly. even curl gets blocked. you need either a real browser (Playwright, Puppeteer) or a TLS-impersonating client like curl_cffi.

if you want a deeper look at Akamai itself, our Akamai bypass guide covers the mechanism in detail. the same techniques apply directly to Booking.com.

install the stack

pip install playwright curl_cffi parsel
playwright install chromium

Playwright launches a real Chromium browser. curl_cffi impersonates Chrome’s TLS fingerprint for the lighter price-checks where you don’t need full JavaScript rendering. parsel parses the HTML.

scrape search results with Playwright

start with a search page that returns hotels for a city and date range.

import asyncio
from playwright.async_api import async_playwright
from parsel import Selector

PROXY = {'server': 'http://gateway.example.com:8000', 'username': 'u', 'password': 'p'}

URL = 'https://www.booking.com/searchresults.html?ss=Singapore&checkin=2026-06-01&checkout=2026-06-03&group_adults=2'

async def scrape_search():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=PROXY)
        ctx = await browser.new_context(
            user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
            locale='en-US',
            viewport={'width': 1366, 'height': 900},
        )
        page = await ctx.new_page()
        await page.goto(URL, wait_until='networkidle')
        await page.wait_for_selector('div[data-testid="property-card"]', timeout=15000)

        html = await page.content()
        await browser.close()

    sel = Selector(text=html)
    hotels = []
    for card in sel.css('div[data-testid="property-card"]'):
        hotels.append({
            'name': card.css('div[data-testid="title"]::text').get(''),
            'location': card.css('span[data-testid="address"]::text').get(''),
            'score': card.css('div[data-testid="review-score"] div::text').get(''),
            'price': card.css('span[data-testid="price-and-discounted-price"]::text').get(''),
            'url': card.css('a[data-testid="title-link"]::attr(href)').get(''),
        })
    return hotels

asyncio.run(scrape_search())

data-testid selectors are the most stable. CSS class names on Booking.com change frequently because their build pipeline auto-generates them. the testid attributes survive UI tweaks because they’re part of the QA harness.

handle the cookie consent banner

first-time visitors see a GDPR banner that blocks page interactions. dismiss it before doing anything.

try:
    await page.click('button#onetrust-accept-btn-handler', timeout=3000)
except:
    pass

wrap it in a try/except because the banner only shows for fresh sessions. if your proxy gives you a sticky session, the cookie persists and the banner doesn’t appear on the next request.

rotate proxies and sessions

Booking.com’s per-IP rate limit is roughly 30-60 requests per hour before Akamai starts challenging you. with a residential pool, you want sticky sessions of 10-15 minutes per IP, then rotate.

async def scrape_many_cities(cities):
    async with async_playwright() as p:
        for city in cities:
            session_id = f'session-{city}'
            proxy = {
                'server': 'http://gateway.example.com:8000',
                'username': f'user-session-{session_id}',
                'password': 'pass',
            }
            browser = await p.chromium.launch(headless=True, proxy=proxy)
            ctx = await browser.new_context()
            page = await ctx.new_page()

            # scrape this city
            ...

            await browser.close()
            await asyncio.sleep(5)

most residential proxy providers let you specify a session ID in the username (user-session-XXX). same session ID = same IP. change the ID to rotate. our Akamai bypass guide covers the fingerprinting layer in more depth.

handle pagination

Booking.com paginates with offset query params. add &offset=25 (or 50, 75, etc.) to the search URL.

for offset in range(0, 250, 25):
    url = f'{base_url}&offset={offset}'
    await page.goto(url, wait_until='networkidle')
    # extract cards

each page returns roughly 25 results. don’t paginate past 1,000 results from the same search; Akamai flags deep pagination as automation. for big crawls, split by city + date pair instead.

scrape an individual hotel page

hotel pages contain pricing per room type, availability, and amenity details.

async def scrape_hotel(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=PROXY)
        ctx = await browser.new_context(locale='en-US')
        page = await ctx.new_page()
        await page.goto(url, wait_until='networkidle')

        await page.wait_for_selector('h2[data-testid="property-header-name"]', timeout=15000)
        html = await page.content()
        await browser.close()

    sel = Selector(text=html)
    return {
        'name': sel.css('h2[data-testid="property-header-name"]::text').get(''),
        'address': sel.css('span[data-testid="address"]::text').get(''),
        'rating': sel.css('div[data-testid="review-score-component"] div::text').get(''),
        'rooms': sel.css('table#hprt-table tr.hprt-table-row').getall(),
    }

the room table is the trickiest part because the markup is legacy (table-based, lots of nested rows for room options). parse it row by row and join with the room-name column.

handle dynamic prices

prices on Booking.com depend on cookies, locale, and currency settings. the same hotel can show different prices to different users in the same city. for accurate price monitoring you need to:

1. set a consistent locale and currency at session start
2. use the same IP geolocation for repeat scrapes (US IP = USD by default)
3. include explicit &selected_currency=USD in the URL
4. compare like-for-like by date range and occupancy

if your IP rotates between countries mid-session, prices will jump because the currency conversion changes. residential pools with country-targeting solve this.

faq

can I scrape Booking.com without Playwright?
yes for static-looking pages, no for anything that involves dynamic price tokens. curl_cffi with Chrome’s TLS fingerprint can fetch some search result HTML, but room-level pricing and availability requires the JavaScript runtime. start with Playwright for reliability, optimize to lighter clients only after you understand which pages are safely fetchable.

what proxies do I need?
residential is the minimum. mobile is overkill unless you’re scraping at very high volume. datacenter IPs get blocked instantly because Akamai recognizes the AS numbers. for provider picks, see our provider comparison.

how do I avoid the Akamai _abck challenge?
use a real browser (Playwright with stealth), don’t disable JavaScript, keep cookies across requests in the same session, and respect rate limits. headers alone won’t pass; the sensor payload requires real DOM execution.

is scraping Booking.com legal?
public price data is generally legal to collect, but Booking.com’s terms of service prohibit automated access. for personal research, low risk. for commercial use, consult a lawyer and consider their official Booking.com Affiliate Partner Program for hotel data instead. for related legal context, our web scraping legal guide covers the broader rules.

why are my scraped prices different from what I see in my browser?
prices vary by IP geolocation, currency, device type, and even browsing history. always pin currency, locale, viewport size, and user agent. if your scraper IP is in Singapore but you want US-resident pricing, get a US residential IP.

how often does Booking.com change selectors?
data-testid attributes are stable across most updates. CSS class names rotate with each deploy (often weekly). build parsers around testids, not classes, and you’ll cut maintenance to once per quarter instead of weekly.

conclusion

scraping Booking.com works in 2026 if you bring real browser automation, residential proxies, and respect for the per-IP rate limits. Akamai is the main obstacle and Playwright (or any real Chromium) handles it transparently as long as you don’t disable JavaScript or strip cookies.

focus on the data-testid selectors, pin your locale and currency, and rotate sticky sessions every 10-15 minutes. that combination keeps you under the radar while collecting clean price data at meaningful volume.

if you’re doing this commercially, consider Booking.com’s official affiliate API or the managed scraping APIs that handle the anti-bot for you. for personal research and price-comparison side projects, the Playwright approach in this guide is plenty.

how to scrape linkedin data without getting banned (2026)

scraping linkedin without bans in 2026 comes down to four things: residential or mobile proxies (never datacenter), aged accounts with established activity, slow request rates (under 80 actions per day per account), and either a managed scraping api or playwright with anti-detection. linkedin actively detects automation. one ip + one fresh account + 200 requests in an hour = ban within 24 hours. this guide covers the legal context, the technical setup, and how to recover when accounts get restricted.

we cover legality first, then the proxy stack, account discipline, browser automation, managed api alternatives, and a 2026 ban-recovery playbook.

is linkedin scraping legal in 2026?

scraping public linkedin data is generally legal in the us under hiq v linkedin (2022) and follow-on rulings. the courts have repeatedly held that scraping public web data is not a violation of the computer fraud and abuse act.

scraping linkedin still violates linkedin’s terms of service. tos violations are not criminal but they give linkedin grounds to ban accounts and pursue civil action against commercial scrapers in some cases.

eu and uk law is stricter. gdpr requires a lawful basis (consent, contract, or legitimate interest) for processing personal data. scraped linkedin data falls under gdpr if it includes eu data subjects. document your lawful basis before processing, and respect data subject rights including erasure requests.

read linkedin’s user agreement for the current commercial-use restrictions. for a deeper read on the legal landscape around lead-gen scraping see our b2b lead generation proxies guide.

what gets you banned in 2026

linkedin’s anti-bot stack flags four signals.

ip pattern. datacenter ips trigger immediately. shared residential ips with known scraper traffic also flag fast. mobile carrier ips have the longest leash.

session pattern. login from a new ip with no warm-up history is suspicious. 50 profile views in 10 minutes is a classic bot signal. clicking through every profile from a search result without scrolling looks robotic.

browser fingerprint. headless chrome without anti-detection patches is detected within minutes. residential proxy + plain selenium = ban in under an hour.

account age and activity. brand new accounts with zero connections and a thin profile that suddenly perform 500 actions trip every alarm. aged accounts with real history get more leniency.

beat all four and bans become rare. miss any one and accounts cycle through faster than you can warm them.

the proxy stack

mobile proxies are the safest tier. linkedin sees thousands of users behind each carrier-grade nat ip, so individual scraping signals are diluted. expect to pay $50 to $150 per port per month.

residential proxies are the value pick. session-rotating residential pools work for most scraping at $4 to $7 per gb. choose providers with sticky sessions of 10+ minutes so a single profile-view session does not change ip mid-flow.

datacenter proxies are unusable for linkedin in 2026. even premium isp proxies (which are residential-issued datacenter ips) get blocked within a few requests.

assign one proxy per linkedin account. never share an ip across multiple accounts. linkedin’s session correlation flags shared ips fast.

account discipline

aged accounts are non-negotiable in 2026. linkedin treats accounts under 6 months old with no activity as bots by default. for production scraping you need accounts with at least 100 connections, a complete profile, posted content from real timestamps, and a normal usage history.

three options for account supply.

option 1: warm your own. spend 30 to 60 days on each account: login, scroll, accept connections, post once a week, like a few posts daily. boring but the accounts last.

option 2: buy aged accounts. resellers sell 1-year-old accounts with 500+ connections for $50 to $200. quality varies wildly. budget for replacement.

option 3: managed scraping apis. let bright data, apify, or proxycurl handle the account problem entirely. you pay per query, they handle bans on their side. cleanest for production.

never run more than 80 to 120 actions per account per day. one action = one profile view, one search, or one connection request. above that, ban risk spikes hard.

browser automation: playwright with anti-detection

for self-managed scraping, playwright with anti-detection patches is the baseline.

from playwright.sync_api import sync_playwright
import time
import random

PROXY = {
    "server": "http://proxy.example.com:8080",
    "username": "user-session-abc123",
    "password": "pass",
}

def scrape_profile(profile_url, session_cookie):
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            proxy=PROXY,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
            ],
        )
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/127.0.0.0 Safari/537.36",
            viewport={"width": 1920, "height": 1080},
            locale="en-US",
            timezone_id="America/New_York",
        )
        context.add_cookies([{
            "name": "li_at",
            "value": session_cookie,
            "domain": ".linkedin.com",
            "path": "/",
        }])

        page = context.new_page()
        page.goto(profile_url, wait_until="networkidle")
        time.sleep(random.uniform(2, 5))

        page.mouse.wheel(0, 600)
        time.sleep(random.uniform(1, 3))
        page.mouse.wheel(0, 800)
        time.sleep(random.uniform(2, 4))

        name = page.locator("h1").inner_text()
        headline = page.locator(".text-body-medium.break-words").first.inner_text()

        browser.close()
        return {"name": name, "headline": headline, "url": profile_url}

the --disable-blink-features=AutomationControlled flag removes the most obvious headless tell. sleeps and mouse-wheel events simulate human pacing. timezone, locale, and user-agent match a typical us desktop user.

for stronger anti-detection, use playwright-stealth or a real antidetect browser like adspower or gologin. plain playwright is detectable by sophisticated fingerprinting.

for the broader python scraping context see our web scraping with python guide.

sticky sessions across the scrape session

linkedin tracks ip across a session. switching ip mid-session looks like account hijacking and triggers a security challenge.

def session_username(account_id):
    """build a sticky username for residential providers that support it."""
    return f"user-session-{account_id}"

def proxy_for_account(account_id):
    return {
        "server": "http://proxy.example.com:8080",
        "username": session_username(account_id),
        "password": "pass",
    }

most residential providers (smartproxy, oxylabs, soax) support session usernames that pin a single residential ip for 10 to 30 minutes. use the same session id for the duration of the linkedin scrape, then rotate when the session expires naturally.

rate limits in practice

based on 6 months of data across 30 aged accounts running through residential proxies in 2026, here is what stayed unbanned:

• profile views: under 80 per day per account
• searches: under 25 per day per account
• connection requests: under 15 per day per account (lifetime cap of 100 per week)
• messages to connections: under 50 per day per account
• session length: 30 to 90 minutes per session, 1 to 2 sessions per day
• gap between sessions: at least 4 hours

push past these and ban rates spike. stay below them and accounts last 6 to 12 months on average before any restriction.

managed scraping apis: the easier path

self-managed linkedin scraping is a job. you maintain account warming, proxy rotation, anti-detection patches, and ban recovery. for many teams the time cost beats the api cost.

managed options in 2026:

bright data linkedin dataset. pre-scraped public profiles. updated continuously. you query by url or company. roughly $0.001 to $0.01 per record depending on volume. no scraping risk on your side.

apify linkedin scraper actors. pay per actor run. simpler than building your own; still subject to linkedin’s anti-bot. 2026 prices: roughly $1 to $3 per 1,000 results.

proxycurl. linkedin profile, company, and job api. enterprise-friendly with response-time slas. $0.10 to $0.30 per profile lookup at typical volumes.

phantombuster. no-code linkedin automation. covers scraping plus connection requests and messaging. see our breakdown in outscraper vs phantombuster vs hunter.io.

for production teams, the managed apis are the right choice unless you need volume that exceeds their rate limits or you are scraping data they do not offer.

what to do when an account gets restricted

linkedin restricts accounts in stages: warning, partial restriction (no search, no messages), full restriction (login redirects to verification), then permanent ban.

at warning stage: stop all automation for 7 to 14 days. log in manually from a regular browser on the same proxy. do normal user activities (scroll, like 1 to 2 posts, accept 1 connection). most accounts recover.

at partial restriction: same playbook plus complete identity verification if linkedin asks (selfie, government id). if you skip verification, the account moves to full restriction. for accounts you bought, this is usually game over.

at full restriction: usually unrecoverable without verification. for managed-api stacks, this is on the api provider, not you.

at permanent ban: replace the account. log the proxy + account combo so you do not reuse the proxy for the next account.

ethical and security notes

if you scrape eu data subjects, you must respect erasure requests. publish a privacy policy that lists linkedin as a data source and provides a removal email. process removals within 30 days.

never scrape data behind a login that requires special permission (closed groups, private messages, premium-only fields). that crosses into the cfaa unauthorized-access territory in the us and is a clear gdpr violation in the eu.

cold email or cold dm using scraped data still requires lawful basis in the eu and uk and a clear opt-out everywhere. a working email is a tool, not a license.

faq

can i scrape public linkedin profiles legally?

in the us, public profile data scraping is generally legal under hiq v linkedin, but it violates linkedin’s tos. in the eu and uk, gdpr requires a lawful basis even for public data when it identifies a person. always document your basis and offer opt-out.

what proxies should i use for linkedin scraping?

mobile proxies are safest. residential proxies with sticky sessions of 10+ minutes are the value pick. datacenter and isp proxies are blocked instantly in 2026. budget $5 to $7 per gb for residential or $50+ per port per month for mobile.

how many requests per day before linkedin bans?

aged accounts on residential proxies tolerate roughly 80 profile views, 25 searches, and 15 connection requests per day. fresh accounts on datacenter ips tolerate maybe 20 to 50 requests before banning.

is selenium or playwright better for linkedin scraping?

playwright is the better default in 2026. its anti-detection options are richer, the api is cleaner, and it handles modern js rendering more reliably. selenium still works but requires more patches to avoid headless detection.

do i need a paid linkedin sales nav account to scrape effectively?

not strictly. public profile scraping works without a paid account. sales nav unlocks deeper search filters and lead lists, which is useful for outbound. paid accounts also tolerate slightly higher rate limits before triggering anti-bot.

should i use a managed linkedin api or build my own scraper?

for under 5,000 profiles per month, managed apis (bright data, apify, proxycurl) are usually cheaper than the engineering plus account management cost. for higher volume or unique fields not in the public datasets, build your own. budget for warming aged accounts, residential proxies, and ongoing maintenance.

the bottom line

linkedin scraping in 2026 is harder than 2022 because linkedin’s anti-bot stack got better. but it is also more accessible because managed datasets cover most common use cases at a per-record price that beats diy.

self-managed approach: aged accounts, residential or mobile proxies (one per account), playwright with anti-detection, conservative rate limits. expect to replace 10 to 20 percent of accounts every quarter.

managed approach: pay $0.001 to $0.30 per record depending on freshness and depth. zero ban exposure. faster time-to-data.

for most teams in 2026 the managed approach wins on total cost. for teams scraping at very high volume or extracting fields managed apis do not surface, the diy stack still has a place. either way, document your gdpr basis and respect opt-outs. it is the difference between a sustainable lead-gen channel and a pile of legal exposure.

web scraping with Node.js: axios, cheerio, puppeteer complete guide (2026)

Node.js is the second most popular language for web scraping after Python, and in 2026 it has caught up on tooling. you get axios for HTTP requests, cheerio for fast HTML parsing, and puppeteer for headless Chrome control. this guide walks through all three with real code, proxy rotation, and the anti-bot patterns that actually work today.

why scrape with Node.js in 2026

Node.js wins when your scraper needs to share code with a frontend, run inside an existing Express or Next.js app, or handle thousands of concurrent requests over a single event loop. its async model is genuinely faster than threaded Python for I/O-bound jobs.

it also gets first-class support from puppeteer (Google maintains it) and the new Chrome DevTools Protocol features land in Node before they land anywhere else. if you scrape JavaScript-heavy sites at scale, Node is the path of least resistance.

the trade-off is the parsing ecosystem. Python’s BeautifulSoup is more forgiving than cheerio when HTML is broken, and pandas is way ahead of anything in JS land for downstream data work. pick Node when concurrency or browser automation matter most.

install the stack

start with a fresh project. you only need three core packages and one helper for proxy support.

mkdir scraper && cd scraper
npm init -y
npm install axios cheerio puppeteer https-proxy-agent

axios handles plain HTTP. cheerio parses HTML with a jQuery-like API. puppeteer launches headless Chrome. https-proxy-agent lets you route axios through HTTP or HTTPS proxies without external config.

most production scrapers use both. axios handles 80% of pages, puppeteer handles the JavaScript-rendered 20%.

scrape static HTML with axios + cheerio

here’s the simplest possible scraper. it pulls book titles and prices from books.toscrape.com (the official sandbox for scraping practice).

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeBooks() {
  const { data } = await axios.get('https://books.toscrape.com/');
  const $ = cheerio.load(data);
  const books = [];

  $('article.product_pod').each((i, el) => {
    books.push({
      title: $(el).find('h3 a').attr('title'),
      price: $(el).find('.price_color').text(),
      stock: $(el).find('.availability').text().trim(),
    });
  });

  return books;
}

scrapeBooks().then(console.log);

axios returns the raw HTML. cheerio loads it into a DOM-like object you query with CSS selectors. the each loop builds an array of objects you can write to JSON or a database.

this pattern handles thousands of requests per minute on a single machine. for sites without anti-bot defenses, it’s all you need.

handle pagination

most listings span multiple pages. wrap the scraper in a loop that follows the next-page link until it disappears.

async function scrapeAllPages() {
  let url = 'https://books.toscrape.com/';
  const all = [];

  while (url) {
    const { data } = await axios.get(url);
    const $ = cheerio.load(data);

    $('article.product_pod').each((i, el) => {
      all.push({
        title: $(el).find('h3 a').attr('title'),
        price: $(el).find('.price_color').text(),
      });
    });

    const next = $('li.next a').attr('href');
    url = next ? new URL(next, url).href : null;
  }

  return all;
}

URL resolution matters. relative links like catalogue/page-2.html break if you concatenate them naively. the URL constructor merges them against the current base.

for sites with offset pagination (?page=1, ?page=2), increment the query param until you get an empty result set. for cursor-based APIs, follow the next-cursor token.

scrape JavaScript-rendered pages with puppeteer

axios fetches HTML that the server returns. it does not run JavaScript. modern SPAs (React, Vue, Next.js) render content client-side, so axios returns an empty shell.

puppeteer launches a real Chrome instance, runs the JavaScript, and gives you the rendered DOM.

const puppeteer = require('puppeteer');

async function scrapeQuotes() {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();

  await page.goto('https://quotes.toscrape.com/js/', {
    waitUntil: 'networkidle2',
  });

  const quotes = await page.$$eval('.quote', (els) =>
    els.map((el) => ({
      text: el.querySelector('.text').innerText,
      author: el.querySelector('.author').innerText,
    }))
  );

  await browser.close();
  return quotes;
}

waitUntil: 'networkidle2' tells puppeteer to wait until network traffic settles. for slower sites, use waitForSelector('.quote') instead so you don’t time out on a busy XHR.

$$eval runs a function inside the browser, so you query the live DOM with regular querySelector calls. the result serializes back to Node.

if you’re picking between puppeteer and the alternatives, see our headless browser deep dive and the selenium vs playwright vs puppeteer comparison.

handle infinite scroll

many listing pages load more results as you scroll. simulate that with page.evaluate.

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let total = 0;
      const distance = 300;
      const timer = setInterval(() => {
        const { scrollHeight } = document.body;
        window.scrollBy(0, distance);
        total += distance;
        if (total >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 200);
    });
  });
}

call autoScroll(page) before extracting data. tune the distance (300px) and interval (200ms) to match how the target site loads chunks. too fast and you’ll miss content, too slow wastes time.

use proxies (essential for any real target)

the moment you scrape Amazon, Google, LinkedIn, or any high-value target, you need rotating proxies. a single IP making 100 requests per minute gets blocked within seconds.

axios with a single proxy:

const { HttpsProxyAgent } = require('https-proxy-agent');

const agent = new HttpsProxyAgent('http://user:pass@proxy.example.com:8000');

const { data } = await axios.get('https://example.com', {
  httpsAgent: agent,
  httpAgent: agent,
  timeout: 15000,
});

for a rotating residential pool, you typically get a single gateway endpoint and the provider rotates IPs per request. mobile proxy gateways work the same way.

puppeteer with a proxy:

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://proxy.example.com:8000'],
});

const page = await browser.newPage();
await page.authenticate({ username: 'user', password: 'pass' });

proxy auth in puppeteer is page-level, not browser-level. call page.authenticate before page.goto.

for a deeper walkthrough on rotation strategies, see rotating proxies and unlimited bandwidth.

avoid common anti-bot triggers

generic blocks happen because your scraper looks nothing like a real browser. fix the obvious tells first.

set a real user agent. axios defaults to axios/1.x which screams bot. rotate through 3-5 modern Chrome strings.

const headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
};

await axios.get(url, { headers });

add randomized delays between requests. 1-3 seconds is enough for most sites. use await new Promise(r => setTimeout(r, 1500 + Math.random() * 1500)).

for puppeteer, install puppeteer-extra-plugin-stealth. it patches the navigator.webdriver flag and dozens of other fingerprint tells in one line.

const puppeteer = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth');
puppeteer.use(Stealth());

stealth alone won’t beat enterprise anti-bot like Cloudflare, DataDome, or Akamai. for those, you need residential proxies plus stealth plus careful request timing. for tougher targets, also see how Python scrapers handle this.

handle errors and retries

network errors are normal. wrap requests in retry logic so a single 503 doesn’t kill your job.

async function fetchWithRetry(url, retries = 3) {
  for (let i = 0; i < retries; i++) {
    try {
      return await axios.get(url, { timeout: 15000 });
    } catch (e) {
      if (i === retries - 1) throw e;
      await new Promise((r) => setTimeout(r, 2 ** i * 1000));
    }
  }
}

exponential backoff (1s, 2s, 4s) is the standard. for 429 responses, respect the Retry-After header if the server sets it.

log everything. failed scrapes are signal, not noise. a sudden spike in 403s means your IP got flagged or the site changed defenses.

save data

for small jobs, write JSON.

const fs = require('fs');
fs.writeFileSync('books.json', JSON.stringify(books, null, 2));

for thousands of rows, stream to a CSV or a database. the csv-stringify package handles escaping. for Postgres, use pg with batch inserts of 500-1000 rows per transaction. for cloud, push to S3 or BigQuery.

never store credentials in your scraper code. use dotenv (npm install dotenv) and a .env file that’s gitignored.

faq

is web scraping with Node.js faster than Python?
for I/O-bound concurrent scraping, yes. Node.js handles thousands of simultaneous connections on a single thread thanks to its event loop. Python needs asyncio or threading to match it, and even then the GIL limits CPU work.

do I need puppeteer or is axios enough?
axios + cheerio is enough for any page that returns full HTML on the first request. open the target site with JavaScript disabled in your browser. if the content you want is still visible, axios works. if the page goes blank, you need puppeteer.

what’s the difference between puppeteer and playwright?
playwright is Microsoft’s fork of puppeteer with multi-browser support (Chromium, Firefox, WebKit) and better auto-waiting. puppeteer only drives Chrome but has tighter integration with Chrome DevTools. both work for scraping. our comparison guide breaks down which to pick.

can I run Node.js scrapers serverless?
yes for axios + cheerio (small footprint, fits in Lambda or Vercel functions). puppeteer is harder because Chrome binaries are 200MB+. use chrome-aws-lambda or the puppeteer-core + @sparticuz/chromium combo, or run puppeteer on a small VM instead.

how do I avoid getting blocked?
rotate residential or mobile proxies, randomize user agents, add 1-3 second delays between requests, and use stealth plugins for puppeteer. for tough targets like Cloudflare or DataDome, you’ll also need to defeat TLS fingerprinting. for any of those defenses, residential IPs are non-negotiable.

is web scraping legal in 2026?
scraping public data is generally legal in the US after the hiQ v. LinkedIn rulings, but terms of service violations and CFAA risk still exist. EU GDPR adds personal-data restrictions. always check the target site’s robots.txt and ToS, and consult a lawyer for commercial use.

conclusion

Node.js gives you a complete scraping stack in three packages. axios for fast HTTP, cheerio for HTML parsing, puppeteer for JavaScript-heavy pages. add a residential proxy pool, stealth plugins, and retry logic, and you have a production scraper.

start with the static stack (axios + cheerio) and only add puppeteer when you actually need it. headless Chrome is 100x slower than HTTP requests and burns way more proxy bandwidth. the cheapest reliable scraper is the one that does the least work per page.

if your scraping needs grow, look at distributed runners like Apify, BullMQ for queues, and managed scraping APIs that handle proxies and CAPTCHAs for you. but for most jobs, the three libraries in this guide will get you 90% of the way there.