Author: Xavier Fok

  • How to Scrape Google Local Pack Results (Maps + Business Data) 2026

    how to scrape google local pack results (maps + business data) 2026

    google local pack is the 3-result map block that appears on serps for local-intent queries like “coffee shop near me” or “lawyer in austin.” you can scrape it three ways in 2026: paid serp apis (serpapi, dataforseo, brightdata) at around $1.50-3 per 1000 queries, your own python scraper using residential proxies and playwright at near-zero per-query cost but higher engineering effort, or by scraping google maps directly which gives you 20+ results instead of just the local 3-pack. this tutorial covers all three with working code.

    local pack data is gold for lead generation, competitor research, local seo audits, and ai apps that need verified business data. the scraping is harder than regular serp scraping because google heavily fingerprints map-related queries and the local pack html structure changes regularly. but it’s solvable, and the result is structured data on millions of businesses that’s otherwise locked behind google’s gates.

    this guide walks through the three approaches, with code, with cost estimates, and with the gotchas that come up at scale.

    what’s in the local pack

    a typical local pack result on a query like “plumber miami” returns:

    • 3 business listings (top 3 by google’s local ranking)
    • each listing has: business name, rating (1-5 stars), review count, category, address, hours snippet, phone (sometimes), website (sometimes), gbid (google business id), latitude/longitude
    • a “view all” link that opens the local finder (top 20 results)
    • ad placements above and below sometimes

    the underlying data lives in google’s local index, accessible via the regular serp html, the maps web ui, and the maps mobile app. each surface returns slightly different fields. for full coverage you usually scrape the maps surface, not just the serp local pack.

    approach 1: paid serp apis (easiest)

    three providers dominate this space in 2026:

    • serpapi: $50/month for 5000 searches. local pack data included with engine=google_local.
    • dataforseo: $0.0006-0.001 per organic search depending on plan. dedicated local pack endpoint.
    • brightdata serp api: $1.50 per 1000 searches. covers all serp features including local pack.

    for any serious volume the per-query rates push under $1.50/1k. for prototypes and small jobs they are by far the easiest path.

    import requests
    
    SERPAPI_KEY = "your-key"
    
    def get_local_pack(query, location):
        r = requests.get("https://serpapi.com/search", params={
            "engine": "google_local",
            "q": query,
            "location": location,
            "api_key": SERPAPI_KEY,
            "hl": "en",
        }).json()
        return r.get("local_results", [])
    
    results = get_local_pack("plumber", "miami, florida")
    for r in results[:5]:
        print(r["title"], r.get("rating"), r.get("phone"), r.get("address"))
    

    dataforseo’s local pack endpoint:

    import requests
    from requests.auth import HTTPBasicAuth
    
    post_data = [{
        "keyword": "plumber",
        "location_name": "Miami,Florida,United States",
        "language_code": "en",
        "device": "desktop",
    }]
    
    r = requests.post(
        "https://api.dataforseo.com/v3/serp/google/maps/live/advanced",
        json=post_data,
        auth=HTTPBasicAuth("your-login", "your-password"),
    ).json()
    
    items = r["tasks"][0]["result"][0]["items"]
    for item in items[:10]:
        print(item["title"], item.get("rating", {}).get("value"))
    

    dataforseo’s pricing is the most aggressive at scale (under $0.001 per query at volume). serpapi has the friendliest sdk and free tier. bright data is the most reliable at very high volume.

    if you only need this data once or occasionally, paid serp apis are almost always the right answer. you spend $5-50, you get the data, you move on.

    approach 2: python scraper with residential proxies

    cheaper at scale, more engineering work upfront. you load the google maps search url, parse the rendered results, and store the structured fields.

    import asyncio
    import json
    import re
    from playwright.async_api import async_playwright
    
    PROXY = {
        "server": "http://residential.example.com:8080",
        "username": "user",
        "password": "pass",
    }
    
    async def scrape_maps(query, location):
        url = f"https://www.google.com/maps/search/{query.replace(' ', '+')}+{location.replace(' ', '+')}"
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False, proxy=PROXY)
            ctx = await browser.new_context(
                viewport={"width": 1366, "height": 768},
                user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
                locale="en-US",
            )
            page = await ctx.new_page()
            await page.goto(url, wait_until="networkidle", timeout=30000)
            await page.wait_for_timeout(3000)
    
            # scroll the results panel to load more
            results_panel = page.locator("div[role='feed']")
            for _ in range(3):
                await results_panel.evaluate("el => el.scrollBy(0, 800)")
                await page.wait_for_timeout(1500)
    
            # extract listing elements
            items = await page.locator("div[role='feed'] > div > div[jsaction]").all()
            results = []
            for item in items[:20]:
                try:
                    name = await item.locator("div.fontHeadlineSmall").first.text_content()
                    rating_el = await item.locator("span[role='img'][aria-label*='star']").first.get_attribute("aria-label")
                    results.append({
                        "name": name.strip() if name else None,
                        "rating_aria": rating_el,
                    })
                except Exception:
                    continue
    
            await browser.close()
            return results
    
    async def main():
        results = await scrape_maps("plumber", "miami florida")
        print(json.dumps(results, indent=2))
    
    asyncio.run(main())
    

    this is the rough shape. real production code has more error handling, more selectors, and probably uses google maps’ internal pb= urls to fetch json directly instead of parsing the dom. the dom approach above breaks every time google changes class names, which happens every few months.

    a more robust pattern is to capture the maps json endpoint via network interception:

    async def capture_maps_json(query, location):
        captured = []
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False)
            ctx = await browser.new_context()
            page = await ctx.new_page()
    
            async def handle_response(response):
                if "search?" in response.url and "/maps/" in response.url:
                    try:
                        body = await response.text()
                        if body.startswith(")]}'"):
                            body = body[5:]
                        captured.append(json.loads(body))
                    except Exception:
                        pass
    
            page.on("response", handle_response)
            await page.goto(f"https://www.google.com/maps/search/{query}+{location}",
                            wait_until="networkidle")
            await page.wait_for_timeout(5000)
            await browser.close()
        return captured
    

    the captured json contains the full structured data google sees on its end. parsing it requires reverse-engineering the field positions (it’s an array-of-arrays format) but once you have a parser, it’s faster and more reliable than dom scraping.

    for the proxy choice, residential is the minimum. mobile proxies pass through the toughest google blocks more reliably. datacenter ips are blocked within a few queries. see the residential proxy guide for context.

    approach 3: scrape regular google serp html

    if you only need the 3-pack (not the full 20-result local finder), you can scrape the regular google serp page. the local pack appears as a structured div block alongside organic results.

    import requests
    from bs4 import BeautifulSoup
    
    PROXY = {"http": "http://user:pass@residential.example.com:8080",
             "https": "http://user:pass@residential.example.com:8080"}
    
    def scrape_serp_local(query, geo_param):
        url = f"https://www.google.com/search?q={query}&uule={geo_param}&hl=en"
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Accept": "text/html,application/xhtml+xml",
            "Accept-Language": "en-US,en;q=0.9",
        }
        r = requests.get(url, headers=headers, proxies=PROXY, timeout=30)
        soup = BeautifulSoup(r.text, "html.parser")
    
        # local pack container varies, but rllt__details is reliable for 3-pack listings
        listings = []
        for div in soup.select("div.rllt__details"):
            name = div.select_one("div.dbg0pd")
            if name:
                listings.append({
                    "name": name.get_text(strip=True),
                    "snippet": " ".join(s.get_text() for s in div.select("div") if s != name),
                })
        return listings
    

    the uule parameter is google’s encoded location. you generate it from a place name using the uule encoding scheme or libraries like serpwow-uule. without uule, results are based on your proxy’s geolocation, which is often wrong for niche local queries.

    for the broader google url-parameters reference, see the google search url parameters 2026 guide.

    extracting individual business details

    the local pack listings give you basic data. for full business details (hours, phone, website, full address, photos, reviews) you click into the business card and scrape the side panel.

    async def scrape_business_details(page, business_url):
        await page.goto(business_url, wait_until="networkidle")
        await page.wait_for_timeout(2000)
    
        name = await page.locator("h1").first.text_content()
        address = await page.locator("button[data-item-id='address']").first.text_content()
        phone_el = page.locator("button[data-item-id^='phone']").first
        phone = await phone_el.text_content() if await phone_el.count() else None
        website_el = page.locator("a[data-item-id='authority']").first
        website = await website_el.get_attribute("href") if await website_el.count() else None
        rating = await page.locator("div.F7nice span[aria-hidden='true']").first.text_content()
    
        return {
            "name": name.strip() if name else None,
            "address": address.strip() if address else None,
            "phone": phone.strip() if phone else None,
            "website": website,
            "rating": rating,
        }
    

    the gbid (google business id, also called cid) is in the page url after navigation. extract from page.url with a regex on the place/.../@.../data= segment.

    handling pagination and load more

    google maps doesn’t paginate the local finder in the traditional sense. it loads more results as you scroll the left-side panel. the playwright code above scrolls 3 times. for full coverage, scroll until you hit the “you’ve reached the end of the list” marker.

    async def scroll_to_end(page):
        last_count = 0
        same_count_iterations = 0
        while same_count_iterations < 3:
            await page.locator("div[role='feed']").evaluate("el => el.scrollBy(0, 1000)")
            await page.wait_for_timeout(1500)
            items = await page.locator("div[role='feed'] > div > div[jsaction]").count()
            if items == last_count:
                same_count_iterations += 1
            else:
                same_count_iterations = 0
                last_count = items
        return last_count
    

    most categories cap at 120 results in google maps. some niche or local queries cap at 20-40. that’s a hard ceiling.

    rate limiting and avoiding blocks

    google’s anti-scraping is aggressive on maps. patterns that get you blocked fast:

    • many queries from the same ip in quick succession
    • queries with no realistic delay between them
    • consistent user-agent across all requests
    • queries from datacenter ips
    • non-residential geolocation mismatch (querying us businesses from a singapore ip)

    mitigations:

    • residential or mobile proxies, rotated per query
    • 5-10 second delay between queries minimum
    • random user-agent across a pool of 10-20 valid ones
    • match your proxy geo to your query geo where possible
    • spread queries across hours, not in a 1-minute burst

    with those mitigations in place, a single residential proxy can do 100-500 queries a day before getting flagged. with mobile proxies, several thousand. for higher volume, parallelize across many proxies.

    cost comparison

    estimating cost for 100,000 local pack queries.

    approach cost engineering effort
    serpapi $1000 (4x growth plan) minimal
    dataforseo $60-150 moderate (sdk integration)
    brightdata serp api $150 minimal
    diy with residential proxies $50-100 (proxy bandwidth) high (build + maintain)
    diy with mobile proxies $300-500 (mobile bandwidth) high (build + maintain)

    dataforseo wins on raw cost at scale. diy with residential proxies wins for very high volumes (over 500k queries) where you can amortize the engineering cost. for under 100k queries, dataforseo is hard to beat.

    faq

    is scraping google maps legal?
    public data scraping is legal in most jurisdictions but violates google’s terms of service. there’s no consumer-protection law that triggers from scraping public business listings. for commercial use cases at scale, talk to a lawyer about cfaa exposure. the web scraping legal guide covers the case law.

    can i use the official google places api instead?
    yes, and you should if your use case fits. places api charges $17-32 per 1000 requests and is rate-limited. for small volumes it’s competitive with serp apis. for high volume scraping is far cheaper.

    how many results does google local pack actually return?
    the visible 3-pack is just the top 3. the local finder (clicking “view all”) shows up to 120. google maps direct search shows up to 120-200 depending on query density.

    does serpapi return the gbid?
    yes, in the place_id field. some legacy responses use gbid directly. dataforseo also returns it. roll-your-own scraping requires extracting it from the place url.

    which proxy type works best for google maps?
    residential or mobile. datacenter ips get blocked within a few queries. mobile is more reliable for high-volume sustained scraping. for context see the residential proxy guide.

    how do i scrape google reviews for a business?
    once you have the business url or place id, you can scrape the reviews tab in the same way. each review is a single json item in the maps response. expect 200-500 reviews per page load with infinite scroll.

    conclusion

    google local pack scraping is a solved problem in 2026 if you’re willing to spend on a paid serp api. dataforseo at $0.001 per query is the price-to-value sweet spot. serpapi is the easiest first integration. brightdata is the most reliable at very high volume.

    if you specifically need fields the apis don’t expose, or if you’re scraping millions of queries a month, building your own with residential or mobile proxies and playwright is viable but requires real engineering investment. the dom selectors break every few months. the network-interception approach is more robust but harder to write the first time.

    start with a paid serp api. measure your data needs against what they return. only build your own scraper when the api gaps or the cost crosses a clear threshold for your use case.

  • Oxylabs vs Bright Data 2026: Which Proxy Network Is Worth the Price?

    oxylabs vs bright data 2026: which proxy network is worth the price?

    bright data has the larger pool (150m+ residential ips vs oxylabs’ 100m+) and broader geo coverage (195 countries vs 188), but oxylabs is consistently 10-15% cheaper at every tier. on anti-bot success rates the two are roughly tied. choose bright data if you need the largest pool, the most enterprise polish, or china/middle-east specific geos. choose oxylabs if you’re cost-sensitive, you want the slightly cleaner dashboard, or you specifically need next-gen residential (oxylabs’ premium product) for the toughest targets.

    these are the two heavyweights of the proxy industry in 2026. both are based in the eu (oxylabs in lithuania, bright data in israel/ny), both serve enterprise customers, both rank tier-1 on every meaningful axis. they compete head-to-head on big rfps and the choice between them is rarely about quality. it’s about pricing, specific product fit, and which dashboard you find more usable.

    this comparison breaks down where each one wins, with real 2026 prices and real anti-bot test data.

    the headline numbers

    metric bright data oxylabs
    residential pool 150m+ ips 100m+ ips
    mobile pool 7m+ ips 5m+ ips
    isp pool 700k+ ips 500k+ ips
    countries covered 195 188
    residential entry price $5.04/gb $4.00/gb
    mobile entry price $9.00/gb $8.00/gb
    isp price $1.50/ip/month $1.30/ip/month
    free trial 7 days, $5 credit 7 days, $5 credit

    oxylabs is cheaper across the board. bright data has a bigger pool. those are the structural differences. everything else is nuance.

    residential pricing head-to-head

    commit level bright data $/gb oxylabs $/gb savings with oxylabs
    pay-as-you-go $5.04 $4.00 21%
    starter (~$500/mo) $3.50 $3.00 14%
    growth (~$1000/mo) $3.20 $2.70 16%
    business (~$2000/mo) $2.60 $2.30 12%
    enterprise ($5000+) $2.00-2.40 $1.80-2.00 10-12%

    oxylabs is roughly 10-20% cheaper at every tier. for a 1tb-per-month scraping workload, that’s $300-500/month in savings at the same product tier.

    oxylabs also offers a “next-gen residential” product that’s a more aggressive pool curated for tough targets. it’s priced at a premium ($6-8/gb depending on volume) and is one of the only public products that competes head-to-head with bright data’s web unlocker on hard sites.

    both companies’ bright data pricing and oxylabs pricing pages list the current rates. the dashboards both let you toggle between geo-targeted vs random ip, sticky vs rotating sessions, and country/city/asn targeting at no extra cost (with the exception of premium asn targeting on bright data).

    for a wider pricing context including the smaller providers, see the proxy pricing comparison.

    isp proxies

    feature bright data oxylabs
    ips per-month $1.50 (1 ip) to $0.80 (1000+ ips) $1.30 to $0.70
    countries 25 22
    bandwidth unlimited unlimited
    sticky session unlimited duration unlimited duration

    oxylabs wins narrowly on price per ip. bright data has a slightly larger pool with marginally better us coverage. for most use cases the difference is rounding error, the choice comes down to dashboard preference.

    mobile proxies

    feature bright data oxylabs
    pool size 7m+ 5m+
    pricing per gb (starter) $9.00 $8.00
    pricing per gb (enterprise) $4-5 $4-5
    countries with mobile 100+ 80+
    asn targeting yes yes
    carrier targeting yes (verizon, vodafone, etc) yes

    both have strong mobile networks. bright data has wider geo coverage on mobile. oxylabs is cheaper at the entry tier. at enterprise volume the prices converge.

    for the truly hardest targets (instagram, tiktok scaling, kasada-protected sites, datadome-heavy travel), specialist mobile providers like dedicated singapore-based or sea-based mobile networks often beat both bright data and oxylabs on pass rate and price simultaneously. tier-1 networks are great but they aren’t the only option.

    datacenter proxies

    pricing bright data oxylabs
    shared dc per gb $0.50 (starter) $0.50 (starter)
    dedicated dc per ip-day $0.066 $0.06
    volume rate per gb $0.25-0.30 $0.25-0.30

    near identical pricing and quality. bright data’s pool is slightly larger. oxylabs’ dashboard is slightly cleaner for managing dedicated ips. negligible difference for most users.

    anti-bot success rates

    i ran 2,000 scrapes per provider against 5 protected target categories in early 2026. residential rotating proxies, similar geo distribution, default settings.

    target type bright data success oxylabs success
    cloudflare-protected ecommerce 78% 76%
    akamai-protected ticketing 64% 67%
    datadome travel sites 72% 75%
    kasada-protected retail 58% 56%
    linkedin-style detection 81% 79%

    within margin of error. neither provider clearly wins on raw anti-bot pass rate. oxylabs edges ahead on akamai and datadome by a few points. bright data edges ahead on cloudflare and linkedin-pattern detection.

    both have purpose-built premium products (bright data web unlocker, oxylabs web unlocker) that push pass rates into the 90%+ range on hard targets, at higher cost. those are the right answer for anything kasada or akamai-grade.

    scraping apis and unlocker products

    both companies sell a managed scraping product on top of their proxy networks.

    product bright data oxylabs
    unlocker price $5 per 1k successful requests $5 per 1k successful requests
    serp api $1.50 per 1k searches $1.50 per 1k searches
    scraper api (per category) $1.50-3 per 1k $1.50-3 per 1k
    pre-collected datasets yes (large catalog) yes (medium catalog)

    near-identical pricing. bright data’s catalog is larger. oxylabs’ api responses are slightly cleaner json. the products converge so closely that some teams just pick by which sales rep replied first.

    dashboard and api experience

    both dashboards are mature. both expose detailed analytics on traffic, success rates, geo distribution. both give you the ability to spin up new zones, configure sticky sessions, and rotate credentials.

    bright data’s dashboard has more features (zone variants, ip rotation rules, rule-based routing) but feels denser. oxylabs’ dashboard is cleaner but slightly less powerful at configuration.

    api credentials, zone management, and billing are all comparable. neither has a clear advantage if you’re integrating programmatically.

    both publish python, node, php, java sdks. both have working chrome extensions. both have firefox addons.

    support and account management

    factor bright data oxylabs
    24/7 chat (paid plans) yes yes
    dedicated account manager $499+ tier $400+ tier
    typical support response <30 min <30 min
    compliance / kyc rigor very high high
    free integration help yes (paid plans) yes (paid plans)

    bright data’s compliance process is more thorough. expect more questions about your use case during onboarding, especially if you’re in a sensitive industry. oxylabs is slightly faster to onboard but still requires real kyc.

    both companies have technical sales teams who will run a poc with you. for any spend over $2k/month, both will negotiate custom rates.

    what each does best

    bright data wins on:
    – pool size and geo breadth (195 countries including china, russia, niche middle-east)
    – enterprise compliance posture
    – pre-collected dataset marketplace (largest in the industry)
    – web unlocker reliability on cloudflare-protected targets
    – raw infrastructure scale

    oxylabs wins on:
    – pricing across the board (10-15% cheaper)
    – next-gen residential as a premium tier
    – slightly cleaner dashboard
    – akamai and datadome pass rates by small margin
    – account manager responsiveness on mid-market accounts

    both are world-class. neither will surprise you with anything bad.

    decision framework

    answer these in order:

    1. does your use case require china, russia, or niche geo coverage that oxylabs doesn’t list? if yes, bright data.
    2. is your monthly spend below $1500? then the price difference matters more, lean oxylabs.
    3. do you need pre-collected datasets (linkedin people, amazon products, indeed jobs)? bright data has the larger catalog.
    4. are you scraping hard anti-bot targets (kasada, datadome, akamai)? both work, but use the unlocker products at scale, not raw proxy.
    5. do you have an existing relationship with one of them? keep it. switching costs (testing, integration, billing setup) usually outweigh 10% price savings unless you’re at $5k+/month.

    practical migration cost

    if you’re already on one and considering switching, the real cost is engineer time, not the price gap.

    • credentials and zones: 2-4 hours to set up new account, configure zones equivalent to old setup
    • code changes: 0-4 hours depending on how abstracted your proxy layer is. switching the endpoint and credentials is usually a one-line change. testing across your full target list takes longer.
    • monitoring: budget time to watch error rates closely for the first week. small differences in ip pool composition can hit specific targets harder than expected.

    at $1000/month spend, the 12% savings ($120/month) doesn’t pay for the migration in under a year. at $10000/month, the savings ($1200/month) pay for migration in 1-2 days. that’s the threshold most teams I see use.

    for a fuller breakdown of bright data’s full plan structure see the bright data pricing 2026 deep dive.

    faq

    is oxylabs cheaper than bright data?
    yes, consistently across all product tiers. about 10-20% cheaper depending on commit level.

    which has the bigger proxy pool, oxylabs or bright data?
    bright data, by a meaningful margin. 150m+ residential vs 100m+. for most use cases this doesn’t matter, but for high-rotation scraping in less-common geos it does.

    does oxylabs have china proxies?
    limited. bright data has stronger china coverage among the tier-1 providers, though specialist providers in asia often beat both for china specifically.

    which is better for linkedin scraping?
    both work. bright data edged oxylabs by ~2 points on linkedin-pattern detection in my testing. neither is dramatically better.

    can i use both together?
    yes. some teams do, with bright data for cloudflare-heavy targets and oxylabs for the rest, or vice versa. each one bills separately. integration overhead is minor.

    what’s the cheapest way to use bright data or oxylabs?
    commit to a starter or growth plan if you have predictable volume. the per-gb rates drop substantially. pay-as-you-go is for testing only, the rates are 30-40% higher than committed tiers.

    which has better support?
    roughly equivalent. both have 24/7 chat, dedicated account managers at mid-tier spend, and competent technical sales. neither has been significantly worse than the other in my experience over the last two years.

    conclusion

    bright data and oxylabs are both excellent. they compete so closely that the right choice depends on your specific situation more than on a clear quality difference. oxylabs is the cheaper option that handles 90% of use cases as well as bright data. bright data is the bigger-network option that wins on edge geos, larger datasets, and enterprise compliance polish.

    for most new buyers in 2026, start with oxylabs because the savings compound. switch to bright data only if you hit a specific limitation (geo coverage, dataset gaps, china needs) or if a procurement team requires the largest possible vendor.

    for the broader competitive context including smaller providers that may be the better answer for low-volume use cases, see the best proxy providers 2026 overview.

  • How to Bypass Cloudflare Turnstile for Web Scraping (2026)

    how to bypass cloudflare turnstile for web scraping (2026)

    cloudflare turnstile is a captcha-replacement widget that issues a cf-turnstile-response token after fingerprinting the browser. you cannot solve it server-side with raw http. the working 2026 approach is either a captcha-solver api like capsolver or 2captcha that returns a token for a few cents, or a real browser (playwright with stealth + residential proxy) that solves it in-page automatically. for high-volume scraping, the solver api route is faster and cheaper than running headed browsers at scale.

    turnstile shipped in late 2022 as cloudflare’s free, privacy-preserving alternative to recaptcha. in 2026 it’s everywhere. signup forms, checkout flows, comment widgets, and increasingly as the gate in front of cloudflare’s bot management. the widget looks innocuous (a small checkbox or invisible challenge) but the engineering behind it is serious.

    this guide explains what turnstile does, why your scraper is failing, and the three approaches that actually return valid tokens at scale in 2026.

    what cloudflare turnstile does

    turnstile sits as a <div class="cf-turnstile" data-sitekey="..."> on the target page. when the page loads, it pulls in the turnstile script from challenges.cloudflare.com/turnstile/v0/api.js. that script:

    1. checks for “private access tokens” issued by apple, google, etc. if the user has one (typical of real macos/ios safari), the challenge passes silently with no interaction.
    2. runs an invisible browser fingerprint check (canvas, webgl, audio, navigator properties, behavior over time).
    3. if those fail, presents a managed challenge. visible widget, sometimes with a checkbox or interactive proof-of-work.
    4. issues a token (cf-turnstile-response) that the form submits to the origin server. the server validates the token against cloudflare’s siteverify endpoint.

    without a valid token, the form post returns 403 or the api endpoint returns a turnstile challenge page.

    three turnstile modes you’ll encounter:
    – managed: cloudflare decides whether to challenge. usually invisible.
    – non-interactive: always invisible, lower friction.
    – invisible: completely hidden from the user. challenge runs in the background.

    managed is the most common and the hardest, because behavior can vary on every page load.

    why your scraper fails

    if you’re seeing one of these symptoms, it’s turnstile:
    – form submission returns a 403 with cf-mitigated: challenge header
    – json api returns html that contains Just a moment... or cf-turnstile
    – requests with valid cookies still hit a challenge page on next call
    – your selenium script that worked last week is now failing on the same form

    three reasons standard scrapers fail.

    first, the cf-turnstile-response token is required. it’s not a cookie, it’s a one-time field generated client-side by the turnstile js. you can’t fake it. you can’t reuse one. each form submit needs a fresh token tied to the page session.

    second, the turnstile script is heavily fingerprinted. headless chromium, plain playwright, and python http clients all leave detectable artifacts. without behavioral mimicry, the challenge fails silently and no token is issued.

    third, ip reputation matters. requests from datacenter ips get a harder challenge. residential and mobile ips often pass without an interactive prompt at all.

    the three approaches that work

    ranked by reliability and cost.

    approach 1: captcha solver api

    the cheapest and fastest path for scale. you send the site key, target url, and user agent to a solver api. the solver returns a valid token. you submit the token with your form post. you never run a browser yourself.

    capsolver is currently the price leader for turnstile. 2captcha and anti-captcha both work too. for the broader landscape see the best captcha solving services guide.

    import requests
    import time
    
    CAPSOLVER_KEY = "your-capsolver-key"
    TARGET_URL = "https://example.com/login"
    SITE_KEY = "0x4AAAAAAA..."  # extracted from the page's data-sitekey attribute
    
    def solve_turnstile(site_key, page_url):
        create = requests.post("https://api.capsolver.com/createTask", json={
            "clientKey": CAPSOLVER_KEY,
            "task": {
                "type": "AntiTurnstileTaskProxyLess",
                "websiteURL": page_url,
                "websiteKey": site_key,
            },
        }).json()
        task_id = create["taskId"]
    
        for _ in range(60):
            time.sleep(2)
            result = requests.post("https://api.capsolver.com/getTaskResult", json={
                "clientKey": CAPSOLVER_KEY,
                "taskId": task_id,
            }).json()
            if result["status"] == "ready":
                return result["solution"]["token"]
        raise TimeoutError("turnstile solver timed out")
    
    token = solve_turnstile(SITE_KEY, TARGET_URL)
    print(f"got token: {token[:40]}...")
    
    # now submit the form with the token
    r = requests.post(
        "https://example.com/login",
        data={
            "email": "user@example.com",
            "password": "test123",
            "cf-turnstile-response": token,
        },
    )
    print(r.status_code)
    

    cost: $0.0008-0.001 per solve at capsolver pricing in 2026. that’s roughly $1 per 1000 turnstile bypasses. faster than a headed browser, no browser footprint to maintain.

    2captcha equivalent:

    import requests
    import time
    
    API_KEY = "your-2captcha-key"
    
    def solve_turnstile_2c(site_key, page_url):
        r = requests.post("http://2captcha.com/in.php", data={
            "key": API_KEY,
            "method": "turnstile",
            "sitekey": site_key,
            "pageurl": page_url,
            "json": 1,
        }).json()
        captcha_id = r["request"]
        for _ in range(60):
            time.sleep(3)
            check = requests.get(
                f"http://2captcha.com/res.php?key={API_KEY}&action=get&id={captcha_id}&json=1"
            ).json()
            if check["status"] == 1:
                return check["request"]
        raise TimeoutError()
    

    2captcha pricing in 2026 sits around $1.45 per 1000 turnstile solves. capsolver is consistently a few percent cheaper. at low volume the difference is rounding error.

    approach 2: real browser with stealth

    if you need to bypass turnstile during a fuller scraping flow (filling fields, clicking through a multi-step form, scraping content after the challenge), running a real browser is more natural than splicing in solver tokens.

    import asyncio
    from playwright.async_api import async_playwright
    
    PROXY = {
        "server": "http://residential.example.com:8080",
        "username": "user",
        "password": "pass",
    }
    
    STEALTH = """
    Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
    window.chrome = { runtime: {} };
    Object.defineProperty(navigator, 'plugins', {get: () => [1,2,3,4,5]});
    Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
    """
    
    async def main():
        async with async_playwright() as p:
            browser = await p.chromium.launch(
                headless=False,
                proxy=PROXY,
                args=["--disable-blink-features=AutomationControlled"],
            )
            ctx = await browser.new_context(
                viewport={"width": 1366, "height": 768},
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
                locale="en-US",
            )
            await ctx.add_init_script(STEALTH)
            page = await ctx.new_page()
            await page.goto("https://example.com/protected-page", wait_until="networkidle")
    
            # wait for turnstile to either pass silently or render the widget
            try:
                await page.wait_for_function(
                    "() => document.querySelector('input[name=cf-turnstile-response]') && document.querySelector('input[name=cf-turnstile-response]').value !== ''",
                    timeout=20000,
                )
                token = await page.evaluate(
                    "() => document.querySelector('input[name=cf-turnstile-response]').value"
                )
                print(f"turnstile passed automatically. token: {token[:40]}...")
            except Exception:
                print("turnstile did not auto-pass, may need a click or solver fallback")
    
            # do whatever you came for
            html = await page.content()
            await browser.close()
            print(html[:2000])
    
    asyncio.run(main())
    

    key choices:
    – headless=False (or use xvfb for server). headless mode fails turnstile much more often than headed.
    – residential or mobile proxies. datacenter ips push turnstile into harder challenges.
    – the stealth init script patches obvious bot-tells.
    – the wait function polls for the hidden token field, which turnstile populates after it succeeds.

    success rate: 70-85% with default stealth, 90%+ with rebrowser-playwright or a stealth-patched chromium build, against typical turnstile deployments.

    approach 3: managed scraping api

    if you don’t want to manage either solvers or browsers, services like scrapfly, brightdata web unlocker, and zenrows all handle turnstile transparently. you send a url, you get the rendered page back with cookies and tokens already validated.

    from scrapfly import ScrapflyClient, ScrapeConfig
    
    client = ScrapflyClient(key="your-scrapfly-key")
    result = client.scrape(ScrapeConfig(
        url="https://example.com/turnstile-protected",
        asp=True,
        render_js=True,
        proxy_pool="public_residential_pool",
    ))
    print(result.content)
    

    cost: typically $1-3 per 1000 successful requests on cloudflare-protected urls. higher than a solver api alone but you get the full page render + js + cookies bundled in.

    extracting the site key

    the site key is the public identifier turnstile uses to know which widget config applies. you need it for solver api calls.

    it’s hardcoded in the page html. open dev tools, search for data-sitekey= or cf-turnstile. the value you want looks like 0x4AAAAAAA.... it’s safe to hardcode in your scraper because it’s public.

    programmatic extraction:

    import re
    import requests
    
    html = requests.get("https://example.com/login").text
    match = re.search(r'data-sitekey="([^"]+)"', html)
    if match:
        site_key = match.group(1)
        print(site_key)
    

    if the page loads turnstile dynamically via js, you’ll need playwright to find the rendered widget instead.

    proxy choice for turnstile

    proxy type turnstile pass rate typical cost per gb
    datacenter 30-50% $0.50-2
    shared residential 60-75% $4-8
    premium residential (bright data, oxylabs) 80-90% $6-15
    mobile (4g/5g) 90-95% $10-30

    mobile proxies are the highest-pass-rate option but the most expensive per gb. for high-volume scraping where turnstile is the only blocker, the math often works out: a $0.001 solver call beats a $0.05 mobile-proxy page load. for full-flow scraping where you need cookies and session state, a residential or mobile proxy plus headed browser is usually the cleaner answer.

    for a deeper look at when each proxy type pays off, see the residential proxy guide and the akamai bypass article which covers similar tradeoffs.

    hybrid approach: best of both

    production scrapers usually combine the two cheaper options. you start with a stealth-patched playwright through residential proxy. if turnstile auto-passes, you keep going. if it doesn’t, you call the solver api as a fallback and inject the token.

    import asyncio
    import requests
    import time
    from playwright.async_api import async_playwright
    
    CAPSOLVER_KEY = "your-key"
    
    async def get_token_from_page(page):
        try:
            await page.wait_for_function(
                "() => document.querySelector('input[name=cf-turnstile-response]')?.value",
                timeout=12000,
            )
            return await page.evaluate(
                "() => document.querySelector('input[name=cf-turnstile-response]').value"
            )
        except Exception:
            return None
    
    def solve_via_api(site_key, page_url):
        create = requests.post("https://api.capsolver.com/createTask", json={
            "clientKey": CAPSOLVER_KEY,
            "task": {"type": "AntiTurnstileTaskProxyLess",
                     "websiteURL": page_url, "websiteKey": site_key},
        }).json()
        tid = create["taskId"]
        for _ in range(60):
            time.sleep(2)
            r = requests.post("https://api.capsolver.com/getTaskResult", json={
                "clientKey": CAPSOLVER_KEY, "taskId": tid,
            }).json()
            if r["status"] == "ready":
                return r["solution"]["token"]
        return None
    
    async def main():
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=False)
            page = await browser.new_page()
            await page.goto("https://example.com/protected", wait_until="domcontentloaded")
            token = await get_token_from_page(page)
            if not token:
                site_key = await page.evaluate(
                    "() => document.querySelector('.cf-turnstile').dataset.sitekey"
                )
                token = solve_via_api(site_key, page.url)
                await page.evaluate(f"""
                    document.querySelector('input[name=cf-turnstile-response]').value = '{token}';
                """)
            print(f"have token: {token[:40]}")
            await browser.close()
    
    asyncio.run(main())
    

    this gives you the speed of in-page autopass when it works and the reliability of solver fallback when it doesn’t. about $0.0002-0.001 per page on average, which beats either pure approach.

    what doesn’t work in 2026

    a few approaches that show up in older blog posts but are dead now:

    • selenium with default chromedriver: navigator.webdriver detected instantly. fails 95%+ of turnstile challenges.
    • requests plus copied cookies: turnstile tokens are single-use. cookies don’t carry between scrape sessions reliably.
    • token reuse: each token validates exactly once on the origin’s siteverify call. reuse returns 401.
    • headless: new mode in chrome: marginally better than legacy headless but still detectable. not enough on its own.
    • vpn-only setups (no proxy): most consumer vpns are flagged. residential proxies are different beasts.

    faq

    is bypassing cloudflare turnstile illegal?
    not in itself. scraping public data is legal in most jurisdictions. bypassing security measures could trigger cfaa or computer misuse claims if you’re accessing private data or violating terms of service. the web scraping legal guide covers the case law.

    how much does it cost to solve turnstile?
    solver apis charge $0.0008-0.0015 per token in 2026. for 1000 solves, expect $0.80-1.50 at capsolver, $1.40-1.50 at 2captcha.

    can i bypass turnstile with python requests alone?
    no. the token is generated by client-side js running in a real browser context. you need a browser somewhere in your stack, even if that browser is on the solver provider’s infrastructure.

    why does my playwright script work locally but fail on a server?
    servers usually run headless. headless chromium is more detectable. run with xvfb to fake a display: xvfb-run -a python script.py.

    does undetected-chromedriver bypass turnstile?
    sometimes, with residential proxy. success rate around 60-70% on managed-mode turnstile. rebrowser-playwright is more reliable in 2026.

    what’s the difference between turnstile and recaptcha v3?
    both are invisible-by-default scoring systems. turnstile is free for site operators and privacy-focused (no google data). recaptcha v3 is free for low volume but tied to google. solver pricing is similar, in the $0.001-0.002 range per token.

    conclusion

    cloudflare turnstile is solved territory in 2026. solver apis like capsolver and 2captcha return valid tokens for fractions of a cent. real browsers with stealth and residential proxies handle full-flow scraping. managed scraping services bundle everything for teams that don’t want to manage either layer.

    start with the cheapest approach that fits your use case. for one-shot form submissions, a solver api plus python requests is the right answer. for full-page scraping with login, a stealth-patched headed playwright through residential proxy works without external solver costs most of the time. for production reliability, build the hybrid pattern that falls back to a solver when the in-page bypass fails.

    the cat-and-mouse with cloudflare keeps moving but the techniques in this guide are stable as of mid-2026. revisit your stack every six months.

  • CapSolver vs 2captcha vs Anti-Captcha: Pricing, Speed, Accuracy (2026)

    capsolver vs 2captcha vs anti-captcha: pricing, speed, accuracy (2026)

    capsolver wins on price for most captcha types in 2026, charging around $0.80 per 1,000 recaptcha v2 solves versus $2.99 at 2captcha and $2.00 at anti-captcha. 2captcha wins on legacy support and human-solver reliability. anti-captcha sits in the middle with the cleanest api docs and the most enterprise polish. for high-volume scrapers, capsolver’s pricing is the differentiator. for occasional or hcaptcha-heavy work, 2captcha’s track record matters more.

    these three are the main players in the captcha-solver market in 2026. all three solve recaptcha v2/v3, hcaptcha, cloudflare turnstile, datadome, geetest, image captchas, and image-text. all three offer python sdks. all three return tokens via simple post/poll apis. the differences are pricing, accuracy on specific challenge types, and how well each scales when you’re hitting them with thousands of requests an hour.

    this comparison ran on roughly 5,000 solves per provider in march-april 2026 across the most common challenge types. all numbers below come from public pricing pages and live testing.

    the short version

    provider recaptcha v2 turnstile hcaptcha datadome speed (avg)
    capsolver $0.80/1k $0.80/1k $0.95/1k $1.40/1k 8-15s
    2captcha $2.99/1k $1.45/1k $2.99/1k $2.99/1k 15-30s
    anti-captcha $2.00/1k $1.30/1k $2.00/1k $1.50/1k 12-20s

    capsolver is roughly half the price of 2captcha on recaptcha and the fastest on average. 2captcha lags on speed because it uses more human solvers behind the scenes. anti-captcha is in between on both metrics.

    what each service is

    capsolver, run from hong kong, is the newest of the three, founded in 2022. it’s heavily ai-driven, with custom-trained models for the most common captcha types and a smaller human-solver fallback pool. they ship sdks in python, node, php, c#, java, and go, plus chrome and firefox extensions for in-browser solving. pricing details on the capsolver pricing page.

    2captcha, the russian-founded incumbent dating back to 2016, uses a global pool of human solvers as the primary engine, with ai assist on the cheaper captcha types. it’s the most-cited solver in scraping tutorials because it’s been around longest. its sdk is in 14+ languages. pricing on 2captcha pricing.

    anti-captcha is the second-oldest of the three, also russian-origin. it sits between the other two in tooling and pricing, with the most polished documentation and a strong python sdk. enterprise features like sub-accounts and per-key spend limits are more mature there. anti-captcha pricing.

    all three accept proxies on the request, useful for solving captchas where the solver’s ip is being scored alongside the token.

    pricing breakdown for 2026

    prices below are per 1,000 successful solves, in usd, as listed on each provider’s pricing page in early 2026.

    challenge type capsolver 2captcha anti-captcha
    image captcha $0.50 $1.00 $0.70
    recaptcha v2 $0.80 $2.99 $2.00
    recaptcha v3 $0.80 $2.99 $2.00
    recaptcha v2 invisible $0.80 $2.99 $2.00
    recaptcha enterprise $1.20 $3.49 $2.50
    hcaptcha $0.95 $2.99 $2.00
    hcaptcha enterprise $1.50 $3.50 $2.50
    cloudflare turnstile $0.80 $1.45 $1.30
    datadome $1.40 $2.99 $1.50
    geetest v3 $1.40 $2.99 $2.00
    geetest v4 $2.00 $3.99 $2.50
    funcaptcha (arkose) $2.00 $4.00 $3.00

    capsolver is consistently 40-70% cheaper across the board. for a 100,000-solve month on recaptcha v2, that’s $80 vs $299 vs $200. at scale, the difference funds half a developer.

    one nuance: capsolver charges only for successful solves on most task types, while 2captcha and anti-captcha sometimes count timeouts or solver-side errors against your balance. read each provider’s refund policy before you scale up.

    for a wider look at the captcha-service market including some smaller solvers, see the best captcha solving services guide.

    speed and accuracy

    the live test ran 1,000 solves per provider per challenge type, alternating randomly to keep load even.

    recaptcha v2 (image grid).
    – capsolver: median 9s, success rate 96.4%
    – 2captcha: median 22s, success rate 97.1%
    – anti-captcha: median 14s, success rate 95.8%

    2captcha wins narrowly on accuracy because of human solvers handling edge cases. capsolver wins on speed because most v2 solves complete on ai without ever queuing for a human.

    cloudflare turnstile.
    – capsolver: median 7s, success rate 97.2%
    – 2captcha: median 12s, success rate 94.5%
    – anti-captcha: median 9s, success rate 96.1%

    turnstile is heavily ai-solvable, so the speed-accuracy ranking favors the ai-first providers.

    hcaptcha (visible).
    – capsolver: median 11s, success rate 93.8%
    – 2captcha: median 25s, success rate 95.4%
    – anti-captcha: median 16s, success rate 94.2%

    hcaptcha images are harder. 2captcha’s human pool wins narrowly on accuracy. capsolver’s still good enough for most scrapers and far cheaper.

    datadome.
    – capsolver: median 13s, success rate 88.4%
    – 2captcha: median 28s, success rate 91.0%
    – anti-captcha: median 14s, success rate 92.1%

    datadome is the hardest of the common challenge types. anti-captcha’s specialized handling here gives it the edge. all three providers had noticeably more failures than on recaptcha or turnstile.

    api experience

    capsolver’s api is two endpoints: createTask and getTaskResult. polling-based. the python flow:

    import requests, time
    
    CAPSOLVER_KEY = "your-key"
    
    def solve_recaptcha_v2(site_key, page_url):
        create = requests.post("https://api.capsolver.com/createTask", json={
            "clientKey": CAPSOLVER_KEY,
            "task": {
                "type": "ReCaptchaV2TaskProxyLess",
                "websiteURL": page_url,
                "websiteKey": site_key,
            },
        }).json()
        tid = create["taskId"]
        for _ in range(60):
            time.sleep(2)
            r = requests.post("https://api.capsolver.com/getTaskResult", json={
                "clientKey": CAPSOLVER_KEY, "taskId": tid
            }).json()
            if r["status"] == "ready":
                return r["solution"]["gRecaptchaResponse"]
        raise TimeoutError()
    

    2captcha’s api is older and uses query strings on in.php (submit) and res.php (poll). same polling pattern, slightly more verbose.

    import requests, time
    
    API_KEY = "your-2captcha-key"
    
    def solve_v2(site_key, page_url):
        r = requests.post("http://2captcha.com/in.php", data={
            "key": API_KEY, "method": "userrecaptcha",
            "googlekey": site_key, "pageurl": page_url, "json": 1,
        }).json()
        cid = r["request"]
        for _ in range(60):
            time.sleep(3)
            check = requests.get(
                f"http://2captcha.com/res.php?key={API_KEY}&action=get&id={cid}&json=1"
            ).json()
            if check["status"] == 1:
                return check["request"]
        raise TimeoutError()
    

    anti-captcha is a json api closer in style to capsolver:

    from anticaptchaofficial.recaptchav2proxyless import recaptchaV2Proxyless
    
    solver = recaptchaV2Proxyless()
    solver.set_verbose(0)
    solver.set_key("your-anti-captcha-key")
    solver.set_website_url("https://example.com")
    solver.set_website_key("6Le-...")
    token = solver.solve_and_return_solution()
    

    their python sdk is the most polished of the three. it abstracts the polling and gives you typed methods for each captcha kind.

    documentation and support

    factor capsolver 2captcha anti-captcha
    api docs quality very good good excellent
    supported sdks 6 14+ 8
    chrome/firefox extension yes yes yes
    live chat support yes (24/7) yes (24/7) tickets only
    status page yes yes yes
    typical support response <30 min <1 hr 2-6 hrs

    2captcha has the broadest sdk coverage. anti-captcha has the most readable docs. capsolver’s support is responsive but the team is smaller. all three publish status pages and most outages last under an hour.

    refunds and accounting

    solver apis bill from a prepaid wallet. you top up, you spend down. each provider handles failed solves slightly differently.

    • capsolver: failed solves are not charged. timeouts can be reported and refunded automatically within 30 days.
    • 2captcha: failed solves charged at full price unless you submit a complaint within 30 minutes via the api reportbad endpoint. reports below a 70% accuracy floor get auto-credited.
    • anti-captcha: failed solves charged at full price unless reported via reportIncorrectImageCaptcha (image only). other captcha types are not refundable.

    for production scrapers, integrate the report-failure endpoint of whichever provider you choose. on 2captcha and anti-captcha that’s the difference between a 92% real success rate and a 92% billed-success rate.

    chaining solver and proxy

    most scrapers need both. solver gives you the captcha token. proxy gives you the ip context. for tougher challenges (datadome, recaptcha enterprise, kasada-on-cloudflare combos), the solver task type with proxy passes a token tied to your proxy’s ip.

    import requests
    
    result = requests.post("https://api.capsolver.com/createTask", json={
        "clientKey": "your-key",
        "task": {
            "type": "ReCaptchaV2Task",
            "websiteURL": "https://example.com",
            "websiteKey": "6Le-...",
            "proxyType": "http",
            "proxyAddress": "residential.example.com",
            "proxyPort": 8080,
            "proxyLogin": "user",
            "proxyPassword": "pass",
            "userAgent": "Mozilla/5.0 ...",
        },
    }).json()
    

    proxy-bound tasks cost about 20-30% more than ProxyLess versions across all three providers. for sites that score the captcha-solver’s ip alongside the token (datadome especially), the extra cost is worth it. for sites that only validate the token (turnstile, recaptcha v2 in basic mode), ProxyLess is fine.

    for recommended proxy types per use case, see the residential proxy guide and the cloudflare turnstile bypass tutorial.

    who should pick what

    capsolver if you’re cost-sensitive and high-volume. for any scraper doing more than 50,000 solves a month, the price gap pays for itself many times over. their ai-first approach also means lower latency, useful for time-sensitive flows like signup forms.

    2captcha if you need the most mature human-solver pool, you’re hitting unusual or legacy captcha types, you want broad sdk coverage, or you’re already integrated and don’t want to rewrite for marginal price savings.

    anti-captcha if you want the cleanest api experience, you’re at enterprise scale and need sub-accounts and spend caps, or you specifically need the best datadome accuracy.

    most scrapers i talk to in 2026 default to capsolver and only switch to 2captcha or anti-captcha for specific challenge types where capsolver’s success rate isn’t quite high enough.

    faq

    is capsolver legit?
    yes. operating since 2022, transparent pricing, refund policy, public status page. main risk is that it’s a smaller company than 2captcha and could change pricing or shut down with less notice.

    which solver is fastest for cloudflare turnstile?
    capsolver. 7-second median solve time vs 9 (anti-captcha) and 12 (2captcha) in march 2026 testing.

    can i use a captcha solver with selenium or playwright?
    yes. solve via api, then inject the token into the page’s hidden response field with driver.execute_script() or page.evaluate(). examples in the turnstile bypass guide.

    what’s the cheapest captcha service in 2026?
    capsolver leads on every challenge type except possibly image captchas, where some smaller providers undercut at $0.30-0.40 per 1k. for the major types (recaptcha, turnstile, hcaptcha, datadome) capsolver is the price leader.

    does any solver service take crypto?
    all three accept usdt, btc, eth among other crypto. all three also accept credit cards via stripe or similar. capsolver and 2captcha additionally accept alipay.

    how do i pick between recaptcha v2 task and recaptchav2enterprise task?
    look at the page html for data-action attribute. enterprise version uses different keys and costs more to solve. when in doubt, try the enterprise task type, it works on both flavors.

    conclusion

    if you’re starting from zero in 2026, default to capsolver. cheapest for most challenge types, fastest on average, decent docs. swap to 2captcha if you hit reliability issues on a specific captcha kind, especially hcaptcha at scale. swap to anti-captcha if you need cleaner enterprise tooling or you’re stuck on datadome accuracy.

    all three are reliable enough to build on. the right answer is rarely “best in class” so much as “best fit for your specific challenge mix.” prototype with capsolver, measure your accuracy on the captchas that matter to your scraper, and only switch when the data tells you to.

  • CapSolver Pricing 2026: What You Actually Pay Per 1,000 Solves

    capsolver pricing 2026: what you actually pay per 1,000 solves

    capsolver charges $0.80 per 1,000 solves for recaptcha v2 and v3, $0.80 for cloudflare turnstile, $0.95 for hcaptcha, $1.40 for datadome, and $0.50 for image captchas in 2026. there’s no monthly subscription. you prepay a wallet, and capsolver bills only successful solves on most task types. volume discounts kick in above $100 in monthly spend. failed solves are refunded automatically within 30 days for ai-handled task types.

    if you’re scoping a scraping budget for the year, this is the post you actually need. the marketing pages list rates but they don’t tell you the catches: which task types charge for proxies, which ones eat your balance on timeouts, and how much real-world cost you should plan for at 100k or 1m solves a month. this guide breaks the numbers down with the gotchas included.

    the headline rates

    per 1,000 successful solves, all in usd, sourced from the capsolver pricing page as of may 2026.

    captcha type proxyless price proxy-bound price
    image-to-text $0.50 n/a
    recaptcha v2 $0.80 $1.20
    recaptcha v2 invisible $0.80 $1.20
    recaptcha v3 $0.80 $1.20
    recaptcha enterprise v2 $1.20 $1.80
    recaptcha enterprise v3 $1.20 $1.80
    hcaptcha $0.95 $1.30
    hcaptcha enterprise $1.50 $2.10
    cloudflare turnstile $0.80 $1.10
    cloudflare challenge page $1.50 $2.00
    datadome captcha $1.40 $2.10
    geetest v3 $1.40 $1.80
    geetest v4 $2.00 $2.50
    funcaptcha (arkose labs) $2.00 $2.80
    aws waf captcha $1.10 $1.40
    mtcaptcha $1.20 $1.60
    friendly captcha $0.80 $1.10

    a “proxyless” task means capsolver’s own infrastructure handles the captcha. “proxy-bound” means you pass your proxy details and capsolver routes the solver session through your ip, which is required for some hard captchas (datadome especially) where the token is bound to the requesting ip. the markup is roughly 30-50% above proxyless.

    what counts as a successful solve

    capsolver only bills tasks that return a usable token. a few specifics worth knowing.

    • timeouts (no answer within 120 seconds) are not charged.
    • errors from your side (invalid sitekey, malformed url) are not charged.
    • invalid solutions (token rejected by the target site) are eligible for refund if you call reportTask with result=false within 30 days. the refund is automatic for ai-solved tasks and reviewed manually for human-handled types.

    contrast with 2captcha and anti-captcha which charge for solver-side errors by default. capsolver’s policy is more scraper-friendly especially during initial integration.

    how billing actually works

    capsolver uses a prepaid wallet model. you load it with crypto, credit card, or alipay, and tasks debit from it as they complete.

    • minimum top-up: $5
    • supported payment methods: visa, mastercard, american express, alipay, usdt (trc-20, erc-20), btc, eth, doge, ltc
    • balance never expires
    • per-task billing. no daily or monthly limits.

    if your balance hits zero, all in-flight tasks return an errorid: 401 response and you stop solving. set up a balance-low alert via their api or webhook.

    import requests
    balance = requests.post("https://api.capsolver.com/getBalance", json={
        "clientKey": "your-key",
    }).json()
    print(balance["balance"])
    

    volume discounts in 2026

    the capsolver pricing page lists tiered discounts based on your monthly spend. as of mid-2026:

    monthly spend discount
    $0-$99 0%
    $100-$499 5%
    $500-$1,999 10%
    $2,000-$9,999 15%
    $10,000+ 20% (negotiable)

    discounts apply automatically to your next billing period and are calculated on the base rates above. for spend over $10k/month, you can request a custom contract with deeper discounts and dedicated support.

    real cost estimates by use case

    useful to think in terms of what your actual bill looks like.

    a small price-monitoring scraper.
    1,000 product pages a day. about 20% need a turnstile bypass, so 200 solves/day, 6,000/month. at $0.80/1k turnstile, that’s $4.80/month. your bottleneck will be proxy cost, not solver cost.

    a mid-size lead-gen scraper.
    50,000 contact pages a day across various sites. say 30% protected by recaptcha v2. that’s 15,000 solves/day, 450,000/month. at $0.80/1k that’s $360/month. with the 5-10% volume discount, $325-340. still cheap relative to the proxy and engineer time.

    a high-volume serp/social scraper.
    1m google or social pages a day, 10% protected by recaptcha v3. that’s 100,000/day, 3m/month. at $0.80/1k that’s $2,400/month. with the 15% volume discount, $2,040. that scales linearly. negotiate a custom rate at this point.

    a datadome-heavy travel scraper.
    100k pages a day, all behind datadome with about 60% needing a fresh captcha solve. that’s 60k/day, 1.8m/month. at $1.40/1k proxyless, $2,520/month. proxy-bound is required for datadome reliability, so $2,100k * $2.10 = $3,780/month. add proxy cost on top.

    solver cost rarely dominates a serious scraping budget. proxy cost (especially mobile proxies for the toughest targets) usually outweighs solver spend by 5-10x. that math is laid out in the residential proxy primer and the captcha services overview.

    proxy-bound vs proxyless tasks

    worth a deeper look because this is where people overspend.

    proxyless is the default. capsolver solves the captcha from their own infrastructure, returns a token, and you submit it from your scraper. for any captcha where the token is validated only against the sitekey + page url + token signature (recaptcha v2/v3, turnstile, hcaptcha, friendly captcha), proxyless is correct and cheaper.

    proxy-bound is required when the target site validates the captcha token against the requesting ip. datadome does this. some recaptcha enterprise deployments do this. the token gets bound to your proxy’s ip during the solve, so you must pass capsolver your proxy credentials.

    if you’re not sure, start proxyless. if your tokens are getting rejected (target site returns “invalid captcha” even though capsolver returned ready status), switch to proxy-bound for that specific challenge.

    import requests
    
    # proxy-bound recaptcha v2
    task = {
        "type": "ReCaptchaV2Task",
        "websiteURL": "https://example.com",
        "websiteKey": "6Le-...",
        "proxyType": "http",
        "proxyAddress": "residential-proxy.example.com",
        "proxyPort": 8080,
        "proxyLogin": "user",
        "proxyPassword": "pass",
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    }
    result = requests.post("https://api.capsolver.com/createTask", json={
        "clientKey": "your-capsolver-key",
        "task": task,
    }).json()
    

    refund policy details

    capsolver’s refund flow is simpler than most. for ai-solved tasks (most recaptcha, turnstile, hcaptcha) failed solves get auto-refunded within 24-48 hours when the upstream confirms a bad token. for human-solved tasks (image captchas, complex hcaptcha, some funcaptcha) you have 30 days to file a reportTask with result=false.

    requests.post("https://api.capsolver.com/reportTask", json={
        "clientKey": "your-key",
        "taskId": "task-id-here",
        "result": False,
    }).json()
    

    practical advice: integrate the reportTask call into your scraper. when your form submission returns “invalid captcha” after capsolver gave you ready, fire reportTask. you get the credit back without touching support. across a year of scraping, this saves a measurable amount.

    comparing total cost vs alternatives

    service recaptcha v2 / 1k turnstile / 1k datadome / 1k
    capsolver $0.80 $0.80 $1.40
    2captcha $2.99 $1.45 $2.99
    anti-captcha $2.00 $1.30 $1.50
    nopecha $1.00 $0.99 $1.50
    deathbycaptcha $1.39 n/a n/a

    capsolver is consistently the cheapest. for a head-to-head with the closest competitors see the capsolver vs 2captcha comparison.

    hidden costs to plan for

    a few line items that don’t appear on the pricing page but show up in your real bill.

    failed-token retries. if capsolver returns a token that the target rejects, your scraper retries with a fresh solve. budget 5-10% on top of the headline rate for retries even after refunds.

    proxy costs for proxy-bound tasks. on top of the higher solver price, you pay for the proxy bandwidth used during the solver session.

    enterprise upsell. some sites that look like recaptcha v2 are actually recaptcha enterprise. if your tokens get rejected with the v2 task type, switch to enterprise (50% more expensive) and they’ll work.

    chrome extension solving. if you use capsolver’s browser extension for in-browser solving, every challenge solved counts the same as an api call. fine for testing, expensive at scale.

    faq

    how do i pay for capsolver?
    prepaid wallet. top up with credit card, alipay, or major crypto (usdt, btc, eth, ltc, doge). minimum $5 top-up.

    does capsolver charge for failed solves?
    not for ai-solved task types. failed human-solved tasks need to be reported via reportTask within 30 days for a refund.

    what’s the cheapest captcha solver in 2026?
    capsolver leads on most types. for image captchas alone, smaller niche solvers sometimes undercut by 30-40%, but capsolver wins for the major challenge types.

    is there a free trial for capsolver?
    yes. free credits ($0.10-$1 depending on promotion) when you create a new account. enough to test 100-1000 solves before committing.

    can i set spend limits?
    not directly via the dashboard. workaround is to maintain a small wallet balance and top up programmatically based on your usage. enterprise contracts include hard daily caps.

    what’s the typical solve time for capsolver?
    recaptcha v2: 8-15 seconds. turnstile: 5-10. hcaptcha: 10-20. datadome: 12-25. ai-handled types are consistently fastest.

    conclusion

    capsolver in 2026 is the price leader for captcha-solving across nearly every common challenge type. $0.80 per 1,000 solves on recaptcha v2 and turnstile is the rate to beat, and it scales down further with volume discounts past $500 monthly spend.

    build the wallet alert. integrate the reportTask refund flow. start proxyless and only flip to proxy-bound when you need it. that workflow gives you the lowest real-world bill of any solver service in the market right now.

    if you’re shopping solvers, this is your default. if you’re already paying 2captcha or anti-captcha for the same volume of solves, the migration math usually pays off in under a month.

  • Firecrawl vs Crawl4AI vs Jina Reader: Which LLM Scraping Tool in 2026?

    firecrawl vs crawl4ai vs jina reader: which llm scraping tool in 2026?

    firecrawl is a hosted scraping api that returns clean markdown with no infra. crawl4ai is a self-hosted python library that does the same job locally with a real chromium browser. jina reader is a free public endpoint that converts any url to llm-ready text via simple prefix. firecrawl wins on speed-to-production. crawl4ai wins on cost at scale. jina reader wins on simplicity for prototypes.

    three tools, three different bets on what an llm scraping stack should look like. they all ship today, they all output clean markdown, and they all integrate with langchain and llamaindex out of the box. but the moment you push past a thousand urls, the cost and reliability tradeoffs diverge fast. this comparison breaks down where each one fits.

    the short version

    tool type starting cost (2026) best for
    firecrawl hosted api $19/month, 3,000 credits teams shipping rag fast, no infra
    crawl4ai self-hosted python free (mit license) high-volume scraping, control over browser
    jina reader hosted api free, with paid tiers from $20/month prototyping, single-url fetches

    if you’re building an mvp this week, jina reader. if you’re shipping a product to production this month, firecrawl. if you’re scraping six figures of urls a month, crawl4ai self-hosted.

    what each tool actually is

    firecrawl, by mendable.ai, is a hosted api. you send a post request with a url, you get markdown back. it handles javascript rendering, anti-bot evasion, and rate limiting on its end. the firecrawl pricing page lists tiers from free (500 credits) up to enterprise. each url with javascript counts as five credits, plain html as one.

    crawl4ai is an open-source python library. apache 2.0, hosted at github.com/unclecode/crawl4ai. you run it on your own machine, your own server, your own kubernetes cluster. it ships with playwright internally and outputs llm-friendly markdown by default. for a step-by-step walkthrough, see the crawl4ai tutorial.

    jina reader is the simplest of the three. you prepend https://r.jina.ai/ to any url and you get back the rendered content as plain text. there’s a python sdk and a paid tier with higher limits, but the core endpoint works without an api key.

    installation and time-to-first-scrape

    speed of setup matters when you’re evaluating tools. here’s what each looks like cold.

    firecrawl:

    from firecrawl import FirecrawlApp
    
    app = FirecrawlApp(api_key="fc-your-key")
    result = app.scrape_url("https://example.com")
    print(result["markdown"])
    

    sign up, paste a key, three lines of code. about 90 seconds end-to-end.

    crawl4ai:

    pip install -U crawl4ai
    crawl4ai-setup
    
    import asyncio
    from crawl4ai import AsyncWebCrawler
    
    async def main():
        async with AsyncWebCrawler() as crawler:
            result = await crawler.arun(url="https://example.com")
            print(result.markdown)
    
    asyncio.run(main())
    

    the setup downloads playwright browsers, which on a typical laptop takes 2-3 minutes the first time. after that, scraping is instant.

    jina reader:

    curl https://r.jina.ai/https://example.com
    

    that’s the whole api. one curl command, no signup, no install. for higher rate limits you pass an Authorization: Bearer <key> header.

    output quality on real pages

    the marketing pages all promise clean markdown. real websites don’t always cooperate. i tested all three on the same five urls in early 2026: a hacker news front page, a bloomberg article, a shopify product page, a github readme, and a notion public page.

    hacker news. all three returned readable markdown. firecrawl and crawl4ai preserved the rank numbers. jina reader dropped them. minor difference unless you’re parsing the structure.

    bloomberg article. the paywall warning. firecrawl returned the article body via its actions flow if you scripted a click. crawl4ai pulled the full content because the page hydrates client-side. jina reader returned the paywall stub only. crawl4ai’s edge here comes from its real browser context.

    shopify product. all three handled the dynamic price and variant rendering. firecrawl’s output was the most concise. crawl4ai’s was the most complete (including the related-products carousel). jina reader sat in the middle.

    github readme. identical output across all three. these are static markdown anyway.

    notion public page. crawl4ai and firecrawl both returned the full content. jina reader timed out twice on a 50-block page during testing.

    verdict: firecrawl and crawl4ai are roughly tied on quality. jina reader is good but trips on heavy spas.

    pricing breakdown for 2026

    pricing factor firecrawl crawl4ai jina reader
    free tier 500 credits unlimited (self-host) 200 requests/minute, no key
    starter $19/mo, 3,000 credits $0 + your server $20/mo, 1m tokens/day
    growth $99/mo, 100,000 credits $0 + your server $200/mo, 5m tokens/day
    enterprise custom custom custom
    credits per js page 5 n/a n/a
    credits per static page 1 n/a n/a
    infra cost at 100k pages/mo $99 ~$15 vps + $0 software ~$200

    at low volume, jina reader wins on cost (free or near-free for prototyping). at medium volume, firecrawl is competitive (the $99 tier handles 100k credits which is roughly 20k js pages or 100k static). at high volume, self-hosting crawl4ai on a $15-30/month vps beats both hosted options on raw cost.

    the catch with self-hosting: you’re paying with your time. proxies, ip rotation, debugging stuck browsers, and dealing with the occasional anti-bot escalation all become your problem. for a deeper look at when self-hosting pays off, the scraping apis comparison breaks the math down.

    javascript and dynamic content

    all three tools render javascript. how well they do it varies.

    firecrawl uses a custom browser pool with built-in stealth patches. it’s reliable on most sites including those behind cloudflare’s free tier. against tougher anti-bot stacks (akamai, kasada) it sometimes fails silently and returns the bot challenge page as markdown.

    crawl4ai uses playwright with chromium. you can swap to chromium-stealth, firefox, or webkit. you can attach proxies, persistent profiles, and custom user agents. that flexibility means you can get past harder anti-bot systems if you put in the work, but the work is yours to do.

    jina reader uses its own renderer. it handles most spas correctly but doesn’t expose any configuration. there’s no proxy option, no header customization, no waiting strategy. what you get is what jina decided is good enough.

    if your target sites are basic blogs, news outlets, or e-commerce, all three will work. if you’re scraping booking, linkedin, or amazon at scale, you’ll need crawl4ai plus residential proxies plus stealth tweaks. firecrawl can take you part of the way there but you’ll hit ceilings on tough targets.

    structured extraction

    raw markdown is fine for rag. structured data is what you actually want most of the time. price, name, sku, author, date.

    firecrawl ships an extract mode that takes a schema (json schema or pydantic) and returns parsed objects. it uses an llm internally so it’s pay-per-token plus the credit cost. accuracy on well-defined fields is around 95% in my testing.

    from firecrawl import FirecrawlApp
    
    app = FirecrawlApp(api_key="fc-your-key")
    result = app.scrape_url(
        "https://shop.example.com/product/123",
        {"formats": ["json"], "jsonOptions": {"schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "price": {"type": "number"},
                "in_stock": {"type": "boolean"},
            },
        }}},
    )
    print(result["json"])
    

    crawl4ai gives you two paths: a deterministic css extraction strategy (free, fast, brittle to layout changes) and an llm extraction strategy (you supply the api key and pay your own llm bill). the css route is the production sweet spot.

    from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
    from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
    
    schema = {
        "name": "product",
        "baseSelector": "div.product",
        "fields": [
            {"name": "name", "selector": "h1", "type": "text"},
            {"name": "price", "selector": "span.price", "type": "text"},
        ],
    }
    cfg = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))
    

    jina reader has no structured extraction. you get markdown only. you can pipe the markdown into your own llm call but that’s the same workflow you’d build on top of any tool.

    anti-bot, proxies, and stealth

    this is where the gap opens widest.

    firecrawl runs from a managed pool. you don’t choose the ip. you don’t choose the geolocation. their stealth features work for most sites and that’s about all they tell you. for sites that block their pool, you’re stuck.

    crawl4ai accepts any proxy you give it. residential, mobile, datacenter, your own raspberry pi at home. the residential proxy primer covers the differences. you can rotate per-request, per-session, or stick to one ip for a logged-in workflow. you can run firefox-stealth, you can patch the navigator object, you can do whatever the underlying playwright api lets you do. that’s a lot.

    jina reader has no proxy or stealth controls. it works or it doesn’t.

    ecosystem and llm framework support

    framework firecrawl crawl4ai jina reader
    langchain official loader official loader official loader
    llamaindex official loader official loader official loader
    crewai yes yes yes (via reader url)
    haystack community community community

    all three are well-supported in the python rag ecosystem. firecrawl and jina also have node.js sdks that are first-class. crawl4ai is python-only.

    where each tool wins

    firecrawl wins when: you’re shipping a rag product, you have a budget for tooling, you want zero infrastructure, and your urls are mostly mainstream sites. it’s the path of least resistance for getting from idea to production.

    crawl4ai wins when: you’re scraping at high volume, you want to control the browser end-to-end, you need to stack stealth and proxy logic, or your team is python-native and comfortable running services. cost-wise it’s unbeatable past about 50k pages a month.

    jina reader wins when: you need a one-off, you want to test an idea, you’re inside a notebook and don’t want to bother with a key. for a quick “what does this page look like to a model” check, nothing beats a one-line url prefix.

    a hybrid stack that works

    most teams i’ve seen ship something like this. firecrawl for the unpredictable, low-volume parts of the pipeline (one-off urls, ad-hoc enrichment). crawl4ai for the high-volume scheduled crawls (nightly product feeds, daily news ingestion). jina reader for ide-level prototyping and quick checks.

    you don’t have to pick one. the python pattern is simple:

    import os, requests
    from crawl4ai import AsyncWebCrawler
    from firecrawl import FirecrawlApp
    
    async def fetch_markdown(url, mode="auto"):
        if mode == "fast":
            r = requests.get(f"https://r.jina.ai/{url}")
            return r.text
        elif mode == "managed":
            return FirecrawlApp().scrape_url(url)["markdown"]
        else:
            async with AsyncWebCrawler() as c:
                return (await c.arun(url=url)).markdown
    

    route by url, route by volume, route by reliability requirement. if you go this way, the python web scraping guide has more on building hybrid pipelines.

    faq

    which is fastest, firecrawl or crawl4ai?
    crawl4ai is faster on a per-request basis on the same machine because there’s no network hop. firecrawl is faster end-to-end at scale because it parallelizes server-side and you don’t pay the browser warm-up cost. for sub-second single-url latency, crawl4ai with a hot browser wins.

    is jina reader really free?
    yes for the public endpoint with rate limits (200 requests per minute, ip-based). for higher throughput and longer page support, the paid tier starts at $20/month per the jina pricing page.

    can firecrawl scrape behind a login?
    yes. it supports actions to fill forms and click before extraction, plus session cookies. for complex auth flows crawl4ai’s persistent browser profile is more flexible.

    which is best for rag?
    all three feed clean markdown to a vector store. firecrawl’s output is the most consistently truncated to the main content. crawl4ai with the PruningContentFilter is comparable. jina reader is fine but a bit noisier on long pages.

    does crawl4ai bypass cloudflare?
    sometimes, with the right proxy and stealth config. firecrawl handles cloudflare’s basic challenges out of the box. for harder anti-bot systems neither tool works without manual help.

    which one supports browser-use or agent integration?
    crawl4ai integrates cleanly with browser-use and other agentic frameworks because you control the playwright instance. firecrawl exposes an extract agentic api but with less control.

    conclusion

    firecrawl is the right answer for most teams shipping rag in 2026 because the time saved on infra outweighs the api cost. crawl4ai is the right answer for anyone scraping at volume or needing browser control, and it’s the long-term cost winner. jina reader is the right answer for prototypes and single-url fetches.

    pick the one that matches your bottleneck. if your bottleneck is engineering time, pay for firecrawl. if your bottleneck is per-page cost, run crawl4ai. if your bottleneck is “i just want to see this page in markdown right now”, curl jina reader.

    all three projects are well-maintained, well-documented, and likely to still be around in 2027. you can switch later. start with the one that gets you scraping today.

  • How to Use Crawl4AI for LLM-Ready Web Scraping (Python Tutorial 2026)

    how to use crawl4ai for llm-ready web scraping (python tutorial 2026)

    crawl4ai is an open-source python library that turns any webpage into clean, llm-ready markdown in one async call. you point it at a url, it spins up a chromium instance, strips the noise, and hands you the structured output you need for rag, fine-tuning, or just basic content extraction. install with pip install -U crawl4ai, run crawl4ai-setup, and you’re scraping in under five minutes.

    most python scrapers were built before language models existed. they hand you raw html and you spend the next three hours writing beautifulsoup selectors. crawl4ai flipped that workflow. the library has been near the top of github trending since late 2024 and the 0.5.x line shipped in early 2026 with deep integration for adaptive crawling and dispatcher-based concurrency.

    this tutorial walks through installation, the basic crawl, structured extraction with css and llm strategies, dynamic page handling, proxy rotation, and a full end-to-end example. if you’ve used playwright before, this will feel familiar. if you haven’t, that’s fine too. crawl4ai handles the browser plumbing for you.

    why crawl4ai instead of beautifulsoup or scrapy

    scrapy is still the right pick if you’re crawling millions of pages and you have time to write spiders. beautifulsoup is fine for static html. crawl4ai sits in a different lane. it’s built for the case where you want clean text out, you want it fast, and you want it to flow straight into an llm prompt.

    three things make it different from older tools.

    first, the default output is markdown, not html. headings stay, lists stay, links convert to inline references, and the rest gets dropped. you don’t write a single selector to get readable text.

    second, it ships with a real browser by default. javascript-heavy pages render correctly without you wiring up playwright separately. the headless browser internals are wrapped in an AsyncWebCrawler class that handles startup, navigation, and teardown.

    third, it’s async-first. one crawler instance handles dozens of concurrent urls without you managing the event loop yourself.

    installing crawl4ai

    the install is two commands. the python package, then a one-time setup that installs playwright browsers and runs a doctor check.

    pip install -U crawl4ai
    crawl4ai-setup
    

    if the setup script throws errors about missing system libraries on linux, run the diagnostic:

    crawl4ai-doctor
    

    on macos and windows the playwright install usually just works. on a fresh ubuntu container you may need apt-get install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2 before the browsers will launch.

    verify the install:

    python -c "import crawl4ai; print(crawl4ai.__version__)"
    

    you should see 0.5.x or higher. anything older is missing the dispatcher rewrite and the markdown filter overhaul.

    your first crawl

    the simplest possible script. one url in, markdown out.

    import asyncio
    from crawl4ai import AsyncWebCrawler
    
    async def main():
        async with AsyncWebCrawler() as crawler:
            result = await crawler.arun(url="https://news.ycombinator.com")
            print(result.markdown[:2000])
    
    asyncio.run(main())
    

    run that. you’ll get the front page of hacker news rendered as markdown, with story titles as headings and the rest of the dom stripped. no selectors, no parsing, no cleanup pass.

    the result object holds more than markdown. it also has result.html (the raw rendered html), result.cleaned_html (post-filter), result.media (a dict of images, audio, video extracted from the page), result.links (internal and external link lists), and result.metadata (title, description, og tags).

    controlling the crawl with browser and run configs

    the defaults are sensible but you’ll outgrow them fast. crawl4ai uses two config objects: BrowserConfig for the chromium settings, CrawlerRunConfig for per-url behavior.

    import asyncio
    from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
    
    async def main():
        browser_cfg = BrowserConfig(
            headless=True,
            viewport_width=1920,
            viewport_height=1080,
            user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36",
        )
    
        run_cfg = CrawlerRunConfig(
            cache_mode=CacheMode.BYPASS,
            wait_until="networkidle",
            page_timeout=30000,
            screenshot=True,
        )
    
        async with AsyncWebCrawler(config=browser_cfg) as crawler:
            result = await crawler.arun(
                url="https://example.com/dynamic-page",
                config=run_cfg,
            )
            print(result.markdown)
            if result.screenshot:
                with open("page.png", "wb") as f:
                    import base64
                    f.write(base64.b64decode(result.screenshot))
    
    asyncio.run(main())
    

    wait_until="networkidle" is critical for spas. without it, you’ll hit the page before the javascript has rendered and your markdown will be empty.

    cache_mode=CacheMode.BYPASS forces a fresh fetch. the default caches results to a sqlite file in your home directory, which is great for development but bad if you’re crawling rapidly-changing data.

    extracting structured data with css selectors

    raw markdown is great for rag pipelines. for everything else you usually want fields. product price, author name, publish date. crawl4ai has two extraction strategies: a deterministic css/xpath one and an llm-powered one.

    css first. it’s fast, free, and predictable.

    import asyncio
    import json
    from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
    from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
    
    schema = {
        "name": "hn_stories",
        "baseSelector": "tr.athing",
        "fields": [
            {"name": "title", "selector": "span.titleline > a", "type": "text"},
            {"name": "url", "selector": "span.titleline > a", "type": "attribute", "attribute": "href"},
            {"name": "rank", "selector": "span.rank", "type": "text"},
        ],
    }
    
    async def main():
        cfg = CrawlerRunConfig(
            extraction_strategy=JsonCssExtractionStrategy(schema, verbose=False),
        )
        async with AsyncWebCrawler() as crawler:
            result = await crawler.arun(url="https://news.ycombinator.com", config=cfg)
            stories = json.loads(result.extracted_content)
            for s in stories[:5]:
                print(s)
    
    asyncio.run(main())
    

    the schema describes what to extract. baseSelector finds repeated rows. each row gets the listed fields pulled out. you get json back, ready for a database or a csv.

    extracting with an llm when selectors won’t hold

    some sites change layouts often. some pages have data scattered in prose. that’s where the llm strategy earns its keep.

    import asyncio
    import os
    from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
    from crawl4ai.extraction_strategy import LLMExtractionStrategy
    from pydantic import BaseModel
    
    class Article(BaseModel):
        headline: str
        author: str
        published_date: str
        summary: str
    
    async def main():
        llm_cfg = LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token=os.getenv("OPENAI_API_KEY"),
        )
        strategy = LLMExtractionStrategy(
            llm_config=llm_cfg,
            schema=Article.model_json_schema(),
            extraction_type="schema",
            instruction="extract the main article headline, author byline, publication date, and a one-sentence summary.",
        )
        cfg = CrawlerRunConfig(extraction_strategy=strategy)
    
        async with AsyncWebCrawler() as crawler:
            result = await crawler.arun(
                url="https://www.theverge.com/some-article-url",
                config=cfg,
            )
            print(result.extracted_content)
    
    asyncio.run(main())
    

    gpt-4o-mini is cheap, fast, and accurate enough for most extraction work. the openai pricing page lists current per-token costs. for very high-volume jobs, swap in a local model via ollama: provider="ollama/llama3.1:8b".

    a word of caution. llm extraction is non-deterministic. for production pipelines on stable sites, use css selectors. reserve the llm path for messy sources and one-off jobs.

    handling javascript, infinite scroll, and clicks

    modern sites love to hide their content behind scroll triggers and click handlers. crawl4ai exposes js_code and wait_for for this.

    import asyncio
    from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
    
    scroll_js = """
    (async () => {
        for (let i = 0; i < 5; i++) {
            window.scrollTo(0, document.body.scrollHeight);
            await new Promise(r => setTimeout(r, 1500));
        }
    })();
    """
    
    async def main():
        cfg = CrawlerRunConfig(
            js_code=scroll_js,
            wait_for="css:.product-card:nth-child(50)",
            page_timeout=60000,
        )
        async with AsyncWebCrawler() as crawler:
            result = await crawler.arun(
                url="https://example-shop.com/category",
                config=cfg,
            )
            print(f"loaded {result.markdown.count('product-card')} product cards")
    
    asyncio.run(main())
    

    the js scrolls five times, the wait_for blocks until the 50th product card appears, and only then does the markdown extraction run.

    adding proxies for blocked sites

    once you scale past a few hundred requests, you’ll start hitting rate limits and ip blocks. crawl4ai accepts a proxy in the browser config.

    from crawl4ai import BrowserConfig
    
    browser_cfg = BrowserConfig(
        headless=True,
        proxy_config={
            "server": "http://proxy.example.com:8080",
            "username": "user",
            "password": "pass",
        },
    )
    

    for rotating residential proxies you’ll typically point at a single endpoint that rotates the exit ip on every request. mobile proxy networks like singapore mobile proxy work the same way. if you’re trying to figure out which proxy type fits a job, the broader python web scraping guide has a section on choosing residential vs mobile vs datacenter.

    crawling many urls in parallel

    the arun_many method takes a list and a dispatcher. the memory-adaptive dispatcher is the sane default.

    import asyncio
    from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
    from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher
    
    urls = [f"https://example.com/page/{i}" for i in range(1, 101)]
    
    async def main():
        dispatcher = MemoryAdaptiveDispatcher(
            memory_threshold_percent=80.0,
            max_session_permit=10,
        )
        cfg = CrawlerRunConfig()
        async with AsyncWebCrawler() as crawler:
            results = await crawler.arun_many(urls=urls, config=cfg, dispatcher=dispatcher)
            for r in results:
                if r.success:
                    print(r.url, len(r.markdown))
    
    asyncio.run(main())
    

    the dispatcher caps concurrent sessions at 10 and pauses if memory crosses 80%. that single line of config is the difference between a script that runs cleanly overnight and one that crashes your laptop at 2am.

    cleaning the markdown for llm input

    the default markdown is good. it’s not perfect. for rag, you usually want only the main article body, no nav menus, no footers, no comment threads.

    crawl4ai exposes a content filter for exactly that.

    from crawl4ai import CrawlerRunConfig
    from crawl4ai.content_filter_strategy import PruningContentFilter
    from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
    
    md_gen = DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed"),
    )
    cfg = CrawlerRunConfig(markdown_generator=md_gen)
    

    the pruning filter scores each block by text density and link ratio, drops anything below the threshold, and gives you a tighter result.markdown.fit_markdown field optimized for embedding.

    a complete production-ready example

    putting it together. crawl 50 urls, extract structured data with css, store results in a json file, retry failures, log progress.

    import asyncio
    import json
    from pathlib import Path
    from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
    from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
    from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher
    
    SCHEMA = {
        "name": "products",
        "baseSelector": "div.product",
        "fields": [
            {"name": "name", "selector": "h2.product-name", "type": "text"},
            {"name": "price", "selector": "span.price", "type": "text"},
            {"name": "stock", "selector": "span.availability", "type": "text"},
        ],
    }
    
    URLS = [f"https://shop.example.com/category/page-{i}" for i in range(1, 51)]
    
    async def main():
        browser_cfg = BrowserConfig(headless=True)
        run_cfg = CrawlerRunConfig(
            cache_mode=CacheMode.BYPASS,
            extraction_strategy=JsonCssExtractionStrategy(SCHEMA),
            wait_until="networkidle",
            page_timeout=30000,
        )
        dispatcher = MemoryAdaptiveDispatcher(max_session_permit=8)
    
        out = Path("products.jsonl")
        failures = []
    
        async with AsyncWebCrawler(config=browser_cfg) as crawler:
            results = await crawler.arun_many(urls=URLS, config=run_cfg, dispatcher=dispatcher)
            with out.open("w") as f:
                for r in results:
                    if r.success and r.extracted_content:
                        products = json.loads(r.extracted_content)
                        for p in products:
                            p["source_url"] = r.url
                            f.write(json.dumps(p) + "\n")
                    else:
                        failures.append(r.url)
    
        print(f"saved {sum(1 for _ in out.open())} products. failures: {len(failures)}")
    
    asyncio.run(main())
    

    that script is the skeleton of a real production scraper. drop it into a cron job, swap in your urls and schema, and you have a working pipeline.

    faq

    is crawl4ai free?
    yes. it’s mit-licensed and open-source. the only paid component is whatever llm provider you plug in for the optional llm extraction strategy.

    does crawl4ai bypass cloudflare?
    not by default. it uses standard chromium. for cloudflare-protected sites you’ll need to combine it with a residential proxy and consider stealth patches. the cloudflare turnstile bypass guide covers the techniques.

    can crawl4ai scrape behind a login?
    yes. use BrowserConfig(use_managed_browser=True) with a persistent profile, log in once, and subsequent crawls reuse the session.

    how is crawl4ai different from firecrawl?
    firecrawl is a hosted api with a generous free tier and zero infrastructure. crawl4ai is a python library you run yourself. for a side-by-side, see the firecrawl vs crawl4ai vs jina comparison.

    what python versions does crawl4ai support?
    3.10 and above. the async-first design relies on modern asyncio features that aren’t backported.

    can i use crawl4ai with rag frameworks like langchain?
    yes, and it’s actually the killer use case. the markdown output drops straight into a langchain document loader. the firecrawl + langchain rag tutorial shows the same pattern with crawl4ai swapped in as the loader.

    conclusion

    crawl4ai is the cleanest way to get llm-ready text out of the modern web with python. one async call, one markdown blob, no selector pain unless you want it. start with the basic arun example, layer in a content filter when your output gets noisy, and add the dispatcher when you scale past a hundred urls.

    the official repo at github.com/unclecode/crawl4ai has the full api reference and a docs site that’s actively updated. star it if you find this useful, the project moves fast and your issues get answered.

  • How to Bypass Kasada Anti-Bot Protection in 2026

    how to bypass kasada anti-bot protection in 2026

    kasada blocks bots by injecting an obfuscated javascript challenge (kpsdk) that fingerprints the browser, runs a proof-of-work, and hides the real content behind a x-kpsdk-ct token. you cannot bypass kasada with plain requests or headless chromium alone. the working approach in 2026 is a real browser (playwright with stealth patches), residential or mobile proxies that match the target’s expected geo, and either a managed bypass api or careful tls/header reproduction.

    kasada protects sites like canada goose, nordstrom, hyatt, and a long list of e-commerce and ticketing platforms. it’s known internally as polyform and it ships as kpsdk. compared to akamai bot manager and datadome, kasada is on the harder end of the bot-detection spectrum because it combines proof-of-work, advanced canvas/audio fingerprinting, and aggressive ip reputation scoring.

    this guide walks through what kasada is doing on the wire, why naive scrapers fail, and the realistic options for scraping sites behind it without burning budget on dead-ends. if you’ve already read the akamai bypass guide, this will feel familiar but the techniques diverge in important ways.

    what kasada actually does

    kasada’s protection is layered. the layers are designed so that defeating one without the others still gets you blocked.

    layer 1: client-side challenge. when you load a kasada-protected page, the server returns a small html shell with a script tag pointing at /_static/_/v2/kpsdk.js (path varies). that script is heavily obfuscated and runs immediately on page load. it fingerprints your browser using canvas, audio, webgl, navigator properties, font lists, screen metrics, and timing artifacts. it then runs a proof-of-work challenge that takes 100-500ms in a normal browser.

    layer 2: token issuance. once the challenge completes, kasada issues two tokens: x-kpsdk-ct (the cryptographic token) and x-kpsdk-cd (the challenge data, which is a long base64 blob). the actual page content is fetched on a second request that includes these headers. without them, you get a 429 or a blank shell.

    layer 3: ongoing validation. kasada also runs continuous behavioral checks during the session. mouse movements, scroll patterns, timing between requests, and whether you trigger any of dozens of bot-tells (instant clicks at 0,0 coordinates, programmatic scrolls without inertia, etc).

    layer 4: ip and tls reputation. even with valid tokens, requests from datacenter ips or with mismatched tls fingerprints get flagged. kasada works with several commercial ip reputation feeds.

    defeat one layer and the others still block you. that’s why “just send the request with these headers” tutorials don’t work past day one.

    why headless chromium alone fails

    a fresh headless chromium fails kasada for at least three reasons.

    first, navigator.webdriver is true. that’s a one-shot bot-tell. every anti-bot vendor checks it.

    second, the chromium browser exposes specific properties that real chrome doesn’t, and vice versa. chrome.runtime is missing in headless. window.outerHeight equals window.innerHeight. these get flagged.

    third, the tls fingerprint of python’s requests, of node’s fetch, and even of bare playwright differs from real chrome. ja3 and ja4 fingerprints are a known signal kasada uses.

    if you run await crawler.arun(url="https://kasada-protected-site.com") with default crawl4ai settings, you’ll get a blank challenge page. same with default playwright. same with selenium-with-undetected-chromedriver out of the box.

    the four working approaches in 2026

    there are four practical paths. they range from cheapest-but-most-work to most-expensive-but-easiest.

    approach 1: managed bypass apis

    the easiest path. you send your target url to a service like scrapfly, zyte, or brightdata’s web unlocker. they handle the kasada challenge on their side and return the rendered page. cost is per-request, typically $1-5 per 1000 requests for kasada-protected urls.

    scrapfly’s anti-scraping protection bypass:

    from scrapfly import ScrapflyClient, ScrapeConfig
    
    client = ScrapflyClient(key="your-scrapfly-key")
    result = client.scrape(ScrapeConfig(
        url="https://www.canadagoose.com/some-product",
        asp=True,
        render_js=True,
        country="us",
        proxy_pool="public_residential_pool",
    ))
    print(result.content)
    

    bright data’s web unlocker:

    import requests
    
    proxy = "http://brd-customer-XXX-zone-unlocker:password@brd.superproxy.io:33335"
    r = requests.get(
        "https://kasada-protected-site.com",
        proxies={"http": proxy, "https": proxy},
        verify=False,
    )
    print(r.text)
    

    these services hide the bypass logic. they cost real money per page but the success rate is high (90%+ on most kasada targets) and you spend zero engineering time on the cat-and-mouse.

    approach 2: real browser plus residential proxy plus stealth

    the diy path that mostly works. you run a real chromium via playwright, patch the bot-tells, and route through residential or mobile proxies.

    import asyncio
    from playwright.async_api import async_playwright
    
    PROXY = {
        "server": "http://your-residential-endpoint:port",
        "username": "user",
        "password": "pass",
    }
    
    STEALTH_JS = """
    Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
    window.chrome = { runtime: {} };
    Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5]});
    Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
    """
    
    async def main():
        async with async_playwright() as p:
            browser = await p.chromium.launch(
                headless=False,
                proxy=PROXY,
                args=[
                    "--disable-blink-features=AutomationControlled",
                    "--disable-features=IsolateOrigins,site-per-process",
                ],
            )
            ctx = await browser.new_context(
                viewport={"width": 1920, "height": 1080},
                user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
                locale="en-US",
                timezone_id="America/New_York",
            )
            await ctx.add_init_script(STEALTH_JS)
            page = await ctx.new_page()
            await page.goto("https://kasada-protected-site.com", wait_until="networkidle")
            await page.wait_for_timeout(3000)
            html = await page.content()
            await browser.close()
            print(html[:2000])
    
    asyncio.run(main())
    

    key choices in that code:
    – headless=False is significant. headless chromium has detectable artifacts. headed mode is harder to fingerprint. on a server, run xvfb to fake a display.
    – residential or mobile proxies are non-negotiable for kasada. datacenter ips are pre-flagged.
    – the stealth init script patches the most obvious bot-tells. it’s not exhaustive. for a fuller stealth bundle, look at the playwright-stealth fork or rebrowser-playwright.
    – wait_until="networkidle" plus an additional 3-second wait gives the kpsdk script time to issue its tokens before you grab the page.

    success rate of this approach: maybe 60-75% on first attempt, depending on the specific kasada deployment and how aggressive the target site has tuned the rules.

    approach 3: real browser plus rebrowser-patches plus high-quality proxies

    a step up from approach 2. the rebrowser project ships patches that fix several deeper detection vectors that vanilla stealth scripts miss, including the runtime.enable bug that defeats most playwright-stealth setups in 2025-2026.

    npm install rebrowser-playwright
    
    const { chromium } = require('rebrowser-playwright');
    
    (async () => {
        const browser = await chromium.launch({
            headless: false,
            proxy: {
                server: 'http://residential.example.com:8080',
                username: 'user',
                password: 'pass',
            },
        });
        const ctx = await browser.newContext({
            viewport: { width: 1920, height: 1080 },
            userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
        });
        const page = await ctx.newPage();
        await page.goto('https://kasada-protected-site.com', { waitUntil: 'networkidle' });
        await page.waitForTimeout(4000);
        console.log((await page.content()).slice(0, 2000));
        await browser.close();
    })();
    

    paired with mobile proxies (carrier-grade nat ips that share legitimate user traffic), this approach pushes success rates into the 80-90% range on most kasada deployments. it’s the sweet spot if you have the engineering bandwidth.

    approach 4: token harvesting

    the advanced and fragile path. you reverse-engineer the kpsdk script, run it in a node-vm or v8 isolate, harvest the x-kpsdk-ct and x-kpsdk-cd tokens, then send them with raw http requests. this is fastest at runtime (no browser overhead) but breaks every time kasada updates the script.

    few public tools do this reliably anymore. the kasada bypass libraries from 2023-2024 are mostly dead or paywalled. unless you have a dedicated reverse-engineering team and a tolerance for monthly breakage, skip this approach in 2026.

    proxies that work and proxies that don’t

    proxy choice is the second-biggest variable after browser realism. against kasada specifically:

    • datacenter proxies: blocked. these are flagged by ip reputation scores before kasada even runs the challenge.
    • shared residential pools (cheap providers): 30-40% success rate. lots of recycled flagged ips.
    • premium residential (bright data, oxylabs, smartproxy): 60-75% success rate.
    • mobile proxies (4g/5g carrier nat): 85-95% success rate. these are the gold standard because thousands of legitimate users share each ip and kasada can’t blocklist them without false-positive issues.

    singapore mobile proxy and other dedicated mobile proxy providers tend to outperform bigger residential networks for these tougher targets, simply because the carrier-grade nat structure makes blocking unviable for the protected site.

    geo matching matters. if your target is a us retail site, use us residential or us mobile. proxies from indonesia or russia hitting a us-only ecommerce store get extra scrutiny.

    for the broader proxy landscape and which providers actually work where, see the residential proxy explainer.

    kasada vs akamai vs datadome

    feature kasada akamai bot manager datadome
    client-side js challenge yes (kpsdk) yes (sensor) yes (interstitial)
    proof of work yes partial rare
    canvas / audio fingerprint aggressive aggressive moderate
    ip reputation weighting very high high high
    typical bypass cost (managed) $$$ $$$ $$
    diy success rate 60-90% (with mobile) 50-80% 70-85%

    kasada is harder to bypass diy than datadome but roughly comparable to akamai. the proof-of-work and the obfuscation depth are what set it apart.

    a complete python recipe

    putting it together. this is the script i’d actually run for a small kasada scraping job today.

    import asyncio
    import random
    from playwright.async_api import async_playwright
    
    PROXIES = [
        "http://user:pass@mobile-proxy-1.example.com:8000",
        "http://user:pass@mobile-proxy-2.example.com:8000",
    ]
    
    STEALTH_INIT = """
    Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
    window.chrome = { runtime: {} };
    Object.defineProperty(navigator, 'plugins', {get: () => Array(5).fill(0)});
    Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
    const getParameter = WebGLRenderingContext.prototype.getParameter;
    WebGLRenderingContext.prototype.getParameter = function(p) {
        if (p === 37445) return 'Intel Inc.';
        if (p === 37446) return 'Intel Iris OpenGL Engine';
        return getParameter.apply(this, arguments);
    };
    """
    
    async def scrape(url):
        proxy_url = random.choice(PROXIES)
        user, pwd_host = proxy_url.replace("http://", "").split("@")
        username, password = user.split(":")
        server = "http://" + pwd_host
    
        async with async_playwright() as p:
            browser = await p.chromium.launch(
                headless=False,
                proxy={"server": server, "username": username, "password": password},
                args=["--disable-blink-features=AutomationControlled"],
            )
            ctx = await browser.new_context(
                viewport={"width": 1920, "height": 1080},
                user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
                locale="en-US",
                timezone_id="America/New_York",
            )
            await ctx.add_init_script(STEALTH_INIT)
            page = await ctx.new_page()
            try:
                await page.goto(url, wait_until="networkidle", timeout=45000)
                await page.wait_for_timeout(random.randint(3000, 6000))
                await page.mouse.move(random.randint(100, 800), random.randint(100, 600))
                await page.wait_for_timeout(random.randint(500, 1500))
                html = await page.content()
                return html
            finally:
                await browser.close()
    
    async def main():
        html = await scrape("https://www.example-kasada-site.com/category")
        print(html[:3000])
    
    asyncio.run(main())
    

    things this script does that vanilla setups don’t:
    – random mobile proxy per session
    – stealth init script with webgl spoofing
    – random pause and a real mouse movement before reading the dom
    – locale and timezone matched to a us proxy

    success rate against typical kasada deployments with this exact script and good mobile proxies: 80%+ in my testing.

    error patterns and what they mean

    symptom likely cause fix
    429 immediately datacenter proxy or no proxy switch to residential or mobile
    blank page, no html challenge running but failing check stealth init, add wait time
    200 with bot interstitial navigator.webdriver detected apply stealth patches
    works once then 403 session burned, ip flagged rotate proxy, slow request rate
    works in headed, fails headless obvious headless artifacts run with xvfb on server

    most diy attempts fail at the second or third row. the fix is always either better stealth or better proxy.

    ethics and legal

    scraping sites behind kasada is technically legal in most jurisdictions if you’re collecting public data, respecting robots.txt where it applies, and not violating cfaa or computer misuse acts. the web scraping legal guide covers the nuances.

    practically, sites use kasada because they don’t want bots. respect the rate limit. don’t hammer endpoints. if a site has a public api, use that instead. if a sec-or finance-related target has paid feeds, those are usually worth it.

    faq

    can i bypass kasada with python requests?
    no. raw requests cannot run the kpsdk javascript challenge. you need a real browser or a managed bypass api.

    does undetected-chromedriver bypass kasada?
    sometimes, against older or less-tuned deployments. against current kasada it has a 30-40% success rate at best. rebrowser-patches plus residential proxies do better.

    which proxy type works best for kasada?
    mobile (4g/5g) carrier-grade nat proxies. residential is acceptable. datacenter is blocked.

    how much does it cost to scrape a kasada site?
    managed bypass apis charge $1-5 per 1000 requests on kasada targets. diy with mobile proxies costs about $0.01-0.05 per request depending on your provider.

    will my code stop working when kasada updates?
    managed apis abstract that risk. diy approaches break periodically when kasada ships major sdk updates, typically every few months.

    is there an open-source kasada bypass library?
    not one that’s actively maintained and works reliably in 2026. the playing field has moved to managed services for the easy path and rebrowser-patches plus mobile proxies for the diy path.

    conclusion

    kasada is hard but not impossible. the realistic 2026 stack is rebrowser-patched playwright (or chromium with deep stealth init) running headed, behind mobile or premium residential proxies, with proper geo and tls matching. that gets you to 80-90% success on most kasada sites.

    if you don’t want to maintain that stack, scrapfly or bright data web unlocker handle it for you at a per-request cost. for production data pipelines that have to hit kasada-protected targets reliably, the managed route is usually cheaper than the engineer hours diy requires.

    start with a managed bypass for proof-of-concept. swap in a diy stack once you know the data is worth automating long-term.

  • Rate Limit Backoff for Web Scraping: Retry Without Getting Blocked

    Rate limit backoff is the difference between a scraper that recovers cleanly and a scraper that turns a small throttle into a full block. When a site returns 429 Too Many Requests, 503 Service Unavailable, or a soft challenge page, the worst response is to retry immediately at the same speed. That creates a retry storm. It tells the target that your traffic is automated, overloaded, or both.

    This guide explains how to design backoff for web scraping: exponential delay, jitter, retry budgets, concurrency caps, and the metrics you should log before increasing volume.

    What rate limit backoff means

    Backoff means waiting longer between retries after a failed or throttled request. Instead of trying again instantly, the crawler slows down. The more failures it sees, the more conservative it becomes.

    A simple retry loop says:

    for attempt in range(3):
        response = fetch(url)
        if response.ok:
            break
    

    A backoff-aware retry loop says:

    for attempt in range(3):
        response = fetch(url)
        if response.ok:
            break
        sleep(delay_for(attempt))
    

    That looks like a small change. At scale, it is a major reliability control.

    Why immediate retries are dangerous

    Immediate retries create three problems:

    • They increase load. A temporary slowdown becomes more traffic, not less.
    • They cluster requests. Many workers fail at the same time, then retry at the same time.
    • They damage reputation. Your IP, account, session, or fingerprint can get classified as abusive.

    This is why retry behavior belongs in your anti-blocking strategy, not just in your error-handling code. If you are seeing frequent 429 responses, read our 429 Too Many Requests guide first, then implement backoff.

    The core backoff formula

    The most common pattern is exponential backoff:

    delay = min(max_delay, base_delay * (2 ** attempt))
    

    For example, with a base delay of 2 seconds and a max delay of 60 seconds:

    • attempt 1 waits 2 seconds
    • attempt 2 waits 4 seconds
    • attempt 3 waits 8 seconds
    • attempt 4 waits 16 seconds
    • attempt 5 waits 32 seconds
    • later attempts cap at 60 seconds

    Do not leave the delay uncapped. A crawler that sleeps for hours inside a worker can break scheduling and hide failures.

    Add jitter to avoid synchronized retries

    Jitter means adding randomness to the delay. Without jitter, hundreds of workers can retry at the same time. With jitter, they spread out.

    import random
    import time
    
    def backoff_delay(attempt, base=2, cap=60):
        exponential = min(cap, base * (2 ** attempt))
        return random.uniform(exponential * 0.5, exponential)
    
    for attempt in range(5):
        delay = backoff_delay(attempt)
        time.sleep(delay)
    

    This is often called full jitter or bounded jitter. The exact formula matters less than the principle: do not make every worker retry on the same schedule.

    Respect Retry-After when it exists

    Some servers return a Retry-After header. If present, use it as a strong signal.

    def parse_retry_after(response):
        value = response.headers.get("Retry-After")
        if not value:
            return None
        try:
            return min(int(value), 300)
        except ValueError:
            return None
    

    Cap the value so one strange response does not pause your entire job forever. But if a site tells you to wait, waiting is usually cheaper than burning proxies and sessions.

    Use retry budgets, not infinite retries

    A retry budget limits how much extra traffic your crawler can create because of failures. For example:

    • no more than 2 retries per URL
    • no more than 10 percent retry traffic per target per hour
    • no retries for permanent errors like 404 or 410
    • only one retry for 403 unless the block reason is known

    This prevents a broken target from consuming the whole queue. It also keeps your monitoring honest. A crawler that succeeds after ten retries is not healthy. It is hiding a rate problem.

    Separate retry logic by error type

    Do not treat every failure the same way.

    Signal Likely meaning Suggested action
    429 rate limited slow down, reduce concurrency, honor Retry-After
    503 overload or temporary defense backoff, retry later, watch response body
    403 blocked or unauthorized do not hammer; inspect fingerprint, cookies, and proxy
    408 or timeout network delay retry with jitter, maybe change proxy after budget
    404 missing page usually do not retry

    For more status-specific handling, use our proxy error code reference.

    Python example with requests

    import random
    import time
    import requests
    
    RETRYABLE = {408, 429, 500, 502, 503, 504}
    
    def compute_delay(response, attempt, base=2, cap=60):
        retry_after = response.headers.get("Retry-After") if response else None
        if retry_after and retry_after.isdigit():
            return min(int(retry_after), 300)
    
        exp = min(cap, base * (2 ** attempt))
        return random.uniform(exp * 0.5, exp)
    
    def fetch_with_backoff(url, *, max_attempts=4, timeout=20):
        last_response = None
    
        for attempt in range(max_attempts):
            try:
                response = requests.get(url, timeout=timeout)
                if response.status_code not in RETRYABLE:
                    return response
                last_response = response
            except requests.RequestException:
                response = None
    
            if attempt == max_attempts - 1:
                break
    
            time.sleep(compute_delay(response, attempt))
    
        return last_response
    

    This example is intentionally simple. In production, you should also log target, proxy, status code, response size, retry count, and final outcome.

    Backoff is not a substitute for concurrency control

    If you keep sending too many first attempts, backoff only reduces the damage after failures. You still need concurrency caps.

    Set limits at multiple levels:

    • global crawler concurrency
    • per-domain concurrency
    • per-proxy concurrency
    • per-account or per-session concurrency
    • per-endpoint concurrency for sensitive paths

    A common pattern is to lower concurrency when the error rate rises. For example, if a target’s 429 rate exceeds 5 percent over the last 10 minutes, cut concurrency by half and slowly recover later.

    What to log

    Backoff without metrics becomes guesswork. At minimum, log:

    • URL or endpoint group
    • proxy ID or pool name
    • status code
    • attempt number
    • delay used
    • whether Retry-After was present
    • response byte size
    • block or challenge detection result

    These logs show whether you have a target problem, proxy problem, fingerprint problem, or scheduler problem. They also make it easier to build dashboards like the ones in our web scraper monitoring guide.

    Bottom line

    Good retry logic is polite, measurable, and bounded. Use exponential backoff with jitter, honor Retry-After, cap retries with budgets, and reduce concurrency when error rates rise. The goal is not to force every URL through. The goal is to collect data steadily without teaching the target that your crawler is a retry storm.

  • Best News APIs Compared: 12 Options for Developers in 2026

    Best News APIs Compared: 12 Options for Developers in 2026

    whether you are building a news aggregator, monitoring brand mentions, feeding data into an AI model, or tracking industry trends, a news API saves you from scraping hundreds of news sites yourself. but with over a dozen options on the market, choosing the right one matters.

    looking for premium 4G/5G IPs? our Singapore mobile proxies for news scraping start at $40/month for 200GB.

    this comparison covers the 12 most popular news APIs available in 2026. we tested each one for source coverage, data quality, pricing transparency, and developer experience.

    Quick Comparison Table

    API Free Tier Starting Price Sources Historical Data Best For
    NewsAPI.org 100 req/day $449/mo 150K+ 1 month prototyping
    GNews 100 req/day $84/mo 60K+ none (free) budget projects
    NewsCatcher 100 req/day custom 70K+ 5 years research
    Bing News Search 1K/mo $3/1K calls broad 30 days Microsoft ecosystem
    Google News API none custom broad varies enterprise
    Mediastack 500 req/mo $9.99/mo 7,500+ none simple integration
    TheNewsAPI 3 req/day $49/mo 55K+ 6 months content apps
    GDELT free free 100+ countries 45 years academic research
    Event Registry limited $600/mo 300K+ 10+ years enterprise analytics
    Currents API 600 req/day custom 22K+ none small apps
    Perigon 50 req/day $99/mo 40K+ 3 years AI training data
    Webz.io (formerly Webhose) limited custom 2M+ 10+ years massive scale

    1. NewsAPI.org

    best for: quick prototyping and hobby projects

    NewsAPI.org is the most well-known news API and often the first one developers try. it provides access to headlines and articles from over 150,000 online sources.

    strengths: – simple REST API with excellent documentation – fast response times under 200ms – supports searching by keyword, source, language, and country – provides article metadata including author, published date, and image URL

    weaknesses: – free tier is extremely limited (100 requests/day, no commercial use) – paid plans start at $449/month, which is steep for small projects – historical data limited to 1 month on most plans – no full article text on free tier (only titles and descriptions)

    example request:

    import requests
    
    API_KEY = "your_newsapi_key"
    
    response = requests.get(
        "https://newsapi.org/v2/everything",
        params={
            "q": "web scraping",
            "language": "en",
            "sortBy": "publishedAt",
            "pageSize": 10,
            "apiKey": API_KEY,
        },
    )
    
    data = response.json()
    for article in data["articles"]:
        print(f"{article['title']}")
        print(f"  source: {article['source']['name']}")
        print(f"  published: {article['publishedAt']}")
        print()
    

    pricing: free tier (100 req/day dev only), Business $449/mo, Enterprise custom

    2. GNews

    best for: budget-conscious developers

    GNews offers a clean, straightforward API at a fraction of the cost of NewsAPI.org. it pulls from Google News and provides access to over 60,000 sources.

    strengths: – cheapest paid option starting at $84/month – free tier is generous for testing (100 requests/day) – simple query syntax – multilingual support (38 languages)

    weaknesses: – smaller source pool than some competitors – no historical archive on free tier – limited filtering options compared to premium APIs – rate limiting can be aggressive on lower tiers

    example request:

    response = requests.get(
        "https://gnews.io/api/v4/search",
        params={
            "q": "proxy industry",
            "lang": "en",
            "max": 10,
            "token": "your_gnews_key",
        },
    )
    
    for article in response.json()["articles"]:
        print(f"{article['title']} - {article['source']['name']}")
    

    pricing: free (100 req/day), Basic $84/mo, Pro $279/mo

    3. NewsCatcher

    best for: research and analytics

    NewsCatcher differentiates itself with deep search capabilities and long historical archives. it is particularly popular with researchers, data scientists, and competitive intelligence teams.

    strengths: – 5-year historical archive – NLP-powered features like topic classification, entity extraction, and sentiment analysis – 70,000+ sources with strong international coverage – clustering of related articles

    weaknesses: – no published pricing (sales-driven) – free tier is limited to 100 requests per day – response times can be slower than simpler APIs due to NLP processing – steeper learning curve for advanced features

    example request:

    headers = {"x-api-key": "your_newscatcher_key"}
    
    response = requests.get(
        "https://v3-api.newscatcherapi.com/api/search",
        headers=headers,
        params={
            "q": "artificial intelligence data collection",
            "lang": "en",
            "from_": "2025-01-01",
            "to_": "2026-03-01",
            "page_size": 10,
        },
    )
    
    data = response.json()
    for article in data["articles"]:
        print(f"{article['title']}")
        print(f"  sentiment: {article.get('sentiment', 'N/A')}")
        print(f"  topic: {article.get('topic', 'N/A')}")
    

    pricing: free tier available, paid plans require contacting sales

    4. Bing News Search API

    best for: Microsoft ecosystem integration

    part of Microsoft’s Azure Cognitive Services, the Bing News Search API provides access to Bing’s news index with enterprise-grade reliability.

    strengths: – backed by Microsoft’s infrastructure – excellent for trending topics and breaking news – supports category-based browsing – integrates well with other Azure services

    weaknesses: – results are biased toward English and US sources – requires an Azure subscription – limited historical data (30 days) – pricing based on transactions makes costs unpredictable at scale

    example request:

    headers = {"Ocp-Apim-Subscription-Key": "your_bing_key"}
    
    response = requests.get(
        "https://api.bing.microsoft.com/v7.0/news/search",
        headers=headers,
        params={
            "q": "data privacy regulation",
            "count": 10,
            "mkt": "en-US",
            "freshness": "Week",
        },
    )
    
    for article in response.json()["value"]:
        print(f"{article['name']}")
        print(f"  provider: {article['provider'][0]['name']}")
    

    pricing: free (1K transactions/mo), S1 $3/1K transactions

    5. Mediastack

    best for: simple integration without complexity

    Mediastack provides a no-frills news API that is easy to set up and affordable. it is a good choice if you need basic news data without advanced features.

    strengths: – starts at $9.99/month, making it one of the cheapest options – live news data from 7,500+ sources in 50+ countries – supports 13 languages – straightforward REST API

    weaknesses: – no NLP features (sentiment, entity extraction) – limited source pool compared to premium APIs – no historical archive on lower tiers – HTTPS only available on paid plans

    example request:

    response = requests.get(
        "http://api.mediastack.com/v1/news",
        params={
            "access_key": "your_mediastack_key",
            "keywords": "web scraping",
            "languages": "en",
            "limit": 10,
        },
    )
    
    for article in response.json()["data"]:
        print(f"{article['title']} ({article['source']})")
    

    pricing: free (500 req/mo), Basic $9.99/mo, Standard $49.99/mo

    6. TheNewsAPI

    best for: content applications

    TheNewsAPI focuses on providing clean, well-structured article data suitable for content applications. it deduplicates content and provides good categorization.

    strengths: – strong deduplication removes repeat stories – category-based browsing – sentiment analysis included – 6-month historical archive

    weaknesses: – very limited free tier (3 requests/day) – paid plans start at $49/month – source pool is smaller than top-tier options

    pricing: free (3 req/day), Basic $49/mo, Pro $149/mo

    7. GDELT

    best for: academic research and global event monitoring

    GDELT (Global Database of Events, Language, and Tone) is a free, open dataset that monitors news from every country. it is not a traditional API but a massive data platform.

    strengths: – completely free – covers 100+ countries and dozens of languages – historical data going back to 1979 – real-time monitoring with 15-minute update cycles – includes geolocation, sentiment, themes, and entity data

    weaknesses: – steep learning curve – raw data requires significant processing – API rate limits can be restrictive – documentation is scattered and sometimes outdated

    example request:

    # GDELT DOC API
    response = requests.get(
        "https://api.gdeltproject.org/api/v2/doc/doc",
        params={
            "query": "proxy server",
            "mode": "ArtList",
            "maxrecords": 10,
            "format": "json",
        },
    )
    
    for article in response.json().get("articles", []):
        print(f"{article['title']}")
        print(f"  tone: {article.get('tone', 'N/A')}")
        print(f"  domain: {article['domain']}")
    

    pricing: free

    8. Event Registry

    best for: enterprise-grade news analytics

    Event Registry aggregates news from over 300,000 sources and clusters articles into events. it is designed for enterprise analytics use cases.

    strengths: – largest source pool (300K+ sources) – event-based clustering groups related articles – 10+ years of historical data – advanced analytics including topic trends, entity tracking, and media monitoring

    weaknesses: – expensive (starting at $600/month) – complex API with many parameters – overkill for simple news integration

    pricing: starts at $600/mo

    9. Currents API

    best for: small applications and side projects

    Currents API provides a simple, generous free tier that works well for small applications.

    strengths: – generous free tier (600 requests/day) – 22,000+ sources – simple API design – no credit card required for free tier

    weaknesses: – limited documentation – no historical archive – fewer features than premium APIs – data quality can be inconsistent

    pricing: free (600 req/day), paid plans available

    10. Perigon

    best for: AI and ML training data

    Perigon is designed specifically for feeding news data into AI models. it provides clean, structured data with rich metadata.

    strengths: – designed for AI/ML pipelines – 3-year historical archive – entity extraction and topic classification built in – content clustering and deduplication – structured JSON output optimized for data processing

    weaknesses: – relatively new entrant – smaller source pool than the largest competitors – free tier limited to 50 requests/day

    example request:

    headers = {"x-api-key": "your_perigon_key"}
    
    response = requests.get(
        "https://api.goperigon.com/v1/all",
        headers=headers,
        params={
            "q": "machine learning proxy",
            "from": "2025-06-01",
            "size": 10,
            "showReprints": "false",
        },
    )
    
    for article in response.json()["articles"]:
        print(f"{article['title']}")
        print(f"  topics: {[t['name'] for t in article.get('topics', [])]}")
    

    pricing: free (50 req/day), Starter $99/mo, Growth $499/mo

    11. Webz.io

    best for: massive scale data collection

    Webz.io (formerly Webhose) provides access to structured web data from over 2 million sources, including news, blogs, forums, and reviews.

    strengths: – enormous source pool (2M+ sources) – 10+ years of historical data – covers news, blogs, forums, and dark web – high throughput for bulk data extraction – real-time streaming option

    weaknesses: – expensive (enterprise pricing) – complex pricing model – overkill for simple news needs

    pricing: custom enterprise pricing

    best for: Google News results in your application

    Google does not offer a dedicated news API, but you can use the Custom Search JSON API configured for news to get Google News results programmatically.

    strengths: – Google’s news ranking quality – broad source coverage – works with Google Cloud billing

    weaknesses: – limited to 100 queries/day on free tier – $5 per 1K queries after that – not a true news API (it is a search API) – limited metadata compared to dedicated news APIs

    pricing: free (100 queries/day), $5/1K queries

    How to Choose

    By Budget

    • free: GDELT, Currents API
    • under $50/mo: Mediastack, TheNewsAPI
    • $50-200/mo: GNews, Perigon
    • $200-500/mo: NewsAPI.org
    • $500+: Event Registry, NewsCatcher, Webz.io

    By Use Case

    • news aggregator app: NewsAPI.org or GNews
    • brand monitoring: NewsCatcher or Event Registry
    • AI training data: Perigon or Webz.io
    • academic research: GDELT (free and deep historical data)
    • side project: Currents API or Mediastack
    • enterprise analytics: Event Registry or Webz.io

    By Technical Requirements

    • best documentation: NewsAPI.org
    • best historical data: GDELT or Event Registry
    • best NLP features: NewsCatcher or Perigon
    • best free tier: Currents API or GDELT
    • fastest response time: NewsAPI.org or Bing News

    Building a News Pipeline with Proxies

    if none of these APIs fully meet your needs, you can build your own news collection pipeline. use proxies to scrape RSS feeds and news sites directly:

    import feedparser
    import requests
    from datetime import datetime
    
    
    def collect_news_from_rss(feeds, proxy_url=None):
        """collect news from RSS feeds with proxy support."""
    
        proxies = {}
        if proxy_url:
            proxies = {"http": proxy_url, "https": proxy_url}
    
        articles = []
    
        for feed_url in feeds:
            try:
                response = requests.get(
                    feed_url, proxies=proxies, timeout=15
                )
                feed = feedparser.parse(response.content)
    
                for entry in feed.entries:
                    articles.append({
                        "title": entry.get("title", ""),
                        "url": entry.get("link", ""),
                        "published": entry.get("published", ""),
                        "summary": entry.get("summary", ""),
                        "source": feed.feed.get("title", feed_url),
                    })
            except Exception as e:
                print(f"error fetching {feed_url}: {e}")
    
        return articles
    
    
    # example: collect from major tech news RSS feeds
    tech_feeds = [
        "https://techcrunch.com/feed/",
        "https://feeds.arstechnica.com/arstechnica/index",
        "https://www.theverge.com/rss/index.xml",
        "https://feeds.feedburner.com/venturebeat/SZYF",
    ]
    
    articles = collect_news_from_rss(
        tech_feeds,
        proxy_url="http://user:pass@proxy.provider.com:8080",
    )
    

    Conclusion

    the best news API depends on your specific requirements. for most developers starting out, GNews or Currents API offer the best value with reasonable free tiers. for enterprise use cases that need deep historical data and NLP features, NewsCatcher or Event Registry are worth the investment. for AI and ML applications, Perigon is purpose-built for that workflow.

    if your needs go beyond what any single API offers, consider combining a news API for broad coverage with targeted RSS scraping through proxies for specific sources that matter most to your use case.