Your cart is currently empty!
Author: Xavier Fok
-
How to Scrape Google Local Pack Results (Maps + Business Data) 2026
how to scrape google local pack results (maps + business data) 2026
google local pack is the 3-result map block that appears on serps for local-intent queries like “coffee shop near me” or “lawyer in austin.” you can scrape it three ways in 2026: paid serp apis (serpapi, dataforseo, brightdata) at around $1.50-3 per 1000 queries, your own python scraper using residential proxies and playwright at near-zero per-query cost but higher engineering effort, or by scraping google maps directly which gives you 20+ results instead of just the local 3-pack. this tutorial covers all three with working code.
local pack data is gold for lead generation, competitor research, local seo audits, and ai apps that need verified business data. the scraping is harder than regular serp scraping because google heavily fingerprints map-related queries and the local pack html structure changes regularly. but it’s solvable, and the result is structured data on millions of businesses that’s otherwise locked behind google’s gates.
this guide walks through the three approaches, with code, with cost estimates, and with the gotchas that come up at scale.
what’s in the local pack
a typical local pack result on a query like “plumber miami” returns:
- 3 business listings (top 3 by google’s local ranking)
- each listing has: business name, rating (1-5 stars), review count, category, address, hours snippet, phone (sometimes), website (sometimes), gbid (google business id), latitude/longitude
- a “view all” link that opens the local finder (top 20 results)
- ad placements above and below sometimes
the underlying data lives in google’s local index, accessible via the regular serp html, the maps web ui, and the maps mobile app. each surface returns slightly different fields. for full coverage you usually scrape the maps surface, not just the serp local pack.
approach 1: paid serp apis (easiest)
three providers dominate this space in 2026:
- serpapi: $50/month for 5000 searches. local pack data included with
engine=google_local. - dataforseo: $0.0006-0.001 per organic search depending on plan. dedicated local pack endpoint.
- brightdata serp api: $1.50 per 1000 searches. covers all serp features including local pack.
for any serious volume the per-query rates push under $1.50/1k. for prototypes and small jobs they are by far the easiest path.
import requests SERPAPI_KEY = "your-key" def get_local_pack(query, location): r = requests.get("https://serpapi.com/search", params={ "engine": "google_local", "q": query, "location": location, "api_key": SERPAPI_KEY, "hl": "en", }).json() return r.get("local_results", []) results = get_local_pack("plumber", "miami, florida") for r in results[:5]: print(r["title"], r.get("rating"), r.get("phone"), r.get("address"))dataforseo’s local pack endpoint:
import requests from requests.auth import HTTPBasicAuth post_data = [{ "keyword": "plumber", "location_name": "Miami,Florida,United States", "language_code": "en", "device": "desktop", }] r = requests.post( "https://api.dataforseo.com/v3/serp/google/maps/live/advanced", json=post_data, auth=HTTPBasicAuth("your-login", "your-password"), ).json() items = r["tasks"][0]["result"][0]["items"] for item in items[:10]: print(item["title"], item.get("rating", {}).get("value"))dataforseo’s pricing is the most aggressive at scale (under $0.001 per query at volume). serpapi has the friendliest sdk and free tier. bright data is the most reliable at very high volume.
if you only need this data once or occasionally, paid serp apis are almost always the right answer. you spend $5-50, you get the data, you move on.
approach 2: python scraper with residential proxies
cheaper at scale, more engineering work upfront. you load the google maps search url, parse the rendered results, and store the structured fields.
import asyncio import json import re from playwright.async_api import async_playwright PROXY = { "server": "http://residential.example.com:8080", "username": "user", "password": "pass", } async def scrape_maps(query, location): url = f"https://www.google.com/maps/search/{query.replace(' ', '+')}+{location.replace(' ', '+')}" async with async_playwright() as p: browser = await p.chromium.launch(headless=False, proxy=PROXY) ctx = await browser.new_context( viewport={"width": 1366, "height": 768}, user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", locale="en-US", ) page = await ctx.new_page() await page.goto(url, wait_until="networkidle", timeout=30000) await page.wait_for_timeout(3000) # scroll the results panel to load more results_panel = page.locator("div[role='feed']") for _ in range(3): await results_panel.evaluate("el => el.scrollBy(0, 800)") await page.wait_for_timeout(1500) # extract listing elements items = await page.locator("div[role='feed'] > div > div[jsaction]").all() results = [] for item in items[:20]: try: name = await item.locator("div.fontHeadlineSmall").first.text_content() rating_el = await item.locator("span[role='img'][aria-label*='star']").first.get_attribute("aria-label") results.append({ "name": name.strip() if name else None, "rating_aria": rating_el, }) except Exception: continue await browser.close() return results async def main(): results = await scrape_maps("plumber", "miami florida") print(json.dumps(results, indent=2)) asyncio.run(main())this is the rough shape. real production code has more error handling, more selectors, and probably uses google maps’ internal
pb=urls to fetch json directly instead of parsing the dom. the dom approach above breaks every time google changes class names, which happens every few months.a more robust pattern is to capture the maps json endpoint via network interception:
async def capture_maps_json(query, location): captured = [] async with async_playwright() as p: browser = await p.chromium.launch(headless=False) ctx = await browser.new_context() page = await ctx.new_page() async def handle_response(response): if "search?" in response.url and "/maps/" in response.url: try: body = await response.text() if body.startswith(")]}'"): body = body[5:] captured.append(json.loads(body)) except Exception: pass page.on("response", handle_response) await page.goto(f"https://www.google.com/maps/search/{query}+{location}", wait_until="networkidle") await page.wait_for_timeout(5000) await browser.close() return capturedthe captured json contains the full structured data google sees on its end. parsing it requires reverse-engineering the field positions (it’s an array-of-arrays format) but once you have a parser, it’s faster and more reliable than dom scraping.
for the proxy choice, residential is the minimum. mobile proxies pass through the toughest google blocks more reliably. datacenter ips are blocked within a few queries. see the residential proxy guide for context.
approach 3: scrape regular google serp html
if you only need the 3-pack (not the full 20-result local finder), you can scrape the regular google serp page. the local pack appears as a structured div block alongside organic results.
import requests from bs4 import BeautifulSoup PROXY = {"http": "http://user:pass@residential.example.com:8080", "https": "http://user:pass@residential.example.com:8080"} def scrape_serp_local(query, geo_param): url = f"https://www.google.com/search?q={query}&uule={geo_param}&hl=en" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", "Accept": "text/html,application/xhtml+xml", "Accept-Language": "en-US,en;q=0.9", } r = requests.get(url, headers=headers, proxies=PROXY, timeout=30) soup = BeautifulSoup(r.text, "html.parser") # local pack container varies, but rllt__details is reliable for 3-pack listings listings = [] for div in soup.select("div.rllt__details"): name = div.select_one("div.dbg0pd") if name: listings.append({ "name": name.get_text(strip=True), "snippet": " ".join(s.get_text() for s in div.select("div") if s != name), }) return listingsthe
uuleparameter is google’s encoded location. you generate it from a place name using the uule encoding scheme or libraries likeserpwow-uule. without uule, results are based on your proxy’s geolocation, which is often wrong for niche local queries.for the broader google url-parameters reference, see the google search url parameters 2026 guide.
extracting individual business details
the local pack listings give you basic data. for full business details (hours, phone, website, full address, photos, reviews) you click into the business card and scrape the side panel.
async def scrape_business_details(page, business_url): await page.goto(business_url, wait_until="networkidle") await page.wait_for_timeout(2000) name = await page.locator("h1").first.text_content() address = await page.locator("button[data-item-id='address']").first.text_content() phone_el = page.locator("button[data-item-id^='phone']").first phone = await phone_el.text_content() if await phone_el.count() else None website_el = page.locator("a[data-item-id='authority']").first website = await website_el.get_attribute("href") if await website_el.count() else None rating = await page.locator("div.F7nice span[aria-hidden='true']").first.text_content() return { "name": name.strip() if name else None, "address": address.strip() if address else None, "phone": phone.strip() if phone else None, "website": website, "rating": rating, }the gbid (google business id, also called cid) is in the page url after navigation. extract from
page.urlwith a regex on theplace/.../@.../data=segment.handling pagination and load more
google maps doesn’t paginate the local finder in the traditional sense. it loads more results as you scroll the left-side panel. the playwright code above scrolls 3 times. for full coverage, scroll until you hit the “you’ve reached the end of the list” marker.
async def scroll_to_end(page): last_count = 0 same_count_iterations = 0 while same_count_iterations < 3: await page.locator("div[role='feed']").evaluate("el => el.scrollBy(0, 1000)") await page.wait_for_timeout(1500) items = await page.locator("div[role='feed'] > div > div[jsaction]").count() if items == last_count: same_count_iterations += 1 else: same_count_iterations = 0 last_count = items return last_countmost categories cap at 120 results in google maps. some niche or local queries cap at 20-40. that’s a hard ceiling.
rate limiting and avoiding blocks
google’s anti-scraping is aggressive on maps. patterns that get you blocked fast:
- many queries from the same ip in quick succession
- queries with no realistic delay between them
- consistent user-agent across all requests
- queries from datacenter ips
- non-residential geolocation mismatch (querying us businesses from a singapore ip)
mitigations:
- residential or mobile proxies, rotated per query
- 5-10 second delay between queries minimum
- random user-agent across a pool of 10-20 valid ones
- match your proxy geo to your query geo where possible
- spread queries across hours, not in a 1-minute burst
with those mitigations in place, a single residential proxy can do 100-500 queries a day before getting flagged. with mobile proxies, several thousand. for higher volume, parallelize across many proxies.
cost comparison
estimating cost for 100,000 local pack queries.
approach cost engineering effort serpapi $1000 (4x growth plan) minimal dataforseo $60-150 moderate (sdk integration) brightdata serp api $150 minimal diy with residential proxies $50-100 (proxy bandwidth) high (build + maintain) diy with mobile proxies $300-500 (mobile bandwidth) high (build + maintain) dataforseo wins on raw cost at scale. diy with residential proxies wins for very high volumes (over 500k queries) where you can amortize the engineering cost. for under 100k queries, dataforseo is hard to beat.
faq
is scraping google maps legal?
public data scraping is legal in most jurisdictions but violates google’s terms of service. there’s no consumer-protection law that triggers from scraping public business listings. for commercial use cases at scale, talk to a lawyer about cfaa exposure. the web scraping legal guide covers the case law.can i use the official google places api instead?
yes, and you should if your use case fits. places api charges $17-32 per 1000 requests and is rate-limited. for small volumes it’s competitive with serp apis. for high volume scraping is far cheaper.how many results does google local pack actually return?
the visible 3-pack is just the top 3. the local finder (clicking “view all”) shows up to 120. google maps direct search shows up to 120-200 depending on query density.does serpapi return the gbid?
yes, in theplace_idfield. some legacy responses usegbiddirectly. dataforseo also returns it. roll-your-own scraping requires extracting it from the place url.which proxy type works best for google maps?
residential or mobile. datacenter ips get blocked within a few queries. mobile is more reliable for high-volume sustained scraping. for context see the residential proxy guide.how do i scrape google reviews for a business?
once you have the business url or place id, you can scrape the reviews tab in the same way. each review is a single json item in the maps response. expect 200-500 reviews per page load with infinite scroll.conclusion
google local pack scraping is a solved problem in 2026 if you’re willing to spend on a paid serp api. dataforseo at $0.001 per query is the price-to-value sweet spot. serpapi is the easiest first integration. brightdata is the most reliable at very high volume.
if you specifically need fields the apis don’t expose, or if you’re scraping millions of queries a month, building your own with residential or mobile proxies and playwright is viable but requires real engineering investment. the dom selectors break every few months. the network-interception approach is more robust but harder to write the first time.
start with a paid serp api. measure your data needs against what they return. only build your own scraper when the api gaps or the cost crosses a clear threshold for your use case.
-
Oxylabs vs Bright Data 2026: Which Proxy Network Is Worth the Price?
oxylabs vs bright data 2026: which proxy network is worth the price?
bright data has the larger pool (150m+ residential ips vs oxylabs’ 100m+) and broader geo coverage (195 countries vs 188), but oxylabs is consistently 10-15% cheaper at every tier. on anti-bot success rates the two are roughly tied. choose bright data if you need the largest pool, the most enterprise polish, or china/middle-east specific geos. choose oxylabs if you’re cost-sensitive, you want the slightly cleaner dashboard, or you specifically need next-gen residential (oxylabs’ premium product) for the toughest targets.
these are the two heavyweights of the proxy industry in 2026. both are based in the eu (oxylabs in lithuania, bright data in israel/ny), both serve enterprise customers, both rank tier-1 on every meaningful axis. they compete head-to-head on big rfps and the choice between them is rarely about quality. it’s about pricing, specific product fit, and which dashboard you find more usable.
this comparison breaks down where each one wins, with real 2026 prices and real anti-bot test data.
the headline numbers
metric bright data oxylabs residential pool 150m+ ips 100m+ ips mobile pool 7m+ ips 5m+ ips isp pool 700k+ ips 500k+ ips countries covered 195 188 residential entry price $5.04/gb $4.00/gb mobile entry price $9.00/gb $8.00/gb isp price $1.50/ip/month $1.30/ip/month free trial 7 days, $5 credit 7 days, $5 credit oxylabs is cheaper across the board. bright data has a bigger pool. those are the structural differences. everything else is nuance.
residential pricing head-to-head
commit level bright data $/gb oxylabs $/gb savings with oxylabs pay-as-you-go $5.04 $4.00 21% starter (~$500/mo) $3.50 $3.00 14% growth (~$1000/mo) $3.20 $2.70 16% business (~$2000/mo) $2.60 $2.30 12% enterprise ($5000+) $2.00-2.40 $1.80-2.00 10-12% oxylabs is roughly 10-20% cheaper at every tier. for a 1tb-per-month scraping workload, that’s $300-500/month in savings at the same product tier.
oxylabs also offers a “next-gen residential” product that’s a more aggressive pool curated for tough targets. it’s priced at a premium ($6-8/gb depending on volume) and is one of the only public products that competes head-to-head with bright data’s web unlocker on hard sites.
both companies’ bright data pricing and oxylabs pricing pages list the current rates. the dashboards both let you toggle between geo-targeted vs random ip, sticky vs rotating sessions, and country/city/asn targeting at no extra cost (with the exception of premium asn targeting on bright data).
for a wider pricing context including the smaller providers, see the proxy pricing comparison.
isp proxies
feature bright data oxylabs ips per-month $1.50 (1 ip) to $0.80 (1000+ ips) $1.30 to $0.70 countries 25 22 bandwidth unlimited unlimited sticky session unlimited duration unlimited duration oxylabs wins narrowly on price per ip. bright data has a slightly larger pool with marginally better us coverage. for most use cases the difference is rounding error, the choice comes down to dashboard preference.
mobile proxies
feature bright data oxylabs pool size 7m+ 5m+ pricing per gb (starter) $9.00 $8.00 pricing per gb (enterprise) $4-5 $4-5 countries with mobile 100+ 80+ asn targeting yes yes carrier targeting yes (verizon, vodafone, etc) yes both have strong mobile networks. bright data has wider geo coverage on mobile. oxylabs is cheaper at the entry tier. at enterprise volume the prices converge.
for the truly hardest targets (instagram, tiktok scaling, kasada-protected sites, datadome-heavy travel), specialist mobile providers like dedicated singapore-based or sea-based mobile networks often beat both bright data and oxylabs on pass rate and price simultaneously. tier-1 networks are great but they aren’t the only option.
datacenter proxies
pricing bright data oxylabs shared dc per gb $0.50 (starter) $0.50 (starter) dedicated dc per ip-day $0.066 $0.06 volume rate per gb $0.25-0.30 $0.25-0.30 near identical pricing and quality. bright data’s pool is slightly larger. oxylabs’ dashboard is slightly cleaner for managing dedicated ips. negligible difference for most users.
anti-bot success rates
i ran 2,000 scrapes per provider against 5 protected target categories in early 2026. residential rotating proxies, similar geo distribution, default settings.
target type bright data success oxylabs success cloudflare-protected ecommerce 78% 76% akamai-protected ticketing 64% 67% datadome travel sites 72% 75% kasada-protected retail 58% 56% linkedin-style detection 81% 79% within margin of error. neither provider clearly wins on raw anti-bot pass rate. oxylabs edges ahead on akamai and datadome by a few points. bright data edges ahead on cloudflare and linkedin-pattern detection.
both have purpose-built premium products (bright data web unlocker, oxylabs web unlocker) that push pass rates into the 90%+ range on hard targets, at higher cost. those are the right answer for anything kasada or akamai-grade.
scraping apis and unlocker products
both companies sell a managed scraping product on top of their proxy networks.
product bright data oxylabs unlocker price $5 per 1k successful requests $5 per 1k successful requests serp api $1.50 per 1k searches $1.50 per 1k searches scraper api (per category) $1.50-3 per 1k $1.50-3 per 1k pre-collected datasets yes (large catalog) yes (medium catalog) near-identical pricing. bright data’s catalog is larger. oxylabs’ api responses are slightly cleaner json. the products converge so closely that some teams just pick by which sales rep replied first.
dashboard and api experience
both dashboards are mature. both expose detailed analytics on traffic, success rates, geo distribution. both give you the ability to spin up new zones, configure sticky sessions, and rotate credentials.
bright data’s dashboard has more features (zone variants, ip rotation rules, rule-based routing) but feels denser. oxylabs’ dashboard is cleaner but slightly less powerful at configuration.
api credentials, zone management, and billing are all comparable. neither has a clear advantage if you’re integrating programmatically.
both publish python, node, php, java sdks. both have working chrome extensions. both have firefox addons.
support and account management
factor bright data oxylabs 24/7 chat (paid plans) yes yes dedicated account manager $499+ tier $400+ tier typical support response <30 min <30 min compliance / kyc rigor very high high free integration help yes (paid plans) yes (paid plans) bright data’s compliance process is more thorough. expect more questions about your use case during onboarding, especially if you’re in a sensitive industry. oxylabs is slightly faster to onboard but still requires real kyc.
both companies have technical sales teams who will run a poc with you. for any spend over $2k/month, both will negotiate custom rates.
what each does best
bright data wins on:
– pool size and geo breadth (195 countries including china, russia, niche middle-east)
– enterprise compliance posture
– pre-collected dataset marketplace (largest in the industry)
– web unlocker reliability on cloudflare-protected targets
– raw infrastructure scaleoxylabs wins on:
– pricing across the board (10-15% cheaper)
– next-gen residential as a premium tier
– slightly cleaner dashboard
– akamai and datadome pass rates by small margin
– account manager responsiveness on mid-market accountsboth are world-class. neither will surprise you with anything bad.
decision framework
answer these in order:
- does your use case require china, russia, or niche geo coverage that oxylabs doesn’t list? if yes, bright data.
- is your monthly spend below $1500? then the price difference matters more, lean oxylabs.
- do you need pre-collected datasets (linkedin people, amazon products, indeed jobs)? bright data has the larger catalog.
- are you scraping hard anti-bot targets (kasada, datadome, akamai)? both work, but use the unlocker products at scale, not raw proxy.
- do you have an existing relationship with one of them? keep it. switching costs (testing, integration, billing setup) usually outweigh 10% price savings unless you’re at $5k+/month.
practical migration cost
if you’re already on one and considering switching, the real cost is engineer time, not the price gap.
- credentials and zones: 2-4 hours to set up new account, configure zones equivalent to old setup
- code changes: 0-4 hours depending on how abstracted your proxy layer is. switching the endpoint and credentials is usually a one-line change. testing across your full target list takes longer.
- monitoring: budget time to watch error rates closely for the first week. small differences in ip pool composition can hit specific targets harder than expected.
at $1000/month spend, the 12% savings ($120/month) doesn’t pay for the migration in under a year. at $10000/month, the savings ($1200/month) pay for migration in 1-2 days. that’s the threshold most teams I see use.
for a fuller breakdown of bright data’s full plan structure see the bright data pricing 2026 deep dive.
faq
is oxylabs cheaper than bright data?
yes, consistently across all product tiers. about 10-20% cheaper depending on commit level.which has the bigger proxy pool, oxylabs or bright data?
bright data, by a meaningful margin. 150m+ residential vs 100m+. for most use cases this doesn’t matter, but for high-rotation scraping in less-common geos it does.does oxylabs have china proxies?
limited. bright data has stronger china coverage among the tier-1 providers, though specialist providers in asia often beat both for china specifically.which is better for linkedin scraping?
both work. bright data edged oxylabs by ~2 points on linkedin-pattern detection in my testing. neither is dramatically better.can i use both together?
yes. some teams do, with bright data for cloudflare-heavy targets and oxylabs for the rest, or vice versa. each one bills separately. integration overhead is minor.what’s the cheapest way to use bright data or oxylabs?
commit to a starter or growth plan if you have predictable volume. the per-gb rates drop substantially. pay-as-you-go is for testing only, the rates are 30-40% higher than committed tiers.which has better support?
roughly equivalent. both have 24/7 chat, dedicated account managers at mid-tier spend, and competent technical sales. neither has been significantly worse than the other in my experience over the last two years.conclusion
bright data and oxylabs are both excellent. they compete so closely that the right choice depends on your specific situation more than on a clear quality difference. oxylabs is the cheaper option that handles 90% of use cases as well as bright data. bright data is the bigger-network option that wins on edge geos, larger datasets, and enterprise compliance polish.
for most new buyers in 2026, start with oxylabs because the savings compound. switch to bright data only if you hit a specific limitation (geo coverage, dataset gaps, china needs) or if a procurement team requires the largest possible vendor.
for the broader competitive context including smaller providers that may be the better answer for low-volume use cases, see the best proxy providers 2026 overview.
-
How to Bypass Cloudflare Turnstile for Web Scraping (2026)
how to bypass cloudflare turnstile for web scraping (2026)
cloudflare turnstile is a captcha-replacement widget that issues a
cf-turnstile-responsetoken after fingerprinting the browser. you cannot solve it server-side with raw http. the working 2026 approach is either a captcha-solver api like capsolver or 2captcha that returns a token for a few cents, or a real browser (playwright with stealth + residential proxy) that solves it in-page automatically. for high-volume scraping, the solver api route is faster and cheaper than running headed browsers at scale.turnstile shipped in late 2022 as cloudflare’s free, privacy-preserving alternative to recaptcha. in 2026 it’s everywhere. signup forms, checkout flows, comment widgets, and increasingly as the gate in front of cloudflare’s bot management. the widget looks innocuous (a small checkbox or invisible challenge) but the engineering behind it is serious.
this guide explains what turnstile does, why your scraper is failing, and the three approaches that actually return valid tokens at scale in 2026.
what cloudflare turnstile does
turnstile sits as a
<div class="cf-turnstile" data-sitekey="...">on the target page. when the page loads, it pulls in the turnstile script fromchallenges.cloudflare.com/turnstile/v0/api.js. that script:- checks for “private access tokens” issued by apple, google, etc. if the user has one (typical of real macos/ios safari), the challenge passes silently with no interaction.
- runs an invisible browser fingerprint check (canvas, webgl, audio, navigator properties, behavior over time).
- if those fail, presents a managed challenge. visible widget, sometimes with a checkbox or interactive proof-of-work.
- issues a token (
cf-turnstile-response) that the form submits to the origin server. the server validates the token against cloudflare’ssiteverifyendpoint.
without a valid token, the form post returns 403 or the api endpoint returns a turnstile challenge page.
three turnstile modes you’ll encounter:
– managed: cloudflare decides whether to challenge. usually invisible.
– non-interactive: always invisible, lower friction.
– invisible: completely hidden from the user. challenge runs in the background.managed is the most common and the hardest, because behavior can vary on every page load.
why your scraper fails
if you’re seeing one of these symptoms, it’s turnstile:
– form submission returns a 403 withcf-mitigated: challengeheader
– json api returns html that containsJust a moment...orcf-turnstile
– requests with valid cookies still hit a challenge page on next call
– your selenium script that worked last week is now failing on the same formthree reasons standard scrapers fail.
first, the
cf-turnstile-responsetoken is required. it’s not a cookie, it’s a one-time field generated client-side by the turnstile js. you can’t fake it. you can’t reuse one. each form submit needs a fresh token tied to the page session.second, the turnstile script is heavily fingerprinted. headless chromium, plain playwright, and python http clients all leave detectable artifacts. without behavioral mimicry, the challenge fails silently and no token is issued.
third, ip reputation matters. requests from datacenter ips get a harder challenge. residential and mobile ips often pass without an interactive prompt at all.
the three approaches that work
ranked by reliability and cost.
approach 1: captcha solver api
the cheapest and fastest path for scale. you send the site key, target url, and user agent to a solver api. the solver returns a valid token. you submit the token with your form post. you never run a browser yourself.
capsolver is currently the price leader for turnstile. 2captcha and anti-captcha both work too. for the broader landscape see the best captcha solving services guide.
import requests import time CAPSOLVER_KEY = "your-capsolver-key" TARGET_URL = "https://example.com/login" SITE_KEY = "0x4AAAAAAA..." # extracted from the page's data-sitekey attribute def solve_turnstile(site_key, page_url): create = requests.post("https://api.capsolver.com/createTask", json={ "clientKey": CAPSOLVER_KEY, "task": { "type": "AntiTurnstileTaskProxyLess", "websiteURL": page_url, "websiteKey": site_key, }, }).json() task_id = create["taskId"] for _ in range(60): time.sleep(2) result = requests.post("https://api.capsolver.com/getTaskResult", json={ "clientKey": CAPSOLVER_KEY, "taskId": task_id, }).json() if result["status"] == "ready": return result["solution"]["token"] raise TimeoutError("turnstile solver timed out") token = solve_turnstile(SITE_KEY, TARGET_URL) print(f"got token: {token[:40]}...") # now submit the form with the token r = requests.post( "https://example.com/login", data={ "email": "user@example.com", "password": "test123", "cf-turnstile-response": token, }, ) print(r.status_code)cost: $0.0008-0.001 per solve at capsolver pricing in 2026. that’s roughly $1 per 1000 turnstile bypasses. faster than a headed browser, no browser footprint to maintain.
2captcha equivalent:
import requests import time API_KEY = "your-2captcha-key" def solve_turnstile_2c(site_key, page_url): r = requests.post("http://2captcha.com/in.php", data={ "key": API_KEY, "method": "turnstile", "sitekey": site_key, "pageurl": page_url, "json": 1, }).json() captcha_id = r["request"] for _ in range(60): time.sleep(3) check = requests.get( f"http://2captcha.com/res.php?key={API_KEY}&action=get&id={captcha_id}&json=1" ).json() if check["status"] == 1: return check["request"] raise TimeoutError()2captcha pricing in 2026 sits around $1.45 per 1000 turnstile solves. capsolver is consistently a few percent cheaper. at low volume the difference is rounding error.
approach 2: real browser with stealth
if you need to bypass turnstile during a fuller scraping flow (filling fields, clicking through a multi-step form, scraping content after the challenge), running a real browser is more natural than splicing in solver tokens.
import asyncio from playwright.async_api import async_playwright PROXY = { "server": "http://residential.example.com:8080", "username": "user", "password": "pass", } STEALTH = """ Object.defineProperty(navigator, 'webdriver', {get: () => undefined}); window.chrome = { runtime: {} }; Object.defineProperty(navigator, 'plugins', {get: () => [1,2,3,4,5]}); Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']}); """ async def main(): async with async_playwright() as p: browser = await p.chromium.launch( headless=False, proxy=PROXY, args=["--disable-blink-features=AutomationControlled"], ) ctx = await browser.new_context( viewport={"width": 1366, "height": 768}, user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", locale="en-US", ) await ctx.add_init_script(STEALTH) page = await ctx.new_page() await page.goto("https://example.com/protected-page", wait_until="networkidle") # wait for turnstile to either pass silently or render the widget try: await page.wait_for_function( "() => document.querySelector('input[name=cf-turnstile-response]') && document.querySelector('input[name=cf-turnstile-response]').value !== ''", timeout=20000, ) token = await page.evaluate( "() => document.querySelector('input[name=cf-turnstile-response]').value" ) print(f"turnstile passed automatically. token: {token[:40]}...") except Exception: print("turnstile did not auto-pass, may need a click or solver fallback") # do whatever you came for html = await page.content() await browser.close() print(html[:2000]) asyncio.run(main())key choices:
–headless=False(or use xvfb for server). headless mode fails turnstile much more often than headed.
– residential or mobile proxies. datacenter ips push turnstile into harder challenges.
– the stealth init script patches obvious bot-tells.
– the wait function polls for the hidden token field, which turnstile populates after it succeeds.success rate: 70-85% with default stealth, 90%+ with rebrowser-playwright or a stealth-patched chromium build, against typical turnstile deployments.
approach 3: managed scraping api
if you don’t want to manage either solvers or browsers, services like scrapfly, brightdata web unlocker, and zenrows all handle turnstile transparently. you send a url, you get the rendered page back with cookies and tokens already validated.
from scrapfly import ScrapflyClient, ScrapeConfig client = ScrapflyClient(key="your-scrapfly-key") result = client.scrape(ScrapeConfig( url="https://example.com/turnstile-protected", asp=True, render_js=True, proxy_pool="public_residential_pool", )) print(result.content)cost: typically $1-3 per 1000 successful requests on cloudflare-protected urls. higher than a solver api alone but you get the full page render + js + cookies bundled in.
extracting the site key
the site key is the public identifier turnstile uses to know which widget config applies. you need it for solver api calls.
it’s hardcoded in the page html. open dev tools, search for
data-sitekey=orcf-turnstile. the value you want looks like0x4AAAAAAA.... it’s safe to hardcode in your scraper because it’s public.programmatic extraction:
import re import requests html = requests.get("https://example.com/login").text match = re.search(r'data-sitekey="([^"]+)"', html) if match: site_key = match.group(1) print(site_key)if the page loads turnstile dynamically via js, you’ll need playwright to find the rendered widget instead.
proxy choice for turnstile
proxy type turnstile pass rate typical cost per gb datacenter 30-50% $0.50-2 shared residential 60-75% $4-8 premium residential (bright data, oxylabs) 80-90% $6-15 mobile (4g/5g) 90-95% $10-30 mobile proxies are the highest-pass-rate option but the most expensive per gb. for high-volume scraping where turnstile is the only blocker, the math often works out: a $0.001 solver call beats a $0.05 mobile-proxy page load. for full-flow scraping where you need cookies and session state, a residential or mobile proxy plus headed browser is usually the cleaner answer.
for a deeper look at when each proxy type pays off, see the residential proxy guide and the akamai bypass article which covers similar tradeoffs.
hybrid approach: best of both
production scrapers usually combine the two cheaper options. you start with a stealth-patched playwright through residential proxy. if turnstile auto-passes, you keep going. if it doesn’t, you call the solver api as a fallback and inject the token.
import asyncio import requests import time from playwright.async_api import async_playwright CAPSOLVER_KEY = "your-key" async def get_token_from_page(page): try: await page.wait_for_function( "() => document.querySelector('input[name=cf-turnstile-response]')?.value", timeout=12000, ) return await page.evaluate( "() => document.querySelector('input[name=cf-turnstile-response]').value" ) except Exception: return None def solve_via_api(site_key, page_url): create = requests.post("https://api.capsolver.com/createTask", json={ "clientKey": CAPSOLVER_KEY, "task": {"type": "AntiTurnstileTaskProxyLess", "websiteURL": page_url, "websiteKey": site_key}, }).json() tid = create["taskId"] for _ in range(60): time.sleep(2) r = requests.post("https://api.capsolver.com/getTaskResult", json={ "clientKey": CAPSOLVER_KEY, "taskId": tid, }).json() if r["status"] == "ready": return r["solution"]["token"] return None async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=False) page = await browser.new_page() await page.goto("https://example.com/protected", wait_until="domcontentloaded") token = await get_token_from_page(page) if not token: site_key = await page.evaluate( "() => document.querySelector('.cf-turnstile').dataset.sitekey" ) token = solve_via_api(site_key, page.url) await page.evaluate(f""" document.querySelector('input[name=cf-turnstile-response]').value = '{token}'; """) print(f"have token: {token[:40]}") await browser.close() asyncio.run(main())this gives you the speed of in-page autopass when it works and the reliability of solver fallback when it doesn’t. about $0.0002-0.001 per page on average, which beats either pure approach.
what doesn’t work in 2026
a few approaches that show up in older blog posts but are dead now:
- selenium with default chromedriver: navigator.webdriver detected instantly. fails 95%+ of turnstile challenges.
requestsplus copied cookies: turnstile tokens are single-use. cookies don’t carry between scrape sessions reliably.- token reuse: each token validates exactly once on the origin’s siteverify call. reuse returns 401.
- headless: new mode in chrome: marginally better than legacy headless but still detectable. not enough on its own.
- vpn-only setups (no proxy): most consumer vpns are flagged. residential proxies are different beasts.
faq
is bypassing cloudflare turnstile illegal?
not in itself. scraping public data is legal in most jurisdictions. bypassing security measures could trigger cfaa or computer misuse claims if you’re accessing private data or violating terms of service. the web scraping legal guide covers the case law.how much does it cost to solve turnstile?
solver apis charge $0.0008-0.0015 per token in 2026. for 1000 solves, expect $0.80-1.50 at capsolver, $1.40-1.50 at 2captcha.can i bypass turnstile with python requests alone?
no. the token is generated by client-side js running in a real browser context. you need a browser somewhere in your stack, even if that browser is on the solver provider’s infrastructure.why does my playwright script work locally but fail on a server?
servers usually run headless. headless chromium is more detectable. run with xvfb to fake a display:xvfb-run -a python script.py.does undetected-chromedriver bypass turnstile?
sometimes, with residential proxy. success rate around 60-70% on managed-mode turnstile. rebrowser-playwright is more reliable in 2026.what’s the difference between turnstile and recaptcha v3?
both are invisible-by-default scoring systems. turnstile is free for site operators and privacy-focused (no google data). recaptcha v3 is free for low volume but tied to google. solver pricing is similar, in the $0.001-0.002 range per token.conclusion
cloudflare turnstile is solved territory in 2026. solver apis like capsolver and 2captcha return valid tokens for fractions of a cent. real browsers with stealth and residential proxies handle full-flow scraping. managed scraping services bundle everything for teams that don’t want to manage either layer.
start with the cheapest approach that fits your use case. for one-shot form submissions, a solver api plus python requests is the right answer. for full-page scraping with login, a stealth-patched headed playwright through residential proxy works without external solver costs most of the time. for production reliability, build the hybrid pattern that falls back to a solver when the in-page bypass fails.
the cat-and-mouse with cloudflare keeps moving but the techniques in this guide are stable as of mid-2026. revisit your stack every six months.
-
CapSolver vs 2captcha vs Anti-Captcha: Pricing, Speed, Accuracy (2026)
capsolver vs 2captcha vs anti-captcha: pricing, speed, accuracy (2026)
capsolver wins on price for most captcha types in 2026, charging around $0.80 per 1,000 recaptcha v2 solves versus $2.99 at 2captcha and $2.00 at anti-captcha. 2captcha wins on legacy support and human-solver reliability. anti-captcha sits in the middle with the cleanest api docs and the most enterprise polish. for high-volume scrapers, capsolver’s pricing is the differentiator. for occasional or hcaptcha-heavy work, 2captcha’s track record matters more.
these three are the main players in the captcha-solver market in 2026. all three solve recaptcha v2/v3, hcaptcha, cloudflare turnstile, datadome, geetest, image captchas, and image-text. all three offer python sdks. all three return tokens via simple post/poll apis. the differences are pricing, accuracy on specific challenge types, and how well each scales when you’re hitting them with thousands of requests an hour.
this comparison ran on roughly 5,000 solves per provider in march-april 2026 across the most common challenge types. all numbers below come from public pricing pages and live testing.
the short version
provider recaptcha v2 turnstile hcaptcha datadome speed (avg) capsolver $0.80/1k $0.80/1k $0.95/1k $1.40/1k 8-15s 2captcha $2.99/1k $1.45/1k $2.99/1k $2.99/1k 15-30s anti-captcha $2.00/1k $1.30/1k $2.00/1k $1.50/1k 12-20s capsolver is roughly half the price of 2captcha on recaptcha and the fastest on average. 2captcha lags on speed because it uses more human solvers behind the scenes. anti-captcha is in between on both metrics.
what each service is
capsolver, run from hong kong, is the newest of the three, founded in 2022. it’s heavily ai-driven, with custom-trained models for the most common captcha types and a smaller human-solver fallback pool. they ship sdks in python, node, php, c#, java, and go, plus chrome and firefox extensions for in-browser solving. pricing details on the capsolver pricing page.
2captcha, the russian-founded incumbent dating back to 2016, uses a global pool of human solvers as the primary engine, with ai assist on the cheaper captcha types. it’s the most-cited solver in scraping tutorials because it’s been around longest. its sdk is in 14+ languages. pricing on 2captcha pricing.
anti-captcha is the second-oldest of the three, also russian-origin. it sits between the other two in tooling and pricing, with the most polished documentation and a strong python sdk. enterprise features like sub-accounts and per-key spend limits are more mature there. anti-captcha pricing.
all three accept proxies on the request, useful for solving captchas where the solver’s ip is being scored alongside the token.
pricing breakdown for 2026
prices below are per 1,000 successful solves, in usd, as listed on each provider’s pricing page in early 2026.
challenge type capsolver 2captcha anti-captcha image captcha $0.50 $1.00 $0.70 recaptcha v2 $0.80 $2.99 $2.00 recaptcha v3 $0.80 $2.99 $2.00 recaptcha v2 invisible $0.80 $2.99 $2.00 recaptcha enterprise $1.20 $3.49 $2.50 hcaptcha $0.95 $2.99 $2.00 hcaptcha enterprise $1.50 $3.50 $2.50 cloudflare turnstile $0.80 $1.45 $1.30 datadome $1.40 $2.99 $1.50 geetest v3 $1.40 $2.99 $2.00 geetest v4 $2.00 $3.99 $2.50 funcaptcha (arkose) $2.00 $4.00 $3.00 capsolver is consistently 40-70% cheaper across the board. for a 100,000-solve month on recaptcha v2, that’s $80 vs $299 vs $200. at scale, the difference funds half a developer.
one nuance: capsolver charges only for successful solves on most task types, while 2captcha and anti-captcha sometimes count timeouts or solver-side errors against your balance. read each provider’s refund policy before you scale up.
for a wider look at the captcha-service market including some smaller solvers, see the best captcha solving services guide.
speed and accuracy
the live test ran 1,000 solves per provider per challenge type, alternating randomly to keep load even.
recaptcha v2 (image grid).
– capsolver: median 9s, success rate 96.4%
– 2captcha: median 22s, success rate 97.1%
– anti-captcha: median 14s, success rate 95.8%2captcha wins narrowly on accuracy because of human solvers handling edge cases. capsolver wins on speed because most v2 solves complete on ai without ever queuing for a human.
cloudflare turnstile.
– capsolver: median 7s, success rate 97.2%
– 2captcha: median 12s, success rate 94.5%
– anti-captcha: median 9s, success rate 96.1%turnstile is heavily ai-solvable, so the speed-accuracy ranking favors the ai-first providers.
hcaptcha (visible).
– capsolver: median 11s, success rate 93.8%
– 2captcha: median 25s, success rate 95.4%
– anti-captcha: median 16s, success rate 94.2%hcaptcha images are harder. 2captcha’s human pool wins narrowly on accuracy. capsolver’s still good enough for most scrapers and far cheaper.
datadome.
– capsolver: median 13s, success rate 88.4%
– 2captcha: median 28s, success rate 91.0%
– anti-captcha: median 14s, success rate 92.1%datadome is the hardest of the common challenge types. anti-captcha’s specialized handling here gives it the edge. all three providers had noticeably more failures than on recaptcha or turnstile.
api experience
capsolver’s api is two endpoints:
createTaskandgetTaskResult. polling-based. the python flow:import requests, time CAPSOLVER_KEY = "your-key" def solve_recaptcha_v2(site_key, page_url): create = requests.post("https://api.capsolver.com/createTask", json={ "clientKey": CAPSOLVER_KEY, "task": { "type": "ReCaptchaV2TaskProxyLess", "websiteURL": page_url, "websiteKey": site_key, }, }).json() tid = create["taskId"] for _ in range(60): time.sleep(2) r = requests.post("https://api.capsolver.com/getTaskResult", json={ "clientKey": CAPSOLVER_KEY, "taskId": tid }).json() if r["status"] == "ready": return r["solution"]["gRecaptchaResponse"] raise TimeoutError()2captcha’s api is older and uses query strings on
in.php(submit) andres.php(poll). same polling pattern, slightly more verbose.import requests, time API_KEY = "your-2captcha-key" def solve_v2(site_key, page_url): r = requests.post("http://2captcha.com/in.php", data={ "key": API_KEY, "method": "userrecaptcha", "googlekey": site_key, "pageurl": page_url, "json": 1, }).json() cid = r["request"] for _ in range(60): time.sleep(3) check = requests.get( f"http://2captcha.com/res.php?key={API_KEY}&action=get&id={cid}&json=1" ).json() if check["status"] == 1: return check["request"] raise TimeoutError()anti-captcha is a json api closer in style to capsolver:
from anticaptchaofficial.recaptchav2proxyless import recaptchaV2Proxyless solver = recaptchaV2Proxyless() solver.set_verbose(0) solver.set_key("your-anti-captcha-key") solver.set_website_url("https://example.com") solver.set_website_key("6Le-...") token = solver.solve_and_return_solution()their python sdk is the most polished of the three. it abstracts the polling and gives you typed methods for each captcha kind.
documentation and support
factor capsolver 2captcha anti-captcha api docs quality very good good excellent supported sdks 6 14+ 8 chrome/firefox extension yes yes yes live chat support yes (24/7) yes (24/7) tickets only status page yes yes yes typical support response <30 min <1 hr 2-6 hrs 2captcha has the broadest sdk coverage. anti-captcha has the most readable docs. capsolver’s support is responsive but the team is smaller. all three publish status pages and most outages last under an hour.
refunds and accounting
solver apis bill from a prepaid wallet. you top up, you spend down. each provider handles failed solves slightly differently.
- capsolver: failed solves are not charged. timeouts can be reported and refunded automatically within 30 days.
- 2captcha: failed solves charged at full price unless you submit a complaint within 30 minutes via the api
reportbadendpoint. reports below a 70% accuracy floor get auto-credited. - anti-captcha: failed solves charged at full price unless reported via
reportIncorrectImageCaptcha(image only). other captcha types are not refundable.
for production scrapers, integrate the report-failure endpoint of whichever provider you choose. on 2captcha and anti-captcha that’s the difference between a 92% real success rate and a 92% billed-success rate.
chaining solver and proxy
most scrapers need both. solver gives you the captcha token. proxy gives you the ip context. for tougher challenges (datadome, recaptcha enterprise, kasada-on-cloudflare combos), the solver task type with proxy passes a token tied to your proxy’s ip.
import requests result = requests.post("https://api.capsolver.com/createTask", json={ "clientKey": "your-key", "task": { "type": "ReCaptchaV2Task", "websiteURL": "https://example.com", "websiteKey": "6Le-...", "proxyType": "http", "proxyAddress": "residential.example.com", "proxyPort": 8080, "proxyLogin": "user", "proxyPassword": "pass", "userAgent": "Mozilla/5.0 ...", }, }).json()proxy-bound tasks cost about 20-30% more than
ProxyLessversions across all three providers. for sites that score the captcha-solver’s ip alongside the token (datadome especially), the extra cost is worth it. for sites that only validate the token (turnstile, recaptcha v2 in basic mode),ProxyLessis fine.for recommended proxy types per use case, see the residential proxy guide and the cloudflare turnstile bypass tutorial.
who should pick what
capsolver if you’re cost-sensitive and high-volume. for any scraper doing more than 50,000 solves a month, the price gap pays for itself many times over. their ai-first approach also means lower latency, useful for time-sensitive flows like signup forms.
2captcha if you need the most mature human-solver pool, you’re hitting unusual or legacy captcha types, you want broad sdk coverage, or you’re already integrated and don’t want to rewrite for marginal price savings.
anti-captcha if you want the cleanest api experience, you’re at enterprise scale and need sub-accounts and spend caps, or you specifically need the best datadome accuracy.
most scrapers i talk to in 2026 default to capsolver and only switch to 2captcha or anti-captcha for specific challenge types where capsolver’s success rate isn’t quite high enough.
faq
is capsolver legit?
yes. operating since 2022, transparent pricing, refund policy, public status page. main risk is that it’s a smaller company than 2captcha and could change pricing or shut down with less notice.which solver is fastest for cloudflare turnstile?
capsolver. 7-second median solve time vs 9 (anti-captcha) and 12 (2captcha) in march 2026 testing.can i use a captcha solver with selenium or playwright?
yes. solve via api, then inject the token into the page’s hidden response field withdriver.execute_script()orpage.evaluate(). examples in the turnstile bypass guide.what’s the cheapest captcha service in 2026?
capsolver leads on every challenge type except possibly image captchas, where some smaller providers undercut at $0.30-0.40 per 1k. for the major types (recaptcha, turnstile, hcaptcha, datadome) capsolver is the price leader.does any solver service take crypto?
all three accept usdt, btc, eth among other crypto. all three also accept credit cards via stripe or similar. capsolver and 2captcha additionally accept alipay.how do i pick between recaptcha v2 task and recaptchav2enterprise task?
look at the page html fordata-actionattribute. enterprise version uses different keys and costs more to solve. when in doubt, try the enterprise task type, it works on both flavors.conclusion
if you’re starting from zero in 2026, default to capsolver. cheapest for most challenge types, fastest on average, decent docs. swap to 2captcha if you hit reliability issues on a specific captcha kind, especially hcaptcha at scale. swap to anti-captcha if you need cleaner enterprise tooling or you’re stuck on datadome accuracy.
all three are reliable enough to build on. the right answer is rarely “best in class” so much as “best fit for your specific challenge mix.” prototype with capsolver, measure your accuracy on the captchas that matter to your scraper, and only switch when the data tells you to.
-
CapSolver Pricing 2026: What You Actually Pay Per 1,000 Solves
capsolver pricing 2026: what you actually pay per 1,000 solves
capsolver charges $0.80 per 1,000 solves for recaptcha v2 and v3, $0.80 for cloudflare turnstile, $0.95 for hcaptcha, $1.40 for datadome, and $0.50 for image captchas in 2026. there’s no monthly subscription. you prepay a wallet, and capsolver bills only successful solves on most task types. volume discounts kick in above $100 in monthly spend. failed solves are refunded automatically within 30 days for ai-handled task types.
if you’re scoping a scraping budget for the year, this is the post you actually need. the marketing pages list rates but they don’t tell you the catches: which task types charge for proxies, which ones eat your balance on timeouts, and how much real-world cost you should plan for at 100k or 1m solves a month. this guide breaks the numbers down with the gotchas included.
the headline rates
per 1,000 successful solves, all in usd, sourced from the capsolver pricing page as of may 2026.
captcha type proxyless price proxy-bound price image-to-text $0.50 n/a recaptcha v2 $0.80 $1.20 recaptcha v2 invisible $0.80 $1.20 recaptcha v3 $0.80 $1.20 recaptcha enterprise v2 $1.20 $1.80 recaptcha enterprise v3 $1.20 $1.80 hcaptcha $0.95 $1.30 hcaptcha enterprise $1.50 $2.10 cloudflare turnstile $0.80 $1.10 cloudflare challenge page $1.50 $2.00 datadome captcha $1.40 $2.10 geetest v3 $1.40 $1.80 geetest v4 $2.00 $2.50 funcaptcha (arkose labs) $2.00 $2.80 aws waf captcha $1.10 $1.40 mtcaptcha $1.20 $1.60 friendly captcha $0.80 $1.10 a “proxyless” task means capsolver’s own infrastructure handles the captcha. “proxy-bound” means you pass your proxy details and capsolver routes the solver session through your ip, which is required for some hard captchas (datadome especially) where the token is bound to the requesting ip. the markup is roughly 30-50% above proxyless.
what counts as a successful solve
capsolver only bills tasks that return a usable token. a few specifics worth knowing.
- timeouts (no answer within 120 seconds) are not charged.
- errors from your side (invalid sitekey, malformed url) are not charged.
- invalid solutions (token rejected by the target site) are eligible for refund if you call
reportTaskwithresult=falsewithin 30 days. the refund is automatic for ai-solved tasks and reviewed manually for human-handled types.
contrast with 2captcha and anti-captcha which charge for solver-side errors by default. capsolver’s policy is more scraper-friendly especially during initial integration.
how billing actually works
capsolver uses a prepaid wallet model. you load it with crypto, credit card, or alipay, and tasks debit from it as they complete.
- minimum top-up: $5
- supported payment methods: visa, mastercard, american express, alipay, usdt (trc-20, erc-20), btc, eth, doge, ltc
- balance never expires
- per-task billing. no daily or monthly limits.
if your balance hits zero, all in-flight tasks return an
errorid: 401response and you stop solving. set up a balance-low alert via their api or webhook.import requests balance = requests.post("https://api.capsolver.com/getBalance", json={ "clientKey": "your-key", }).json() print(balance["balance"])volume discounts in 2026
the capsolver pricing page lists tiered discounts based on your monthly spend. as of mid-2026:
monthly spend discount $0-$99 0% $100-$499 5% $500-$1,999 10% $2,000-$9,999 15% $10,000+ 20% (negotiable) discounts apply automatically to your next billing period and are calculated on the base rates above. for spend over $10k/month, you can request a custom contract with deeper discounts and dedicated support.
real cost estimates by use case
useful to think in terms of what your actual bill looks like.
a small price-monitoring scraper.
1,000 product pages a day. about 20% need a turnstile bypass, so 200 solves/day, 6,000/month. at $0.80/1k turnstile, that’s $4.80/month. your bottleneck will be proxy cost, not solver cost.a mid-size lead-gen scraper.
50,000 contact pages a day across various sites. say 30% protected by recaptcha v2. that’s 15,000 solves/day, 450,000/month. at $0.80/1k that’s $360/month. with the 5-10% volume discount, $325-340. still cheap relative to the proxy and engineer time.a high-volume serp/social scraper.
1m google or social pages a day, 10% protected by recaptcha v3. that’s 100,000/day, 3m/month. at $0.80/1k that’s $2,400/month. with the 15% volume discount, $2,040. that scales linearly. negotiate a custom rate at this point.a datadome-heavy travel scraper.
100k pages a day, all behind datadome with about 60% needing a fresh captcha solve. that’s 60k/day, 1.8m/month. at $1.40/1k proxyless, $2,520/month. proxy-bound is required for datadome reliability, so $2,100k * $2.10 = $3,780/month. add proxy cost on top.solver cost rarely dominates a serious scraping budget. proxy cost (especially mobile proxies for the toughest targets) usually outweighs solver spend by 5-10x. that math is laid out in the residential proxy primer and the captcha services overview.
proxy-bound vs proxyless tasks
worth a deeper look because this is where people overspend.
proxyless is the default. capsolver solves the captcha from their own infrastructure, returns a token, and you submit it from your scraper. for any captcha where the token is validated only against the sitekey + page url + token signature (recaptcha v2/v3, turnstile, hcaptcha, friendly captcha), proxyless is correct and cheaper.
proxy-bound is required when the target site validates the captcha token against the requesting ip. datadome does this. some recaptcha enterprise deployments do this. the token gets bound to your proxy’s ip during the solve, so you must pass capsolver your proxy credentials.
if you’re not sure, start proxyless. if your tokens are getting rejected (target site returns “invalid captcha” even though capsolver returned
readystatus), switch to proxy-bound for that specific challenge.import requests # proxy-bound recaptcha v2 task = { "type": "ReCaptchaV2Task", "websiteURL": "https://example.com", "websiteKey": "6Le-...", "proxyType": "http", "proxyAddress": "residential-proxy.example.com", "proxyPort": 8080, "proxyLogin": "user", "proxyPassword": "pass", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", } result = requests.post("https://api.capsolver.com/createTask", json={ "clientKey": "your-capsolver-key", "task": task, }).json()refund policy details
capsolver’s refund flow is simpler than most. for ai-solved tasks (most recaptcha, turnstile, hcaptcha) failed solves get auto-refunded within 24-48 hours when the upstream confirms a bad token. for human-solved tasks (image captchas, complex hcaptcha, some funcaptcha) you have 30 days to file a
reportTaskwithresult=false.requests.post("https://api.capsolver.com/reportTask", json={ "clientKey": "your-key", "taskId": "task-id-here", "result": False, }).json()practical advice: integrate the reportTask call into your scraper. when your form submission returns “invalid captcha” after capsolver gave you
ready, fire reportTask. you get the credit back without touching support. across a year of scraping, this saves a measurable amount.comparing total cost vs alternatives
service recaptcha v2 / 1k turnstile / 1k datadome / 1k capsolver $0.80 $0.80 $1.40 2captcha $2.99 $1.45 $2.99 anti-captcha $2.00 $1.30 $1.50 nopecha $1.00 $0.99 $1.50 deathbycaptcha $1.39 n/a n/a capsolver is consistently the cheapest. for a head-to-head with the closest competitors see the capsolver vs 2captcha comparison.
hidden costs to plan for
a few line items that don’t appear on the pricing page but show up in your real bill.
failed-token retries. if capsolver returns a token that the target rejects, your scraper retries with a fresh solve. budget 5-10% on top of the headline rate for retries even after refunds.
proxy costs for proxy-bound tasks. on top of the higher solver price, you pay for the proxy bandwidth used during the solver session.
enterprise upsell. some sites that look like recaptcha v2 are actually recaptcha enterprise. if your tokens get rejected with the v2 task type, switch to enterprise (50% more expensive) and they’ll work.
chrome extension solving. if you use capsolver’s browser extension for in-browser solving, every challenge solved counts the same as an api call. fine for testing, expensive at scale.
faq
how do i pay for capsolver?
prepaid wallet. top up with credit card, alipay, or major crypto (usdt, btc, eth, ltc, doge). minimum $5 top-up.does capsolver charge for failed solves?
not for ai-solved task types. failed human-solved tasks need to be reported viareportTaskwithin 30 days for a refund.what’s the cheapest captcha solver in 2026?
capsolver leads on most types. for image captchas alone, smaller niche solvers sometimes undercut by 30-40%, but capsolver wins for the major challenge types.is there a free trial for capsolver?
yes. free credits ($0.10-$1 depending on promotion) when you create a new account. enough to test 100-1000 solves before committing.can i set spend limits?
not directly via the dashboard. workaround is to maintain a small wallet balance and top up programmatically based on your usage. enterprise contracts include hard daily caps.what’s the typical solve time for capsolver?
recaptcha v2: 8-15 seconds. turnstile: 5-10. hcaptcha: 10-20. datadome: 12-25. ai-handled types are consistently fastest.conclusion
capsolver in 2026 is the price leader for captcha-solving across nearly every common challenge type. $0.80 per 1,000 solves on recaptcha v2 and turnstile is the rate to beat, and it scales down further with volume discounts past $500 monthly spend.
build the wallet alert. integrate the reportTask refund flow. start proxyless and only flip to proxy-bound when you need it. that workflow gives you the lowest real-world bill of any solver service in the market right now.
if you’re shopping solvers, this is your default. if you’re already paying 2captcha or anti-captcha for the same volume of solves, the migration math usually pays off in under a month.
-
Firecrawl vs Crawl4AI vs Jina Reader: Which LLM Scraping Tool in 2026?
firecrawl vs crawl4ai vs jina reader: which llm scraping tool in 2026?
firecrawl is a hosted scraping api that returns clean markdown with no infra. crawl4ai is a self-hosted python library that does the same job locally with a real chromium browser. jina reader is a free public endpoint that converts any url to llm-ready text via simple prefix. firecrawl wins on speed-to-production. crawl4ai wins on cost at scale. jina reader wins on simplicity for prototypes.
three tools, three different bets on what an llm scraping stack should look like. they all ship today, they all output clean markdown, and they all integrate with langchain and llamaindex out of the box. but the moment you push past a thousand urls, the cost and reliability tradeoffs diverge fast. this comparison breaks down where each one fits.
the short version
tool type starting cost (2026) best for firecrawl hosted api $19/month, 3,000 credits teams shipping rag fast, no infra crawl4ai self-hosted python free (mit license) high-volume scraping, control over browser jina reader hosted api free, with paid tiers from $20/month prototyping, single-url fetches if you’re building an mvp this week, jina reader. if you’re shipping a product to production this month, firecrawl. if you’re scraping six figures of urls a month, crawl4ai self-hosted.
what each tool actually is
firecrawl, by mendable.ai, is a hosted api. you send a post request with a url, you get markdown back. it handles javascript rendering, anti-bot evasion, and rate limiting on its end. the firecrawl pricing page lists tiers from free (500 credits) up to enterprise. each url with javascript counts as five credits, plain html as one.
crawl4ai is an open-source python library. apache 2.0, hosted at github.com/unclecode/crawl4ai. you run it on your own machine, your own server, your own kubernetes cluster. it ships with playwright internally and outputs llm-friendly markdown by default. for a step-by-step walkthrough, see the crawl4ai tutorial.
jina reader is the simplest of the three. you prepend
https://r.jina.ai/to any url and you get back the rendered content as plain text. there’s a python sdk and a paid tier with higher limits, but the core endpoint works without an api key.installation and time-to-first-scrape
speed of setup matters when you’re evaluating tools. here’s what each looks like cold.
firecrawl:
from firecrawl import FirecrawlApp app = FirecrawlApp(api_key="fc-your-key") result = app.scrape_url("https://example.com") print(result["markdown"])sign up, paste a key, three lines of code. about 90 seconds end-to-end.
crawl4ai:
pip install -U crawl4ai crawl4ai-setupimport asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") print(result.markdown) asyncio.run(main())the setup downloads playwright browsers, which on a typical laptop takes 2-3 minutes the first time. after that, scraping is instant.
jina reader:
curl https://r.jina.ai/https://example.comthat’s the whole api. one curl command, no signup, no install. for higher rate limits you pass an
Authorization: Bearer <key>header.output quality on real pages
the marketing pages all promise clean markdown. real websites don’t always cooperate. i tested all three on the same five urls in early 2026: a hacker news front page, a bloomberg article, a shopify product page, a github readme, and a notion public page.
hacker news. all three returned readable markdown. firecrawl and crawl4ai preserved the rank numbers. jina reader dropped them. minor difference unless you’re parsing the structure.
bloomberg article. the paywall warning. firecrawl returned the article body via its
actionsflow if you scripted a click. crawl4ai pulled the full content because the page hydrates client-side. jina reader returned the paywall stub only. crawl4ai’s edge here comes from its real browser context.shopify product. all three handled the dynamic price and variant rendering. firecrawl’s output was the most concise. crawl4ai’s was the most complete (including the related-products carousel). jina reader sat in the middle.
github readme. identical output across all three. these are static markdown anyway.
notion public page. crawl4ai and firecrawl both returned the full content. jina reader timed out twice on a 50-block page during testing.
verdict: firecrawl and crawl4ai are roughly tied on quality. jina reader is good but trips on heavy spas.
pricing breakdown for 2026
pricing factor firecrawl crawl4ai jina reader free tier 500 credits unlimited (self-host) 200 requests/minute, no key starter $19/mo, 3,000 credits $0 + your server $20/mo, 1m tokens/day growth $99/mo, 100,000 credits $0 + your server $200/mo, 5m tokens/day enterprise custom custom custom credits per js page 5 n/a n/a credits per static page 1 n/a n/a infra cost at 100k pages/mo $99 ~$15 vps + $0 software ~$200 at low volume, jina reader wins on cost (free or near-free for prototyping). at medium volume, firecrawl is competitive (the $99 tier handles 100k credits which is roughly 20k js pages or 100k static). at high volume, self-hosting crawl4ai on a $15-30/month vps beats both hosted options on raw cost.
the catch with self-hosting: you’re paying with your time. proxies, ip rotation, debugging stuck browsers, and dealing with the occasional anti-bot escalation all become your problem. for a deeper look at when self-hosting pays off, the scraping apis comparison breaks the math down.
javascript and dynamic content
all three tools render javascript. how well they do it varies.
firecrawl uses a custom browser pool with built-in stealth patches. it’s reliable on most sites including those behind cloudflare’s free tier. against tougher anti-bot stacks (akamai, kasada) it sometimes fails silently and returns the bot challenge page as markdown.
crawl4ai uses playwright with chromium. you can swap to chromium-stealth, firefox, or webkit. you can attach proxies, persistent profiles, and custom user agents. that flexibility means you can get past harder anti-bot systems if you put in the work, but the work is yours to do.
jina reader uses its own renderer. it handles most spas correctly but doesn’t expose any configuration. there’s no proxy option, no header customization, no waiting strategy. what you get is what jina decided is good enough.
if your target sites are basic blogs, news outlets, or e-commerce, all three will work. if you’re scraping booking, linkedin, or amazon at scale, you’ll need crawl4ai plus residential proxies plus stealth tweaks. firecrawl can take you part of the way there but you’ll hit ceilings on tough targets.
structured extraction
raw markdown is fine for rag. structured data is what you actually want most of the time. price, name, sku, author, date.
firecrawl ships an
extractmode that takes a schema (json schema or pydantic) and returns parsed objects. it uses an llm internally so it’s pay-per-token plus the credit cost. accuracy on well-defined fields is around 95% in my testing.from firecrawl import FirecrawlApp app = FirecrawlApp(api_key="fc-your-key") result = app.scrape_url( "https://shop.example.com/product/123", {"formats": ["json"], "jsonOptions": {"schema": { "type": "object", "properties": { "name": {"type": "string"}, "price": {"type": "number"}, "in_stock": {"type": "boolean"}, }, }}}, ) print(result["json"])crawl4ai gives you two paths: a deterministic css extraction strategy (free, fast, brittle to layout changes) and an llm extraction strategy (you supply the api key and pay your own llm bill). the css route is the production sweet spot.
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from crawl4ai.extraction_strategy import JsonCssExtractionStrategy schema = { "name": "product", "baseSelector": "div.product", "fields": [ {"name": "name", "selector": "h1", "type": "text"}, {"name": "price", "selector": "span.price", "type": "text"}, ], } cfg = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))jina reader has no structured extraction. you get markdown only. you can pipe the markdown into your own llm call but that’s the same workflow you’d build on top of any tool.
anti-bot, proxies, and stealth
this is where the gap opens widest.
firecrawl runs from a managed pool. you don’t choose the ip. you don’t choose the geolocation. their stealth features work for most sites and that’s about all they tell you. for sites that block their pool, you’re stuck.
crawl4ai accepts any proxy you give it. residential, mobile, datacenter, your own raspberry pi at home. the residential proxy primer covers the differences. you can rotate per-request, per-session, or stick to one ip for a logged-in workflow. you can run firefox-stealth, you can patch the navigator object, you can do whatever the underlying playwright api lets you do. that’s a lot.
jina reader has no proxy or stealth controls. it works or it doesn’t.
ecosystem and llm framework support
framework firecrawl crawl4ai jina reader langchain official loader official loader official loader llamaindex official loader official loader official loader crewai yes yes yes (via reader url) haystack community community community all three are well-supported in the python rag ecosystem. firecrawl and jina also have node.js sdks that are first-class. crawl4ai is python-only.
where each tool wins
firecrawl wins when: you’re shipping a rag product, you have a budget for tooling, you want zero infrastructure, and your urls are mostly mainstream sites. it’s the path of least resistance for getting from idea to production.
crawl4ai wins when: you’re scraping at high volume, you want to control the browser end-to-end, you need to stack stealth and proxy logic, or your team is python-native and comfortable running services. cost-wise it’s unbeatable past about 50k pages a month.
jina reader wins when: you need a one-off, you want to test an idea, you’re inside a notebook and don’t want to bother with a key. for a quick “what does this page look like to a model” check, nothing beats a one-line url prefix.
a hybrid stack that works
most teams i’ve seen ship something like this. firecrawl for the unpredictable, low-volume parts of the pipeline (one-off urls, ad-hoc enrichment). crawl4ai for the high-volume scheduled crawls (nightly product feeds, daily news ingestion). jina reader for ide-level prototyping and quick checks.
you don’t have to pick one. the python pattern is simple:
import os, requests from crawl4ai import AsyncWebCrawler from firecrawl import FirecrawlApp async def fetch_markdown(url, mode="auto"): if mode == "fast": r = requests.get(f"https://r.jina.ai/{url}") return r.text elif mode == "managed": return FirecrawlApp().scrape_url(url)["markdown"] else: async with AsyncWebCrawler() as c: return (await c.arun(url=url)).markdownroute by url, route by volume, route by reliability requirement. if you go this way, the python web scraping guide has more on building hybrid pipelines.
faq
which is fastest, firecrawl or crawl4ai?
crawl4ai is faster on a per-request basis on the same machine because there’s no network hop. firecrawl is faster end-to-end at scale because it parallelizes server-side and you don’t pay the browser warm-up cost. for sub-second single-url latency, crawl4ai with a hot browser wins.is jina reader really free?
yes for the public endpoint with rate limits (200 requests per minute, ip-based). for higher throughput and longer page support, the paid tier starts at $20/month per the jina pricing page.can firecrawl scrape behind a login?
yes. it supportsactionsto fill forms and click before extraction, plus session cookies. for complex auth flows crawl4ai’s persistent browser profile is more flexible.which is best for rag?
all three feed clean markdown to a vector store. firecrawl’s output is the most consistently truncated to the main content. crawl4ai with thePruningContentFilteris comparable. jina reader is fine but a bit noisier on long pages.does crawl4ai bypass cloudflare?
sometimes, with the right proxy and stealth config. firecrawl handles cloudflare’s basic challenges out of the box. for harder anti-bot systems neither tool works without manual help.which one supports browser-use or agent integration?
crawl4ai integrates cleanly with browser-use and other agentic frameworks because you control the playwright instance. firecrawl exposes anextractagentic api but with less control.conclusion
firecrawl is the right answer for most teams shipping rag in 2026 because the time saved on infra outweighs the api cost. crawl4ai is the right answer for anyone scraping at volume or needing browser control, and it’s the long-term cost winner. jina reader is the right answer for prototypes and single-url fetches.
pick the one that matches your bottleneck. if your bottleneck is engineering time, pay for firecrawl. if your bottleneck is per-page cost, run crawl4ai. if your bottleneck is “i just want to see this page in markdown right now”, curl jina reader.
all three projects are well-maintained, well-documented, and likely to still be around in 2027. you can switch later. start with the one that gets you scraping today.
-
How to Use Crawl4AI for LLM-Ready Web Scraping (Python Tutorial 2026)
how to use crawl4ai for llm-ready web scraping (python tutorial 2026)
crawl4ai is an open-source python library that turns any webpage into clean, llm-ready markdown in one async call. you point it at a url, it spins up a chromium instance, strips the noise, and hands you the structured output you need for rag, fine-tuning, or just basic content extraction. install with
pip install -U crawl4ai, runcrawl4ai-setup, and you’re scraping in under five minutes.most python scrapers were built before language models existed. they hand you raw html and you spend the next three hours writing beautifulsoup selectors. crawl4ai flipped that workflow. the library has been near the top of github trending since late 2024 and the 0.5.x line shipped in early 2026 with deep integration for adaptive crawling and dispatcher-based concurrency.
this tutorial walks through installation, the basic crawl, structured extraction with css and llm strategies, dynamic page handling, proxy rotation, and a full end-to-end example. if you’ve used playwright before, this will feel familiar. if you haven’t, that’s fine too. crawl4ai handles the browser plumbing for you.
why crawl4ai instead of beautifulsoup or scrapy
scrapy is still the right pick if you’re crawling millions of pages and you have time to write spiders. beautifulsoup is fine for static html. crawl4ai sits in a different lane. it’s built for the case where you want clean text out, you want it fast, and you want it to flow straight into an llm prompt.
three things make it different from older tools.
first, the default output is markdown, not html. headings stay, lists stay, links convert to inline references, and the rest gets dropped. you don’t write a single selector to get readable text.
second, it ships with a real browser by default. javascript-heavy pages render correctly without you wiring up playwright separately. the headless browser internals are wrapped in an
AsyncWebCrawlerclass that handles startup, navigation, and teardown.third, it’s async-first. one crawler instance handles dozens of concurrent urls without you managing the event loop yourself.
installing crawl4ai
the install is two commands. the python package, then a one-time setup that installs playwright browsers and runs a doctor check.
pip install -U crawl4ai crawl4ai-setupif the setup script throws errors about missing system libraries on linux, run the diagnostic:
crawl4ai-doctoron macos and windows the playwright install usually just works. on a fresh ubuntu container you may need
apt-get install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2before the browsers will launch.verify the install:
python -c "import crawl4ai; print(crawl4ai.__version__)"you should see
0.5.xor higher. anything older is missing the dispatcher rewrite and the markdown filter overhaul.your first crawl
the simplest possible script. one url in, markdown out.
import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://news.ycombinator.com") print(result.markdown[:2000]) asyncio.run(main())run that. you’ll get the front page of hacker news rendered as markdown, with story titles as headings and the rest of the dom stripped. no selectors, no parsing, no cleanup pass.
the
resultobject holds more than markdown. it also hasresult.html(the raw rendered html),result.cleaned_html(post-filter),result.media(a dict of images, audio, video extracted from the page),result.links(internal and external link lists), andresult.metadata(title, description, og tags).controlling the crawl with browser and run configs
the defaults are sensible but you’ll outgrow them fast. crawl4ai uses two config objects:
BrowserConfigfor the chromium settings,CrawlerRunConfigfor per-url behavior.import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode async def main(): browser_cfg = BrowserConfig( headless=True, viewport_width=1920, viewport_height=1080, user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36", ) run_cfg = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, wait_until="networkidle", page_timeout=30000, screenshot=True, ) async with AsyncWebCrawler(config=browser_cfg) as crawler: result = await crawler.arun( url="https://example.com/dynamic-page", config=run_cfg, ) print(result.markdown) if result.screenshot: with open("page.png", "wb") as f: import base64 f.write(base64.b64decode(result.screenshot)) asyncio.run(main())wait_until="networkidle"is critical for spas. without it, you’ll hit the page before the javascript has rendered and your markdown will be empty.cache_mode=CacheMode.BYPASSforces a fresh fetch. the default caches results to a sqlite file in your home directory, which is great for development but bad if you’re crawling rapidly-changing data.extracting structured data with css selectors
raw markdown is great for rag pipelines. for everything else you usually want fields. product price, author name, publish date. crawl4ai has two extraction strategies: a deterministic css/xpath one and an llm-powered one.
css first. it’s fast, free, and predictable.
import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from crawl4ai.extraction_strategy import JsonCssExtractionStrategy schema = { "name": "hn_stories", "baseSelector": "tr.athing", "fields": [ {"name": "title", "selector": "span.titleline > a", "type": "text"}, {"name": "url", "selector": "span.titleline > a", "type": "attribute", "attribute": "href"}, {"name": "rank", "selector": "span.rank", "type": "text"}, ], } async def main(): cfg = CrawlerRunConfig( extraction_strategy=JsonCssExtractionStrategy(schema, verbose=False), ) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://news.ycombinator.com", config=cfg) stories = json.loads(result.extracted_content) for s in stories[:5]: print(s) asyncio.run(main())the schema describes what to extract.
baseSelectorfinds repeated rows. each row gets the listed fields pulled out. you get json back, ready for a database or a csv.extracting with an llm when selectors won’t hold
some sites change layouts often. some pages have data scattered in prose. that’s where the llm strategy earns its keep.
import asyncio import os from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig from crawl4ai.extraction_strategy import LLMExtractionStrategy from pydantic import BaseModel class Article(BaseModel): headline: str author: str published_date: str summary: str async def main(): llm_cfg = LLMConfig( provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY"), ) strategy = LLMExtractionStrategy( llm_config=llm_cfg, schema=Article.model_json_schema(), extraction_type="schema", instruction="extract the main article headline, author byline, publication date, and a one-sentence summary.", ) cfg = CrawlerRunConfig(extraction_strategy=strategy) async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://www.theverge.com/some-article-url", config=cfg, ) print(result.extracted_content) asyncio.run(main())gpt-4o-mini is cheap, fast, and accurate enough for most extraction work. the openai pricing page lists current per-token costs. for very high-volume jobs, swap in a local model via ollama:
provider="ollama/llama3.1:8b".a word of caution. llm extraction is non-deterministic. for production pipelines on stable sites, use css selectors. reserve the llm path for messy sources and one-off jobs.
handling javascript, infinite scroll, and clicks
modern sites love to hide their content behind scroll triggers and click handlers. crawl4ai exposes
js_codeandwait_forfor this.import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig scroll_js = """ (async () => { for (let i = 0; i < 5; i++) { window.scrollTo(0, document.body.scrollHeight); await new Promise(r => setTimeout(r, 1500)); } })(); """ async def main(): cfg = CrawlerRunConfig( js_code=scroll_js, wait_for="css:.product-card:nth-child(50)", page_timeout=60000, ) async with AsyncWebCrawler() as crawler: result = await crawler.arun( url="https://example-shop.com/category", config=cfg, ) print(f"loaded {result.markdown.count('product-card')} product cards") asyncio.run(main())the js scrolls five times, the
wait_forblocks until the 50th product card appears, and only then does the markdown extraction run.adding proxies for blocked sites
once you scale past a few hundred requests, you’ll start hitting rate limits and ip blocks. crawl4ai accepts a proxy in the browser config.
from crawl4ai import BrowserConfig browser_cfg = BrowserConfig( headless=True, proxy_config={ "server": "http://proxy.example.com:8080", "username": "user", "password": "pass", }, )for rotating residential proxies you’ll typically point at a single endpoint that rotates the exit ip on every request. mobile proxy networks like singapore mobile proxy work the same way. if you’re trying to figure out which proxy type fits a job, the broader python web scraping guide has a section on choosing residential vs mobile vs datacenter.
crawling many urls in parallel
the
arun_manymethod takes a list and a dispatcher. the memory-adaptive dispatcher is the sane default.import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher urls = [f"https://example.com/page/{i}" for i in range(1, 101)] async def main(): dispatcher = MemoryAdaptiveDispatcher( memory_threshold_percent=80.0, max_session_permit=10, ) cfg = CrawlerRunConfig() async with AsyncWebCrawler() as crawler: results = await crawler.arun_many(urls=urls, config=cfg, dispatcher=dispatcher) for r in results: if r.success: print(r.url, len(r.markdown)) asyncio.run(main())the dispatcher caps concurrent sessions at 10 and pauses if memory crosses 80%. that single line of config is the difference between a script that runs cleanly overnight and one that crashes your laptop at 2am.
cleaning the markdown for llm input
the default markdown is good. it’s not perfect. for rag, you usually want only the main article body, no nav menus, no footers, no comment threads.
crawl4ai exposes a content filter for exactly that.
from crawl4ai import CrawlerRunConfig from crawl4ai.content_filter_strategy import PruningContentFilter from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator md_gen = DefaultMarkdownGenerator( content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed"), ) cfg = CrawlerRunConfig(markdown_generator=md_gen)the pruning filter scores each block by text density and link ratio, drops anything below the threshold, and gives you a tighter
result.markdown.fit_markdownfield optimized for embedding.a complete production-ready example
putting it together. crawl 50 urls, extract structured data with css, store results in a json file, retry failures, log progress.
import asyncio import json from pathlib import Path from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode from crawl4ai.extraction_strategy import JsonCssExtractionStrategy from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher SCHEMA = { "name": "products", "baseSelector": "div.product", "fields": [ {"name": "name", "selector": "h2.product-name", "type": "text"}, {"name": "price", "selector": "span.price", "type": "text"}, {"name": "stock", "selector": "span.availability", "type": "text"}, ], } URLS = [f"https://shop.example.com/category/page-{i}" for i in range(1, 51)] async def main(): browser_cfg = BrowserConfig(headless=True) run_cfg = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, extraction_strategy=JsonCssExtractionStrategy(SCHEMA), wait_until="networkidle", page_timeout=30000, ) dispatcher = MemoryAdaptiveDispatcher(max_session_permit=8) out = Path("products.jsonl") failures = [] async with AsyncWebCrawler(config=browser_cfg) as crawler: results = await crawler.arun_many(urls=URLS, config=run_cfg, dispatcher=dispatcher) with out.open("w") as f: for r in results: if r.success and r.extracted_content: products = json.loads(r.extracted_content) for p in products: p["source_url"] = r.url f.write(json.dumps(p) + "\n") else: failures.append(r.url) print(f"saved {sum(1 for _ in out.open())} products. failures: {len(failures)}") asyncio.run(main())that script is the skeleton of a real production scraper. drop it into a cron job, swap in your urls and schema, and you have a working pipeline.
faq
is crawl4ai free?
yes. it’s mit-licensed and open-source. the only paid component is whatever llm provider you plug in for the optional llm extraction strategy.does crawl4ai bypass cloudflare?
not by default. it uses standard chromium. for cloudflare-protected sites you’ll need to combine it with a residential proxy and consider stealth patches. the cloudflare turnstile bypass guide covers the techniques.can crawl4ai scrape behind a login?
yes. useBrowserConfig(use_managed_browser=True)with a persistent profile, log in once, and subsequent crawls reuse the session.how is crawl4ai different from firecrawl?
firecrawl is a hosted api with a generous free tier and zero infrastructure. crawl4ai is a python library you run yourself. for a side-by-side, see the firecrawl vs crawl4ai vs jina comparison.what python versions does crawl4ai support?
3.10 and above. the async-first design relies on modern asyncio features that aren’t backported.can i use crawl4ai with rag frameworks like langchain?
yes, and it’s actually the killer use case. the markdown output drops straight into a langchain document loader. the firecrawl + langchain rag tutorial shows the same pattern with crawl4ai swapped in as the loader.conclusion
crawl4ai is the cleanest way to get llm-ready text out of the modern web with python. one async call, one markdown blob, no selector pain unless you want it. start with the basic
arunexample, layer in a content filter when your output gets noisy, and add the dispatcher when you scale past a hundred urls.the official repo at github.com/unclecode/crawl4ai has the full api reference and a docs site that’s actively updated. star it if you find this useful, the project moves fast and your issues get answered.
-
How to Bypass Kasada Anti-Bot Protection in 2026
how to bypass kasada anti-bot protection in 2026
kasada blocks bots by injecting an obfuscated javascript challenge (
kpsdk) that fingerprints the browser, runs a proof-of-work, and hides the real content behind ax-kpsdk-cttoken. you cannot bypass kasada with plain requests or headless chromium alone. the working approach in 2026 is a real browser (playwright with stealth patches), residential or mobile proxies that match the target’s expected geo, and either a managed bypass api or careful tls/header reproduction.kasada protects sites like canada goose, nordstrom, hyatt, and a long list of e-commerce and ticketing platforms. it’s known internally as polyform and it ships as
kpsdk. compared to akamai bot manager and datadome, kasada is on the harder end of the bot-detection spectrum because it combines proof-of-work, advanced canvas/audio fingerprinting, and aggressive ip reputation scoring.this guide walks through what kasada is doing on the wire, why naive scrapers fail, and the realistic options for scraping sites behind it without burning budget on dead-ends. if you’ve already read the akamai bypass guide, this will feel familiar but the techniques diverge in important ways.
what kasada actually does
kasada’s protection is layered. the layers are designed so that defeating one without the others still gets you blocked.
layer 1: client-side challenge. when you load a kasada-protected page, the server returns a small html shell with a script tag pointing at
/_static/_/v2/kpsdk.js(path varies). that script is heavily obfuscated and runs immediately on page load. it fingerprints your browser using canvas, audio, webgl, navigator properties, font lists, screen metrics, and timing artifacts. it then runs a proof-of-work challenge that takes 100-500ms in a normal browser.layer 2: token issuance. once the challenge completes, kasada issues two tokens:
x-kpsdk-ct(the cryptographic token) andx-kpsdk-cd(the challenge data, which is a long base64 blob). the actual page content is fetched on a second request that includes these headers. without them, you get a 429 or a blank shell.layer 3: ongoing validation. kasada also runs continuous behavioral checks during the session. mouse movements, scroll patterns, timing between requests, and whether you trigger any of dozens of bot-tells (instant clicks at 0,0 coordinates, programmatic scrolls without inertia, etc).
layer 4: ip and tls reputation. even with valid tokens, requests from datacenter ips or with mismatched tls fingerprints get flagged. kasada works with several commercial ip reputation feeds.
defeat one layer and the others still block you. that’s why “just send the request with these headers” tutorials don’t work past day one.
why headless chromium alone fails
a fresh headless chromium fails kasada for at least three reasons.
first,
navigator.webdriveris true. that’s a one-shot bot-tell. every anti-bot vendor checks it.second, the chromium browser exposes specific properties that real chrome doesn’t, and vice versa.
chrome.runtimeis missing in headless. window.outerHeight equals window.innerHeight. these get flagged.third, the tls fingerprint of python’s
requests, of node’sfetch, and even of bare playwright differs from real chrome. ja3 and ja4 fingerprints are a known signal kasada uses.if you run
await crawler.arun(url="https://kasada-protected-site.com")with default crawl4ai settings, you’ll get a blank challenge page. same with default playwright. same with selenium-with-undetected-chromedriver out of the box.the four working approaches in 2026
there are four practical paths. they range from cheapest-but-most-work to most-expensive-but-easiest.
approach 1: managed bypass apis
the easiest path. you send your target url to a service like scrapfly, zyte, or brightdata’s web unlocker. they handle the kasada challenge on their side and return the rendered page. cost is per-request, typically $1-5 per 1000 requests for kasada-protected urls.
scrapfly’s anti-scraping protection bypass:
from scrapfly import ScrapflyClient, ScrapeConfig client = ScrapflyClient(key="your-scrapfly-key") result = client.scrape(ScrapeConfig( url="https://www.canadagoose.com/some-product", asp=True, render_js=True, country="us", proxy_pool="public_residential_pool", )) print(result.content)bright data’s web unlocker:
import requests proxy = "http://brd-customer-XXX-zone-unlocker:password@brd.superproxy.io:33335" r = requests.get( "https://kasada-protected-site.com", proxies={"http": proxy, "https": proxy}, verify=False, ) print(r.text)these services hide the bypass logic. they cost real money per page but the success rate is high (90%+ on most kasada targets) and you spend zero engineering time on the cat-and-mouse.
approach 2: real browser plus residential proxy plus stealth
the diy path that mostly works. you run a real chromium via playwright, patch the bot-tells, and route through residential or mobile proxies.
import asyncio from playwright.async_api import async_playwright PROXY = { "server": "http://your-residential-endpoint:port", "username": "user", "password": "pass", } STEALTH_JS = """ Object.defineProperty(navigator, 'webdriver', {get: () => undefined}); window.chrome = { runtime: {} }; Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5]}); Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']}); """ async def main(): async with async_playwright() as p: browser = await p.chromium.launch( headless=False, proxy=PROXY, args=[ "--disable-blink-features=AutomationControlled", "--disable-features=IsolateOrigins,site-per-process", ], ) ctx = await browser.new_context( viewport={"width": 1920, "height": 1080}, user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", locale="en-US", timezone_id="America/New_York", ) await ctx.add_init_script(STEALTH_JS) page = await ctx.new_page() await page.goto("https://kasada-protected-site.com", wait_until="networkidle") await page.wait_for_timeout(3000) html = await page.content() await browser.close() print(html[:2000]) asyncio.run(main())key choices in that code:
–headless=Falseis significant. headless chromium has detectable artifacts. headed mode is harder to fingerprint. on a server, run xvfb to fake a display.
– residential or mobile proxies are non-negotiable for kasada. datacenter ips are pre-flagged.
– the stealth init script patches the most obvious bot-tells. it’s not exhaustive. for a fuller stealth bundle, look at the playwright-stealth fork or rebrowser-playwright.
–wait_until="networkidle"plus an additional 3-second wait gives the kpsdk script time to issue its tokens before you grab the page.success rate of this approach: maybe 60-75% on first attempt, depending on the specific kasada deployment and how aggressive the target site has tuned the rules.
approach 3: real browser plus rebrowser-patches plus high-quality proxies
a step up from approach 2. the rebrowser project ships patches that fix several deeper detection vectors that vanilla stealth scripts miss, including the runtime.enable bug that defeats most playwright-stealth setups in 2025-2026.
npm install rebrowser-playwrightconst { chromium } = require('rebrowser-playwright'); (async () => { const browser = await chromium.launch({ headless: false, proxy: { server: 'http://residential.example.com:8080', username: 'user', password: 'pass', }, }); const ctx = await browser.newContext({ viewport: { width: 1920, height: 1080 }, userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36', }); const page = await ctx.newPage(); await page.goto('https://kasada-protected-site.com', { waitUntil: 'networkidle' }); await page.waitForTimeout(4000); console.log((await page.content()).slice(0, 2000)); await browser.close(); })();paired with mobile proxies (carrier-grade nat ips that share legitimate user traffic), this approach pushes success rates into the 80-90% range on most kasada deployments. it’s the sweet spot if you have the engineering bandwidth.
approach 4: token harvesting
the advanced and fragile path. you reverse-engineer the kpsdk script, run it in a node-vm or v8 isolate, harvest the
x-kpsdk-ctandx-kpsdk-cdtokens, then send them with raw http requests. this is fastest at runtime (no browser overhead) but breaks every time kasada updates the script.few public tools do this reliably anymore. the kasada bypass libraries from 2023-2024 are mostly dead or paywalled. unless you have a dedicated reverse-engineering team and a tolerance for monthly breakage, skip this approach in 2026.
proxies that work and proxies that don’t
proxy choice is the second-biggest variable after browser realism. against kasada specifically:
- datacenter proxies: blocked. these are flagged by ip reputation scores before kasada even runs the challenge.
- shared residential pools (cheap providers): 30-40% success rate. lots of recycled flagged ips.
- premium residential (bright data, oxylabs, smartproxy): 60-75% success rate.
- mobile proxies (4g/5g carrier nat): 85-95% success rate. these are the gold standard because thousands of legitimate users share each ip and kasada can’t blocklist them without false-positive issues.
singapore mobile proxy and other dedicated mobile proxy providers tend to outperform bigger residential networks for these tougher targets, simply because the carrier-grade nat structure makes blocking unviable for the protected site.
geo matching matters. if your target is a us retail site, use us residential or us mobile. proxies from indonesia or russia hitting a us-only ecommerce store get extra scrutiny.
for the broader proxy landscape and which providers actually work where, see the residential proxy explainer.
kasada vs akamai vs datadome
feature kasada akamai bot manager datadome client-side js challenge yes (kpsdk) yes (sensor) yes (interstitial) proof of work yes partial rare canvas / audio fingerprint aggressive aggressive moderate ip reputation weighting very high high high typical bypass cost (managed) $$$ $$$ $$ diy success rate 60-90% (with mobile) 50-80% 70-85% kasada is harder to bypass diy than datadome but roughly comparable to akamai. the proof-of-work and the obfuscation depth are what set it apart.
a complete python recipe
putting it together. this is the script i’d actually run for a small kasada scraping job today.
import asyncio import random from playwright.async_api import async_playwright PROXIES = [ "http://user:pass@mobile-proxy-1.example.com:8000", "http://user:pass@mobile-proxy-2.example.com:8000", ] STEALTH_INIT = """ Object.defineProperty(navigator, 'webdriver', {get: () => undefined}); window.chrome = { runtime: {} }; Object.defineProperty(navigator, 'plugins', {get: () => Array(5).fill(0)}); Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']}); const getParameter = WebGLRenderingContext.prototype.getParameter; WebGLRenderingContext.prototype.getParameter = function(p) { if (p === 37445) return 'Intel Inc.'; if (p === 37446) return 'Intel Iris OpenGL Engine'; return getParameter.apply(this, arguments); }; """ async def scrape(url): proxy_url = random.choice(PROXIES) user, pwd_host = proxy_url.replace("http://", "").split("@") username, password = user.split(":") server = "http://" + pwd_host async with async_playwright() as p: browser = await p.chromium.launch( headless=False, proxy={"server": server, "username": username, "password": password}, args=["--disable-blink-features=AutomationControlled"], ) ctx = await browser.new_context( viewport={"width": 1920, "height": 1080}, user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", locale="en-US", timezone_id="America/New_York", ) await ctx.add_init_script(STEALTH_INIT) page = await ctx.new_page() try: await page.goto(url, wait_until="networkidle", timeout=45000) await page.wait_for_timeout(random.randint(3000, 6000)) await page.mouse.move(random.randint(100, 800), random.randint(100, 600)) await page.wait_for_timeout(random.randint(500, 1500)) html = await page.content() return html finally: await browser.close() async def main(): html = await scrape("https://www.example-kasada-site.com/category") print(html[:3000]) asyncio.run(main())things this script does that vanilla setups don’t:
– random mobile proxy per session
– stealth init script with webgl spoofing
– random pause and a real mouse movement before reading the dom
– locale and timezone matched to a us proxysuccess rate against typical kasada deployments with this exact script and good mobile proxies: 80%+ in my testing.
error patterns and what they mean
symptom likely cause fix 429 immediately datacenter proxy or no proxy switch to residential or mobile blank page, no html challenge running but failing check stealth init, add wait time 200 with bot interstitial navigator.webdriver detected apply stealth patches works once then 403 session burned, ip flagged rotate proxy, slow request rate works in headed, fails headless obvious headless artifacts run with xvfb on server most diy attempts fail at the second or third row. the fix is always either better stealth or better proxy.
ethics and legal
scraping sites behind kasada is technically legal in most jurisdictions if you’re collecting public data, respecting robots.txt where it applies, and not violating cfaa or computer misuse acts. the web scraping legal guide covers the nuances.
practically, sites use kasada because they don’t want bots. respect the rate limit. don’t hammer endpoints. if a site has a public api, use that instead. if a sec-or finance-related target has paid feeds, those are usually worth it.
faq
can i bypass kasada with python requests?
no. raw requests cannot run the kpsdk javascript challenge. you need a real browser or a managed bypass api.does undetected-chromedriver bypass kasada?
sometimes, against older or less-tuned deployments. against current kasada it has a 30-40% success rate at best. rebrowser-patches plus residential proxies do better.which proxy type works best for kasada?
mobile (4g/5g) carrier-grade nat proxies. residential is acceptable. datacenter is blocked.how much does it cost to scrape a kasada site?
managed bypass apis charge $1-5 per 1000 requests on kasada targets. diy with mobile proxies costs about $0.01-0.05 per request depending on your provider.will my code stop working when kasada updates?
managed apis abstract that risk. diy approaches break periodically when kasada ships major sdk updates, typically every few months.is there an open-source kasada bypass library?
not one that’s actively maintained and works reliably in 2026. the playing field has moved to managed services for the easy path and rebrowser-patches plus mobile proxies for the diy path.conclusion
kasada is hard but not impossible. the realistic 2026 stack is rebrowser-patched playwright (or chromium with deep stealth init) running headed, behind mobile or premium residential proxies, with proper geo and tls matching. that gets you to 80-90% success on most kasada sites.
if you don’t want to maintain that stack, scrapfly or bright data web unlocker handle it for you at a per-request cost. for production data pipelines that have to hit kasada-protected targets reliably, the managed route is usually cheaper than the engineer hours diy requires.
start with a managed bypass for proof-of-concept. swap in a diy stack once you know the data is worth automating long-term.
-
Rate Limit Backoff for Web Scraping: Retry Without Getting Blocked
Rate limit backoff is the difference between a scraper that recovers cleanly and a scraper that turns a small throttle into a full block. When a site returns
429 Too Many Requests,503 Service Unavailable, or a soft challenge page, the worst response is to retry immediately at the same speed. That creates a retry storm. It tells the target that your traffic is automated, overloaded, or both.This guide explains how to design backoff for web scraping: exponential delay, jitter, retry budgets, concurrency caps, and the metrics you should log before increasing volume.
What rate limit backoff means
Backoff means waiting longer between retries after a failed or throttled request. Instead of trying again instantly, the crawler slows down. The more failures it sees, the more conservative it becomes.
A simple retry loop says:
for attempt in range(3): response = fetch(url) if response.ok: breakA backoff-aware retry loop says:
for attempt in range(3): response = fetch(url) if response.ok: break sleep(delay_for(attempt))That looks like a small change. At scale, it is a major reliability control.
Why immediate retries are dangerous
Immediate retries create three problems:
- They increase load. A temporary slowdown becomes more traffic, not less.
- They cluster requests. Many workers fail at the same time, then retry at the same time.
- They damage reputation. Your IP, account, session, or fingerprint can get classified as abusive.
This is why retry behavior belongs in your anti-blocking strategy, not just in your error-handling code. If you are seeing frequent
429responses, read our 429 Too Many Requests guide first, then implement backoff.The core backoff formula
The most common pattern is exponential backoff:
delay = min(max_delay, base_delay * (2 ** attempt))For example, with a base delay of 2 seconds and a max delay of 60 seconds:
- attempt 1 waits 2 seconds
- attempt 2 waits 4 seconds
- attempt 3 waits 8 seconds
- attempt 4 waits 16 seconds
- attempt 5 waits 32 seconds
- later attempts cap at 60 seconds
Do not leave the delay uncapped. A crawler that sleeps for hours inside a worker can break scheduling and hide failures.
Add jitter to avoid synchronized retries
Jitter means adding randomness to the delay. Without jitter, hundreds of workers can retry at the same time. With jitter, they spread out.
import random import time def backoff_delay(attempt, base=2, cap=60): exponential = min(cap, base * (2 ** attempt)) return random.uniform(exponential * 0.5, exponential) for attempt in range(5): delay = backoff_delay(attempt) time.sleep(delay)This is often called full jitter or bounded jitter. The exact formula matters less than the principle: do not make every worker retry on the same schedule.
Respect Retry-After when it exists
Some servers return a
Retry-Afterheader. If present, use it as a strong signal.def parse_retry_after(response): value = response.headers.get("Retry-After") if not value: return None try: return min(int(value), 300) except ValueError: return NoneCap the value so one strange response does not pause your entire job forever. But if a site tells you to wait, waiting is usually cheaper than burning proxies and sessions.
Use retry budgets, not infinite retries
A retry budget limits how much extra traffic your crawler can create because of failures. For example:
- no more than 2 retries per URL
- no more than 10 percent retry traffic per target per hour
- no retries for permanent errors like
404or410 - only one retry for
403unless the block reason is known
This prevents a broken target from consuming the whole queue. It also keeps your monitoring honest. A crawler that succeeds after ten retries is not healthy. It is hiding a rate problem.
Separate retry logic by error type
Do not treat every failure the same way.
Signal Likely meaning Suggested action 429 rate limited slow down, reduce concurrency, honor Retry-After 503 overload or temporary defense backoff, retry later, watch response body 403 blocked or unauthorized do not hammer; inspect fingerprint, cookies, and proxy 408 or timeout network delay retry with jitter, maybe change proxy after budget 404 missing page usually do not retry For more status-specific handling, use our proxy error code reference.
Python example with requests
import random import time import requests RETRYABLE = {408, 429, 500, 502, 503, 504} def compute_delay(response, attempt, base=2, cap=60): retry_after = response.headers.get("Retry-After") if response else None if retry_after and retry_after.isdigit(): return min(int(retry_after), 300) exp = min(cap, base * (2 ** attempt)) return random.uniform(exp * 0.5, exp) def fetch_with_backoff(url, *, max_attempts=4, timeout=20): last_response = None for attempt in range(max_attempts): try: response = requests.get(url, timeout=timeout) if response.status_code not in RETRYABLE: return response last_response = response except requests.RequestException: response = None if attempt == max_attempts - 1: break time.sleep(compute_delay(response, attempt)) return last_responseThis example is intentionally simple. In production, you should also log target, proxy, status code, response size, retry count, and final outcome.
Backoff is not a substitute for concurrency control
If you keep sending too many first attempts, backoff only reduces the damage after failures. You still need concurrency caps.
Set limits at multiple levels:
- global crawler concurrency
- per-domain concurrency
- per-proxy concurrency
- per-account or per-session concurrency
- per-endpoint concurrency for sensitive paths
A common pattern is to lower concurrency when the error rate rises. For example, if a target’s
429rate exceeds 5 percent over the last 10 minutes, cut concurrency by half and slowly recover later.What to log
Backoff without metrics becomes guesswork. At minimum, log:
- URL or endpoint group
- proxy ID or pool name
- status code
- attempt number
- delay used
- whether
Retry-Afterwas present - response byte size
- block or challenge detection result
These logs show whether you have a target problem, proxy problem, fingerprint problem, or scheduler problem. They also make it easier to build dashboards like the ones in our web scraper monitoring guide.
Bottom line
Good retry logic is polite, measurable, and bounded. Use exponential backoff with jitter, honor
Retry-After, cap retries with budgets, and reduce concurrency when error rates rise. The goal is not to force every URL through. The goal is to collect data steadily without teaching the target that your crawler is a retry storm. -
Best News APIs Compared: 12 Options for Developers in 2026
Best News APIs Compared: 12 Options for Developers in 2026
whether you are building a news aggregator, monitoring brand mentions, feeding data into an AI model, or tracking industry trends, a news API saves you from scraping hundreds of news sites yourself. but with over a dozen options on the market, choosing the right one matters.
looking for premium 4G/5G IPs? our Singapore mobile proxies for news scraping start at $40/month for 200GB.
this comparison covers the 12 most popular news APIs available in 2026. we tested each one for source coverage, data quality, pricing transparency, and developer experience.
Quick Comparison Table
API Free Tier Starting Price Sources Historical Data Best For NewsAPI.org 100 req/day $449/mo 150K+ 1 month prototyping GNews 100 req/day $84/mo 60K+ none (free) budget projects NewsCatcher 100 req/day custom 70K+ 5 years research Bing News Search 1K/mo $3/1K calls broad 30 days Microsoft ecosystem Google News API none custom broad varies enterprise Mediastack 500 req/mo $9.99/mo 7,500+ none simple integration TheNewsAPI 3 req/day $49/mo 55K+ 6 months content apps GDELT free free 100+ countries 45 years academic research Event Registry limited $600/mo 300K+ 10+ years enterprise analytics Currents API 600 req/day custom 22K+ none small apps Perigon 50 req/day $99/mo 40K+ 3 years AI training data Webz.io (formerly Webhose) limited custom 2M+ 10+ years massive scale 1. NewsAPI.org
best for: quick prototyping and hobby projects
NewsAPI.org is the most well-known news API and often the first one developers try. it provides access to headlines and articles from over 150,000 online sources.
strengths: – simple REST API with excellent documentation – fast response times under 200ms – supports searching by keyword, source, language, and country – provides article metadata including author, published date, and image URL
weaknesses: – free tier is extremely limited (100 requests/day, no commercial use) – paid plans start at $449/month, which is steep for small projects – historical data limited to 1 month on most plans – no full article text on free tier (only titles and descriptions)
example request:
import requests API_KEY = "your_newsapi_key" response = requests.get( "https://newsapi.org/v2/everything", params={ "q": "web scraping", "language": "en", "sortBy": "publishedAt", "pageSize": 10, "apiKey": API_KEY, }, ) data = response.json() for article in data["articles"]: print(f"{article['title']}") print(f" source: {article['source']['name']}") print(f" published: {article['publishedAt']}") print()pricing: free tier (100 req/day dev only), Business $449/mo, Enterprise custom
2. GNews
best for: budget-conscious developers
GNews offers a clean, straightforward API at a fraction of the cost of NewsAPI.org. it pulls from Google News and provides access to over 60,000 sources.
strengths: – cheapest paid option starting at $84/month – free tier is generous for testing (100 requests/day) – simple query syntax – multilingual support (38 languages)
weaknesses: – smaller source pool than some competitors – no historical archive on free tier – limited filtering options compared to premium APIs – rate limiting can be aggressive on lower tiers
example request:
response = requests.get( "https://gnews.io/api/v4/search", params={ "q": "proxy industry", "lang": "en", "max": 10, "token": "your_gnews_key", }, ) for article in response.json()["articles"]: print(f"{article['title']} - {article['source']['name']}")pricing: free (100 req/day), Basic $84/mo, Pro $279/mo
3. NewsCatcher
best for: research and analytics
NewsCatcher differentiates itself with deep search capabilities and long historical archives. it is particularly popular with researchers, data scientists, and competitive intelligence teams.
strengths: – 5-year historical archive – NLP-powered features like topic classification, entity extraction, and sentiment analysis – 70,000+ sources with strong international coverage – clustering of related articles
weaknesses: – no published pricing (sales-driven) – free tier is limited to 100 requests per day – response times can be slower than simpler APIs due to NLP processing – steeper learning curve for advanced features
example request:
headers = {"x-api-key": "your_newscatcher_key"} response = requests.get( "https://v3-api.newscatcherapi.com/api/search", headers=headers, params={ "q": "artificial intelligence data collection", "lang": "en", "from_": "2025-01-01", "to_": "2026-03-01", "page_size": 10, }, ) data = response.json() for article in data["articles"]: print(f"{article['title']}") print(f" sentiment: {article.get('sentiment', 'N/A')}") print(f" topic: {article.get('topic', 'N/A')}")pricing: free tier available, paid plans require contacting sales
4. Bing News Search API
best for: Microsoft ecosystem integration
part of Microsoft’s Azure Cognitive Services, the Bing News Search API provides access to Bing’s news index with enterprise-grade reliability.
strengths: – backed by Microsoft’s infrastructure – excellent for trending topics and breaking news – supports category-based browsing – integrates well with other Azure services
weaknesses: – results are biased toward English and US sources – requires an Azure subscription – limited historical data (30 days) – pricing based on transactions makes costs unpredictable at scale
example request:
headers = {"Ocp-Apim-Subscription-Key": "your_bing_key"} response = requests.get( "https://api.bing.microsoft.com/v7.0/news/search", headers=headers, params={ "q": "data privacy regulation", "count": 10, "mkt": "en-US", "freshness": "Week", }, ) for article in response.json()["value"]: print(f"{article['name']}") print(f" provider: {article['provider'][0]['name']}")pricing: free (1K transactions/mo), S1 $3/1K transactions
5. Mediastack
best for: simple integration without complexity
Mediastack provides a no-frills news API that is easy to set up and affordable. it is a good choice if you need basic news data without advanced features.
strengths: – starts at $9.99/month, making it one of the cheapest options – live news data from 7,500+ sources in 50+ countries – supports 13 languages – straightforward REST API
weaknesses: – no NLP features (sentiment, entity extraction) – limited source pool compared to premium APIs – no historical archive on lower tiers – HTTPS only available on paid plans
example request:
response = requests.get( "http://api.mediastack.com/v1/news", params={ "access_key": "your_mediastack_key", "keywords": "web scraping", "languages": "en", "limit": 10, }, ) for article in response.json()["data"]: print(f"{article['title']} ({article['source']})")pricing: free (500 req/mo), Basic $9.99/mo, Standard $49.99/mo
6. TheNewsAPI
best for: content applications
TheNewsAPI focuses on providing clean, well-structured article data suitable for content applications. it deduplicates content and provides good categorization.
strengths: – strong deduplication removes repeat stories – category-based browsing – sentiment analysis included – 6-month historical archive
weaknesses: – very limited free tier (3 requests/day) – paid plans start at $49/month – source pool is smaller than top-tier options
pricing: free (3 req/day), Basic $49/mo, Pro $149/mo
7. GDELT
best for: academic research and global event monitoring
GDELT (Global Database of Events, Language, and Tone) is a free, open dataset that monitors news from every country. it is not a traditional API but a massive data platform.
strengths: – completely free – covers 100+ countries and dozens of languages – historical data going back to 1979 – real-time monitoring with 15-minute update cycles – includes geolocation, sentiment, themes, and entity data
weaknesses: – steep learning curve – raw data requires significant processing – API rate limits can be restrictive – documentation is scattered and sometimes outdated
example request:
# GDELT DOC API response = requests.get( "https://api.gdeltproject.org/api/v2/doc/doc", params={ "query": "proxy server", "mode": "ArtList", "maxrecords": 10, "format": "json", }, ) for article in response.json().get("articles", []): print(f"{article['title']}") print(f" tone: {article.get('tone', 'N/A')}") print(f" domain: {article['domain']}")pricing: free
8. Event Registry
best for: enterprise-grade news analytics
Event Registry aggregates news from over 300,000 sources and clusters articles into events. it is designed for enterprise analytics use cases.
strengths: – largest source pool (300K+ sources) – event-based clustering groups related articles – 10+ years of historical data – advanced analytics including topic trends, entity tracking, and media monitoring
weaknesses: – expensive (starting at $600/month) – complex API with many parameters – overkill for simple news integration
pricing: starts at $600/mo
9. Currents API
best for: small applications and side projects
Currents API provides a simple, generous free tier that works well for small applications.
strengths: – generous free tier (600 requests/day) – 22,000+ sources – simple API design – no credit card required for free tier
weaknesses: – limited documentation – no historical archive – fewer features than premium APIs – data quality can be inconsistent
pricing: free (600 req/day), paid plans available
10. Perigon
best for: AI and ML training data
Perigon is designed specifically for feeding news data into AI models. it provides clean, structured data with rich metadata.
strengths: – designed for AI/ML pipelines – 3-year historical archive – entity extraction and topic classification built in – content clustering and deduplication – structured JSON output optimized for data processing
weaknesses: – relatively new entrant – smaller source pool than the largest competitors – free tier limited to 50 requests/day
example request:
headers = {"x-api-key": "your_perigon_key"} response = requests.get( "https://api.goperigon.com/v1/all", headers=headers, params={ "q": "machine learning proxy", "from": "2025-06-01", "size": 10, "showReprints": "false", }, ) for article in response.json()["articles"]: print(f"{article['title']}") print(f" topics: {[t['name'] for t in article.get('topics', [])]}")pricing: free (50 req/day), Starter $99/mo, Growth $499/mo
11. Webz.io
best for: massive scale data collection
Webz.io (formerly Webhose) provides access to structured web data from over 2 million sources, including news, blogs, forums, and reviews.
strengths: – enormous source pool (2M+ sources) – 10+ years of historical data – covers news, blogs, forums, and dark web – high throughput for bulk data extraction – real-time streaming option
weaknesses: – expensive (enterprise pricing) – complex pricing model – overkill for simple news needs
pricing: custom enterprise pricing
12. Google News API (Custom Search)
best for: Google News results in your application
Google does not offer a dedicated news API, but you can use the Custom Search JSON API configured for news to get Google News results programmatically.
strengths: – Google’s news ranking quality – broad source coverage – works with Google Cloud billing
weaknesses: – limited to 100 queries/day on free tier – $5 per 1K queries after that – not a true news API (it is a search API) – limited metadata compared to dedicated news APIs
pricing: free (100 queries/day), $5/1K queries
How to Choose
By Budget
- free: GDELT, Currents API
- under $50/mo: Mediastack, TheNewsAPI
- $50-200/mo: GNews, Perigon
- $200-500/mo: NewsAPI.org
- $500+: Event Registry, NewsCatcher, Webz.io
By Use Case
- news aggregator app: NewsAPI.org or GNews
- brand monitoring: NewsCatcher or Event Registry
- AI training data: Perigon or Webz.io
- academic research: GDELT (free and deep historical data)
- side project: Currents API or Mediastack
- enterprise analytics: Event Registry or Webz.io
By Technical Requirements
- best documentation: NewsAPI.org
- best historical data: GDELT or Event Registry
- best NLP features: NewsCatcher or Perigon
- best free tier: Currents API or GDELT
- fastest response time: NewsAPI.org or Bing News
Building a News Pipeline with Proxies
if none of these APIs fully meet your needs, you can build your own news collection pipeline. use proxies to scrape RSS feeds and news sites directly:
import feedparser import requests from datetime import datetime def collect_news_from_rss(feeds, proxy_url=None): """collect news from RSS feeds with proxy support.""" proxies = {} if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} articles = [] for feed_url in feeds: try: response = requests.get( feed_url, proxies=proxies, timeout=15 ) feed = feedparser.parse(response.content) for entry in feed.entries: articles.append({ "title": entry.get("title", ""), "url": entry.get("link", ""), "published": entry.get("published", ""), "summary": entry.get("summary", ""), "source": feed.feed.get("title", feed_url), }) except Exception as e: print(f"error fetching {feed_url}: {e}") return articles # example: collect from major tech news RSS feeds tech_feeds = [ "https://techcrunch.com/feed/", "https://feeds.arstechnica.com/arstechnica/index", "https://www.theverge.com/rss/index.xml", "https://feeds.feedburner.com/venturebeat/SZYF", ] articles = collect_news_from_rss( tech_feeds, proxy_url="http://user:pass@proxy.provider.com:8080", )Conclusion
the best news API depends on your specific requirements. for most developers starting out, GNews or Currents API offer the best value with reasonable free tiers. for enterprise use cases that need deep historical data and NLP features, NewsCatcher or Event Registry are worth the investment. for AI and ML applications, Perigon is purpose-built for that workflow.
if your needs go beyond what any single API offers, consider combining a news API for broad coverage with targeted RSS scraping through proxies for specific sources that matter most to your use case.