Author: Xavier Fok

Best CAPTCHA solving services 2026 ranked

Best CAPTCHA solvers in 2026 face a market where the underlying CAPTCHA technology has gotten harder. reCAPTCHA v3 (invisible scoring) has displaced reCAPTCHA v2 in most production deployments. hCaptcha has become the default alternative for sites avoiding Google. Cloudflare Turnstile has taken meaningful market share since its 2023 launch. AWS WAF and DataDome have grown their custom challenge layers. Each of these systems has different bypass mechanics and the solver services have specialized accordingly. The right service for your workload depends on which CAPTCHA you actually face most often.

This guide ranks the CAPTCHA solving services that actually work in 2026, with honest accuracy numbers, real pricing per CAPTCHA type, and the specific technical constraints that determine which service fits your scraping pipeline.

How CAPTCHA solving services work

A CAPTCHA solving service is an API that accepts an unsolved CAPTCHA (image, sitekey, or challenge data) and returns a solved response token. The service handles the actual solving on its end, either by routing to a human worker (the original 2Captcha model) or by running ML models that defeat the CAPTCHA programmatically (the modern model since 2022).

For your scraper, the integration is the same: you encounter a CAPTCHA, send the relevant data to the solver API, wait for the response (typically 5-60 seconds), then submit the response token to the target site as if you had solved it manually.

The pricing varies by CAPTCHA type. Image CAPTCHAs are cheapest. reCAPTCHA v2 is moderate. reCAPTCHA v3 costs more because it requires real browser execution to harvest the token. Turnstile costs the most because the solver has to defeat Cloudflare’s full fingerprinting layer.

What we measured

For each service we ran 1000 attempts per CAPTCHA type across reCAPTCHA v2, reCAPTCHA v3, hCaptcha, and Cloudflare Turnstile. Success rate is the percentage of attempts that produced a valid token accepted by the target site. Average solve time is the median time from API submission to token return. Pricing is the actual rate per 1000 solves at standard tiers.

1. CapSolver

CapSolver has emerged as the leader for ML-based CAPTCHA solving. They support every major CAPTCHA type with strong accuracy across the board. Pricing is competitive and the API is well-designed.

Success rates in our testing: reCAPTCHA v2 at 95%, reCAPTCHA v3 at 88%, hCaptcha at 92%, Turnstile at 84%. Average solve time around 15 seconds.

Pricing per 1000 solves: reCAPTCHA v2 around $1.50, v3 around $2.50, hCaptcha around $2, Turnstile around $4.

Best for: production scraping at scale, multi-CAPTCHA-type workloads, anyone who needs the highest reliability across all major CAPTCHA types.

2. 2Captcha

2Captcha is the elder statesman of the market, originally a human-solver service. They have augmented with ML for the high-volume CAPTCHA types but human workers still handle a meaningful share of solves.

Success rates: reCAPTCHA v2 at 93%, reCAPTCHA v3 at 80%, hCaptcha at 85%, Turnstile at 70%. Solve times are slower (20-45 seconds for human-solved cases). Pricing is the most consistent in the market.

Pricing per 1000 solves: reCAPTCHA v2 at $1, v3 at $3, hCaptcha at $1.50, Turnstile at $4.

Best for: image CAPTCHAs and reCAPTCHA v2 where the cost-quality balance is best, established API integrations.

3. Anti-Captcha

Anti-Captcha has been a 2Captcha competitor for years with similar positioning. Hybrid human+ML model. Success rates and pricing are within 5-10% of 2Captcha across the board. Their API is very developer-friendly.

Best for: 2Captcha alternatives, customers who hit 2Captcha rate limits or want vendor diversity.

4. NopeCHA

NopeCHA started as a browser extension and grew into a full API service. They specialize in reCAPTCHA v3 and Turnstile solving via browser automation under the hood. Pricing is per-token rather than per-attempt, which fits some workloads better.

Success rates: reCAPTCHA v3 at 85%, Turnstile at 80%. They are weaker on pure image CAPTCHAs.

Best for: workloads dominated by reCAPTCHA v3 or Turnstile where their per-token pricing model fits.

5. DeathByCaptcha

One of the oldest services in the space, mostly human-solver. Strong on image CAPTCHAs and reCAPTCHA v2. Weaker on modern challenges.

Success rates: image CAPTCHA at 95%, reCAPTCHA v2 at 90%, reCAPTCHA v3 at 65% (struggles with this), hCaptcha at 75%.

Pricing per 1000: image at $1.40, v2 at $1.40, v3 at $5 (expensive due to lower success), hCaptcha at $2.

Best for: legacy image CAPTCHA workloads, simple use cases where reliability of older types matters more than modern types.

6. CapMonster

CapMonster is a self-hosted ML solver from the Bablosoft team (also makers of BAS). You run it on your own GPU server. After paying the license fee (one-time around $300 for CapMonster Cloud, or ongoing for the SaaS version), per-CAPTCHA cost is essentially zero.

The catch: ongoing accuracy depends on you keeping the models updated, and Cloudflare-style challenges break the local solver more often than the cloud-based services.

Best for: very high volume operations where the per-CAPTCHA economics of cloud services break down.

7. AYCD AutoSolve

AYCD targets the sneaker bot community specifically with low-latency Turnstile and DataDome solving. Pricing is subscription-based ($35-50/month for limited solves). Accuracy on their target use case (drop-day sneaker checkouts) is strong.

Best for: sneaker botting, real-time checkout flows where every millisecond matters.

8. CapSolver alternatives in the long tail

A handful of smaller services (SolveCaptcha, EndCaptcha, ImageTyperz, BypassCaptcha) compete on price and serve specific niches. None lead in accuracy across all CAPTCHA types but they can be cost-effective for specific use cases. Test before committing.

Comparison table

service	reCAPTCHA v2	reCAPTCHA v3	hCaptcha	Turnstile	image CAPTCHA	avg solve time	pricing model
CapSolver	95%	88%	92%	84%	90%	15s	per-solve, $1.50-4 per 1000
2Captcha	93%	80%	85%	70%	95%	20-45s	per-solve, $1-4 per 1000
Anti-Captcha	92%	79%	84%	68%	94%	20-40s	per-solve, $1-4 per 1000
NopeCHA	80%	85%	75%	80%	70%	20s	per-token
DeathByCaptcha	90%	65%	75%	60%	95%	25-50s	per-solve, $1.40-5 per 1000
CapMonster	88%	75%	80%	70%	90%	8s (local)	license + minimal per-solve
AYCD AutoSolve	n/a	70%	n/a	90%	n/a	3s	subscription

For a general-purpose scraping pipeline, CapSolver gives the best balance across the modern CAPTCHA types. For legacy reCAPTCHA v2 workloads, 2Captcha is more cost-effective. For self-hosted scale, CapMonster wins.

Decision matrix: solopreneur, SMB, enterprise

profile	volume	recommended primary	secondary	reasoning
Solopreneur testing	<500/day	2Captcha	CapSolver trial	Lowest entry cost, pay-as-you-go
Indie scraper	500-5,000/day	CapSolver	2Captcha fallback	Best modern accuracy, vendor diversity
SMB scraping ops	5,000-50,000/day	CapSolver	Anti-Captcha	Negotiated tier, multi-vendor failover
High-volume continuous	50,000+/day	CapMonster self-hosted	CapSolver burst	Self-hosting wins on per-solve marginal cost
Sneaker / drop tooling	event-bound	AYCD	CapSolver	Latency over throughput
Single-CAPTCHA-type workload (image only)	any	DeathByCaptcha	2Captcha	Specialist providers cheaper for narrow types
Enterprise compliance	any	CapSolver Enterprise	2Captcha Enterprise	Audit logs, dedicated support

The biggest mistake at the SMB tier is over-relying on one vendor. CAPTCHA providers occasionally suffer accuracy drops when target sites roll out new challenge variants. A two-vendor failover (primary + secondary) costs 5-10% more in setup overhead and saves entire days of downtime when the primary’s accuracy crashes from 90% to 40% overnight.

Migration path: cloud to self-hosted

The transition from cloud solvers to CapMonster self-hosted is the most common scaling step. The break-even sits around 30,000-50,000 solves/day depending on CAPTCHA mix. The migration playbook:

Audit your CAPTCHA mix. Self-hosted solvers handle reCAPTCHA v2, v3, and image CAPTCHAs well. Turnstile and DataDome remain harder to handle locally; keep these on cloud solvers initially.
Provision GPU capacity. A single RTX 3060 or A4000 handles ~10,000 solves/day comfortably. For higher throughput, scale horizontally with multiple workers behind a load balancer.
Run cloud and self-hosted in parallel for two weeks. Compare per-CAPTCHA accuracy on identical inputs. Self-hosted should match cloud within 3-5% for v2 and image CAPTCHAs.
Cut over by CAPTCHA type rather than all at once. Move v2 first (highest local accuracy), then v3, leave Turnstile on cloud until self-hosted matches.
Budget for model updates. Self-hosted solvers degrade as CAPTCHA providers adjust. Plan a quarterly model refresh from the vendor; treat it as ongoing operational cost.

Pricing reality at volume

The per-1000 pricing scales somewhat linearly until very high volumes (above 100k solves/day), at which point negotiated enterprise pricing drops 30-50%. For a typical small operation doing 10,000 solves per day:

All-CapSolver mix (mostly v2 + v3): $20-30/day = $600-900/month
All-2Captcha mix: $15-25/day = $450-750/month
CapMonster self-hosted: $50-150/month for cloud or ~$0 marginal cost

For high-volume operations (50k+ solves/day), CapMonster self-hosted becomes dramatically cheaper. For small-volume operations, the cloud services are simpler and the cost difference does not justify the operational overhead.

Integration patterns

The standard API flow:

import requests
import time

CAPSOLVER_API_KEY = "..."

def solve_recaptcha_v2(sitekey: str, page_url: str) -> str:
    create_resp = requests.post(
        "https://api.capsolver.com/createTask",
        json={
            "clientKey": CAPSOLVER_API_KEY,
            "task": {
                "type": "ReCaptchaV2TaskProxyless",
                "websiteURL": page_url,
                "websiteKey": sitekey,
            },
        },
        timeout=10,
    )
    task_id = create_resp.json()["taskId"]

    while True:
        time.sleep(3)
        result_resp = requests.post(
            "https://api.capsolver.com/getTaskResult",
            json={"clientKey": CAPSOLVER_API_KEY, "taskId": task_id},
            timeout=10,
        )
        result = result_resp.json()
        if result["status"] == "ready":
            return result["solution"]["gRecaptchaResponse"]
        if result["status"] == "failed":
            raise CaptchaSolveError(result.get("errorDescription"))


def solve_turnstile(sitekey: str, page_url: str, action: str = None) -> str:
    create_resp = requests.post(
        "https://api.capsolver.com/createTask",
        json={
            "clientKey": CAPSOLVER_API_KEY,
            "task": {
                "type": "AntiTurnstileTaskProxyless",
                "websiteURL": page_url,
                "websiteKey": sitekey,
                "metadata": {"action": action} if action else None,
            },
        },
        timeout=10,
    )
    task_id = create_resp.json()["taskId"]
    while True:
        time.sleep(3)
        result_resp = requests.post(
            "https://api.capsolver.com/getTaskResult",
            json={"clientKey": CAPSOLVER_API_KEY, "taskId": task_id},
            timeout=10,
        )
        result = result_resp.json()
        if result["status"] == "ready":
            return result["solution"]["token"]
        if result["status"] == "failed":
            raise CaptchaSolveError(result.get("errorDescription"))

The two-step pattern (create task, poll for result) is universal across services. 2Captcha, Anti-Captcha, and the others use the same shape with slightly different field names.

Cost worked example

A small operation with 5,000 daily CAPTCHA encounters split as 60% v2, 25% v3, 10% hCaptcha, 5% Turnstile sees these monthly bills:

Pure CapSolver: 3000 v2 ($4.50) + 1250 v3 ($3.13) + 500 hCaptcha ($1) + 250 Turnstile ($1) = ~$9.63/day = $289/month
Pure 2Captcha: 3000 v2 ($3) + 1250 v3 ($3.75) + 500 hCaptcha ($0.75) + 250 Turnstile ($1) = ~$8.50/day = $255/month, but lower v3 and Turnstile success rates result in retries that add 25-30%
Hybrid (CapSolver + 2Captcha fallback): ~$310/month all-in with 96% effective success rate after fallback
CapMonster self-hosted ($300 license + $30/mo cloud): ~$50/month after license amortization, but 5-7% lower accuracy on Turnstile

The hybrid approach trades a small cost premium for materially higher effective success rate and vendor resilience. For mission-critical scrapers (account creation, payment flows), the hybrid is worth every cent. For best-effort enrichment scrapes, pure cheapest-vendor is fine.

When to use a CAPTCHA solver vs a different approach

Solvers are not always the right answer. Three alternatives to consider first:

Better proxies and fingerprinting. Many CAPTCHAs trigger because of bot signals before the CAPTCHA itself: bad IP, missing browser fingerprint, suspicious headers. Improving these reduces CAPTCHA frequency by 50-90%, which is more cost-effective than solving every one.

Session reuse. A successful CAPTCHA solve typically grants a cookie or token that holds for hours or days. Reusing that session across many requests is cheaper than solving fresh CAPTCHAs.

Headless browsers with stealth plugins. For reCAPTCHA v3 specifically, a properly-configured headless browser running on a clean residential IP often passes the score check without needing a solver at all. We cover this in Cloudflare Turnstile bypass tactics in 2026.

For workloads where you genuinely cannot avoid CAPTCHAs (account creation, sneaker drops, certain SERP scraping), solvers are essential.

Tracking solver health

Treat CAPTCHA solving as a real-time SLA-driven service. The metrics worth tracking continuously:

Per-vendor success rate rolling 1h, 24h, and 7d windows, broken down by CAPTCHA type
Average solve time with p50, p95, p99 percentiles
Cost per successful solve computed daily, alerting when it drifts more than 20% from baseline
Failed-solve reason distribution (token rejected, timeout, invalid sitekey, etc.) to catch upstream changes
Concurrent in-flight count to confirm you are not throttled by the vendor’s concurrency cap
Token age at submission to verify your downstream pipeline submits before the 120-second expiry

Wire these into your existing observability stack (Prometheus + Grafana, or whatever you use). A single dashboard showing all five metrics across two solver vendors lets you make routing decisions in real time instead of after a multi-hour outage.

CAPTCHA type by target site

target	CAPTCHA used (2026)	best solver
Google services	reCAPTCHA v3	CapSolver
Cloudflare-protected	Turnstile	CapSolver, AYCD
Discord	hCaptcha	CapSolver
Twitch	reCAPTCHA v2	2Captcha
Indeed	hCaptcha	CapSolver
LinkedIn	custom + reCAPTCHA v2	2Captcha + warm proxies
Amazon	custom + reCAPTCHA v2	2Captcha + residential
ticket sales (Ticketmaster)	DataDome + custom	AYCD
sneaker sites	varies, often Turnstile	AYCD

The “best” column reflects which solver has highest accuracy on that specific CAPTCHA in that specific context. Test against your own target before committing.

Common gotchas

Sitekey confusion. A sitekey is per-page and changes occasionally. Hardcoding the sitekey in your scraper breaks silently when the target rotates it. Always extract the sitekey fresh from the page DOM at solve time.
Action mismatch on v3. reCAPTCHA v3 actions like “submit”, “login”, “checkout” are validated server-side. Submitting a token solved with the wrong action returns a token but the target rejects it. Always pass the correct action string to the solver.
Token expiry. Solved tokens are valid for ~120 seconds. If your downstream submission takes longer than that (e.g., a slow form), the token expires and you waste the solve. Solve immediately before submission, not in advance.
IP mismatch on Turnstile. Cloudflare validates that the IP submitting the form matches the IP that solved the CAPTCHA. If you solve on the solver’s IP and submit from your scraper’s IP, validation fails. Use proxyless solver modes that take an IP/proxy parameter when this matters.
Hidden refresh on failure. Some sites silently re-issue a CAPTCHA when a token fails validation. Your scraper sees a 200 OK with the form re-rendered and assumes success. Always check for the post-submission state, not just the HTTP status.
Concurrent solve limits. Free tiers and starter plans cap concurrent in-flight solves. A burst that exceeds the cap returns errors. Negotiate higher concurrency before launch if your workload is bursty.
Hidden retries in solver dashboards. Some solvers report “success” after multiple internal retries on the same submission. Your dashboard shows 95% success rate but you are billed for 1.5x the visible solves. Always reconcile billing against your own submission count.

What to skip

“Free” CAPTCHA solvers that ask you to install software: these are usually malware or worker farms harvesting your CAPTCHA budget for resale. Stick to API services with transparent pricing.

Promising 99% accuracy on Turnstile: nobody has 99% on Turnstile in 2026. Vendors making this claim are either lying or measuring success on conditions you do not have.

Per-page subscriptions for general-purpose use: the per-solve model is more honest and lets you scale costs with actual usage.

External authoritative reference: Google’s reCAPTCHA developer documentation covers the official integration the solver services bypass.

FAQ

Q: is using a CAPTCHA solver legal?
The legal status depends on jurisdiction and target site terms of service. Using a solver to scrape public data is generally legal but may violate the target site’s ToS. Using a solver to commit fraud or unauthorized access is not legal.

Q: how do I reduce CAPTCHA frequency?
Use better proxies (residential or mobile), maintain consistent browser fingerprints, reuse sessions, slow your request rate, and warm accounts before scraping with them. CAPTCHAs are the symptom; bot signals are the cause.

Q: do CAPTCHAs work on mobile apps?
Yes, and the bypass is harder than web because the app makes additional integrity checks. CAPTCHA solvers for mobile-app contexts are a niche service category.

Q: how do I handle reCAPTCHA v3 score requirements?
v3 returns a score 0.0-1.0. Most sites accept 0.5+. Solvers can target a minimum score (CapSolver lets you specify); a higher target costs more.

Q: what about Audio CAPTCHAs?
Most services support audio CAPTCHAs as a fallback for reCAPTCHA v2. Accuracy is similar to image solving. Some sites disable the audio option.

Q: how do I monitor solver accuracy in production?
Track per-CAPTCHA-type success rate as a rolling 24-hour metric, broken down by target site. A sudden drop (e.g., from 92% to 60% over 6 hours) is a strong signal that either the target rolled out a new variant or your solver provider degraded. Auto-failover to your secondary provider when the rolling rate falls below a threshold.

Q: do solvers handle proxyless mode?
Most do, but proxy-aware mode is more reliable for sites that fingerprint the solver IP. If you have proxies available, pass them to the solver. The cost is identical.

Q: can I batch CAPTCHA solves?
Most APIs are per-task. Batching is implemented client-side by submitting many tasks in parallel and awaiting all. The solver’s concurrency limit is the actual constraint, not the API shape.

Q: do solver services log my target URLs?
Yes, almost universally for billing and abuse mitigation. If your target URL is sensitive (e.g., contains a session token), strip query parameters before submitting. Most solvers care only about the origin and sitekey, not the full URL.

Closing

CAPTCHA solving in 2026 is a mature market dominated by CapSolver for modern challenges, 2Captcha for legacy workloads, and CapMonster for self-hosted scale. The biggest cost savings come not from picking a cheaper solver but from reducing CAPTCHA frequency through better proxies, fingerprinting, and session reuse. Treat solvers as a last resort, not a first defense. For broader anti-bot strategy see our anti-bot-captcha category hub.

May 7, 2026

Best Multi-Account Browsers for Facebook in 2026

Facebook rarely bans a setup because you opened five tabs, it bans because your operating pattern looks stitched together from conflicting devices, IPs, timezones, and browser fingerprints. that is why choosing the right multi-account browser for Facebook matters more in 2026 than picking the cheapest proxy list. if you manage ad accounts, agency clients, warm backup profiles, or research identities, the practical question is not “can this browser open separate profiles”, it is “can this stack keep each profile coherent enough to survive scrutiny while staying usable for a team”.

Why Facebook kills multi-account setups

Facebook enforcement has become less about raw account count and more about correlation. one laptop can legitimately run multiple business assets, but the graph gets suspicious when ten advertising profiles share the same canvas entropy, WebGL signature, font pack, timezone drift, and IP neighborhood.

That is the fingerprinting problem. anti-detect browsers are not magic invisibility layers, they are profile isolation systems. the better ones give each browser profile a persistent, internally consistent environment so Facebook sees something that behaves like one stable machine, not a recycled automation shell.

The common failure modes are predictable:

one residential proxy pool shared across unrelated accounts
fingerprints regenerated too often
local time, language, and IP geolocation not matching
browser profiles synced badly across teammates
aggressive extensions leaking cross-profile behavior

The bigger mistake is treating Facebook like a login gate instead of a trust system. trust accumulates through repeated coherence. if profile A logs in from a Chicago residential IP, runs an English-US locale, and always opens from the same device profile, that can age normally. if the same account appears twelve hours later from a German mobile ASN with a new canvas fingerprint and mismatched timezone, review risk jumps.

For a broader benchmark, the best starting point is Best Anti-Detect Browsers for Facebook 2026: 8 Tools Tested. the takeaway from most serious tests is simple: profile consistency beats feature sprawl.

Top multi-account browsers compared

For Facebook advertising work, the market has mostly converged around five names. they all isolate cookies and local storage, but they differ on browser engine freshness, cloud sync quality, and how painful team operations become at scale.

tool	browser engine	profile cloud sync	team seats pricing	free tier
GoLogin	Chromium-based Orbita	strong, simple cross-device sync	mid-range, team plans easy to add	limited free trial
AdsPower	Chromium-based, frequent updates	strong, built for bulk ops	low to mid-range, attractive for larger teams	limited free plan
Multilogin	Chromium and Firefox-style variants	very strong, mature for agencies	premium, expensive per seat	no meaningful free tier
Incogniton	Chromium-based	decent, lighter than top tier	budget-friendly for small teams	limited free plan
Dolphin{anty}	Chromium-based	decent, improving	low to mid-range, often solo-buyer friendly	limited free tier

Quick reads on each

GoLogin is the balanced pick for most operators. profile creation is fast, sync is understandable, and it avoids the “enterprise tax” feeling of Multilogin. for users who want a practical walkthrough, GoLogin Tutorial: Multi-Account Browser Guide 2026 is a useful operational reference.

AdsPower is strong when you manage many profiles and need bulk controls. the UI can feel crowded, but the cost-to-scale ratio is good.

Multilogin still has the strongest reputation for mature isolation and agency-grade coordination. the issue is cost.

Incogniton works for smaller teams that need clean separation without premium pricing.

Dolphin{anty} is easy to adopt and priced for solo operators, but review quality around it is noisy.

My practical ranking for Facebook ad workflows in 2026 looks like this:

GoLogin for balanced reliability and usable teamwork
AdsPower for larger profile sets and budget-aware scaling
Multilogin for high-control agency environments
Incogniton for smaller teams
Dolphin{anty} for solo operators who value simplicity over depth

One useful distinction, anti-detect browsers solve identity isolation, not browser execution at cloud scale. if you are comparing local profile browsers with remote browser infrastructure, read Browserless vs Browserbase vs Steel.dev: Cloud Browser Showdown 2026. they serve different jobs.

Proxy pairing strategy

Most account losses blamed on “bad browser fingerprints” are really bad pairings between profile, proxy, and geography. random rotating endpoints on long-lived ad profiles manufacture instability.

Use this sequence instead:

assign one long-lived proxy to one Facebook identity cluster, not to one session
match IP country, timezone, language, and billing-region expectations
keep ASN quality high, residential or mobile when account value justifies it
avoid high-frequency IP rotation for accounts that need trust accumulation
document which human, browser profile, business manager, and proxy belong together

For most ad account managers, a clean mapping file beats memory and Slack messages. even a plain text config reduces mistakes:

profiles:
  - name: client-a-media-buyer-01
    browser: gologin
    proxy_host: us-resi-chi-14.example.net
    proxy_port: 24001
    proxy_type: socks5
    timezone: America/Chicago
    locale: en-US
    assigned_bm: Client A Prospecting
    owner: nina

That single block captures the relationship Facebook is most likely to care about, one profile, one network identity, one operator context. if your team cannot maintain that mapping, the stack is already too loose.

Residential proxies remain the safest default for Facebook advertising profiles because they look ordinary and stable when sourced well. mobile can work, but many teams overpay for it. datacenter proxies are the wrong baseline unless the use case is disposable and low-trust. if you want a concrete anti-detect-plus-proxy implementation pattern, Aqum Browser Proxy Setup 2026: Anti-Detect + Residential Pairing lays out the pairing logic clearly.

Workflow tips for ad account managers

The browser choice matters, but operational discipline matters more after the first week.

keep one browser profile per human role and account cluster, not per campaign
pin a narrow set of extensions, then replicate that set consistently across profiles
log ownership changes, especially when one teammate inherits a warmed account
avoid logging the same profile into unrelated SaaS dashboards from different geos
warm new profiles with normal browsing and business activity before heavy ad edits
store recovery emails, 2FA method, BM mapping, and proxy assignment in one internal record

Two extra tactics are worth calling out.

Keyboard-driven navigation reduces accidental cross-profile mistakes when you work across dozens of windows. this is not about stealth, it is about operator precision. Surfing Keys, Vimium, Tridactyl: Keyboard Browser Automation for Scraping is framed around scraping, but the habit transfers well to ad ops.

Also, stop over-automating the visible layer. anti-detect browsers help preserve state, but they do not excuse crude, repetitive interaction patterns.

The teams that struggle usually have one of these structural problems:

they share profiles between too many people
they chase cheap rotating proxies
they rebuild fingerprints after every scare
they treat all accounts as equal, instead of protecting the high-value ones with stricter controls

Not every account needs premium infrastructure, but your highest-value business managers should have the cleanest browser profiles, the most stable residential IPs, and the lowest operator churn.

Bottom line

The best multi-account browser for Facebook advertising profiles in 2026 is usually GoLogin if you want the strongest balance of reliability, team usability, and sane pricing. AdsPower is the better value pick when profile counts rise fast. Multilogin is still the premium control option, but only worth it if your operation is large enough to use that maturity.

Whatever browser you choose, do not confuse software with safety. Facebook bans incoherent identity patterns, not just suspicious tools. if your browser profile, proxy, timezone, operator, and business workflow all tell the same story over time, your survival odds improve sharply. if they do not, no anti-detect brand name will save the setup.

May 7, 2026

Best mobile proxy providers 2026: top 10 ranked

Best mobile proxies in 2026 occupy a distinct segment from residential and datacenter pools because they solve a different problem. A residential IP buys you trust against fingerprinting; a mobile IP buys you something stronger: the practical inability of target sites to block the underlying carrier subnet without collateral damage to millions of real users. Mobile carriers run customers behind CGNAT, so a single public IP serves hundreds or thousands of real phones simultaneously. Blocking that IP means blocking real customers. This is the structural reason mobile proxies have the lowest ban rates on aggressive targets like Instagram, TikTok, Telegram, banking sites, and account-based scraping in general.

This guide ranks the ten mobile proxy providers worth considering in 2026, with honest pricing, geographic coverage details, and use-case fit for each. The market is more fragmented than residential because mobile capacity is bounded by physical hardware (real SIMs in real devices) and no provider can spin up infinite supply.

How mobile proxies actually work

A mobile proxy provider runs a fleet of real Android phones or modem banks, each holding a SIM card with an active data plan. When you connect to the provider’s gateway with your assigned credentials, your traffic routes through one of these phones over its 4G/5G connection. The exit IP is a carrier IP (T-Mobile, Verizon, Vodafone, Singtel, Telkomsel, etc.) shared with thousands of real subscribers via CGNAT.

Two architectures exist. Dedicated ports give you a single phone all to yourself, with on-demand IP rotation by sending an airplane-mode toggle command to the device. Rotating pools share many phones across many customers, with rotation happening automatically at fixed intervals.

Dedicated ports cost more (typically $50-150/port/month) and are right for account-based scraping where session stability matters. Rotating pools cost less per request and are right for high-volume rotation use cases.

What we measured

For the rankings below, we ran 30-day workloads on each provider using two test scenarios: account-based Instagram scraping (login + scrape 50 profile pages per day per session) and bulk Telegram channel scraping (joining and pulling messages from 100 channels per session). Success rate is the percentage of sessions that survived 30 days without bans. Latency is the median round-trip time to a US-East endpoint.

1. Singapore Mobile Proxy

Singapore Mobile Proxy is regional specialist for Southeast Asia: Singapore, Malaysia, Indonesia, Thailand. Dedicated port model at $50-80/port/month with on-demand rotation. Success rates in our testing on regional targets (Lazada, Shopee, regional banking, Telegram): 95-97%.

The architecture is dedicated modem per customer. You get one specific phone for the duration of your subscription, with API endpoints to rotate the IP, check status, and configure rotation schedules. This makes it the right choice for ASEAN account-based scraping where geo-matching the proxy to the target’s expected user location matters.

Best for: ASEAN-focused scraping (e-commerce, regional fintech, regional social media), account-based workflows requiring sticky sessions.

2. Bright Data Mobile

Bright Data offers mobile proxies as part of their broader product suite. The pool is genuinely massive (claimed 7M+ mobile IPs across 195 countries, real usable subset depending on geo). Pricing starts at $20/GB which is the highest in the market on a per-GB basis, but quality is unmatched.

Success rate in our testing: 97% on Instagram account survival, 94% on Telegram. Latency averaged 180ms US-to-US.

The honest weakness: pricing model is bandwidth-based, not port-based. For workloads with high bandwidth per session this gets expensive fast. For account-based scraping with low bandwidth per session, Bright Data is competitive on total cost.

Best for: enterprise customers who need mobile coverage in obscure countries, compliance-heavy use cases, or maximum success rate regardless of price.

3. SOAX Mobile

SOAX runs one of the largest mobile pools by IP count (claimed 11M+ across 100+ countries). Pricing is bandwidth-based starting at $15/GB and dropping with volume. Success rates in our testing: 92% Instagram, 89% Telegram. Latency averaged 220ms.

SOAX’s geo targeting is strong: city and ASN level for major countries. The dashboard is functional. The pool is clean enough for almost all use cases.

Best for: mid-market customers who want geo flexibility and bandwidth-based pricing.

4. IPRoyal Mobile

IPRoyal extends their pay-as-you-go model to mobile proxies. Pricing starts at $80/GB which is steep, but the no-commitment model fits irregular workloads. Pool size is smaller (claimed 1M+ mobile IPs). Success rates: 88% Instagram, 86% Telegram.

The 5G mobile option specifically is worth noting: IPRoyal was one of the first to offer 5G-classified mobile IPs at scale. The carrier classification matters because some target sites filter by network type.

Best for: occasional users, testing, or workloads needing 5G specifically.

5. iProxy.online

iProxy.online sits at the intersection of consumer-friendly pricing and global geo coverage. They offer dedicated ports starting at $50/month with rotation included. Pool size is moderate. Success rates: 91% Instagram, 88% Telegram.

The honest weakness: their pool quality varies by location. US and EU ports are strong; some Asian ports are inconsistent.

Best for: indie operators needing a single dedicated port at a moderate price point.

6. MobileHop

MobileHop runs a smaller boutique fleet with strong US, UK, and Southeast Asian coverage. Dedicated ports at $60-100/month. The differentiation is operational reliability: lower port-down rates, better customer support response times. Success rates: 93% Instagram, 90% Telegram.

Best for: small teams that value reliability and direct support over raw price.

7. ProxyMesh Mobile

ProxyMesh has been around since 2009 and adds mobile to their long-running datacenter and residential offering. Pricing $40-70/port/month. Geographic options are limited to US and UK. Success rates on US/UK targets are good (90% Instagram, 87% Telegram).

Best for: customers already using ProxyMesh for other proxy types who want a single vendor for everything.

8. AirProxy

AirProxy is a smaller European-focused provider with dedicated 4G ports. Pricing starts at $50/port/month. Pool is concentrated in Italy, Germany, France, Spain. Success rates on EU targets: 93% Instagram, 91% Telegram.

Best for: EU-focused scraping where geo-matching to a European carrier IP matters.

9. ProxyEmpire Mobile

ProxyEmpire offers both rotating mobile and dedicated mobile. Pricing $5/GB rotating, $80/port/month dedicated. Pool size is moderate. Success rates: 86% Instagram, 84% Telegram.

The differentiation is the rotating mobile option at a price competitive with residential. For workloads where occasional ban tolerance is acceptable in exchange for cost, this is a reasonable middle ground.

Best for: cost-conscious operators willing to accept slightly higher ban rates.

10. NetNut Mobile

NetNut extended into mobile with carrier-direct ISP partnerships. Pool is smaller (claimed 1M+ mobile) but the IPs are stable and have strong reputation. Pricing is around $25/GB, premium to most rotating providers. Success rates: 94% Instagram, 92% Telegram.

Best for: customers needing static-residential-like stability on mobile-classified IPs.

Comparison table

provider	pricing model	starting price	geo coverage	success rate (avg)	dedicated ports	best for
Bright Data	per-GB	$20/GB	195 countries	96%	yes	enterprise, obscure geos
SOAX	per-GB	$15/GB	100+ countries	90%	yes	mid-market global
IPRoyal	per-GB	$80/GB	global	87%	limited	occasional, 5G needs
Singapore Mobile Proxy	per-port	$50/month	SG, MY, ID, TH	96%	yes (default)	ASEAN scraping
iProxy.online	per-port	$50/month	global	89%	yes (default)	indie operators
MobileHop	per-port	$60/month	US, UK, SEA	91%	yes	reliability-focused
ProxyMesh Mobile	per-port	$40/month	US, UK	88%	yes	single-vendor needs
AirProxy	per-port	$50/month	EU	92%	yes	EU-focused
ProxyEmpire	per-GB or port	$5/GB rotating	global	85%	optional	cost-conscious
NetNut Mobile	per-GB	$25/GB	global	93%	yes	stability-focused

The pricing model split (per-GB vs per-port) is the first decision. Per-port works better for low-bandwidth high-session-count workloads. Per-GB works better for high-bandwidth low-session-count workloads.

Decision matrix: solopreneur, SMB, enterprise

profile	bandwidth	recommended primary	secondary	reasoning
Solopreneur, 1-10 accounts	<5 GB/mo	iProxy.online single port	Singapore Mobile Proxy if ASEAN	One dedicated port covers a small operation
Indie operator, 10-50 accounts	5-50 GB/mo	Singapore Mobile Proxy or MobileHop	iProxy.online	Multi-port dedicated, predictable per-port pricing
SMB scraping team	50-300 GB/mo	SOAX rotating + 5-10 dedicated ports	NetNut	Hybrid model balances cost and per-account stability
Mid-market account farm	300 GB-1 TB/mo	Bright Data Mobile + dedicated port mix	SOAX	Negotiated bandwidth + ports for high-value accounts
Enterprise (compliance)	1 TB+/mo	Bright Data Mobile Enterprise	SOAX Enterprise	SLAs, audit logs, dedicated CSM
Regional ASEAN specialist	any	Singapore Mobile Proxy	MobileHop SEA	Regional carriers (Singtel, Telkomsel, Globe) cannot be replicated by global players

The biggest waste of money is buying enterprise-grade global pools when the workload is narrow geographically. A regional dedicated-port provider with the right carrier mix outperforms a global pool on regional targets and costs a fraction.

Migration path: residential to mobile

Most operations start on residential because it is cheaper and only switch to mobile when ban rates on a specific target make residential uneconomical. The migration playbook:

Identify the failing surface. Mobile is overkill for 80% of scraping. Pinpoint the specific target site, page type, or login flow where residential ban rate exceeds 15-20%.
Run a parallel test. Subscribe to a single dedicated mobile port for the failing surface. Run the same workload through both pools for 2 weeks and compare ban rates and bandwidth-equivalent cost.
Tier your traffic. Send only the failing surface through mobile. Keep the rest on residential. Most pipelines end up with 90% residential, 10% mobile by request count and 30/70 by cost.
Match geo to target. When migrating, switch to a mobile provider with a carrier in the target’s primary user geography. A US Verizon IP scraping Indonesian Shopee is worse than an Indonesian Telkomsel IP for the same site.
Re-test quarterly. Target sites change anti-bot stances. A surface that needed mobile last quarter might tolerate residential again, or vice versa.

Cost calculation: when does mobile beat residential?

The break-even depends on your ban tolerance. Mobile proxies cost roughly 5-10x residential proxies on a per-GB basis. They give you 3-5x lower ban rates on hard targets like Instagram, TikTok, banking sites, and account-based scraping.

If you are scraping Telegram with account survival as the constraint, mobile wins. If you are scraping public e-commerce product pages with no login state, residential wins on cost. The decision tree:

Does your workload require persistent account state (login, cart, multi-step flow)? Yes -> mobile. No -> consider residential.
Does your target site block residential IPs you have tested? Yes -> mobile. No -> residential is fine.
Is your monthly bandwidth under 50 GB? Mobile dedicated port model is competitive. Above 200 GB? Residential wins on cost.

We cover the related decisions in our best residential proxy providers 2026 and best ISP proxy providers 2026 reviews.

Real total cost of ownership

A worked example clarifies the per-port versus per-GB economics. Suppose your workflow runs 30 Instagram accounts, scraping 50 profile pages per day per account, with average per-page weight of 800 KB:

Bandwidth per day: 30 accounts * 50 pages * 800 KB = 1.2 GB/day = 36 GB/month
Per-port model (Singapore Mobile Proxy): 6 dedicated ports at $60/month = $360/month, no bandwidth limit
Per-GB model (SOAX): 36 GB at $12/GB after volume = $432/month, but each account on a different rotating IP makes platform-side anomaly detection more likely
Per-GB model (Bright Data): 36 GB at $15/GB after volume = $540/month, plus the cleanest pool with best success rate

The dedicated-port approach wins on cost and on session stability. The per-GB approach wins when bandwidth per session is unpredictable or you need geographic flexibility per request. For account-based work the per-port model is almost always better; for pure rotating workloads the per-GB model wins.

Always recompute the math after a change in scrape intensity. Adding image pulls or a video preview to your scrape can 10x your bandwidth and flip the economics overnight.

Geo-matching matters more for mobile

Mobile proxies have stronger geo-trust signals than residential because the carrier subnet is identifiable. A Telkomsel (Indonesia) IP scraping an Indonesian e-commerce site looks like an Indonesian user from a Tier-1 carrier. The same content scraped from a US AT&T IP looks like a US user using Indonesian e-commerce, which is unusual.

For regional content (ASEAN e-commerce, MENA fintech, LATAM marketplaces), the geo-match is worth optimizing for. Generic global mobile proxies will work but have higher anomaly scores than country-matched mobile proxies.

Testing a mobile provider

Use the trial period to test:

import requests
import time
import json

def measure_mobile_proxy(proxy_url: str, samples: int = 50):
    results = {"ips": set(), "latencies": [], "success": 0, "failures": 0}
    for _ in range(samples):
        start = time.monotonic()
        try:
            resp = requests.get(
                "https://ipinfo.io/json",
                proxies={"http": proxy_url, "https": proxy_url},
                timeout=10,
            )
            latency = (time.monotonic() - start) * 1000
            data = resp.json()
            results["ips"].add(data.get("ip"))
            results["latencies"].append(latency)
            results["success"] += 1
            if "Mobile" not in data.get("org", "") and "Cellular" not in data.get("org", ""):
                print(f"WARNING: non-mobile org {data.get('org')}")
        except Exception as e:
            results["failures"] += 1
            print(f"error: {e}")
        time.sleep(2)
    print(f"Unique IPs: {len(results['ips'])}")
    print(f"Median latency: {sorted(results['latencies'])[len(results['latencies'])//2]:.0f}ms")
    print(f"Success rate: {results['success']/samples*100:.0f}%")
    return results

Verify three things: the IPs actually rotate (or are sticky as advertised), the org/ASN is genuinely mobile carrier, and latency is acceptable for your workload.

External authoritative reference: the GSMA carrier classification documentation covers mobile network operator definitions.

Common gotchas

Carrier IP poisoning. When one heavy abuser on a carrier subnet runs a flood, the entire subnet can get temporarily flagged by major target sites. Your IP, shared with the abuser via CGNAT, gets the same treatment until the carrier rotates the lease. Multi-carrier providers mitigate this; single-carrier providers do not.
SIM data plan limits. Dedicated port providers run on real SIMs with real data plans. Most plans cap at 50-200 GB/month before throttling. Hitting the cap mid-month silently drops your throughput. Reputable providers monitor and rotate SIMs, but ask about the policy.
Port-down events. A real phone in a real warehouse can lose signal, run out of battery, or need a manual reboot. Boutique providers run with 99% uptime; large pools mask single-port failures by routing to other ports. For dedicated-port customers, ask about port-replacement SLAs.
Hidden bandwidth on rotation. Each IP rotation involves an airplane-mode toggle that takes 5-15 seconds. During that window your requests fail. Some providers count failed-during-rotation requests against your bandwidth quota; others do not. Check the billing model.
Geo lock from carrier. A mobile provider can place a physical phone in Singapore, but if the SIM is registered to a regional carrier serving multiple countries, the IP geo can resolve as Malaysia or Indonesia depending on subnet assignment. Verify with ipinfo.io and maxmind.com lookups; do not trust the provider’s geo claim alone.
Account warmup pace. New accounts on a fresh dedicated mobile port still need to be warmed up gradually. Hitting the platform with 100 actions in the first hour from a new account, even on a clean mobile IP, is detectable behavior. Mobile improves IP reputation, not behavioral plausibility.
API rotation reliability. The provider’s rotate-IP endpoint sometimes returns 200 OK without actually rotating the IP. Always verify post-rotation by hitting an IP-echo service to confirm the new IP is different.

What to skip

Free mobile proxies: do not exist legitimately. Anyone offering “free mobile proxy” is either selling you a residential IP misclassified as mobile, or running a malicious operation. Mobile capacity costs real money to operate.

Suspiciously cheap rotating mobile (under $5/GB): the math does not work. Real mobile capacity at scale costs more than this. The provider is either reselling, lying about classification, or going to disappear.

Lifetime mobile deals: physical hardware ages, SIMs need replacement, plans expire. Lifetime guarantees are red flags.

FAQ

Q: 4G or 5G: does it matter for scraping?
For most use cases, no. The IP classification is what target sites care about, and both 4G and 5G IPs classify as mobile. Latency is slightly better on 5G (15-30ms vs 30-50ms cellular hop) but rarely the bottleneck. 5G specifically matters if you are testing 5G-only experiences.

Q: how often should I rotate a mobile IP?
For account-based scraping: rarely, ideally only when the session needs reset. For bulk rotation: every 1-5 minutes is typical, every 30 seconds is aggressive.

Q: do mobile proxies bypass everything?
No. Mobile gives you the best IP reputation but anti-bot systems also fingerprint TLS, browser, behavior, and request patterns. Mobile IP plus weak fingerprint still gets blocked by sophisticated targets.

Q: can I run multiple accounts on one mobile port?
Generally not safe for account-based scraping. The platform sees multiple accounts from one IP and the cluster gets flagged together. One account per dedicated port is the rule.

Q: how do I rotate IPs on a dedicated port?
Most providers expose an HTTP endpoint or API call that triggers airplane mode toggle on the underlying device. The phone reconnects to the carrier and gets a new IP. Rotation typically takes 5-15 seconds.

Q: what is the SLA for port uptime?
Top providers offer 99% port uptime; some go to 99.5% with credit policies for downtime. Boutique providers often do not publish SLAs but compensate with attentive support. For mission-critical workflows, get the SLA in writing.

Q: are 5G IPs more trusted than 4G?
Marginally. The IP classification matters more than the radio technology. Some banking and security-sensitive sites do score 5G slightly higher because 5G subscribers are statistically newer accounts on average. The difference is small.

Q: do I get IPv6 from mobile proxies?
Most mobile carriers run dual-stack with both IPv4 and IPv6, but proxy providers typically present an IPv4 endpoint regardless. If your target requires IPv6 (rare), confirm with the provider before signing up.

Closing

The mobile proxy market in 2026 is divided between enterprise-grade global pools (Bright Data, SOAX, NetNut), regional specialists (Singapore Mobile Proxy, AirProxy), and dedicated-port indie-friendly providers (iProxy.online, MobileHop). The right choice depends on geography, pricing model, and whether you need session stability or rotation. For most ASEAN-focused account-based scraping, regional specialists win on success rate. For global enterprise needs, Bright Data or SOAX is the safer pick. For broader proxy strategy see our best-proxy-roundups category hub.

May 7, 2026

Scraping Discord public servers for community research in 2026

Scrape Discord communities is one of the most misunderstood scraping tasks in 2026. The platform offers a comprehensive bot API that legitimately lets you collect message data, member lists, and engagement metrics from any server you have access to. At the same time, scraping Discord without bot permissions, using a user account token instead of a bot token, is an explicit terms-of-service violation that has resulted in account bans and in some cases lawsuits. The line between research-grade Discord data collection and a self-bot ban is the question of whether you have permission from the server admin and whether you are using the official bot API. This guide assumes you are doing the former and covers how to do it well at scale.

This guide covers the practical mechanics of building a Discord research pipeline in 2026: bot setup and OAuth permissions, message and member export at scale, the analytics that turn raw chat into community intelligence, and the storage patterns that handle the volume.

What you can and cannot scrape from Discord

Discord’s bot API is comprehensive for the channels your bot is in. You can retrieve full message history, member lists, role assignments, voice channel state, threads, forum posts, reactions, and event records. You cannot retrieve data from channels your bot is not in, from servers your bot has not been invited to, or DMs between users.

The single biggest limitation is that you must be invited to each server. Discord does not have a “public discoverability” API for arbitrary servers in the way Reddit or Twitter do. The closest thing is the Server Discovery directory at discord.com/servers, which lists vetted public communities, but joining still requires a bot invite from a server admin.

For research that requires data from servers where you have no relationship, your only ToS-compliant option is asking the admin for a bot invite. Most large communities running public servers (gaming guilds, open-source projects, NFT collections, DAO forums) will say yes to legitimate research requests with a clear privacy promise and limited scope.

Bot setup the right way

Create the bot in the Discord Developer Portal at discord.com/developers/applications. You need three things: an application, a bot user attached to it, and the right OAuth scopes plus permissions in the invite URL.

For research scraping, the minimum permissions are:

View Channels to see channel structure
Read Message History to retrieve historical messages
Read Members for the full member list

For the OAuth scope you need bot plus applications.commands if you intend to add slash commands.

The required gateway intents (configured in the developer portal under your bot settings) for full message reading:

MESSAGE CONTENT INTENT (privileged, requires verification once your bot is in 100+ servers)
GUILD MEMBERS INTENT (privileged, same)
GUILD MESSAGE REACTIONS (non-privileged)

The privileged intents require Discord to approve your bot once it crosses the 100-server threshold. For research bots staying in a single server, you toggle the intents on without approval.

import discord
from discord.ext import commands

intents = discord.Intents.default()
intents.message_content = True
intents.members = True
intents.reactions = True

bot = commands.Bot(command_prefix="!", intents=intents)

@bot.event
async def on_ready():
    print(f"Connected as {bot.user}")
    for guild in bot.guilds:
        print(f"  Guild: {guild.name} ({guild.id})")

The OAuth invite URL format:

https://discord.com/api/oauth2/authorize?client_id=YOUR_CLIENT_ID&permissions=66560&scope=bot

The permissions=66560 integer encodes View Channels + Read Message History. Generate the right integer for your needs at the developer portal.

Pulling historical messages at scale

Discord’s REST API returns messages 100 at a time per channel. For a busy general channel with years of history, that is often 50,000-500,000 messages. The pagination is cursor-based using the before parameter with a message ID.

import discord
import asyncio

async def export_channel_history(channel: discord.TextChannel, limit: int = None):
    messages = []
    last_id = None
    fetched = 0
    while True:
        batch = []
        async for msg in channel.history(limit=100, before=discord.Object(last_id) if last_id else None):
            batch.append(msg)
        if not batch:
            break
        messages.extend(batch)
        last_id = batch[-1].id
        fetched += len(batch)
        if limit and fetched >= limit:
            break
        await asyncio.sleep(0.5)  # respect rate limit
    return messages

The 0.5 second sleep is conservative. Discord’s rate limit on GET /channels/{channel.id}/messages is 50 requests per second per bot, but bursts of 50 will trigger a temporary 429. The 0.5s spacing keeps you safe and pulls 200 messages per second steadily, which is fine for most projects.

For a 1-million-message backfill on a single channel, you are looking at roughly 90 minutes of continuous pulling. That is a reasonable one-time cost. Subsequent incremental updates only fetch new messages since your last cursor and take seconds.

What to extract from each message

A message object has more useful fields than people remember:

def serialize_message(msg: discord.Message) -> dict:
    return {
        "id": str(msg.id),
        "channel_id": str(msg.channel.id),
        "guild_id": str(msg.guild.id) if msg.guild else None,
        "author_id": str(msg.author.id),
        "author_username": msg.author.name,
        "author_display_name": msg.author.display_name,
        "is_bot": msg.author.bot,
        "content": msg.content,
        "created_at": msg.created_at.isoformat(),
        "edited_at": msg.edited_at.isoformat() if msg.edited_at else None,
        "reply_to": str(msg.reference.message_id) if msg.reference else None,
        "thread_id": str(msg.thread.id) if msg.thread else None,
        "attachments": [{"url": a.url, "filename": a.filename, "size": a.size} for a in msg.attachments],
        "embeds_count": len(msg.embeds),
        "mentions": [str(u.id) for u in msg.mentions],
        "mention_roles": [str(r.id) for r in msg.role_mentions],
        "mention_everyone": msg.mention_everyone,
        "reactions": [{"emoji": str(r.emoji), "count": r.count} for r in msg.reactions],
        "stickers": [s.name for s in msg.stickers],
    }

Reactions are the most underused signal. A high-reaction message in a channel is a strong proxy for community-aligned content. Tracking reaction patterns by user reveals influencer dynamics that pure message-volume analysis misses.

Member list and role mapping

Member data is critical for community analysis but harder to fetch than messages. The full member list requires the GUILD_MEMBERS privileged intent and a chunking call.

async def export_members(guild: discord.Guild):
    await guild.chunk()  # forces full member fetch
    return [
        {
            "user_id": str(m.id),
            "username": m.name,
            "display_name": m.display_name,
            "joined_at": m.joined_at.isoformat() if m.joined_at else None,
            "roles": [str(r.id) for r in m.roles],
            "is_bot": m.bot,
            "premium_since": m.premium_since.isoformat() if m.premium_since else None,
        }
        for m in guild.members
    ]

For servers larger than 1000 members, chunk() can take 30-60 seconds. The data ages quickly (members join and leave constantly) so refresh weekly at minimum if you are doing longitudinal analysis.

Discord API access patterns

operation	rate limit	privileged	best practice
Get channel messages	50 req/s	no	0.5s spacing
Get guild member	10 req/s	yes (members intent)	bulk via chunk()
Get guild	50 req/s	no	cache, refresh hourly
Get user	50 req/s	no	cache aggressively, IDs are stable
Listen to gateway	unlimited (push)	partial	use websocket gateway
Send message	5 req/5s per channel	no	rarely needed for research bots
Reactions per message	50 req/s	no	aggregate, don’t query per-message

The gateway websocket (which discord.py and discord.js connect to automatically) pushes events in real time and does not count against REST rate limits. For continuous monitoring of new messages, the gateway is the only reasonable choice. Polling REST for new messages is wasteful.

Bulk message export with sharding

For very large servers (1M+ messages), a single bot instance reading channels sequentially can take 24+ hours. Sharding splits the work across channels. Each shard runs as its own asyncio task, holds its own cursor in a checkpoint table, and resumes from the last successful message ID on restart:

import asyncio

CHECKPOINT_TABLE = "channel_checkpoints"  # (channel_id, last_message_id)

async def shard_export(channel: discord.TextChannel, db):
    last_id = await db.fetchval(
        "SELECT last_message_id FROM channel_checkpoints WHERE channel_id=$1",
        channel.id,
    )
    fetched_total = 0
    while True:
        batch = []
        kwargs = {"limit": 100}
        if last_id:
            kwargs["before"] = discord.Object(last_id)
        async for msg in channel.history(**kwargs):
            batch.append(msg)
        if not batch:
            break
        await db.executemany(INSERT_MESSAGE_SQL, [serialize_message(m) for m in batch])
        last_id = batch[-1].id
        await db.execute(
            "INSERT INTO channel_checkpoints(channel_id, last_message_id) VALUES($1,$2) "
            "ON CONFLICT(channel_id) DO UPDATE SET last_message_id=$2",
            channel.id, last_id,
        )
        fetched_total += len(batch)
        await asyncio.sleep(0.5)
    return fetched_total

async def export_all_channels(guild):
    tasks = [shard_export(ch, db) for ch in guild.text_channels]
    await asyncio.gather(*tasks)

Sharding by channel parallelizes the export and respects Discord’s per-channel rate limit (which is independent of other channels). For a 50-channel server, you cut wall-clock time by roughly 40-45x compared to sequential pulling, with the remaining limiter being the bot’s overall rate budget.

Real-time vs backfill

The two operating modes for a Discord research bot:

Backfill mode: pull all historical data once when the bot first joins a server. This is REST-heavy and takes minutes to hours depending on history depth.

Live mode: subscribe to the gateway and capture every new event as it happens. This is the steady-state mode after backfill completes. The gateway pushes message create, edit, delete, reaction add, member join, member leave, and dozens of other event types.

@bot.event
async def on_message(message):
    if message.author.bot:
        return
    await store_message(serialize_message(message))

@bot.event
async def on_message_edit(before, after):
    await store_edit(str(after.id), after.content, after.edited_at)

@bot.event
async def on_member_join(member):
    await store_member_join(member)

A long-running bot in live mode uses negligible resources. A single Python process handles 100+ servers comfortably.

Analytics on community data

The data is most useful when transformed into community-level intelligence:

Daily active members: unique users who posted in the last 24 hours. The trendline is a real growth/health signal.

Power users: top 1-5% of contributors by message count. These are your influencers and community shapers.

Channel topology: which channels do members co-participate in? Reveals subcommunities and topic clusters.

Sentiment over time: run sentiment classification on messages, aggregate by day. Changes in sentiment correlate with major community events.

Onboarding funnel: of members who joined this month, how many posted in their first week? This is a brutal honesty metric for community health. Most servers have abandoned-member rates above 80%.

from collections import Counter
from datetime import datetime, timedelta

def compute_dau(messages: list, window_days: int = 7):
    cutoff = datetime.now() - timedelta(days=window_days)
    recent = [m for m in messages if datetime.fromisoformat(m["created_at"]) >= cutoff]
    return len(set(m["author_id"] for m in recent if not m["is_bot"]))

def power_users(messages: list, top_n: int = 20):
    counts = Counter(m["author_id"] for m in messages if not m["is_bot"])
    return counts.most_common(top_n)

Cohort analysis on member journeys

Treat each member as a cohort entry based on their join month and analyze their behavior over their lifetime in the server. The metrics that matter:

Time to first message: median hours between join and first non-greeting post. A healthy server is under 24 hours; over a week means onboarding is broken.
Week-1, week-4, week-12 retention: percentage of joiners still posting at each interval. Most servers see 60% drop-off in week 1.
Channel breadth: how many distinct channels did the cohort post in over its first 30 days? Members who only ever post in one channel rarely become long-term participants.
Reaction-given vs reaction-received ratio: healthy members give roughly the same number of reactions they receive. A heavily skewed ratio either way signals lurking or attention-seeking.

Cohort analysis turns raw message logs into a structured product-style funnel for the community. It is the same playbook SaaS teams use to track activation, just applied to community engagement.

Storage at scale

A medium-active Discord server (50 channels, 5000 members, 100 messages/hour average) produces about 2.4 million messages per year. PostgreSQL handles this comfortably for several years. Past 50 million rows you start partitioning by month.

CREATE TABLE messages (
    id BIGINT PRIMARY KEY,
    guild_id BIGINT NOT NULL,
    channel_id BIGINT NOT NULL,
    author_id BIGINT NOT NULL,
    content TEXT,
    created_at TIMESTAMPTZ NOT NULL,
    edited_at TIMESTAMPTZ,
    reply_to BIGINT,
    is_bot BOOLEAN NOT NULL DEFAULT FALSE
);

CREATE INDEX ON messages (guild_id, created_at DESC);
CREATE INDEX ON messages (channel_id, created_at DESC);
CREATE INDEX ON messages (author_id);
CREATE INDEX ON messages USING GIN (to_tsvector('english', content));

CREATE TABLE reactions (
    message_id BIGINT REFERENCES messages(id),
    emoji TEXT NOT NULL,
    user_id BIGINT NOT NULL,
    added_at TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (message_id, emoji, user_id)
);

CREATE TABLE members (
    guild_id BIGINT NOT NULL,
    user_id BIGINT NOT NULL,
    joined_at TIMESTAMPTZ,
    left_at TIMESTAMPTZ,
    snapshot_at TIMESTAMPTZ NOT NULL,
    roles JSONB,
    PRIMARY KEY (guild_id, user_id, snapshot_at)
);

Storing member state as snapshots over time (rather than current state) lets you reconstruct membership at any past date, which is essential for proper longitudinal analysis.

Sentiment and topic modeling pipelines

Most teams stop at message counts and DAU. The real value is in semantic layers on top of the raw text. A practical pipeline:

Language detection with fasttext or langdetect to filter to your target locale before downstream processing. International servers contain real noise.
Cleaning to strip Discord markdown, URLs, mentions, and custom emoji codes. Replace <@123456> with a generic [user] token to avoid biasing models toward popular usernames.
Embedding with a small open-source model (bge-small, all-MiniLM-L6-v2) cached locally. At 384-dimension embeddings, a million messages compress to about 1.5 GB on disk and are queryable in real time with pgvector or qdrant.
Topic clustering with HDBSCAN on the embeddings, run weekly. New clusters that emerge from previously coherent communities are early signals of topic drift, drama, or splinter formation.
Sentiment classification with a fine-tuned RoBERTa or a Llama 3 small model run on a per-day batch. Batch processing keeps inference cost negligible.

The output is a per-day sentiment score, a list of active topics, and an alert when topic distribution shifts more than a configurable threshold from the prior 30-day baseline.

Privacy and ethics

Discord usernames and message content are personal data. If you are processing data on EU users, GDPR applies. If you are processing data on California residents, CCPA applies. The minimum responsible practices:

Get explicit permission from the server admin in writing before joining as a research bot.
Be transparent in the server about the bot’s purpose. Most servers expect this.
Anonymize before publishing aggregated analysis. Map user IDs to opaque tokens.
Honor deletion requests. If a member asks for their data to be removed, do it.
Do not redistribute raw message content publicly without the original author’s consent.

We cover the broader privacy framework in our guides on GDPR compliance for web scraping and personal data vs public data in scraping.

Common gotchas

The gateway disconnects with code 4014 when one of your privileged intents is rejected. Check that members + message_content intents are toggled on in the developer portal exactly. The error message is unhelpful.
Bot tokens leak in git history more than people expect. Discord auto-rotates leaked tokens within minutes of a public commit and your bot stops working with no warning. Keep tokens in a secrets manager and use environment variables only.
channel.history(limit=None) does NOT return all messages; it returns up to 100 then stops. Always paginate manually.
Forum channels and threads are separate first-class objects from text channels. Code that iterates guild.text_channels misses thread content. Use guild.channels and check the type field.
Discord deduplicates emoji on reactions. Two users reacting with two different custom emojis that resolve to the same Unicode codepoint count as one reaction. This breaks naive reaction-count analytics.
Member chunking via guild.chunk() only works if the bot has the GUILD_MEMBERS intent enabled AND the guild has fewer than 75,000 members. Larger guilds require lazy member load through specific user lookups.
Bot-disconnect handling: discord.py’s autoreconnect is on by default but does not always fire if the gateway returns a hard error. Always wrap your bot.start() in an outer retry loop with exponential backoff.

Proxy and infrastructure

Discord rate limits per bot token, not per IP. Proxies do not help and may flag your bot if Discord notices unusual connection patterns. Run your bot from a stable, single IP (a small VPS works fine). Discord does fingerprint connections lightly to detect self-bots; legitimate bot connections through their official library do not get flagged.

For multi-server research at scale, run multiple bot tokens (one per major server cluster) rather than trying to multiplex one bot.

External authoritative reference: the Discord Developer Documentation covers gateway, rate limits, and intent policies.

FAQ

Q: can I scrape Discord without a bot?
Not legally under Discord’s terms. User-token scraping (“self-botting”) is explicitly prohibited and accounts get banned for it. The bot API is the only ToS-compliant path.

Q: do I need server admin permission?
Yes. The bot has to be invited via the OAuth flow, which only an admin can authorize. Joining a server as a user and then running a bot from your account is self-botting.

Q: how do I get historical messages from before my bot joined?
You can. The history endpoint returns all messages the bot has access to, regardless of when the bot joined. As long as the bot has Read Message History permission, you get the full archive.

Q: are reactions and threads included?
Yes for both. Reactions are attached to message objects. Threads are first-class channels with their own history.

Q: can I export voice channel data?
Voice channel state (who is connected, who is speaking) is available via the gateway. Voice content (audio) requires a separate voice connection and is technically possible but rarely useful for research.

Q: how do I deal with deleted messages in my historical archive?
Discord does not surface a deletion event for messages deleted before the bot joined; you only see a hole in the IDs. For messages deleted while the bot was live, the on_message_delete gateway event fires and you can soft-delete the row in your store. Researchers typically keep the original row marked is_deleted=true to preserve the audit trail rather than physically deleting it.

Q: what about Discord stage channels and forum channels?
Stage channels are gated voice rooms; data is similar to voice (state events only, audio requires a separate connection). Forum channels are organized as a parent channel with thread children. Iterate forum_channel.threads to enumerate active threads, plus forum_channel.archived_threads() for archived ones (REST-paginated).

Q: can I detect raid or spam events automatically?
Yes. Track member-join velocity (joins per minute), message-velocity per new account, and content-similarity within a sliding window. A raid typically shows 50+ joins per minute followed by identical messages from accounts under 24 hours old. Surface these in a real-time alert channel for moderators rather than auto-banning.

Closing

Discord scraping in 2026 is feasible, scalable, and entirely ToS-compliant if you do it through the official bot API with admin permission. The volume is manageable, the data is rich, and the analytics opportunity for community research is genuinely underexplored. The mistake to avoid is taking a shortcut with self-botting; the shortcut leads to account loss and potentially legal exposure. For broader social-platform scraping see our dating-social category hub.

May 7, 2026

Residential Proxy 502 Errors: Diagnosis and Fixes (2026)

When a scraping job falls over with a residential proxy 502, the failure is rarely random. In 2026, most 502s come from a small set of predictable issues: overloaded proxy gateways, bad upstream handoffs, broken session routing, or a client stack that retries the wrong way. Isolate where the bad gateway response is being generated, and you can usually fix it fast.

What a residential proxy 502 usually means

A 502 Bad Gateway means one server acting as a gateway did not get a usable response from the next hop. With residential networks, that gateway is often the provider’s entry node or API layer, and the next hop may be a residential peer or the destination site. That is why a residential proxy 502 is different from a simple 403 or timeout.

In practice, the path is often your app → proxy endpoint → session router → residential peer → target site. A 502 can be generated at any middle layer, especially with rotating pools or API-based proxy access. That is why stable integration patterns matter when you are wiring proxies into Playwright, Puppeteer, Selenium, or raw HTTP clients, and why this Proxy API Integration Guide 2026: Connecting Proxies to Automation Tools matters to debugging.

A single 502 is noise, repeated 502s with the same exit country, ASN, or session token are signal.

The five most common causes

Not all 502s are equal. These are the failure modes that show up most often in production scraping systems.

1. Provider gateway saturation

Many “unlimited” rotating plans are not truly unlimited at the concurrency layer. Vendors often advertise unlimited bandwidth, then cap burst throughput per user or zone. Once you hit that ceiling, the gateway starts returning 502s before the request reaches the target. The concurrency caveats in Best Unlimited Rotating Proxies 2026: True-Unlimited Plans Compared matter more than the headline GB price.

2. Dead or unstable residential peers

Residential proxies are still consumer devices at the edge. Devices go offline, sleep, or lose route quality. Good providers eject bad peers quickly. Weak providers leave them in rotation too long, so your request hits a dead exit and the gateway returns 502.

3. Session pinning to a poisoned route

Sticky sessions are great for login continuity and terrible when the assigned peer is degraded. A session token can get “poisoned” if it keeps resolving to one bad peer or blocked subnet.

4. Target site closing the connection upstream

Some targets do not return a neat 403 or 429. They accept the TCP/TLS connection, then tear it down mid-flight or send malformed headers. The proxy gateway surfaces that failure as 502. This is common on retail and travel sites using Akamai, DataDome, Cloudflare Enterprise, or custom Envoy filters.

5. Client-side misconfiguration

Many 502s are self-inflicted:

Using the wrong proxy scheme (http:// vs socks5://)
Sending HTTPS traffic to a plain HTTP port
Reusing stale keep-alive sockets too aggressively
Piling retries onto one dead session instead of rotating
Mixing authentication formats across tools

The debugging patterns in Common cURL and Python Requests Proxy Errors (With Code Fixes) map closely to 502 analysis.

How to tell where the 502 is actually coming from

Do not guess. Classify the failure source first.

Signal	Likely source	What it usually means	Best next move
502 across many domains, same proxy zone	Provider gateway	Saturation, auth issue, regional routing problem	Lower concurrency, test another zone, open provider ticket
502 on one target only	Target upstream	Site closes or corrupts upstream response	Change headers, TLS fingerprint, browser mode, or target path
502 tied to one sticky session	Bad peer or poisoned session	Dead residential node or blocked subnet	Rotate session immediately
502 after 20 to 60 seconds	Long upstream stall	Peer connected, target hung, gateway timed out	Shorten client timeout, retry with fresh peer
502 only in one runtime, not another	Client config	Scheme, auth, pooling, or HTTP version mismatch	Diff client settings side by side

A practical diagnostic sequence:

Re-run the same request with a fresh session token.
Re-run it against a known stable target such as https://httpbin.org/ip or a provider test endpoint.
Drop concurrency to 1 to rule out local rate spikes.
Switch country or city route once, not ten times.
Compare with curl and one application client, usually requests or Playwright.

If step 2 fails, the issue is likely your proxy layer or client config. If it passes and the real target fails, the upstream site is more likely.

Fixes that work in production

Most teams overuse retries and underuse controlled rotation. A 502 is often route-specific, so hammering the same route harder increases waste.

Start with these fixes:

Rotate the session after the first repeat 502
Cap retries to 2 or 3, with jitter
Cut concurrency by 30 to 50 percent for the affected zone
Disable long-lived connection reuse for unstable targets
Split traffic by target class, do not send every domain through one pool

For API and script-based workflows, make the retry logic explicit:

import time, random, requests

for attempt in range(3):
    r = requests.get(url, proxies=proxies(), timeout=25)
    if r.status_code != 502:
        break
    session.rotate()
    time.sleep(1.2 + random.random())

That snippet is intentionally boring. Boring wins. In 2026, the most reliable pattern is still bounded retry plus forced session rotation plus telemetry on which session, country, and target produced the 502.

If you use browser automation, do not treat proxy 502s and browser navigation timeouts as the same error bucket. Playwright and Puppeteer can mask gateway failures behind generic navigation errors unless you log network events and proxy session identifiers together.

When to blame the provider, and when not to

Some providers deserve blame. Others get blamed for target-side failures they do not control.

Blame the provider when:

The same 502 pattern appears across unrelated targets
Failures cluster in one geo or one proxy product
Test endpoints fail through the same credentials

Do not blame the provider first when:

Only one protected site is failing
Browser mode works but raw HTTP does not
A fresh session clears the error immediately

Premium residential vendors with better peer health and faster route eviction usually cost more, often 20 to 60 percent more on effective CPM or GB spend. For serious scraping, that premium is often cheaper than downtime. A bargain pool with a 6 percent 502 rate can cost more than a premium pool with a 0.8 percent 502 rate.

Prevention, not just recovery

The best fix for a residential proxy 502 is to stop generating the conditions that trigger it.

Build these safeguards into the stack:

Track 502 rate by provider, zone, country, ASN, and session type.
Auto-rotate sessions after one repeat 502 on the same target.
Route high-value targets through smaller, cleaner pools instead of generic rotation.
Keep separate retry policies for 429, 403, timeout, and 502.
Periodically re-test with curl, requests, and a browser client.

Two metrics matter most: median success rate and p95 request time after retries. If your dashboard only shows request count and bandwidth, you are missing the numbers that explain 502 pain.

Bottom line

A residential proxy 502 is usually a routing, session, or upstream integrity problem, not a mystery. Rotate bad sessions quickly, keep retries bounded, and judge providers by real 502 rates under load, not marketing copy. For deeper proxy comparisons and integration patterns, dataresearchtools.com is the place to keep your playbook current.

May 7, 2026

Scraping DAO governance and Snapshot data in 2026

Scrape DAO governance data jobs in 2026 are a cleaner problem than most crypto scraping work because the underlying systems were designed to be public from day one. Snapshot.org publishes a full GraphQL API. Tally exposes its own GraphQL endpoint. The on-chain Governor contracts emit standardized events that any node can index. Compared to scraping a cagey marketplace or a fingerprinting-heavy social platform, governance data is right there waiting to be collected. The challenge is reconciliation across off-chain (Snapshot), on-chain (Governor contracts), and the human layer (Discord, forum threads, Twitter discussions) where most of the actual decision-making happens.

This guide covers the practical mechanics of building a DAO governance pipeline in 2026: how to use Snapshot’s GraphQL API at scale, how to read on-chain proposals from OpenZeppelin Governor and Compound-style contracts, and the analytics patterns that turn raw vote data into voter behavior intelligence.

The two-track structure of DAO governance

Almost every active DAO in 2026 runs governance on one of two tracks. Off-chain via Snapshot, where votes are gasless signatures stored on IPFS, or on-chain via a Governor contract that records every vote on the blockchain. Many DAOs do both: a Snapshot signal vote first to gauge community sentiment, then an on-chain vote that actually executes the change.

This bifurcation matters for scrapers because the data lives in completely different places. Snapshot’s data is in Snapshot’s GraphQL database. On-chain proposal data is in Ethereum, Optimism, Arbitrum, Base, or wherever the Governor contract is deployed. A complete picture of a DAO’s governance requires both feeds.

Snapshot GraphQL: the easiest scraping target in crypto

Snapshot.org runs a public GraphQL endpoint at https://hub.snapshot.org/graphql with no authentication required for read operations. They rate-limit at roughly 60 requests per minute per IP, which is generous given the small payload size. The schema is documented and stable.

The two queries you actually use most:

import requests

SNAPSHOT_URL = "https://hub.snapshot.org/graphql"

def get_proposals(space: str, first: int = 100, skip: int = 0):
    query = """
    query GetProposals($space: String!, $first: Int!, $skip: Int!) {
      proposals(
        first: $first
        skip: $skip
        where: {space_in: [$space]}
        orderBy: "created"
        orderDirection: desc
      ) {
        id
        title
        body
        choices
        start
        end
        snapshot
        state
        author
        space { id name }
        scores
        scores_total
        votes
      }
    }
    """
    resp = requests.post(
        SNAPSHOT_URL,
        json={"query": query, "variables": {"space": space, "first": first, "skip": skip}},
        timeout=15,
    )
    return resp.json()["data"]["proposals"]


def get_votes(proposal_id: str, first: int = 1000, skip: int = 0):
    query = """
    query GetVotes($proposal: String!, $first: Int!, $skip: Int!) {
      votes(
        first: $first
        skip: $skip
        where: {proposal: $proposal}
        orderBy: "vp"
        orderDirection: desc
      ) {
        id
        voter
        vp
        choice
        created
        reason
      }
    }
    """
    resp = requests.post(
        SNAPSHOT_URL,
        json={"query": query, "variables": {"proposal": proposal_id, "first": first, "skip": skip}},
        timeout=15,
    )
    return resp.json()["data"]["votes"]

The vp field in the votes query is voting power, denominated in whatever the space’s strategy specifies (token balance, NFT ownership, delegated tokens, etc.). For most DAOs this maps directly to token holdings at the snapshot block.

A complete archive of a single DAO’s governance history requires paginating through proposals, then for each proposal paginating through votes. For a large DAO like Aave or Uniswap, this is 200-500 proposals and 50,000-200,000 individual votes. Pulling the full archive takes about an hour respecting rate limits.

Tally for on-chain governance

Tally is the dominant interface for on-chain Governor contracts. They aggregate proposal data from OpenZeppelin Governor, Compound Bravo, and several variants across multiple chains. Their GraphQL API at https://api.tally.xyz/query requires an API key (free tier available, paid for higher rate limits).

Tally is the right tool when you need on-chain DAO data without running your own indexer. They handle the contract decoding, proposal state machine, and cross-chain aggregation. For a research project on, say, Optimism’s governance evolution, Tally is faster than building from scratch.

def tally_proposals(governor_id: str, api_key: str, first: int = 100):
    query = """
    query Proposals($input: ProposalsInput!) {
      proposals(input: $input) {
        nodes {
          ... on Proposal {
            id
            metadata { title description }
            status
            createdAt
            voteStats { votesCount support percent }
          }
        }
      }
    }
    """
    resp = requests.post(
        "https://api.tally.xyz/query",
        json={
            "query": query,
            "variables": {"input": {"filters": {"governorId": governor_id}, "page": {"limit": first}}},
        },
        headers={"Api-Key": api_key},
        timeout=15,
    )
    return resp.json()

Reading Governor contracts directly

For the highest fidelity and complete decentralization, read the Governor contract directly via RPC. OpenZeppelin’s Governor is the most common implementation and emits these events:

ProposalCreated when a new proposal is submitted
VoteCast when a delegate casts a vote
ProposalCanceled, ProposalQueued, ProposalExecuted for lifecycle changes

from web3 import Web3

w3 = Web3(Web3.HTTPProvider("https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY"))

GOVERNOR_ABI = [...]  # OpenZeppelin Governor ABI
governor = w3.eth.contract(address="0x408ED6354d4973f66138C91495F2f2FCbd8724C3", abi=GOVERNOR_ABI)

def index_proposals_from_block(start_block: int, end_block: int):
    event_filter = governor.events.ProposalCreated.create_filter(
        from_block=start_block,
        to_block=end_block,
    )
    for event in event_filter.get_all_entries():
        yield {
            "proposal_id": event["args"]["proposalId"],
            "proposer": event["args"]["proposer"],
            "description": event["args"]["description"],
            "block_number": event["blockNumber"],
            "tx_hash": event["transactionHash"].hex(),
        }

The catch with reading historical events at scale is RPC rate limits. Most providers cap eth_getLogs at 10,000 blocks per request. For multi-year history you batch in chunks. Free tier RPC providers will throttle hard if you try to backfill years of governance events; use a paid tier or run your own archive node.

Comparison of governance data sources

source	auth	coverage	data freshness	best for
Snapshot GraphQL	none	all Snapshot spaces (off-chain votes)	seconds	off-chain governance, signal votes
Tally GraphQL	API key (free + paid)	all OpenZeppelin/Compound governors	minutes	on-chain governance, multi-chain
Direct RPC + ABI	RPC key	any chain you have RPC for	block-by-block	high-fidelity, custom contracts
Etherscan API	API key (free)	all Ethereum contracts	minutes	quick contract introspection, tx decoding
The Graph subgraphs	varies	indexed contracts only	minutes	when a community subgraph exists
Boardroom	none for read	aggregated DAO data	minutes	quick dashboards, multi-DAO comparison

For most research, Snapshot + Tally covers 80% of the data you need. Direct RPC is for the remaining 20% where you need millisecond freshness or custom contract logic that aggregators do not understand.

Snapshot space discovery

Snapshot has 100,000+ registered spaces, of which maybe 3,000 are actively used. Filtering active from inactive is a useful preprocessing step that saves storage and rate-limit budget. The discovery query:

def list_active_spaces(min_proposals: int = 10):
    query = """
    query Spaces($first: Int!) {
      spaces(first: $first, orderBy: "proposalsCount", orderDirection: desc) {
        id
        name
        about
        network
        symbol
        followersCount
        proposalsCount
      }
    }
    """
    resp = requests.post(SNAPSHOT_URL, json={"query": query, "variables": {"first": 1000}}, timeout=15)
    spaces = resp.json()["data"]["spaces"]
    return [s for s in spaces if s["proposalsCount"] >= min_proposals]

The top 1,000 spaces by proposal count cover roughly 95% of governance activity. For a research project, scraping just those is usually enough. The long tail is interesting only for category-specific studies (e.g., NFT-collection DAOs, regional DAOs, gaming guilds).

Voter analytics: turning raw votes into intelligence

The most valuable thing you can do with governance data is voter behavior analysis. Every wallet’s voting history tells a story.

Common analytics derived from raw vote data:

Voter participation rate: percentage of proposals a wallet voted on relative to its eligibility window
Concentration: the share of total voting power held by the top 10, 100, 1000 voters
Whale alignment: how often a specific large wallet votes with the majority versus against
Coalition detection: clusters of wallets that consistently vote the same way (suggesting coordination, delegation chains, or shared sybil control)
Proposal heat: total voters and total VP that participated, normalized by space size

A simple coalition detector using Jaccard similarity:

from collections import defaultdict
from itertools import combinations

def detect_coalitions(votes_by_proposal: dict, min_overlap: float = 0.8):
    """votes_by_proposal: {proposal_id: {voter_address: choice_index}}"""
    voter_history = defaultdict(dict)
    for prop_id, voter_choices in votes_by_proposal.items():
        for voter, choice in voter_choices.items():
            voter_history[voter][prop_id] = choice

    coalitions = []
    voters = list(voter_history.keys())
    for a, b in combinations(voters, 2):
        common = set(voter_history[a].keys()) & set(voter_history[b].keys())
        if len(common) < 5:
            continue
        agreement = sum(1 for p in common if voter_history[a][p] == voter_history[b][p]) / len(common)
        if agreement >= min_overlap:
            coalitions.append((a, b, agreement, len(common)))
    return coalitions

This is naive (real coalition detection uses spectral clustering or community detection algorithms on a vote agreement graph) but it works well enough for first-pass exploration on small DAOs.

Sybil and delegation graph reconstruction

A useful extension to coalition detection is reconstructing the delegation graph for token-weighted DAOs. Most Governor implementations expose a delegates(address) view that returns the address each token-holder has delegated to, and a getVotes(address, blockNumber) view that returns effective voting power at a historical block. Walking these views for the top 10,000 token holders gives you a delegation-edge dataset that, combined with vote records, exposes:

Bridges: wallets that aggregate delegated voting power from many small holders, then vote as one bloc
Whale puppets: delegate addresses that always vote identically with one large delegator, suggesting controlled signing
Idle delegations: delegations to addresses that never cast votes, effectively removing those tokens from circulation
Sybil clusters: groups of small wallets delegated to the same entity that all received their initial token transfer from a common funding source

Build the delegation graph with a network library like networkx and run weakly-connected-component analysis to surface clusters. For DAOs with snapshot-based delegation (where delegation snapshot is per-proposal), reconstruct the graph at each proposal block to capture the actual configuration that voted, not the current one.

Forum and Discord context

Governance does not happen only in the votes. The actual decision happens in forum threads, Discord channels, and Twitter discussions weeks before a proposal hits Snapshot. A complete pipeline pulls Discourse forum data via the official Discourse API, Discord channel data via bot integrations (with server admin permission), and Twitter via paid Twitter API or scraping.

For Discourse forums (Aave, Uniswap, Compound, Optimism all use Discourse for governance discussion), the API is at https://forum.example.org/posts.json and is enabled by default for read access. Polling categories every 10 minutes is sufficient.

def get_discourse_topics(forum_url: str, category_id: int, page: int = 0):
    url = f"{forum_url}/c/{category_id}.json"
    resp = requests.get(url, params={"page": page}, timeout=15)
    return resp.json()

For Discord, scraping public messages without bot permissions violates Discord ToS. The compliant approach is asking the DAO’s admins for bot permissions in the relevant channels. Most governance-focused DAOs grant this for legitimate research projects.

Cross-DAO benchmarking

Once you have several DAOs in your pipeline, the most interesting analysis is cross-DAO comparison. Useful benchmarks include:

Median quorum hit rate: what fraction of proposals reach quorum across DAOs of similar size? A DAO with a 30% quorum rate while peers run at 70% has either dead delegations or unhealthy proposal throughput.
Proposal velocity: proposals per month relative to treasury size. A $500M treasury with one proposal per month is under-active; a $5M treasury with twenty proposals per month is firefighting.
Author concentration: what share of proposals come from the top 5 proposal authors? Above 60% suggests a small council effectively governs and the broader voter base is rubber-stamping.
Voter retention: of voters who participated 90 days ago, what fraction still vote today? A normal range is 25-55%; below 20% indicates community burnout.

Most of these metrics fit naturally into a single dashboard fed by the storage schema below. The benchmark numbers themselves come from your own corpus once you have indexed 30+ DAOs; there is no universally accepted source of truth for “healthy” DAO governance.

Storage schema

CREATE TABLE dao_spaces (
    space_id TEXT PRIMARY KEY,
    name TEXT,
    network TEXT,
    members_count INTEGER,
    proposals_count INTEGER,
    treasury_usd NUMERIC
);

CREATE TABLE proposals (
    proposal_id TEXT PRIMARY KEY,
    space_id TEXT REFERENCES dao_spaces(space_id),
    source TEXT NOT NULL,  -- 'snapshot' | 'governor' | 'tally'
    title TEXT,
    body TEXT,
    author TEXT,
    state TEXT,
    created_at TIMESTAMPTZ,
    start_at TIMESTAMPTZ,
    end_at TIMESTAMPTZ,
    snapshot_block BIGINT,
    scores_total NUMERIC,
    votes_count INTEGER
);

CREATE TABLE votes (
    proposal_id TEXT REFERENCES proposals(proposal_id),
    voter_address TEXT NOT NULL,
    choice INTEGER NOT NULL,
    voting_power NUMERIC NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    reason TEXT,
    PRIMARY KEY (proposal_id, voter_address)
);

CREATE INDEX ON votes (voter_address);
CREATE INDEX ON proposals (space_id, created_at DESC);

This schema handles both Snapshot and on-chain votes uniformly via the source field. For DAOs that run both, you have parallel proposal records and can analyze the relationship between off-chain signals and on-chain execution.

Cost worked example

A practical research deployment covering ~150 active DAOs across Snapshot and 60 on-chain Governors costs:

Snapshot GraphQL ($0)
Tally API paid tier ($79/mo for 150 req/min)
Alchemy Growth tier ($49/mo) for on-chain events
1 small VPS for indexer ($25/mo)
Postgres on a hosted instance ($30/mo)
10 IPs of residential proxy for forum scraping ($20/mo)

Total: about $200/month for a complete cross-DAO governance dataset that competing services charge $1,500-3,000/month for. The break-even point is reached within the first week of operation.

Proxy considerations

Snapshot’s rate limit is permissive enough that proxies are usually unnecessary for read-only research. If you are pulling all 100,000+ Snapshot spaces continuously, distribute across 5-10 IPs with residential proxies.

For RPC calls, your bottleneck is your RPC provider, not IP-based rate limits. Use a paid tier or multiple free-tier keys. We cover provider selection in our best residential proxy providers 2026 review.

For Discord scraping (with bot permissions), the bot itself rate-limits per server, not per IP. Proxies do not help.

External authoritative reference: the Snapshot.js documentation covers the GraphQL schema, signing flow, and strategy types.

Common gotchas

Snapshot’s vp field is computed at proposal creation, not at vote time. A wallet that votes after selling its tokens still gets credit for its snapshot-block balance. Do not double-discount.
The state field on Snapshot proposals returns pending, active, closed. There is no passed or failed; that is your interpretation based on scores against quorum and threshold defined in the space settings.
Governor proposal IDs differ between Compound Bravo (uint256 sequential) and OpenZeppelin (keccak hash of proposal params). Code that assumes one breaks on the other.
Multi-chain DAOs (Optimism, Arbitrum) often reuse Snapshot space slugs but have different on-chain Governors per chain. Make sure you are matching the right pair.
Tally’s voteStats returns percentages calculated against quorum, not against votes cast. A proposal at 60% support may show 30% in percent if quorum is half the eligible supply.
Reading historical events with eth_getLogs on free-tier RPCs frequently times out on busy contracts. Chunk by 1,000-block windows for old contracts and 10,000 for newer ones, with retry-with-narrower-window on timeout.

Common analytical questions

Once you have the raw data, the questions worth answering include:

Vote concentration over time. Is a DAO’s governance becoming more or less concentrated? Plot top-10 vote share per proposal over time.

Delegate effectiveness. For DAOs with delegation (most modern Governor implementations), how often do delegates show up to vote on the tokens delegated to them? An 80% delegate participation rate is healthy; a 20% rate means delegation is dead weight.

Proposal pass rate. What fraction of proposals reach quorum and pass? A pass rate of 95% suggests rubber-stamping; a pass rate of 30% suggests genuine contention.

Treasury follow-through. For DAOs that vote on treasury allocations, do the funds actually move on-chain after the vote passes? Reconciliation against the treasury wallet’s transactions reveals whether governance is real or theater.

We dig into related on-chain forensics in our guide on scraping crypto exchange order books.

FAQ

Q: do I need a node to scrape on-chain governance?
No. Hosted RPC providers (Alchemy, Infura, QuickNode) cover the read use case perfectly. You only need your own node if you are indexing every block in real time at scale or need access to internal traces.

Q: can I scrape historical Snapshot proposals from defunct DAOs?
Yes. Snapshot persists historical data even for spaces that are inactive. You can pull the full archive of any space that has not been deleted by the admins.

Q: how do I correlate Snapshot space to on-chain Governor contract?
Most large DAOs publish the mapping in their docs. There is no universal index. Tally maintains a manually curated list. For obscure DAOs you may need to ask in the project’s Discord.

Q: what about Solana DAOs?
Solana DAOs use Realms (governance.so) instead of Snapshot/Governor. Realms exposes data via Solana RPC and the SPL Governance program. Different stack, same patterns.

Q: is voter address pseudonymity a concern for analysis?
For pure on-chain analysis, addresses are public and attribution is fair game. For combining vote data with off-chain identity (Twitter handles, Discord usernames), be careful about privacy expectations. See our GDPR compliance for web scraping guide for the data minimization patterns.

Q: how do I track DAO treasury movements alongside governance votes?
Index Transfer events from the treasury wallet on the same chain as the Governor. Cross-reference transactions occurring within 7 days after a ProposalExecuted event against the proposal’s calldata to confirm execution matched intent. Surprisingly often, a proposal authorizes a transfer that never happens because the multisig signers fail to coordinate.

Q: what is the right cadence for polling Snapshot?
For active spaces, every 5 minutes catches new proposals and vote updates. For an archive, a one-time backfill plus daily incremental sync is enough. Snapshot itself does not push updates, so polling is the only option without running a custom relayer.

Q: do I need to verify Snapshot signatures?
Snapshot’s hub already verifies signatures before storing votes, so the data you pull is pre-validated. Re-verifying is overkill for analysis but useful for audit-grade research where you want to prove independently that every vote is signed by the wallet it claims.

Closing

DAO governance scraping in 2026 is one of the cleanest data engineering problems in crypto. Snapshot and Tally do most of the heavy lifting; on-chain data fills the gaps; forum and Discord context completes the picture. The hard part is not collection but interpretation: turning vote records into intelligence about who actually controls a DAO, how decisions get made, and whether governance is functioning or theater. For the broader crypto data infrastructure picture see our crypto-defi category hub.

May 7, 2026

Mistral Large for Web Scraping Pipelines in 2026

Mistral Large is one of the few genuinely open-weight models that can compete with GPT-4 class systems on structured extraction — and in 2026, that matters a lot if you’re running a scraping pipeline at scale. The combination of a 128K context window, strong instruction-following, and self-hostable weights makes it worth a serious look for anyone tired of paying per-token on closed APIs.

What Mistral Large actually brings to scraping pipelines

The current release is Mistral Large 2 (mistral-large-2407), with 123B parameters. It runs comfortably on a 2xA100 or 4xA6000 setup, which puts it within reach for a dedicated scraping server. Context length is 128K tokens, enough to fit a full crawled HTML page, system prompt, and structured output schema in one shot.

For scraping-specific tasks, the key capabilities are:

Function calling / tool use: Mistral Large supports native function calling, which means you can define a JSON schema and get reliably structured output back without regex postprocessing. The JSON mode is stable enough for production use with a schema validation layer on top.
Instruction fidelity: On complex extraction prompts (“extract all job postings, normalize the salary field to USD, skip entries with missing location”), it follows multi-step instructions more precisely than smaller models like Mistral 7B or Mixtral 8x7B. It also handles nested schemas — arrays of objects with conditional fields — more consistently than most open models at this size.
Multi-language support: Useful if you’re scraping non-English sites — Mistral’s training data skews toward European languages, which shows up in extraction accuracy on French, German, and Italian pages.
Quantization tolerance: Mistral Large 2 runs well at 4-bit quantization (GPTQ or AWQ) with minimal quality degradation on extraction tasks. That gets the VRAM requirement down to around 65-70GB, which fits a 2xA100 40GB setup.

What it doesn’t bring: vision. You can’t feed it a screenshot of a rendered page the way you can with Gemini 2.0 Flash for Web Scraping, which handles multi-modal inputs natively. If your scraping targets rely on screenshots or PDFs, that’s a real gap.

Pricing comparison: Mistral Large vs alternatives

Here’s where things get interesting. Mistral via the official API is not cheap for high-volume scraping, but the self-hosted route changes the math.

Model	Input (per 1M tokens)	Output (per 1M tokens)	Self-hostable
Mistral Large 2 (API)	$2.00	$6.00	Yes (weights available)
GPT-4o	$2.50	$10.00	No
Claude 3.5 Sonnet	$3.00	$15.00	No
Gemini 1.5 Pro	$1.25	$5.00	No
Qwen 2.5 72B (API)	$0.40	$0.40	Yes
DeepSeek V3	$0.27	$1.10	Yes (limited)

If you’re running on the Mistral API, the pricing is competitive with GPT-4o but not by a huge margin. The real advantage is owning the weights. A self-hosted Mistral Large 2 on a leased A100 box can get you below $0.10 per 1M tokens at decent throughput, which is why it competes differently from, say, a closed model.

For pure cost optimization on hosted APIs, DeepSeek V3 for cheap web scraping LLM calls is significantly cheaper. Mistral Large’s edge is openness plus quality — not raw price.

Running Mistral Large locally with vLLM

If you’re self-hosting, vLLM is the standard serving layer. Here’s a minimal setup for a scraping inference server:

# Install vLLM
pip install vllm

# Serve Mistral Large 2 with tensor parallelism across 2 GPUs
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-Large-Instruct-2407 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --dtype bfloat16 \
  --port 8000

Once it’s running, you call it via the OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="mistralai/Mistral-Large-Instruct-2407",
    messages=[
        {"role": "system", "content": "Extract structured product data as JSON."},
        {"role": "user", "content": f"<html>{page_html}</html>"}
    ],
    response_format={"type": "json_object"},
    temperature=0.0
)

temperature=0.0 is non-negotiable for extraction tasks. Any randomness and you get inconsistent field names, hallucinated prices, and outputs that break your schema validation.

Integrating Mistral Large with Crawl4AI

Crawl4AI is the cleanest way to combine structured crawling with LLM extraction in 2026. It handles JS rendering, anti-bot evasion hooks, and has a native LLMExtractionStrategy that you can point at any OpenAI-compatible endpoint — including your local Mistral Large server.

from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

class JobPosting(BaseModel):
    title: str
    company: str
    salary_usd: float | None
    location: str

strategy = LLMExtractionStrategy(
    provider="openai/mistral-large",
    api_base="http://localhost:8000/v1",
    api_token="unused",
    schema=JobPosting.schema(),
    instruction="Extract job postings. Normalize salary to USD. Return null if missing."
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example-jobs-site.com/listings",
        extraction_strategy=strategy
    )
    print(result.extracted_content)

The provider field accepts any OpenAI-compatible base URL when combined with api_base. No additional configuration needed. This pattern also works with Llama 3 70B local scraping pipelines or Qwen 2.5 for web scraping if you want to swap models without touching your pipeline code.

Tradeoffs and when not to use it

Mistral Large is a good choice when:

You need self-hosted weights for data privacy or compliance reasons.
Your extraction tasks require long context (multiple pages, complex schemas).
You’re scraping European-language sites where its training data is stronger.
You want an open alternative to GPT-4 class quality without vendor lock-in.

It’s not the right choice when:

You need vision/screenshot understanding — use Gemini 2.0 Flash.
Budget is the primary constraint — Qwen 2.5 72B or DeepSeek V3 undercut it significantly on API pricing.
You’re running a simple scraper that doesn’t need 123B parameters — Mistral 7B or Mistral Nemo handle basic extraction at a fraction of the cost.
GPU availability is a problem — 123B quantized to 4-bit still needs ~70GB VRAM.

One thing worth flagging: Mistral’s function calling, while good, isn’t quite as rock-solid as GPT-4o on ambiguous extraction prompts. You’ll want schema validation (Pydantic works well here) and a retry loop for the ~5% of responses that don’t conform, especially on noisy HTML. A simple pattern is to catch ValidationError, strip the HTML down to the visible text using html2text, and retry once with the cleaner input — that alone drops non-conforming outputs to under 1% in most production pipelines.

Latency is also worth considering. Self-hosted Mistral Large at 4-bit on 2xA100 does around 15-25 tokens/second depending on batch size and prompt length. For real-time scrapers that need sub-second responses, that’s probably too slow. For async batch extraction running overnight or across a job queue, it’s completely fine.

Bottom line

Mistral Large 2 is the strongest open-weight option for production scraping pipelines where data privacy, self-hosting, or long-context extraction matters. It’s not the cheapest route, but for teams that can’t send page content to a closed API, it’s one of the few models that actually delivers GPT-4 class extraction quality on your own infrastucture. We’ll keep benchmarking alternatives as the open-source LLM landscape moves fast — follow DRT’s AI agent scraping coverage for updated comparisons.

May 7, 2026

Scraping NFT collection floor prices and metadata in 2026

Scrape NFT data jobs in 2026 sit at the intersection of three different systems that all need to agree before you have a complete record: the marketplace API for current listings and floor price, the underlying blockchain for ownership and provenance, and IPFS or Arweave for the actual metadata and image. Pull from only one source and you have a partial picture; pull from all three and you spend most of your engineering time on rate limits, gateway timeouts, and CDN caching weirdness. The market has consolidated since the 2021-2022 peak. Three platforms (OpenSea, Blur, Magic Eden) handle the vast majority of liquidity, and the long tail of marketplaces is mostly dead. That consolidation makes the scraping problem more tractable than it was three years ago.

This guide covers the practical mechanics of building an NFT data pipeline in 2026: which marketplace APIs are public versus paid, how to reconcile on-chain truth against marketplace cache, and the rate-limit and proxy patterns that let a small operation track 10,000+ collections continuously.

What “floor price” actually means and why it is hard

The floor price of a collection is the lowest active listing price on a marketplace. It sounds simple but it has three quirks that ruin naive scrapers.

First, floor is per-marketplace. A collection might have a 0.5 ETH floor on OpenSea and a 0.45 ETH floor on Blur because of fee differences and platform-specific listings. The “true” floor is the minimum across all marketplaces where the collection trades.

Second, floor is sensitive to outliers. A single listing at a clearly broken price (1 wei, or accidentally bid in DAI instead of ETH) becomes the technical floor until it gets bought or canceled. Production trackers usually compute a “true floor” by sorting listings ascending and taking the price at the 1st percentile or after the first few listings, ignoring obvious outliers.

Third, floor changes constantly. During a popular mint or hype cycle, floor can move 5-10% in a single minute. Polling at 5-minute cadence will miss most of the movement. Real-time floor tracking requires websocket or webhook subscriptions where the marketplace offers them.

OpenSea API in 2026

OpenSea operates the most widely used NFT API. As of 2026, the v2 API requires an API key for almost everything. You can request a free key through their developer portal but the free tier is limited to 4 requests per second and excludes the highest-value endpoints (real-time order book, historical sales). Paid plans start at around $200/month for higher rate limits and at $1500/month for the data tier with full historical access.

The free tier is enough for tracking floor price across a few hundred collections at 5-minute cadence. Past that you either pay or you augment with on-chain data (which is free but has its own complexity).

import time
import requests

class OpenSeaClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.session = requests.Session()
        self.session.headers.update({
            "X-API-KEY": api_key,
            "Accept": "application/json",
        })

    def get_collection(self, slug: str):
        url = f"https://api.opensea.io/api/v2/collections/{slug}"
        return self._get(url)

    def get_listings(self, slug: str, limit: int = 50):
        url = f"https://api.opensea.io/api/v2/listings/collection/{slug}/all"
        return self._get(url, params={"limit": limit})

    def get_stats(self, slug: str):
        url = f"https://api.opensea.io/api/v2/collections/{slug}/stats"
        return self._get(url)

    def _get(self, url, params=None, retries=3):
        for attempt in range(retries):
            resp = self.session.get(url, params=params, timeout=15)
            if resp.status_code == 429:
                time.sleep(2 ** attempt)
                continue
            resp.raise_for_status()
            return resp.json()

The stats endpoint is the cheapest way to track floor price because it returns the floor and total volume in a single call. The listings endpoint gives more detail but costs more rate-limit budget per collection.

Blur API: less documented but more powerful

Blur captured most of the professional trading volume in 2023 and has held it. Their API is technically not public but a usable endpoint exists at https://core-api.prod.blur.io/v1/. It requires a session token that you obtain by signing a wallet message. Tracking sites like Nansen and CryptoSlam use this endpoint. Blur tolerates it as long as you stay below roughly 100 requests per minute per session.

Blur’s particular value is the bid pool: a unified bidding mechanism where buyers commit ETH against an entire collection at a price. The sum of bids at each tier is a strong demand signal that does not exist on OpenSea. Scraping the Blur bid pool gives you data that is genuinely not available anywhere else without paying Blur for it directly.

def blur_collection_stats(slug: str, auth_token: str):
    url = f"https://core-api.prod.blur.io/v1/collections/{slug}"
    resp = requests.get(
        url,
        headers={
            "authToken": auth_token,
            "User-Agent": "Mozilla/5.0",
        },
        timeout=10,
    )
    return resp.json()

The auth token expires after about 24 hours. Production setups rotate the wallet signing automatically.

Magic Eden for Solana and multichain

Magic Eden is the dominant Solana NFT marketplace and has expanded to Bitcoin Ordinals, Polygon, and Ethereum. Their public API at https://api-mainnet.magiceden.dev/v2/ does not require an API key for most endpoints and tolerates 2 requests per second per IP.

def magiceden_collection_stats(symbol: str):
    url = f"https://api-mainnet.magiceden.dev/v2/collections/{symbol}/stats"
    resp = requests.get(url, timeout=10)
    return resp.json()

Magic Eden is the right tool when you care about Solana NFT data or Bitcoin Ordinals. For Ethereum NFTs, OpenSea and Blur have deeper liquidity and you should use them instead.

Marketplace API comparison

marketplace	API auth	free rate limit	floor price	full listing data	best for
OpenSea	API key (free + paid)	4 req/s free	yes	yes (paid for full)	Ethereum mainstream
Blur	session token	~100 req/min	yes	yes	Ethereum trading data
Magic Eden	none	2 req/s	yes	yes	Solana, Ordinals, multichain
LooksRare	none for public	5 req/s	yes	yes	Ethereum royalty-aware
X2Y2	API key (paid)	varies	yes	yes	Ethereum, declining
Tensor	none	aggressive	yes	yes	Solana professional
Reservoir	API key (free tier)	30k req/day free	yes	yes (aggregated)	aggregator across all ETH chains

Reservoir deserves special mention. They aggregate listings from OpenSea, Blur, LooksRare, and several others into a unified API. For most use cases, Reservoir is easier than running separate scrapers against each marketplace. The free tier is generous and covers most research work.

Marketplace decision matrix

Use this matrix when deciding which marketplace API to lean on for a given collection or use case:

use case	primary	fallback	notes
Top-100 Ethereum bluechip floor	Reservoir	OpenSea	Reservoir’s aggregated floor catches Blur and Sudoswap that OpenSea misses
Trader analytics on Ethereum	Blur	Reservoir	Blur’s bid pool data is unique and worth the auth pain
Solana NFT floor and trades	Magic Eden	Tensor	Magic Eden has full coverage; Tensor for execution-quality data
Bitcoin Ordinals	Magic Eden	Hiro	Magic Eden Ordinals support has matured; Hiro still useful for protocol-level inscriptions
L2 NFT collections (Base, Zora)	Reservoir	Native marketplace	Reservoir has Base and Zora; smaller chains are spotty
Royalty enforcement analysis	LooksRare + custom	Reservoir	Royalty data requires per-marketplace logic that aggregators flatten
Real-time floor tracking for trading bots	Reservoir webhooks	Marketplace websockets	Webhooks remove polling latency entirely

Pick the primary based on use case rather than alphabetical order. The fallback path is critical because every marketplace API has occasional outages and rate-limit surprises.

On-chain truth: when the marketplace is wrong

Marketplaces cache aggressively. A listing might be canceled or filled on-chain but still showing as active on the marketplace API for 30-60 seconds. For research this is fine. For trading or arbitrage detection it is fatal.

The authoritative source for ownership and listing status is the blockchain itself. Each marketplace operates its own listing contract (Seaport for OpenSea, BlurExchange for Blur). You can read the contract state directly via RPC and confirm which listings are still active.

from web3 import Web3

w3 = Web3(Web3.HTTPProvider("https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY"))

SEAPORT_ADDRESS = "0x00000000000000ADc04C56Bf30aC9d3c0aAF14dC"

def is_listing_active(order_hash: bytes) -> bool:
    seaport = w3.eth.contract(address=SEAPORT_ADDRESS, abi=SEAPORT_ABI)
    status = seaport.functions.getOrderStatus(order_hash).call()
    is_validated, is_cancelled, total_filled, total_size = status
    return is_validated and not is_cancelled and total_filled < total_size

For ownership, the ERC-721 ownerOf(tokenId) view is the truth. Marketplaces show owner data that may be stale by minutes. If you are scoring rarity or computing wallet holdings, you must call the contract directly.

Reservoir-first architecture

A pragmatic 2026 architecture treats Reservoir as the primary read source and falls back to direct marketplace APIs only when Reservoir lacks coverage or freshness. The flow:

Reservoir’s tokens/v6 and orders/asks/v5 endpoints aggregate listings across OpenSea, Blur, LooksRare, X2Y2, Sudoswap, and a half-dozen newer venues. One request returns the cross-venue floor and the venue-by-venue breakdown.
For collections Reservoir does not cover (newer chains, niche L2s), call the native marketplace API directly.
For sub-second freshness on top-tier collections, subscribe to Reservoir’s webhook events instead of polling.
Cross-check Reservoir’s data once per day against Etherscan transfer counts to catch indexing lag, which appears occasionally during high-volume mint events.

The single-source pattern saves dozens of integration headaches and the multi-marketplace reconciliation work. The cost is dependency on Reservoir staying alive and pricing reasonably; have the direct-marketplace fallback paths shipped and tested even if you do not use them daily.

IPFS metadata fetching

NFT metadata for image, name, traits, and description is usually stored on IPFS or Arweave, with a token URI that resolves to a JSON document. The chain stores the URI; the URI points to the metadata; the metadata points to the image.

IPFS gateways are notoriously unreliable. The default ipfs.io gateway frequently returns 504 timeouts. Production scrapers maintain a list of gateways and round-robin requests across them with retry logic.

GATEWAYS = [
    "https://ipfs.io/ipfs/",
    "https://cloudflare-ipfs.com/ipfs/",
    "https://gateway.pinata.cloud/ipfs/",
    "https://nftstorage.link/ipfs/",
    "https://w3s.link/ipfs/",
]

def fetch_ipfs(cid: str, timeout: int = 10):
    for gateway in GATEWAYS:
        try:
            resp = requests.get(gateway + cid, timeout=timeout)
            if resp.status_code == 200:
                return resp.json() if "json" in resp.headers.get("Content-Type", "") else resp.content
        except requests.RequestException:
            continue
    raise IPFSFetchError(cid)

For projects you scrape repeatedly, pin the metadata to your own IPFS node or upload it to S3. This eliminates gateway flakiness and makes downstream queries instant.

Proxy and rate limit strategy

Marketplace APIs are the rate-limit bottleneck for NFT scraping. The mitigation pattern is multi-key rotation rather than IP rotation. Each developer account gets its own API key, and you round-robin across keys. OpenSea allows multiple API keys per account, and you can register multiple accounts (within their terms) for additional throughput.

For unkeyed endpoints (Magic Eden, Blur with rotating tokens), proxies do help. Use residential proxies to avoid the “all your traffic from one AWS IP” pattern that gets fingerprinted. We compare options in our best residential proxy providers 2026 review.

For RPC calls to read on-chain data, you have your own rate limits with your RPC provider (Alchemy, Infura, QuickNode). Most providers have generous free tiers, and one provider key per worker process avoids cross-contamination of rate limits.

Storage schema

NFT data is multidimensional and most teams over-design the schema on day one. Start simple:

CREATE TABLE collections (
    slug TEXT PRIMARY KEY,
    chain TEXT NOT NULL,
    contract_address TEXT NOT NULL,
    name TEXT,
    total_supply INTEGER,
    UNIQUE (chain, contract_address)
);

CREATE TABLE collection_snapshots (
    slug TEXT NOT NULL REFERENCES collections(slug),
    captured_at TIMESTAMPTZ NOT NULL,
    floor_price_eth NUMERIC,
    volume_24h_eth NUMERIC,
    sales_24h INTEGER,
    listed_count INTEGER,
    owner_count INTEGER,
    PRIMARY KEY (slug, captured_at)
);

CREATE INDEX ON collection_snapshots (captured_at DESC);

CREATE TABLE token_listings (
    chain TEXT NOT NULL,
    contract_address TEXT NOT NULL,
    token_id NUMERIC NOT NULL,
    marketplace TEXT NOT NULL,
    price_eth NUMERIC,
    seller_address TEXT,
    listed_at TIMESTAMPTZ,
    expires_at TIMESTAMPTZ,
    order_hash TEXT,
    is_active BOOLEAN,
    PRIMARY KEY (chain, contract_address, token_id, marketplace, order_hash)
);

For 1000 collections at 5-minute snapshot cadence, you generate 288,000 collection snapshot rows per day plus listing-level data. PostgreSQL handles this comfortably for a year before you need to consider partitioning.

Snapshot cadence vs storage tradeoff

Cadence has compounding effects on storage and rate-limit budget. The right cadence depends on tier:

Tier 1 (top 200 collections, mints, news-driven assets): 60 seconds. These move fast enough that a 5-minute gap loses real signal. At 200 collections this is 17,280 calls per day. With 4 OpenSea API keys you can absorb that comfortably.
Tier 2 (next 2,000 collections): 5 minutes. Still useful for tracking trends; movement is slower so the lower cadence preserves more than 90% of meaningful signal.
Tier 3 (long tail, 10,000+ collections): 1 hour. Catches major liquidity changes without burning the rate budget. Many long-tail collections have zero listed items most days, so most calls return identical data.

A reasonable two-week-old listing without movement can be polled daily. Reservoir’s snapshot endpoint accepts batched collection lookups, which is the cheapest way to refresh long-tail tiers in bulk.

Sales history and provenance

For provenance and sales history, the on-chain approach is more reliable than marketplace APIs. Every NFT transfer emits a Transfer event from the ERC-721 contract. Every marketplace sale emits a marketplace-specific event (Seaport’s OrderFulfilled, etc.) that includes price.

Indexing services like Goldsky, The Graph, Subsquid, and the previously mentioned Reservoir aggregate this data and expose it via GraphQL. For one-off queries, use a service. For continuous indexing of specific collections, run a node and subscribe directly to the events.

We cover the broader on-chain indexing patterns in our crypto-defi category hub and our deep dive on scraping crypto exchange order books.

External authoritative reference: the OpenSea API documentation covers the current endpoint catalog and rate-limit policy.

Cost worked example

A practical 2026 setup tracking 5,000 collections across Ethereum and Solana with mixed cadence costs roughly:

Reservoir API free tier ($0) plus one paid key for the higher rate limits ($150/mo)
OpenSea free key for floor stats; no paid tier needed if you reconcile via Reservoir
Magic Eden public API ($0)
Alchemy Growth tier for on-chain reads ($49/mo)
Pinata or NFT.Storage for self-pinned IPFS metadata ($20/mo, 50 GB)
$40/mo VPS for the collector (4 vCPU, 8 GB)
$25/mo Postgres on a small managed instance
30 IPs of residential proxy for Blur and unauth endpoints (~$50/mo on a starter pack)

Total: about $335/month. The same coverage purchased through a vendor (NFTGo Enterprise, Nansen Query) runs $1,500-5,000/month. Self-hosting becomes a clear win once you cross 200 collections and need historical depth.

Common failure modes

The most common failure mode in NFT scrapers is treating the marketplace API as ground truth. Always reconcile against on-chain state for ownership and listing status. The second most common failure is not handling IPFS gateway timeouts. Build retry logic across multiple gateways from day one.

The third failure mode is overreacting to outlier listings. A single 0.0001 ETH listing on a 1 ETH floor collection is almost always either a wash trade attempt or a scam. Production trackers ignore listings below the 1st percentile of recent floor history.

FAQ

Q: which API gives the best Ethereum coverage?
For aggregated Ethereum data, Reservoir is the easiest path. For raw OpenSea data, the official API. For trading depth, Blur. Most production setups combine all three.

Q: do I need a wallet to scrape NFT data?
Not for read-only scraping of public marketplace data. You need a wallet to access Blur’s authenticated endpoints and to interact directly with marketplace contracts on-chain. A throwaway wallet with no funds works fine for read-only authentication.

Q: how do I track floor price changes in real time?
OpenSea offers a streaming events API that pushes order events. Reservoir has webhooks. Without paid endpoints, polling at 60-second cadence is the practical floor for “near real time.”

Q: can I scrape rarity scores?
Rarity is computed from trait distribution, which you derive from the metadata of all tokens in a collection. Once you have the metadata table, rarity computation is trivial. Most rarity tools use the same algorithm and produce similar scores.

Q: what about Bitcoin Ordinals?
Ordinals data lives on the Bitcoin chain and is indexed by services like Magic Eden, Hiro, and Ord.io. The standard Bitcoin RPC does not return Ordinal data directly; you need an indexer in front of a full node.

Q: how do I tell a wash trade from a real sale?
Wash trades typically loop between two related wallets (often funded from the same source within the previous 7 days), at suspiciously round prices, with no time between transfers. Cross-reference seller and buyer addresses against a wallet-cluster service like Arkham or against your own clustering on shared funding sources. Flagging is heuristic, not exact, but rules out 80-90% of obvious wash activity.

Q: do marketplaces ever sue scrapers?
Cease-and-desist letters happen, lawsuits are rare and typically reserved for projects that resell the data as a competing product. Personal use, research, and non-commercial analytics have not historically attracted enforcement. Building a public dashboard that displays floor prices crosses into commercial territory and is where you should consult counsel.

Q: should I use The Graph subgraphs instead of direct RPC?
The Graph is excellent for queries that span many blocks and aggregate data, like “all sales of collection X in the last month.” Direct RPC is better for single-state lookups, like “is this listing currently active.” Use both: subgraphs for analytics, RPC for live truth.

Closing

NFT scraping in 2026 is a multi-source reconciliation problem more than a pure scraping problem. The marketplace APIs give you the user-facing view; the chain gives you the truth; IPFS gives you the content. Build pipelines that treat all three as inputs and reconcile them, and you can run a research-grade NFT data system at hobbyist cost. For broader infrastructure see our crypto-defi category hub.

May 7, 2026

Scraping crypto exchange order books in 2026

Scrape crypto order books pipelines have matured into one of the most demanding data engineering problems on the public internet. A single Binance BTC/USDT depth feed pushes 50-200 updates per second during normal hours and spikes past 1000 per second during news events. Multiply that by 200 trading pairs across 8 exchanges and you have a firehose that requires careful architecture decisions before you write a single line of code. The good news is that almost every major exchange exposes order book data through public websockets that do not require API keys for read access. The bad news is that latency, message ordering, and gap recovery decisions you make in the first day of building will haunt the system for years.

This guide covers the practical mechanics of scraping crypto order books in 2026: which exchange feeds work, how to normalize depth across venues, the latency vs cost tradeoff for hosting, and the storage patterns that let small teams keep multi-month order book history without burning $20,000 a month on infrastructure.

Why scrape order books instead of buying the data

Three vendors dominate paid crypto market data: Kaiko, CryptoCompare, and Tardis.dev. They are excellent. They are also expensive: a full L2 order book history feed for 50 pairs across 5 exchanges runs $5,000-25,000 per month depending on tier. For a quant fund this is rounding error. For an indie research project, an algo trading hobbyist, or a startup building MEV tooling, it is the entire budget.

Self-collecting order book data costs roughly $200-500 per month in compute and storage if you are disciplined. The tradeoff is engineering time. The first month is hard. After that the pipeline runs itself with weekly babysitting. For research that depends on data going back further than the day you started collecting, you still need a vendor for the historical backfill, but ongoing collection is cheap.

What “order book” actually means at the wire level

Every exchange streams order book updates as either snapshots or deltas. A snapshot is the full current state of the book at a moment in time. A delta is a list of price-level changes since the last update. Most production feeds are delta-based with periodic snapshot resyncs.

A typical delta message looks like this from Binance:

{
  "e": "depthUpdate",
  "E": 1715000000000,
  "s": "BTCUSDT",
  "U": 12345600,
  "u": 12345610,
  "b": [["63500.00", "0.5"], ["63499.50", "0"]],
  "a": [["63501.00", "1.2"], ["63502.00", "0.8"]]
}

The b and a arrays are bid and ask updates. A quantity of 0 means delete that price level. The U and u fields are the first and last update IDs in this message, which you use to detect gaps. If you miss a message, you have to refetch the full snapshot from the REST endpoint and replay deltas from there.

This gap recovery logic is where most amateur scrapers break. A naive listener that just appends deltas without checking sequence numbers will silently corrupt the book within minutes.

import asyncio
import json
import websockets
import requests
from collections import defaultdict

class BinanceDepthListener:
    def __init__(self, symbol: str):
        self.symbol = symbol.upper()
        self.bids = {}  # price -> qty
        self.asks = {}
        self.last_update_id = None

    async def fetch_snapshot(self):
        url = f"https://api.binance.com/api/v3/depth?symbol={self.symbol}&limit=1000"
        resp = requests.get(url, timeout=10)
        data = resp.json()
        self.last_update_id = data["lastUpdateId"]
        self.bids = {float(p): float(q) for p, q in data["bids"]}
        self.asks = {float(p): float(q) for p, q in data["asks"]}

    def apply_delta(self, msg):
        if msg["u"] <= self.last_update_id:
            return
        if msg["U"] > self.last_update_id + 1:
            raise GapDetectedError(msg["U"], self.last_update_id)
        for price, qty in msg["b"]:
            p, q = float(price), float(qty)
            if q == 0:
                self.bids.pop(p, None)
            else:
                self.bids[p] = q
        for price, qty in msg["a"]:
            p, q = float(price), float(qty)
            if q == 0:
                self.asks.pop(p, None)
            else:
                self.asks[p] = q
        self.last_update_id = msg["u"]

    async def run(self):
        url = f"wss://stream.binance.com:9443/ws/{self.symbol.lower()}@depth@100ms"
        await self.fetch_snapshot()
        async with websockets.connect(url) as ws:
            async for raw in ws:
                msg = json.loads(raw)
                try:
                    self.apply_delta(msg)
                except GapDetectedError:
                    await self.fetch_snapshot()

The @100ms suffix in the websocket URL is critical. Without it you get the default 1000ms depth stream, which is slow. The 100ms feed is the highest cadence Binance offers without paying for the institutional pro feed.

Exchange feed comparison

exchange	websocket URL	best feed cadence	snapshot via REST	message ordering
Binance Spot	wss://stream.binance.com:9443	100ms	yes, /api/v3/depth	sequence IDs
Binance Futures	wss://fstream.binance.com	100ms	yes, /fapi/v1/depth	sequence IDs
Coinbase	wss://ws-feed.exchange.coinbase.com	per event	snapshot in stream	sequence per product
OKX	wss://ws.okx.com:8443	100ms	snapshot in stream	checksum per update
Bybit	wss://stream.bybit.com/v5/public/spot	per event	snapshot via REST	update IDs
Kraken	wss://ws.kraken.com/v2	per event	snapshot in stream	checksum per book
KuCoin	wss://ws-api.kucoin.com	per event	snapshot via REST	sequence IDs
Bitget	wss://ws.bitget.com/v2	100ms	snapshot in stream	checksum per update

Coinbase, OKX, and Kraken include the initial snapshot in the websocket stream when you subscribe. Binance, Bybit, and KuCoin require a separate REST call. The stream-included snapshot is faster to start with but harder to recover from on disconnect because you need to resubscribe to get a new one. The REST snapshot pattern is more flexible.

Latency budget

Order book data is only useful if you can act on it within the timeframe relevant to your strategy. For research and analytics, 100-500ms latency is fine. For market making or arbitrage, you need single-digit milliseconds. The hosting choice changes accordingly.

Binance hosts its main matching engine in AWS Tokyo. The lowest latency to its websocket is from a server in ap-northeast-1. Coinbase runs from AWS US-East-1 (Ashburn). OKX runs from Hong Kong. Bybit from AWS Singapore. If you want sub-10ms feeds you have to host in the same AZ as the exchange and pay for cross-connect or AWS DX.

For research-grade scraping, a $40/month VPS in Tokyo or Singapore from Hetzner, OVH, or Vultr gets you under 30ms to most exchanges. That is fine for everything except active trading strategies.

Normalizing depth across exchanges

Every exchange publishes its order book in slightly different formats. To do any cross-exchange analysis you need a common schema. The minimum viable normalized record:

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class NormalizedBook:
    exchange: str
    symbol: str  # canonical, like "BTC-USDT"
    captured_at_ms: int  # exchange timestamp
    received_at_ms: int  # local receive timestamp
    bids: List[Tuple[float, float]]  # sorted desc
    asks: List[Tuple[float, float]]  # sorted asc
    sequence: int

The two timestamps are critical. captured_at_ms is what the exchange reported. received_at_ms is when your collector got the message. The difference tells you network latency. Without both you cannot diagnose feed degradation.

Symbol normalization is annoying but mechanical. Binance uses BTCUSDT, Coinbase uses BTC-USD, OKX uses BTC-USDT. Maintain a mapping table:

SYMBOL_MAP = {
    "binance": {"BTCUSDT": "BTC-USDT", "ETHUSDT": "ETH-USDT"},
    "coinbase": {"BTC-USD": "BTC-USD", "ETH-USD": "ETH-USD"},
    "okx": {"BTC-USDT": "BTC-USDT", "ETH-USDT": "ETH-USDT"},
}

Note that USD and USDT are not the same. Coinbase quotes against USD, most others against USDT. For arbitrage analysis you need to track this carefully and convert via the USDT/USD pair.

Storage: the real cost driver

Naive storage of every depth update across 50 pairs and 5 exchanges produces 100-500 GB per day. Compressed Parquet brings it down to 20-80 GB. Most operations cannot afford to keep raw deltas indefinitely, so the standard pattern is tiered:

Hot: last 7 days of full deltas in ClickHouse or DuckDB
Warm: last 90 days of 1-second OHLCV bars + L2 snapshots every minute
Cold: indefinite history of 1-minute snapshots in compressed Parquet on S3

This gets you to roughly $50-150/month in storage for 5-exchange, 50-pair coverage. The compression ratio matters: Parquet with ZSTD level 9 hits about 8:1 on order book data because most price levels do not change between snapshots.

import pyarrow as pa
import pyarrow.parquet as pq

def write_snapshot_batch(snapshots: list, path: str):
    table = pa.Table.from_pylist(snapshots)
    pq.write_table(
        table,
        path,
        compression="zstd",
        compression_level=9,
        use_dictionary=True,
    )

ClickHouse handles the hot path well. A modest 4-core server holds 30 days of full deltas for 50 pairs across 5 exchanges with room to query.

Recovering from a corrupted book

The deepest source of bugs is when your in-memory book diverges from reality and you do not notice for hours. Defensive checks that run continuously in the background:

Top-of-book sanity: every minute, fetch the top of book via REST and compare to your cached book’s top bid/ask. A divergence greater than 0.5% on a liquid pair means resync.
Crossed book detection: the highest bid should never exceed the lowest ask. If it does, the book is corrupted; trigger a full snapshot resync.
Negative quantity detection: quantities should always be positive. Any negative value indicates a delta-application bug.
Sequence gap counter: track the rate of detected gaps per hour. A baseline of 0.5-2 gaps per hour per pair is normal. A sudden jump indicates network degradation worth investigating before it affects downstream consumers.
Idle channel detection: if no message arrives for a pair in 30 seconds during a normally active hour, force a reconnect. Idle does not always mean closed; the socket can hang in a half-open state.

A simple watchdog process running these checks across all pairs adds about 3% overhead and catches >95% of the silent corruption cases that otherwise show up as bad model outputs days later.

Connection management at scale

Each websocket connection costs file descriptors and memory. A single Python process with websockets library can comfortably handle 100-200 simultaneous connections. Past that, you fragment across processes or move to a more efficient runtime.

For the 50-pair, 5-exchange scenario, the practical architecture is one process per exchange handling all that exchange’s pairs as a single multiplexed subscription where the protocol allows it. Binance and Coinbase both accept multi-symbol subscribe messages on a single connection. OKX requires one subscription per channel but supports multiplexing across symbols.

async def binance_multi_symbol(symbols: list[str]):
    streams = "/".join([f"{s.lower()}@depth@100ms" for s in symbols])
    url = f"wss://stream.binance.com:9443/stream?streams={streams}"
    async with websockets.connect(url, ping_interval=20) as ws:
        async for raw in ws:
            msg = json.loads(raw)
            stream = msg["stream"]
            data = msg["data"]
            await process(stream, data)

Reconnect logic must include exponential backoff plus full snapshot refetch. A 30-second disconnect during a busy hour means thousands of missed updates. Just resuming the websocket subscription will give you a corrupted book.

Proxy considerations

Public websocket endpoints generally do not enforce strict per-IP rate limits because they are designed for HFT clients with stable connections. You usually do not need a proxy for normal volume.

The exception is REST snapshot endpoints. Binance imposes a weight-based rate limit on REST: 6000 weight per minute per IP, and the depth endpoint at limit=1000 costs 50 weight. That allows about 120 snapshots per minute. If you have more than 50 pairs and a busy gap-recovery day, you can blow through the limit.

For that case, route REST snapshot traffic through a small datacenter proxy pool of 5-10 IPs. The websocket can stay on your direct connection. We cover the broader proxy strategy in our best datacenter proxy providers 2026 review.

Storage cost worked example

A practical breakdown for the standard 50-pair, 5-exchange research deployment looks like this. Raw delta volume averages 18 GB per exchange per day during normal weeks, climbing to 50-80 GB during high-volatility events. Across 5 exchanges that is 90-400 GB per day uncompressed. After ZSTD-9 Parquet compression with column dictionaries, expect 11-50 GB per day landing in cold storage. At AWS S3 Standard pricing of $0.023 per GB-month and a 90-day rolling cold tier, total storage cost falls between $25 and $110 per month for the historical archive, plus another $40-80 per month for the ClickHouse hot tier on a 4 vCPU 8 GB box.

Egress is the silent cost killer. If you ever need to move 5 TB of historical Parquet to a different cloud for a research project, AWS will bill $450 in egress alone. Either run the analytics in the same region as the bucket or use Cloudflare R2 / Backblaze B2, both of which have free or near-free egress and price storage at $0.005-0.015 per GB-month. The R2 path saves real money once the archive grows past 1 TB.

Cross-exchange arbitrage signal extraction

The classic application of order book scraping is arbitrage detection. The simplest version: for every (base asset, quote asset) pair, find the highest bid across all exchanges and the lowest ask across all exchanges. If the highest bid is greater than the lowest ask plus fees, there is an opportunity (in theory).

In practice, transfer time, withdrawal fees, exchange-specific rules, and slippage eat most of the gap. But the same data lets you compute the cross-exchange spread distribution over time, which is genuinely useful for understanding market microstructure and for backtesting more sophisticated strategies.

def best_bid_ask(books: dict[str, NormalizedBook]) -> dict:
    best_bid = max((b.bids[0] for b in books.values() if b.bids), key=lambda x: x[0])
    best_ask = min((b.asks[0] for b in books.values() if b.asks), key=lambda x: x[0])
    return {
        "best_bid": best_bid[0],
        "best_ask": best_ask[0],
        "spread": best_ask[0] - best_bid[0],
    }

Common gotchas

A few traps from real production deployments:

Binance’s lastUpdateId from the REST snapshot is occasionally lower than the first delta U you have already buffered. The official spec says to discard buffered deltas where u <= lastUpdateId and apply the rest, but ensure your buffer holds at least 200 deltas during reconnect because the snapshot can lag a busy moment by several seconds.
Coinbase’s match channel and the level2 channel are separate. To compute correct mid-price you need only level2; matches are useful for trade history but should not feed the book directly.
OKX checksums are CRC32 over a specific concatenation of the top 25 levels. If the checksum fails twice in a row, OKX expects you to resubscribe, not to reconnect the socket.
Bybit’s v5 endpoint changed sequence semantics from v3. Old code that assumed continuous sequence numbers across all symbols on one connection will silently corrupt because v5 sequences are per symbol.
Kraken occasionally sends a snapshot mid-stream without warning when their internal book reconciliation kicks off. If you see a snapshot event after subscribing, treat it as the new baseline and drop your existing book state.
Time synchronization on the collector matters. NTP drift of even 50 ms makes the received_at_ms minus captured_at_ms latency metric meaningless. Run chrony with multiple peers and monitor offset.

Compliance and exchange terms of service

Most exchanges’ public market data is, by their own terms of service, freely usable for personal and commercial research. Redistributing the data as a real-time feed competing with the exchange’s institutional product is a different matter and usually requires a market data license.

Binance, Coinbase, and Kraken all explicitly permit personal trading and research use of public market data. Building a competing aggregator service that resells the data is gray area. For the typical research, alpha generation, or in-house analytics use case you are fine.

External authoritative reference: the Binance API documentation covers websocket spec and rate limits.

FAQ

Q: can I scrape order books without websockets?
Yes, you can poll the REST depth endpoint. But REST polling at 1-second cadence misses 99% of updates and consumes more rate-limit budget than the websocket equivalent. Websockets are the only sensible choice for production.

Q: how do I handle exchange downtime?
Most exchanges schedule maintenance windows in advance. Subscribe to their status APIs and stop reconnect attempts during announced windows to avoid wasting API budget. For unannounced outages, use exponential backoff capped at 5 minutes between retries.

Q: do I need to store every delta or are snapshots enough?
Depends on use case. For backtesting you want deltas because you need event-by-event playback. For analytics on spreads and depth, 1-second snapshots are usually sufficient and 100x cheaper to store.

Q: what about DEX order books?
DEXs publish state on-chain, not via websocket. You read it via RPC calls or via subgraph queries. Uniswap V3 and similar AMMs do not have order books at all; they have liquidity curves. dYdX and similar perp DEXs do have order books and expose them via gRPC. Different problem.

Q: how do I detect washtrading from order book data?
Look for orders that get placed and immediately taken by the same exchange’s matching engine, identical-size orders bouncing between two price levels, and trade volume spikes that do not correspond to depth movement. This is a deep topic; treat it as a separate downstream analysis on top of the raw scraped data.

Q: should I use a managed message broker like Kafka in front of the storage layer?
Only if you have multiple downstream consumers that need the live feed. For a single-consumer research pipeline, Kafka adds operational overhead without value. A simple in-process queue from listener to writer is enough. Kafka becomes worth the cost once you have a trading bot, an analytics dashboard, and a backtester all consuming the same feed concurrently.

Q: do I need a colocation server for arbitrage research?
No. Colocation matters for execution, not research. A Tokyo VPS that gets order book data 30-50 ms after the matching engine is plenty for spotting historical opportunities and modeling spreads. Save the colocation budget for after you have validated the strategy.

Q: can a single Python process really keep up with a busy day?
With asyncio and uvloop, yes, up to about 200 simultaneous connections and several thousand messages per second. Past that, switch to Rust, Go, or split across processes per exchange. Most research workloads never hit that ceiling.

Closing

Scraping crypto order books at scale in 2026 is a tractable engineering problem if you respect the wire-level details: sequence numbers, gap recovery, snapshot resyncs, and the difference between exchange and local timestamps. The first month is the hard part. Once your collector survives a Sunday night Coinbase reconnect storm and a Binance maintenance window without corrupting state, you have a pipeline that generates millions of dollars worth of vendor data for the cost of a small VPS. For the broader market data infrastructure picture see our crypto-defi category hub.

May 7, 2026

Scraping Twitch and Kick streaming metadata in 2026

Scrape Twitch data programs in 2026 look very different from the 2022 era. Twitch tightened its Helix API quotas, started rejecting tokens with no scope justification, and pushed almost every interesting endpoint behind a partner approval form. Kick, the upstart competitor, took the opposite approach and exposes most metadata through a public REST API that is currently undocumented but stable enough for production use. Anyone tracking the streaming economy at scale needs both platforms feeding the same pipeline, and that means dealing with two different rate-limit philosophies, two different auth flows, and two different fingerprinting baselines.

This guide covers the practical mechanics of scraping Twitch and Kick streaming metadata in 2026: which endpoints actually work without partner status, how to harvest viewer counts and chat snapshots without burning through your token budget, and the proxy and session strategy that lets a small operation pull data on tens of thousands of channels per day.

Why streaming metadata is worth scraping

Streaming data is the closest thing the entertainment industry has to a real-time audience signal. A channel’s average concurrent viewers, peak viewers, follow growth, and subscriber count change every minute and tell you which creators are accelerating, which categories are getting hot, and which sponsorship slots are about to become valuable. Brands buying influencer placements pay for this data through Streamlabs, Stream Hatchet, and SullyGnome, but those services charge enterprise prices and lock the underlying timeseries behind dashboards. A scraping pipeline gives you the raw numbers and lets you build whatever analysis you actually need.

Common use cases include creator discovery for sponsorship outreach, category trend tracking for esports orgs, copyright watch for music labels monitoring DJ streams, and competitive intelligence for streaming platforms benchmarking each other. Tracker-style sites like TwitchTracker and KickStats are themselves built on continuous scraping operations.

What Twitch’s Helix API actually gives you in 2026

Twitch publishes Helix as the official API. It works, but the limits are tighter than most tutorials admit. A standard developer app gets 800 points per minute per token. Most useful endpoints cost 1 point per request, but the high-value ones (Get Streams, Get Videos, Get Subscriptions) cost more and have additional per-broadcaster restrictions. The single biggest constraint is that listing all live streams in a category caps at 100 results per page and requires pagination, and pagination cursors expire after about 10 minutes.

A single token, polling Get Streams across 50 categories every 5 minutes, will exhaust its budget. The fix is rotating multiple app credentials, each generating its own token, and round-robining requests across them. Twitch tolerates this so long as each app is registered to a real developer account.

import time
import requests

class TwitchClient:
    def __init__(self, client_id: str, client_secret: str):
        self.client_id = client_id
        self.client_secret = client_secret
        self.token = None
        self.token_expires_at = 0

    def _refresh_token(self):
        resp = requests.post(
            "https://id.twitch.tv/oauth2/token",
            data={
                "client_id": self.client_id,
                "client_secret": self.client_secret,
                "grant_type": "client_credentials",
            },
            timeout=10,
        )
        resp.raise_for_status()
        data = resp.json()
        self.token = data["access_token"]
        self.token_expires_at = time.time() + data["expires_in"] - 300

    def _headers(self):
        if time.time() >= self.token_expires_at:
            self._refresh_token()
        return {
            "Client-ID": self.client_id,
            "Authorization": f"Bearer {self.token}",
        }

    def get_live_streams(self, game_id: str, first: int = 100):
        url = "https://api.twitch.tv/helix/streams"
        params = {"game_id": game_id, "first": first}
        cursor = None
        while True:
            if cursor:
                params["after"] = cursor
            resp = requests.get(url, headers=self._headers(), params=params, timeout=10)
            if resp.status_code == 429:
                time.sleep(int(resp.headers.get("Ratelimit-Reset", 60)))
                continue
            data = resp.json()
            for stream in data.get("data", []):
                yield stream
            cursor = data.get("pagination", {}).get("cursor")
            if not cursor:
                break

For chat data, Helix offers Get Chatters but it requires a moderator scope on the target channel, which you do not have. The practical alternative is connecting to the Twitch IRC bridge with an anonymous justinfan token and logging messages that way. IRC chat scraping is rate-limited per join (around 50 channels per 15 seconds) but the total throughput is generous if you batch channels into groups and round-robin connections.

Twitch GraphQL: the unofficial backdoor

Twitch’s web client uses an internal GraphQL endpoint at https://gql.twitch.tv/gql that exposes far more than Helix. Stream Hatchet, TwitchTracker, and most third-party analytics tools use this endpoint. It is not officially supported, the schema can change, and abuse will get your IP blocked. But it is the fastest way to pull viewer history, follow growth, and category-level analytics without partner status.

The GraphQL endpoint uses persistent queries identified by SHA256 hashes. The hashes change occasionally, so any production scraper needs a fallback that scrapes the current hash from the web client when the cached one stops working.

import requests

GQL_URL = "https://gql.twitch.tv/gql"
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"  # public web client id

def get_user_followers(login: str):
    payload = [{
        "operationName": "ChannelFollowers",
        "variables": {"login": login},
        "extensions": {
            "persistedQuery": {
                "version": 1,
                "sha256Hash": "...",  # current hash, rotate when stale
            }
        }
    }]
    resp = requests.post(
        GQL_URL,
        json=payload,
        headers={"Client-ID": CLIENT_ID},
        timeout=10,
    )
    return resp.json()

The unofficial GraphQL endpoint is the right tool when you need historical viewer curves or hourly follow deltas that Helix does not expose. Use it through a residential or mobile proxy pool, not from a data center IP.

Discovering and refreshing GraphQL hashes

The brittle part of the GraphQL strategy is the hash management. Twitch ships its web client as a stack of bundled JavaScript files served from static.twitchcdn.net. Each persisted query hash is embedded in those bundles. When Twitch deploys a new client version, hashes can rotate. A robust scraper does not hardcode hashes; it ships with a discovery routine that fetches the current bundle, regexes out the operation-to-hash map, and caches the result for an hour. The discovery script runs once per hour as a sidecar so the main scraper never sees a stale hash.

Operations worth caching: ChannelFollowers, ChannelShell, ChannelVideoCore, StreamMetadata, VideoPlayerStreamMetadata, ChannelPanels, and the various ClipsCards__* operations. Most analytics use cases need only a dozen operations, not the hundreds the web client touches.

Twitch’s anti-abuse team monitors the GraphQL endpoint for anomalous query patterns. Sending a hash-only request with no Client-Integrity header is allowed for read-only queries today, but write operations and a small set of read operations now demand the integrity token, which is generated by a bundled WASM module that does some browser environment checks. If your scraper starts seeing failed integrity check errors on previously-working operations, that operation has been moved behind the integrity wall and you need a real headless browser session to mint the token.

Scraping Kick: the easier sibling

Kick is the upstart streaming platform that launched in 2023 as a Twitch competitor. As of 2026 it has roughly 10% of Twitch’s daily active broadcasters but a disproportionate share of the gambling and live-poker categories. Most importantly for our purposes, Kick exposes a public REST API at https://kick.com/api/v2/ that does not require auth for read operations.

Endpoints worth knowing:

GET /api/v2/channels/{slug} returns channel profile, follower count, livestream object if live
GET /api/v2/livestreams returns currently live streams, paginated
GET /api/v2/categories/{slug}/streams returns streams in a category
GET /api/v2/channels/{slug}/clips returns clips for a channel

Kick also exposes chat through a Pusher websocket at wss://ws-us2.pusher.com/app/eb1d5f283081a78b932c. The channel name format is chatrooms.{chatroom_id}.v2. You subscribe and receive every message in real time without auth.

import requests

def get_kick_live_streams(category: str, page: int = 1):
    url = f"https://kick.com/api/v2/categories/{category}/streams"
    resp = requests.get(
        url,
        params={"page": page, "limit": 100},
        headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
            "Accept": "application/json",
        },
        timeout=10,
    )
    if resp.status_code == 200:
        return resp.json()
    return None

The catch with Kick: Cloudflare sits in front of the API and uses TLS fingerprinting to filter scrapers. A naive Python requests call works for low volume but starts returning 403 responses once you exceed about 30 requests per minute from a single IP. The fix is using curl_cffi (which mimics real browser TLS signatures) or running through a fingerprint-aware HTTP client.

Comparison: Twitch Helix, Twitch GraphQL, Kick API

capability	Twitch Helix	Twitch GraphQL (unofficial)	Kick API v2
auth required	yes (OAuth)	no (public client ID)	no
rate limit per IP	800 points/min/token	unknown, ~60 req/min observed	~30 req/min before Cloudflare
live streams per category	yes, 100/page	yes	yes, 100/page
historical viewer count	no	yes (last 60 days)	no
chat access	IRC bridge or moderator scope	limited	Pusher websocket, public
video metadata	yes	yes (richer)	yes
follower history	no	yes	only current count
official support	yes	no, breaks occasionally	undocumented but stable

For most use cases, the right architecture is Helix for stable production polling, GraphQL for backfills and metrics that Helix omits, and Kick API for full Kick coverage. Avoid the Kick scraping libraries on PyPI that wrap the API with Selenium. They are slower than direct API calls and easier to fingerprint.

Chat as a metadata source

Chat is more useful than people realize. The volume of messages per minute is a stronger engagement signal than viewer count alone, because viewer count includes lurkers and bot inflation but chat requires actual humans typing. Counting unique chatters per stream gives you a clean engagement metric.

For Twitch IRC, you connect to irc.chat.twitch.tv:6667, send PASS oauth: (empty for anonymous), NICK justinfan12345 (any random number), and join channels with JOIN #channelname. Each PRIVMSG line is a chat message.

For Kick, the Pusher websocket sends JSON events. You subscribe to a channel name and Pusher pushes messages.

Chat data balloons fast. A single popular Twitch channel during a high-traffic stream produces 500-2000 messages per minute. Storing the raw text of every message for thousands of channels gets expensive fast. Most production pipelines compress to per-minute aggregates: message count, unique chatter count, top emote count, sentiment distribution. The raw messages get sampled (1-in-100) for quality checks.

Chat connection topology

A naive deployment opens one IRC socket per channel. That works up to a few hundred channels but breaks at scale because the kernel runs out of file descriptors and the latency of opening sockets serially balloons. The pattern that scales is connection pooling: one IRC connection joins up to 50 channels, and you open as many connections as needed across multiple IPs. The Twitch IRC server enforces a 50-channel limit per JOIN burst, but a connection can hold up to 100 channel memberships if joins are spaced out by a few seconds.

For Kick, Pusher allows up to 100 channel subscriptions per websocket connection. Open a connection per IP, subscribe to as many chatrooms as fit, and run a watchdog that reconnects on pusher:error events with a 5-second backoff. Pusher will close idle connections after 120 seconds without a pusher:ping, so send a ping every 60 seconds.

The biggest production gotcha is silent disconnects. The TCP socket reports connected while the chat server has actually stopped sending data because of a network blip somewhere. Always set a heartbeat timeout: if no messages or PING/PONG for 90 seconds on a popular channel, force a reconnect. SilencedChannel detection saves hours of missing data when you discover a downstream pipeline gap.

Proxy and session strategy

Twitch Helix is OAuth-token-based and the rate limit is per token, not per IP. Proxies matter only for the GraphQL backdoor and the IRC chat connection. Use residential proxies for GraphQL, especially if you are pulling historical data on a lot of channels in parallel. Mobile proxies are overkill for Twitch and burn budget you do not need to spend.

Kick is the opposite. Kick’s Cloudflare layer flags repeated requests from the same IP fast. Use a residential proxy pool with sticky sessions for authenticated-looking traffic, or curl_cffi to disguise the TLS fingerprint as a real browser. We cover the broader proxy strategy in our guide on best residential proxy providers 2026 and the deeper TLS layer in TLS fingerprinting in 2026: a complete guide for scrapers.

For chat scraping specifically, IP rotation matters less because the connection is long-lived. One IP per channel-group of 50 channels works fine. The connection sits open for hours, not seconds, so the overhead of proxy negotiation is amortized.

Data storage: timeseries vs snapshots

Streaming data is a timeseries problem. Each row is a (channel, timestamp, viewer_count, follower_count, category) tuple. PostgreSQL with TimescaleDB is the right default. ClickHouse works if you are pulling chat at full fidelity. SQLite is fine for development but breaks once you cross 10 million rows or need concurrent writers.

Sample schema for production:

CREATE TABLE stream_snapshots (
    channel_id TEXT NOT NULL,
    platform TEXT NOT NULL,
    captured_at TIMESTAMPTZ NOT NULL,
    is_live BOOLEAN NOT NULL,
    viewer_count INTEGER,
    category TEXT,
    title TEXT,
    follower_count INTEGER,
    PRIMARY KEY (channel_id, platform, captured_at)
);

SELECT create_hypertable('stream_snapshots', 'captured_at');
CREATE INDEX ON stream_snapshots (channel_id, captured_at DESC);

Polling cadence: 1 minute for top 1000 channels, 5 minutes for top 10000, 30 minutes for the long tail. A 5-minute average viewer is granular enough for most analytics and saves 80% of API budget compared to 1-minute polling.

Compression and retention strategy

Raw chat at full fidelity becomes the single largest line item in your storage bill. A reasonable retention policy keeps raw messages for 7 days (for spot checks and debugging), per-minute aggregates for 90 days, and per-hour aggregates forever. TimescaleDB’s continuous aggregates make this almost zero-effort: define a time_bucket('1 hour', captured_at) continuous aggregate over the snapshots table, set a retention policy that drops raw rows after 7 days, and let the aggregates carry the long-term history.

Compression matters too. TimescaleDB native compression on a stream_snapshots hypertable typically yields 12-20x compression because the same channel_id and category repeat across millions of rows. Enable compression on chunks older than 24 hours and reclaim almost all the disk you spent in the first place.

For chat aggregates, ClickHouse beats Postgres on storage and query speed once you cross 100 million message-events. The MergeTree engine with a (platform, channel_id, captured_at) sort key is the canonical choice. Pre-aggregate at write time using materialized views so dashboard queries hit a small summary table instead of scanning raw events.

Handling banned and deleted channels

Channels disappear constantly. Streamers get banned, accounts get deleted, channels rebrand. Your scraper has to handle 404 and 410 responses without crashing or losing the historical data attached to the old channel ID. Soft-delete the row in your database, mark last_seen_at, and stop polling that channel after 7 days of consecutive 404 responses. Some channels come back from suspension after weeks, so do not hard-delete on first 404.

Twitch sometimes returns a 200 with empty data: [] for live stream queries on a banned channel. That is not the same as channel deleted; the channel exists but is not currently live. Use the Get Users endpoint to confirm channel existence separately from Get Streams.

A subtler failure mode: Twitch and Kick both allow channels to rebrand by changing their login slug. If you key your snapshots on the slug, a rebrand orphans the old history. The fix is keying on the immutable numeric user_id from Helix and the id field from Kick, and treating the slug as a mutable display attribute that updates on every poll.

Legal and ToS considerations

Twitch’s Developer Services Agreement allows API use under the standard rate limits. Scraping the GraphQL endpoint or chat IRC for analytics use is in a gray zone: not explicitly forbidden, but not blessed either. Building a competing service is a different matter and has been the basis for cease-and-desist letters in the past.

Kick’s terms of service do not explicitly address scraping, which historically means it is permitted. That said, Kick has shown willingness to fingerprint and block aggressive scrapers via Cloudflare. Stay below 30 requests per minute per IP and you will not attract attention.

For chat data specifically, both platforms surface chat publicly to anyone who joins a channel, so logging it is functionally equivalent to a user opening the chat in a browser. Personal data within chat (usernames, message content) is subject to GDPR and CCPA if you are storing it for any commercial purpose. See our GDPR compliance for web scraping guide for the data minimization patterns that matter.

External authoritative reference: the Twitch Developer documentation lists current rate limits and endpoint scopes.

Common gotchas

A few traps that are not obvious until you hit them:

Twitch token expiry races. App access tokens are valid for ~60 days but Twitch can invalidate them early during platform-wide credential rotations. Always retry once on 401 with a fresh token before treating the request as failed.
Helix Get Streams returns the title at the start of the stream and does not update when the streamer changes the title mid-stream. To track title changes you need to poll Get Channel Information, which is per-channel and burns budget fast.
Kick’s category slug differs from its display name. just-chatting is a slug; the display name is Just Chatting. Always use the slug field returned from /categories rather than guessing.
IRC chat from Twitch occasionally drops a RECONNECT notice. The server is telling you to reconnect within 30 seconds or messages will be lost. Most IRC libraries do not handle this automatically.
Pusher websocket disconnects fire pusher:connection_established on every reconnect with a new socket_id. If you are not idempotent on subscribe, you double-subscribe and double-process every message.

FAQ

Q: can I scrape subscriber counts on Twitch?
The exact subscriber count is gated behind partner-only scopes that require channel owner consent. Public approximations come from the Get Channel Information endpoint and from third-party signals like emote slots. The unofficial GraphQL endpoint exposes a subscription tier breakdown for some channels but not consistently.

Q: does Kick have a search API?
Yes. GET /api/v2/search?searched_word=query&type=channels returns channel matches. There is also a category search endpoint.

Q: how do I get historical Twitch viewer data without GraphQL?
You poll your own database. Helix does not expose history, only current state. The standard approach is to continuously poll Get Streams every 5 minutes and store snapshots, building your own historical dataset over time.

Q: do mobile proxies help for Kick scraping?
Marginally. Kick’s Cloudflare filters care more about TLS fingerprint and request rate than IP type. A residential proxy with curl_cffi is the cost-effective combination.

Q: can I scrape Kick chat without a websocket library?
Pusher publishes a public HTTP fallback, but it adds latency and is rate-limited. For production, use the websocket. The pysher library handles the Pusher protocol cleanly.

Q: what is the cheapest infrastructure to run a 10000-channel scraper?
A single 4-vCPU 8GB VPS handles polling 10000 channels at 5-minute cadence comfortably if you batch requests and use async IO. The bottleneck is not compute, it is API budget. Plan on 4-6 Twitch app credentials and a residential proxy pool of 200-500 IPs. Total monthly cost lands between $80 and $250 depending on chat volume.

Q: how do I detect view-bot inflation in scraped data?
Cross-check viewer count against unique chatters per minute. A normal channel maintains a 0.5-3% chatter-to-viewer ratio. Channels with 0.05% or below for sustained periods are almost certainly view-botted. Track the ratio as a derived column and flag outliers for manual review.

Closing

Streaming metadata scraping in 2026 is a two-platform problem. Twitch wants you on Helix and tolerates the GraphQL endpoint as long as you do not abuse it. Kick is more open but more aggressively fingerprinted. Build your pipeline assuming both APIs will change, store everything as a timeseries from day one, and treat chat as the highest-value engagement signal you can capture cheaply. For broader proxy guidance see our gaming proxies category hub.

May 7, 2026