Your cart is currently empty!
Author: Xavier Fok
-
How to Scrape iCIMS Career Sites (2026)
iCIMS powers career portals for thousands of mid-to-enterprise employers, and scraping it at scale is harder than it looks. The platform serves job listings through JavaScript-rendered pages, enforces bot detection via Cloudflare and custom fingerprinting, and rotates URL structures across tenant subdomains. if you’re building a talent pipeline, competitive intelligence feed, or labor market dataset, here’s what actually works in 2026.
How iCIMS Structures Its Job Data
iCIMS career sites follow a predictable tenant subdomain pattern:
https://{company}.icims.com/jobs/search. each company gets its own subdomain, but the underlying HTML skeleton is consistent across tenants. job detail pages live at/jobs/{job_id}/joband contain the full description, location, department, and requisition metadata in both rendered HTML and a partially-hydrated JSON blob embedded in atag.that structured data block is your fastest extraction path. it follows the
JobPostingschema.org spec and includestitle,hiringOrganization,jobLocation,datePosted, anddescriptionwithout needing a headless browser for the detail page. the search listing page is the hard part -- it's React-rendered and paginated via XHR calls to a private REST API.Reverse-Engineering the Search API
the real extraction leverage comes from the iCIMS job search XHR endpoint, not the DOM. open DevTools on any
{tenant}.icims.com/jobs/searchpage, filter by XHR, and you'll see calls to:GET /jobs/search?ss=1&searchLocation=&searchCategory=&searchZip=&searchRadius=50&searchPositionType=&applyOnline=1&in_iframe=1that endpoint returns paginated JSON with
searchResultscontaining job IDs, titles, locations, and department labels. you can drive pagination withstartrowandmaxrowsparameters (default 10, max 25 per call). extract the IDs, then hit the detail endpoint for full content.import httpx, time TENANT = "yourcompany" BASE = f"https://{TENANT}.icims.com/jobs" def fetch_jobs(start=0, max_rows=25): params = { "ss": 1, "startrow": start, "maxrows": max_rows, "searchPositionType": "", "applyOnline": 1, "in_iframe": 1 } r = httpx.get(f"{BASE}/search", params=params, headers={ "User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)", "Referer": f"https://{TENANT}.icims.com/jobs/search" }, timeout=15) r.raise_for_status() return r.json() def fetch_detail(job_id): r = httpx.get(f"{BASE}/{job_id}/job", timeout=15) from bs4 import BeautifulSoup import json, re soup = BeautifulSoup(r.text, "html.parser") ld = soup.find("script", {"type": "application/ld+json"}) return json.loads(ld.string) if ld else {}add a 1-2 second delay between requests per tenant. iCIMS rate-limits by IP and will return 429s if you hammer a single subdomain.
Anti-Bot Layers and How to Route Around Them
iCIMS deployments vary in how aggressively they're protected. here's a practical breakdown:
protection layer frequency bypass approach Cloudflare challenge page ~40% of tenants residential proxy + TLS fingerprint match IP rate limiting (429) universal throttle + rotating proxies User-Agent fingerprinting moderate browser-like UA string + accept headers CAPTCHA on search rare (<5%) headless browser + solver Referrer checking common always set Refererto the search pagefor most tenants, httpx with a realistic User-Agent and a proper Referer header is enough. for Cloudflare-protected tenants, you'll need residential IPs. datacenter IPs get challenged or blocked outright on ~40% of iCIMS deployments.
the scraping patterns here are similar to what you'd encounter with How to Scrape Taleo Career Sites at Scale (2026) -- both platforms sit behind enterprise-grade CDN layers with tenant-level variance in protection strictness.
Scaling Across Hundreds of Tenants
scraping one iCIMS tenant is straightforward. scraping 500 of them for a labor market feed requires a different architecture:
- build a tenant discovery list -- iCIMS doesn't publish a directory, but you can source subdomains from job board aggregators, LinkedIn company pages, and certificate transparency logs (
crt.shquery:%.icims.com). - deduplicate subdomains and validate them with a HEAD request before adding to your queue.
- run per-tenant scrapers in parallel, but cap concurrency per IP to 2-3 tenants at a time.
- use a rotating residential proxy pool so each subdomain sees requests from varied IPs. per-tenant sticky sessions for 5-10 minutes prevent cookie invalidation mid-crawl.
- store raw JSON from the search API separately from parsed detail records so you can re-parse without re-fetching.
key fields to extract per job:
job_id,tenant,title,location.city,location.state,location.country,department,employment_type,date_posted,description_html. if you want structured skill extraction, run the raw description through an LLM after collection.for teams building similar pipelines against ATS platforms, the approaches covered in How to Scrape Ashby Career Sites for Talent Pipelines (2026) and How to Scrape Personio Career Sites (2026) show how the same tenant-discovery and schema-extraction pattern generalizes across vendors.
Common Errors and What They Mean
- 429 Too Many Requests -- you're hitting the rate limiter. back off 30-60 seconds, add jitter, reduce concurrency on that tenant.
- 403 Forbidden -- IP is blocked or Cloudflare challenge triggered. rotate to a residential proxy and retry with a fresh session.
- Empty
searchResultsarray -- tenant uses a custom iCIMS build with a different API path. fall back to DOM scraping the listing page directly. - Malformed JSON in ld+json block -- some older iCIMS tenants have invalid JSON in the schema.org tag. use
json.loadsinside a try/except and fall back to BeautifulSoup field extraction. - Redirect to login page -- the job is no longer active. log it as expired and skip.
this error taxonomy overlaps significantly with what you see in other structured-data scraping contexts. if you're also pulling product catalog data, the same proxy rotation and error handling patterns apply in environments like How to Scrape Amazon Best Sellers Across 18 Marketplaces (2026), where IP reputation is equally critical.
one underrated issue: iCIMS occasionally injects a
redirect for bots that don't execute JavaScript. if your scraper returns a page with zero job listings but a valid 200 status, check whether you landed on the noscript fallback. the giveaway is atag in the response body.the same JavaScript-rendering challenge comes up in entirely different verticals -- the JS-heavy listing pages in How to Scrape Latin American Real Estate Sites (Imovelweb, Mercado Libre) use almost identical anti-scrape patterns at the CDN layer.
Bottom Line
for most iCIMS tenants, the XHR search API plus schema.org ld+json extraction gets you clean, structured job data without a headless browser. residential proxies are only necessary for the ~40% of tenants running Cloudflare. scale across hundreds of tenants with a discovery pipeline built on crt.sh and careful per-IP throttling. this site covers these ATS and structured-data scraping targets in depth -- bookmark it if you're building any kind of labor market or recruiting intelligence feed.
Related guides on dataresearchtools.com
- build a tenant discovery list -- iCIMS doesn't publish a directory, but you can source subdomains from job board aggregators, LinkedIn company pages, and certificate transparency logs (
-
How to Scrape Ashby Career Sites for Talent Pipelines (2026)
Ashby has quietly become the ATS of choice for fast-growing startups and Series B+ companies, which makes scraping Ashby career sites a high-signal move for talent pipeline builders, recruiting agencies, and competitive intelligence teams. The problem is that Ashby’s job boards aren’t served from a single domain — each company self-hosts under a pattern like
jobs.ashbyhq.com/— and the rendering is React-based, which trips up naive scrapers.How Ashby Job Pages Are Structured
Every Ashby career site follows a predictable URL schema:
https://jobs.ashbyhq.com/{company-slug} https://jobs.ashbyhq.com/{company-slug}/{job-id}The listing page renders a JSON payload into the DOM, but Ashby also exposes a public API endpoint that returns structured job data without JavaScript rendering:
GET https://api.ashbyhq.com/posting-api/job-board/{company-slug}This is the cleanest extraction path. The response is JSON with fields like
title,team,location,isRemote,employmentType, andapplicationFormDefinition. No authentication required, no browser needed.import httpx, json SLUG = "linear" # replace with target company slug resp = httpx.get(f"https://api.ashbyhq.com/posting-api/job-board/{SLUG}", timeout=15) data = resp.json() for job in data.get("jobs", []): print(job["title"], "|", job.get("location", {}).get("name"), "|", job["id"])Run this against a list of target slugs and you have a structured talent pipeline feed in minutes.
Finding Company Slugs at Scale
The slug discovery problem is where most pipelines break. There’s no public directory of all Ashby customers, so you need to build your own list.
Three approaches that work in 2026:
- Google dork:
site:jobs.ashbyhq.comreturns thousands of indexed subpaths. Paginate through results and extract the slug from the URL path. - LinkedIn scrape: Filter companies by ATS tech stack using tools like Clay or Phantom Buster, which surface ATS provider from careers page redirects.
- Common Crawl: Query the March 2026 crawl for
jobs.ashbyhq.comhostnames and extract unique slugs from theurlcolumn in Athena or BigQuery.
For a talent agency scraping 500+ companies, a seeded Common Crawl query gives the highest coverage per compute dollar.
Anti-Bot Behaviour and Rate Limits
The posting API (
api.ashbyhq.com/posting-api) is intentionally public and low-friction. Ashby wants jobs indexed. That said, hammering it with concurrent requests will get your IP soft-blocked within minutes.Realistic limits from testing:
Behaviour Observed limit Concurrent requests (same IP) ~5 before 429s appear Requests per minute (single IP) ~60 sustained Cooldown after 429 30-90 seconds User-agent rejection Not enforced on API Bot detection on HTML pages Cloudflare Turnstile (varies by company) The HTML job listing pages (
jobs.ashbyhq.com) are a different story. Some companies enable Cloudflare Turnstile on the front-end, which means rendering them requires a headless browser or a Turnstile solver. For bulk data extraction, stick to the API — avoid the HTML path entirely unless you need application form fields that aren’t exposed in the JSON.Rotate IPs per company slug, not per request. A residential proxy pool with 1 request per slug per session keeps your fingerprint clean and stays well within Ashby’s tolerance. If you’re also scraping other ATS platforms in the same pipeline — say, Recruitee or Personio — use separate proxy sessions per provider to avoid cross-contamination of block signals.
Normalising Ashby Data for Cross-ATS Pipelines
Raw Ashby output doesn’t map cleanly to other ATS schemas. If you’re building a unified talent intelligence feed that also pulls from iCIMS or Taleo, normalisation is the unglamorous work that determines whether your pipeline is actually useful.
Ashby-specific fields to watch:
location.namecan be"Remote", a city, or a hybrid string like"New York, NY (Hybrid)"— parse these consistentlyemploymentTypeuses Ashby’s own enum:"FullTime","PartTime","Contract","Temporary"— remap to your schemateamis a nested object withidandname, not a flat stringcompensationTierappears only when the company has salary transparency enabled — treat it as optional
A canonical schema across ATS providers should use ISO 3166-1 alpha-2 for country codes, a
remote_typeenum (full,hybrid,none), and Unix timestamps forposted_at. Ashby’screatedAtfield is UTC ISO 8601, which is straightforward to convert.The same normalisation discipline applies when you’re pulling structured data from completely different verticals — the schema design lessons in How to Scrape Latin American Real Estate Sites cover multi-source field unification patterns that transfer directly to multi-ATS pipelines.
Running This at Scale
For a production pipeline covering 1,000+ Ashby companies, the architecture is straightforward:
- Orchestration: Temporal or a simple cron on a VPS — Ashby jobs don’t change by the minute, so daily or twice-daily refreshes are enough
- Queue: Redis or SQS with one task per company slug
- Workers: 10-20 concurrent workers, each with a dedicated residential IP session
- Storage: Postgres with a
ats_jobstable and a(company_slug, job_id, scraped_at)composite key for deduplication - Change detection: Hash the job list per slug on each run and only emit events when the hash changes
Short bullet checklist before going to production:
- Confirm the slug list covers your target company set (test 10 manually)
- Set
httpxtimeout to 15s and retry twice with exponential backoff on 5xx - Log 429s with the slug and timestamp — patterns reveal which companies have extra rate protection
- Store raw JSON alongside normalised rows — Ashby’s schema has changed twice in the past 18 months
For monitoring, track the ratio of slugs returning zero jobs vs. a non-empty list. A sudden spike in zero-job responses usually means your IP pool is blocked, not that all your targets froze hiring simultaneously.
Bottom Line
The Ashby posting API is genuinely scraper-friendly — use it instead of rendering HTML, rotate IPs at the slug level, and invest the saved complexity into normalisation and deduplication. If you’re building a serious multi-ATS talent pipeline, Ashby is one of the easier integrations; the harder work is schema consistency across providers. DRT covers ATS scraping patterns, proxy infrastructure, and data pipeline design in depth — the same principles apply whether you’re pulling from five job boards or five hundred.
Related guides on dataresearchtools.com
- Google dork:
-
How to Scrape Personio Career Sites (2026)
Personio career sites are a goldmine for recruiting intelligence and job market analysis — and because Personio powers HR for thousands of European SMBs, scraping their public job boards gives you structured hiring data that isn’t reliably indexed on LinkedIn or Indeed. This guide covers how to scrape Personio career sites in 2026, including the two main URL patterns, anti-bot posture, and a working Python approach.
How Personio Career Pages Are Structured
Personio hosts career pages on two URL patterns:
https://{company}.jobs.personio.de/(German-hosted, common for EU companies)https://{company}.jobs.personio.com/(global variant)
Some companies embed the widget on their own domain via an iframe or JavaScript snippet, but the underlying data still comes from Personio’s API. Each job listing URL follows the pattern
/job/{id}, and the index page loads a full JSON payload on the client side — which is your primary extraction target.The job listing data is injected into the page as a
window.__NUXT__server-side rendered object, or in newer deployments, fetched from a JSON API endpoint athttps://{company}.jobs.personio.com/api/v1/jobs(no auth required). That API endpoint is the clean path — you bypass HTML parsing entirely.Extracting the JSON API Directly
The undocumented but stable endpoint returns a JSON array of all open roles:
import httpx import json COMPANY = "yourcompany" BASE = f"https://{COMPANY}.jobs.personio.com" def fetch_jobs(): r = httpx.get( f"{BASE}/api/v1/jobs", headers={"Accept": "application/json", "Accept-Language": "en"}, timeout=15, ) r.raise_for_status() return r.json() # list of job dicts jobs = fetch_jobs() for job in jobs: print(job["id"], job["name"], job.get("department", {}).get("name"))Each job object contains
id,name,department,office,employment_type,created_at, and ajob_descriptionsarray with HTML content blocks. For full JD text, you’ll need a second call to/job/{id}or parse thejob_descriptionskey already in the response.If the
/api/v1/jobsendpoint returns 404, fall back to scraping the rendered HTML and extracting theJSON blob, then parsing the nested state tree.Anti-Bot Posture and Rate Limits
Personio's job pages are relatively permissive compared to enterprise ATS platforms. There's no Cloudflare challenge on most subdomains, and Akamai is not present. That said, a few behaviors to watch for:
- Rate limiting: Aggressive crawling (more than ~30 req/min) triggers 429s from their CDN. Space requests with a 2-4 second jitter.
- User-Agent checks: The bare
python-httpxUA gets blocked on some subdomains. Use a realistic browser UA string. - Geo-blocking: Some Personio customers restrict their career page to specific regions. If you're scraping a German Mittelstand company from a US IP, you may get a redirect or empty results.
For the geo issue, residential proxies from a German or European IP pool resolve it cleanly. This is the same pattern used when scraping ATS platforms like SmartRecruiters hiring pages -- the target's CDN sees a local visitor rather than a datacenter range.
Issue Symptom Fix 429 Too Many Requests Burst of requests blocked Add jitter (2-4s), reduce concurrency Empty JSON array []response but page shows jobsSwitch Accept-Language header to de404 on /api/v1/jobsOlder Personio tenant Parse __NUXT_DATA__from HTMLGeo-block redirect 302 to /not-availableUse EU residential proxy iframe embed Jobs on company domain, not Personio Trace network tab for Personio API origin Parsing Job Detail Pages
If you need structured job descriptions (not just titles), here's the hierarchy to expect inside
job_descriptions:- Each object has a
namefield (section heading like "Your Role", "Requirements") - The
valuefield contains raw HTML - Nested
job_descriptionsarrays appear for grouped sections
A clean parse using BeautifulSoup:
from bs4 import BeautifulSoup def extract_text_blocks(job: dict) -> dict: sections = {} for block in job.get("job_descriptions", []): heading = block.get("name", "body") html = block.get("value", "") text = BeautifulSoup(html, "html.parser").get_text(separator="\n").strip() sections[heading] = text return sectionsThis is cleaner than scraping the rendered page and avoids the flaky CSS selectors that break when Personio updates their frontend. Similar structured-extraction approaches work well for Recruitee job pages, which also expose a JSON-first data layer.
Scaling Across Multiple Companies
If you're building a broader job market dataset -- tracking hiring velocity, team growth, or competitive intelligence -- you'll need to crawl many Personio tenants, not just one.
The main challenge is discovery. There's no public Personio company directory. Practical sourcing approaches:
- Search
site:jobs.personio.deorsite:jobs.personio.comon Google to surface active tenants - Pull from LinkedIn company pages (many link directly to their Personio career site)
- Use Crunchbase or Apollo to filter EU SMBs, then probe the Personio subdomain pattern
Once you have a list of subdomains, the
/api/v1/jobscall is identical across all tenants. A simple async crawler withhttpx.AsyncClientand a semaphore of 5-10 concurrent workers handles hundreds of companies per hour without triggering rate limits.For pipelines that need to stay current, schedule daily or weekly crawls and diff against your previous snapshot. New
created_attimestamps flag fresh postings; disappearing job IDs signal closed roles. Platforms like Ashby career sites use a similar versioned-listing model, making diff-based freshness detection a reusable pattern across ATS targets.If your use case extends beyond job data into broader HR intelligence -- org structure signals, headcount trends, or location expansion -- consider pairing Personio data with iCIMS career site scrapes to cover the mid-market US segment that Personio doesn't reach.
One note on scope: Personio job data is B2B recruiting intelligence. If your pipeline is pivoting toward real estate or property listings across emerging markets, the extraction patterns here generalize, though the tooling differs -- DRT has separate coverage on scraping Latin American real estate sites like Imovelweb and Mercado Libre for those use cases.
Handling Edge Cases
A few Personio-specific quirks that'll bite you in production:
- Multilingual listings: A job may have separate entries for
deandenlanguage variants. They share the same base ID but differ by locale parameter. Filter by?language=enquery param on the API call. - Department normalization: Department names are free-text set by the employer. "Engineering", "Tech", "R&D", and "Product & Engineering" all need to be mapped in your downstream schema.
- Employment type inconsistency: Some tenants use "Full-time" vs "Vollzeit" vs custom strings. Normalize to a canonical enum before storing.
- Deleted vs. filled roles: Personio removes filled roles from the index immediately. If you need historical data, snapshot on every crawl.
Bottom Line
Personio's
/api/v1/jobsendpoint is the right entry point -- skip the HTML, go straight to the JSON, add a realistic UA and EU residential proxy for geo-sensitive tenants, and run async crawls with conservative concurrency. The data is clean, consistently structured, and requires no authentication. For broader ATS coverage or freshness-detection patterns across European hiring markets, dataresearchtools.com covers the full stack of platforms engineers actually encounter in production pipelines.Related guides on dataresearchtools.com
-
How to Scrape Recruitee Pages for Lead Sourcing (2026)
Recruitee powers career pages for thousands of mid-market companies across Europe and North America, and its consistent URL structure makes it one of the more approachable ATS targets for lead sourcing at scale. If you’re building a list of companies actively hiring in a specific role, location, or tech stack, scraping Recruitee pages gives you a real-time signal that job boards like LinkedIn lag by days.
Understanding Recruitee’s URL and API Structure
Every Recruitee career site follows the same pattern:
https://{company}.recruitee.com/for the public jobs page, andhttps://{company}.recruitee.com/api/offers/for the JSON feed. That API endpoint is the main event — it returns structured job data without any rendering requirement.A typical response from
/api/offers/looks like this:{ "offers": [ { "id": 182934, "title": "Senior Data Engineer", "department": "Engineering", "location": "Amsterdam, Netherlands", "remote": true, "created_at": "2026-04-12T08:00:00Z", "career_url": "https://acme.recruitee.com/o/senior-data-engineer" } ] }No authentication, no token rotation, just a clean GET. For most companies this endpoint returns 200 with
Content-Type: application/json. It doesn’t paginate (all offers come back in one call), which keeps the scraper simple.The harder part is getting the list of company subdomains to query. There’s no public Recruitee directory, so you need to seed your target list from a separate source: Apollo, Crunchbase, or a curated vertical list. This is conceptually similar to the approach covered in How to Scrape Yellow Pages Business Data, where you build a domain list first, then loop your scraper over it.
Building the Scraper
Use
httpxwith async for throughput. Recruitee doesn’t aggressively rate-limit individual company subdomains, but if you’re hitting 1,000+ subdomains in one run, you’ll want concurrency caps and retry logic.import asyncio import httpx async def fetch_offers(client: httpx.AsyncClient, slug: str) -> dict: url = f"https://{slug}.recruitee.com/api/offers/" try: r = await client.get(url, timeout=10) if r.status_code == 200: return {"slug": slug, "offers": r.json().get("offers", [])} except (httpx.TimeoutException, httpx.RequestError): pass return {"slug": slug, "offers": []} async def main(slugs: list[str]): async with httpx.AsyncClient(follow_redirects=True) as client: tasks = [fetch_offers(client, s) for s in slugs] return await asyncio.gather(*tasks)Run this with a semaphore (limit to 20-30 concurrent) and you can process 5,000 companies in under 10 minutes on a standard VPS. The
follow_redirects=Truematters — some companies migrate away from Recruitee and the old subdomain 301s somewhere unhelpful.Useful fields to extract per offer:
title,department,location,remote,created_at,career_url. Thecreated_atfield is the most valuable for freshness filtering — jobs posted in the last 14 days indicate active hiring, which is a strong lead qualifier.Anti-Bot Considerations and Proxy Use
The JSON API endpoint is low-friction for most companies, but if you’re scraping the HTML career pages (to capture structured data not in the API, like required skills parsed from job descriptions), you’ll hit Cloudflare on some subdomains. Recruitee’s default configuration doesn’t block the API path, but aggressive crawling of the HTML listings will get your IP flagged.
For the API-only approach, residential proxies are overkill — a rotating datacenter pool at ~3 req/s per IP is fine. If you’re parsing HTML job descriptions at scale, use residential or mobile IPs and add a randomized delay between 1.5 and 4 seconds. Keep your User-Agent consistent with a recent Chrome build.
Compared to more heavily protected ATS platforms, Recruitee is relatively open:
ATS Platform API Available Cloudflare Present Auth Required JS Rendering Needed Recruitee Yes ( /api/offers/)Sometimes (HTML only) No No (API) Workday No public API Yes Yes Yes SmartRecruiters Partial Yes Sometimes Yes Personio No public API Varies No No Ashby Yes ( /api/job-board/)Minimal No No Workday is the most locked down by far — as covered in How to Scrape Workday Career Sites at Scale (2026), it requires full browser automation and per-tenant URL discovery. Recruitee is closer to Ashby in permissiveness, which is why it’s a good starting point if you’re new to ATS scraping.
Enriching and Qualifying the Lead Data
Raw job posting data alone is a weak lead signal. You need to layer on company-level attributes to prioritize outreach. A useful enrichment stack:
- Resolve the company slug to a domain using the Clearbit or Hunter enrichment API
- Cross-reference against your CRM to filter out existing customers or known churned accounts
- Pull headcount and funding stage from Apollo or Crunchbase to segment by company size
- Score by job recency — offers posted within the last 7 days get the highest priority
- Filter by department if you’re targeting specific buyers (e.g., only “Engineering” or “Data” roles indicate a technical buyer)
What the Recruitee API doesn’t give you is department headcount or seniority distribution across all open roles. For that, you’d need to aggregate across multiple job posts. If a company has 8 open engineering roles across data/ML/backend, that’s a much stronger signal than one generic posting.
This enrichment workflow mirrors what’s needed when targeting other ATS sources. How to Scrape SmartRecruiters Hiring Pages (2026) and How to Scrape Personio Career Sites (2026) cover similar enrichment approaches for those platforms — the enrichment logic is largely portable once you have normalized offer records.
Handling Edge Cases
A few things that will break a naive scraper:
- Subdomain not found (404/NXDOMAIN): Some companies deactivate their Recruitee account but the subdomain persists in your seed list. Catch DNS failures separately from HTTP errors and flag them for removal.
- Empty offers array: A company may have a valid Recruitee account with zero active postings. Log these separately — they’re worth re-checking in 30 days rather than discarding.
- Non-English job descriptions: Recruitee is popular in the Netherlands, Germany, and Poland. If your downstream NLP pipeline assumes English, add a language detection step (langdetect or fasttext) before parsing.
- Custom domains: Some companies configure a custom domain (e.g.,
jobs.acme.com) that proxies to Recruitee. The API path still works:https://jobs.acme.com/api/offers/. Check the page source for the Recruitee widget script to confirm.
How to Scrape Ashby Career Sites for Talent Pipelines (2026) documents a nearly identical custom-domain issue — it’s a common pattern across modern ATS platforms that support white-labeling.
Bottom Line
Recruitee’s
/api/offers/endpoint is the cleanest ATS scraping target available right now — no auth, no JS rendering, structured JSON out of the box. The real work is in building a quality seed list of company subdomains and enriching the output into actionable lead records. Start with a focused vertical (SaaS companies in the Netherlands, for example), validate your pipeline on 500 companies before scaling, and re-run the scrape weekly for fresh hiring signals. DRT covers this class of infrastructure scraping targets in depth — if you’re building a full multi-ATS pipeline, bookmark the full series.Related guides on dataresearchtools.com
-
How to Scrape SmartRecruiters Hiring Pages (2026)
It looks like write permission to
~/Desktop/drt-articles/is being blocked. can you grant access to that path, or let me know an alternative folder to save to?Related guides on dataresearchtools.com
-
How to Scrape Workday Career Sites at Scale (2026)
Workday career sites are some of the most frustrating scrape targets in the job data space — heavily JavaScript-rendered, rate-limited per IP, and protected by Cloudflare or Akamai depending on the employer. If you’re building a job aggregator, a recruiting intelligence tool, or a competitive headcount tracker, you need a reliable pipeline that handles Workday’s quirks without burning through proxies or getting your IP ranges blocked inside 48 hours.
How Workday Serves Job Data
Workday career pages follow a consistent pattern. The public-facing URL is typically
https://. The page shell loads via a React-based SPA, then fetches job listings through a GraphQL-like REST endpoint:.wd1.myworkdayjobs.com/en-US/ /jobs GET https://<company>.wd1.myworkdayjobs.com/wday/cxs/<company>/<tenant>/jobsThe payload is a POST with a JSON body:
{ "limit": 20, "offset": 0, "searchText": "", "locations": [], "categories": [] }This endpoint is unauthenticated for most public career sites, which makes it the cleanest extraction path. Skip Selenium entirely for the initial crawl — hit the API directly with
httpxorrequests, paginate by incrementingoffsetby 20, and parse thejobPostingsarray in the response. You’ll get title, location, requisition ID, and a relative URL per listing.The detail page for each job is a second request:
GET /jobDetails?jobPostingId=. This returns full description HTML in a JSON field. Parse it withBeautifulSouporlxmland you’re done.Anti-Bot Layers and Where They Kick In
The direct API approach works until it doesn’t. Workday deploys different protection stacks depending on the employer’s contract tier:
Protection Layer Trigger Symptoms Cloudflare Bot Management High request velocity from single IP 403 with CF ray header Akamai Bot Manager Headless browser fingerprint Empty response body or redirect loop Workday rate limiting >50 req/min per IP 429 with Retry-AfterheaderTenant-level blocks Repeated scraping of same tenant 503 or silent empty results For Fortune 500 employers — think Salesforce, JPMorgan, or Deloitte — you will hit Cloudflare or Akamai. For mid-market companies, basic IP rotation is usually sufficient.
Rotate residential or mobile proxies, not datacenter IPs. Workday’s bot scoring is sensitive to ASN reputation. A Singapore or US residential pool with 5-10 second request delays per IP handles the majority of mid-market tenants without triggering blocks. Unlike simpler ATS platforms such as Lever and Greenhouse, Workday applies consistent bot scoring across all employer tenants, so you can’t exploit per-tenant gaps.
Scaling Across Thousands of Workday Tenants
The harder engineering problem is discovery: finding all Workday tenants worth scraping. There is no public tenant directory.
Three practical approaches:
- Seed from LinkedIn company pages. Filter companies by ATS using tools like Apify’s LinkedIn Company Scraper or a custom crawler, then check for the
wd1.myworkdayjobs.compattern in theCareerslink. - Use Google dorks:
site:wd1.myworkdayjobs.com -site:myworkday.comreturns indexed tenant subdomains. Export 100-200 at a time, deduplicate, and build your tenant list. - Buy a commercial dataset. Revelio Labs and Coresignal both maintain ATS-tagged company datasets. $500-2000 gets you a CSV with Workday tenant slugs for 15,000+ companies.
Once you have tenants, the crawl architecture matters. A naive sequential crawler will take weeks at scale. Use a job queue (Celery + Redis, or RQ) with per-tenant rate limiting. Set a max concurrency of 1 request per tenant per minute and run 50-100 workers. At that rate, 10,000 tenants with an average of 30 job postings each is a ~6-hour full crawl.
import httpx import time def fetch_jobs(tenant: str, company: str, offset: int = 0) -> dict: url = f"https://{company}.wd1.myworkdayjobs.com/wday/cxs/{company}/{tenant}/jobs" payload = {"limit": 20, "offset": offset, "searchText": "", "locations": [], "categories": []} headers = { "Content-Type": "application/json", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36", } r = httpx.post(url, json=payload, headers=headers, timeout=15) r.raise_for_status() return r.json()For tenants behind Akamai, swap
httpxfor a Playwright or Camoufox session that passes browser fingerprinting. Keep headless sessions warm across multiple requests to the same tenant rather than spinning up a new context per page — cold browser fingerprints score worse than warm ones.Storing and Deduplicating Job Postings
Job postings are volatile. The same requisition ID appears across multiple crawl cycles, and companies close and reopen roles. Your schema needs a few things:
requisition_id+tenant_slugas a composite unique keyfirst_seen_atandlast_seen_attimestamps for freshness trackingis_activeboolean flipped to false when a job disappears from the feed- A
raw_jsoncolumn for the full response payload so you can reparse without re-crawling
PostgreSQL with a partial index on
(tenant_slug, is_active)handles tens of millions of rows without issue. If you’re running this alongside other ATS scrapers — SmartRecruiters or Recruitee for example — normalize job records into a single canonical schema with anats_sourcefield. Cross-ATS analysis gets much easier when the data model is unified from day one.Proxy Selection for Workday at Scale
Not all proxy types perform equally against Workday’s bot stack:
- Residential rotating proxies: best default. US residential pools (Brightdata, Oxylabs, Smartproxy) handle 90% of tenants. Expect $3-8 per GB.
- Mobile proxies: highest trust score, best for Cloudflare-protected Fortune 500 tenants. More expensive at $15-25/GB, but failure rates drop significantly. The same logic applies when scraping boutique recruitment sites that use shared Cloudflare plans.
- Datacenter proxies: avoid entirely for Workday. Block rates exceed 60% even on premium providers.
- ISP/static residential: middle ground. Good for low-volume, high-fidelity scraping of a fixed tenant list.
Proxy rotation strategy matters as much as proxy type. The same principles that apply to review site scraping hold here: use sticky sessions per tenant (not per request), keep session duration under 10 minutes, and retire any IP that returns a 429 or 403 for a minimum of 30 minutes before reassignment.
Key failure signals to handle in your retry logic:
429withRetry-After: back off for the specified duration, then retry with a fresh IP403with CF ray header: rotate IP immediately, add 5s jitter before retry- Empty
jobPostingsarray with HTTP 200: silent block, treat as soft failure, retry after 15 minutes - Connection timeout: infrastructure issue or hard IP block, retire IP for 1 hour
Bottom Line
Workday’s direct JSON API is your fastest path to structured job data — skip the browser automation unless you’re targeting the Cloudflare tier of employers. Pair it with residential or mobile proxy rotation, a per-tenant rate limiter, and a deduplication schema built around
requisition_id. At 50-100 workers, you can maintain a fresh, full-coverage dataset across 10,000+ tenants on a single cloud instance. DRT covers the full ATS scraping stack across all major platforms if you’re building a multi-source job data pipeline.Related guides on dataresearchtools.com
- How to Scrape Boutique Recruitment Site Postings (2026)
- How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
- How to Scrape SmartRecruiters Hiring Pages (2026)
- How to Scrape Recruitee Pages for Lead Sourcing (2026)
- Pillar: How Proxies Help Scrape Reviews at Scale: Yelp, Google, Trustpilot (2026)
- Seed from LinkedIn company pages. Filter companies by ATS using tools like Apify’s LinkedIn Company Scraper or a custom crawler, then check for the
-
How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
Lever and Greenhouse power job listings for thousands of tech companies, and scraping them programmatically is one of the cleanest data collection tasks you can do in 2026 — both platforms expose structured APIs and predictable URL patterns that make bulk extraction far more reliable than scraping legacy ATS systems.
Why Lever and Greenhouse Are Scraper-Friendly
Neither platform is trying to hide its job data. Greenhouse publishes a public JSON board API that requires no authentication for read access. Lever similarly exposes a public posting endpoint per employer. Both are designed this way intentionally: companies want their listings indexed and aggregated.
That said, “scraper-friendly” doesn’t mean “rate-limit-free.” Both platforms throttle aggressive crawlers, and Greenhouse’s newer board configurations increasingly route through Cloudflare. If you’re building a high-volume aggregator, you’ll hit walls that a naive requests loop won’t survive. For context on how other ATS stacks compare, see How to Scrape Workday Career Sites at Scale (2026) — Workday is the opposite of Greenhouse: no public API, full JS rendering required.
Greenhouse API: The Right Way to Pull Job Data
Greenhouse’s Job Board API is the cleanest starting point. Each company has a unique board token, and the base endpoint is:
GET https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=trueThe
content=trueparameter pulls full job descriptions. Without it, you only get metadata.import httpx import time BOARD_TOKENS = ["stripe", "airbnb", "figma", "notion"] def fetch_greenhouse_jobs(token: str) -> list[dict]: url = f"https://boards-api.greenhouse.io/v1/boards/{token}/jobs" r = httpx.get(url, params={"content": "true"}, timeout=15) r.raise_for_status() return r.json().get("jobs", []) all_jobs = [] for token in BOARD_TOKENS: jobs = fetch_greenhouse_jobs(token) all_jobs.extend(jobs) time.sleep(1.2) # stay under rate limits print(f"Collected {len(all_jobs)} listings")Each job record returns
id,title,location,updated_at,content(HTML),departments,offices, andmetadata. The department and office arrays are particularly useful for org-structure analysis or filtering by function.Finding the board token for a given company is the only friction point. Most companies embed it in their careers page URL (
boards.greenhouse.io/{token}) or in the page source as a data attribute. A simple regex against the page source finds it in under a second.Lever API: Postings and Department Filters
Lever’s public posting endpoint follows the same pattern:
GET https://api.lever.co/v0/postings/{company_slug}?mode=jsonThe
mode=jsonparameter is essential — without it, Lever returns HTML. Lever also supports department and team filtering via query params, which makes targeted extraction much cleaner than post-processing a full dump.Useful query parameters:
?department=Engineering— filter by department?team=Backend— filter by team?commitment=Full-time— filter by job type?mode=json&limit=50&offset=0— pagination (default page size is 25)
Lever responses include
id,text(job title),categories(department, team, location, commitment),description(HTML),descriptionPlain,lists(responsibilities and requirements as structured arrays),salaryRange, andapplyUrl. ThesalaryRangefield is populated for US roles when companies opt into transparency.Handling Scale: Board Tokens in Bulk
If you’re building a jobs aggregator covering thousands of companies, the bottleneck is token/slug discovery, not the API calls themselves. A practical pipeline looks like this:
- Seed a company list from a SaaS review aggregator (G2, Capterra). If you want structured data from those platforms, How to Scrape G2.com and Capterra SaaS Reviews Programmatically covers the extraction path in detail.
- For each company, check careers page URLs for Greenhouse or Lever patterns.
- Validate the token/slug returns a 200 before adding it to your active list.
- Run nightly delta syncs using
updated_atfiltering rather than full re-pulls.
Delta syncing matters because both APIs are fast but not free under load. Greenhouse returns
updated_atper job; Lever returnscreatedAtandupdatedAt. A daily sync that only fetches listings updated in the last 24 hours reduces your request volume by 80-90% on a mature dataset.Platform Comparison
Feature Greenhouse Lever Public API Yes, JSON Yes, JSON Auth required No (read) No (read) Salary data Rare US roles, opt-in Department filtering Post-process Native query param Cloudflare protection Increasing Minimal Pagination style Single response Limit/offset HTML job descriptions Yes ( content=true)Yes + plain text Greenhouse is better for companies with complex location/department hierarchies. Lever is cleaner for filtering by commitment type or team before pulling content.
When the API Breaks Down
Both APIs have edge cases that will quietly return garbage without throwing errors:
- Deleted listings return 200: Greenhouse keeps deleted jobs in responses for up to 72 hours with no flag. Check
updated_atrecency and watch for emptydepartmentsarrays as a signal. - Lever pagination silently truncates: If a company has more than 250 listings, Lever’s API stops at 250 without a “next page” indicator in some configurations. Always check if your result count equals a round number and retry with offset.
- Cloudflare 1020 on Greenhouse: This is an access denied, not a rate limit. Rotating residential IPs fixes it. Data center IPs increasingly trigger 1020 on high-traffic boards.
- Board token changes: Companies occasionally reorg and migrate to a new board slug. Build dead-link detection into your pipeline.
For ATS platforms without public APIs, the extraction approach differs substantially. How to Scrape SmartRecruiters Hiring Pages (2026) covers a platform that requires a hybrid API-plus-DOM strategy, and How to Scrape Recruitee Pages for Lead Sourcing (2026) covers another mid-market ATS popular in Europe. If your target companies use smaller regional systems, How to Scrape Boutique Recruitment Site Postings (2026) is the right starting point for less structured targets.
Bottom line
Greenhouse and Lever are the easiest ATS platforms to scrape at scale in 2026 — both expose well-documented public APIs that return structured JSON with no login required. Start with the API before reaching for a browser automation tool. Use delta syncing by
updated_atto keep costs low, rotate residential IPs to handle Cloudflare friction on high-traffic Greenhouse boards, and build dead-slug detection from day one. DRT covers the full ATS ecosystem if you need extraction guides for the other major platforms your targets use.Related guides on dataresearchtools.com
-
How to Scrape Boutique Recruitment Site Postings (2026)
Boutique recruitment sites — niche job boards, regional talent platforms, and vertical-specific hiring portals — are some of the richest, least-contested sources of hiring signal you can scrape in 2026. Unlike LinkedIn or Indeed, they rarely invest in serious bot mitigation, but they also lack the clean APIs that mainstream ATS platforms expose. That gap is exactly why scraping boutique recruitment site postings requires a different playbook from what you’d use on enterprise systems.
What “boutique recruitment site” actually means
The category is broad. It includes:
- Vertical job boards (tech-only boards like Wellfound, finance-specific boards like eFinancialCareers, legal boards like Lawjobs)
- Regional platforms (JobsDB in Southeast Asia, StepStone in Europe, JobStreet across APAC)
- Staffing agency portals that publish live client postings on their own subdomains
- White-label ATS installs that don’t expose a public API (TeamTailor, Breezy HR, Pinpoint)
The staffing agency portals are the most valuable for competitive intelligence: they reveal which companies are hiring before those roles hit aggregators. The tradeoff is that each portal has a custom HTML structure with no predictable schema.
Fingerprinting the stack before writing a scraper
Before touching Playwright or BeautifulSoup, spend ten minutes identifying the underlying tech. Open DevTools, check the Network tab for XHR requests, and look at the page source for giveaway class names or API subdomains.
Common patterns you’ll hit:
Signal Likely stack Best extraction method /api/jobs?page=XHR callsCustom REST API Direct HTTP requests /__api/widgets/endpointsTeamTailor JSON from undocumented widget API data-listing-idattributesCustom CMS HTML parse with CSS selectors jobs.lever.coiframe embedLever See dedicated Lever guide Cloudflare challenge on first load Any stack Residential/mobile proxy + browser render If you see Lever or Greenhouse embeds, you’re better served by the documented approach in How to Scrape Lever and Greenhouse Job Boards Programmatically (2026) rather than treating the host site as a scrape target. Similarly, if the careers page redirects to a
mycompany.workday.comsubdomain, the How to Scrape Workday Career Sites at Scale (2026) guide covers that path specifically.Extraction strategies by site type
Static or server-rendered HTML
Regional job boards (JobStreet, StepStone, and smaller country-level clones) are often server-rendered with paginated listing pages. These are the simplest to scrape: a
requests+lxmlloop with polite delays is enough for low-volume pulls.import httpx from lxml import html import time BASE = "https://example-jobboard.com/jobs" HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)"} def scrape_page(page_num: int) -> list[dict]: resp = httpx.get(f"{BASE}?page={page_num}", headers=HEADERS, timeout=15) tree = html.fromstring(resp.text) jobs = [] for card in tree.cssselect(".job-card"): jobs.append({ "title": card.cssselect(".job-title")[0].text_content().strip(), "company": card.cssselect(".company-name")[0].text_content().strip(), "url": card.cssselect("a")[0].get("href"), }) return jobs for i in range(1, 20): print(scrape_page(i)) time.sleep(2)Adjust the CSS selectors per site. Keep delays above 1.5 seconds per page — boutique boards have thin infrastructure and you’ll trigger rate limits or kill their server faster than a big platform would notice.
JavaScript-rendered pages with a hidden JSON API
TeamTailor and Breezy HR render listings client-side but pull from an internal REST API. The fastest approach is to intercept that API call rather than rendering the DOM.
For TeamTailor specifically, the widget endpoint follows a predictable pattern:
https://career.teamtailor.com/api/widget/v1/jobs?company_id=XXXX. The company ID is in the page source. Hit that endpoint directly and skip the browser entirely.For Breezy HR portals (
company.breezy.hr), the job list loads fromhttps://company.breezy.hr/json. Again, direct HTTP, no rendering needed.Sites that require a browser
If the page loads a Cloudflare managed challenge or uses bot-detection fingerprinting (mouse movement checks, canvas fingerprinting, TLS JA3 matching), you need a real browser context. Playwright with stealth patches is the standard tool here.
Key configuration points:
- Set a realistic viewport (1440×900, not headless defaults)
- Randomize the User-Agent per session using a current Chrome string
- Add random delays between scroll events (50-200ms)
- Rotate IPs between sessions, not between requests
IP rotation matters more than most people expect. Many boutique boards block on subnet-level reputation, not individual IPs. A residential or mobile proxy pool will pass where datacenter IPs fail — mobile IPs in particular score well on the carrier trust signals that Cloudflare uses.
Handling pagination and deduplication
Boutique sites paginate in three ways: query string (
?page=2), cursor-based (?after=TOKEN), and infinite scroll that fires a new XHR on viewport entry.Numbered pagination is easy to loop over. Cursor-based requires extracting the next-page token from each response, usually in a
meta.next_cursororlinks.nextfield. Infinite scroll requires Playwright — listen for the network request the page fires when you scroll to 80% of the page height, then extract the URL pattern and replay it as direct HTTP.For deduplication across runs, hash the canonical job URL. Don’t hash on title plus company because boutique boards repost expired roles constantly. The URL is stable; the surrounding metadata drifts.
Structuring the output for downstream use
Recruitment data has a short shelf life — a posting can close in 48 hours. Design your schema with a
first_seen_atandlast_seen_attimestamp pair so downstream consumers can calculate posting age and detect removals without keeping a separate diff log.Minimum viable schema:
job_id(hashed URL or extracted native ID)title,company,location,posted_atdescription_html(raw, don’t strip yet)source_url,first_seen_at,last_seen_atstatus(active / closed)
Keep the raw HTML in
description_html. Structured extraction (skills, salary, seniority) should be a separate pass — either a regex pipeline or an LLM extraction step — so you can reprocess historical data without re-scraping.If you are scraping a staffing agency portal specifically for lead sourcing rather than job intelligence, the data model in How to Scrape Recruitee Pages for Lead Sourcing (2026) maps the same fields to a CRM-ready format. And if your target has migrated to SmartRecruiters, How to Scrape SmartRecruiters Hiring Pages (2026) covers the platform-specific quirks.
Bottom line
Boutique recruitment sites reward patient, targeted work: identify the stack first, hit the JSON API directly when available, and only bring a browser when you genuinely need one. Rotate mobile proxies for anything behind Cloudflare — datacenter ranges will get you blocked in minutes on sites this small. DRT covers the full recruitment scraping stack from ATS platforms to niche boards, so bookmark the guides linked above if you’re building a broader pipeline.
Related guides on dataresearchtools.com
- How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
- How to Scrape Workday Career Sites at Scale (2026)
- How to Scrape SmartRecruiters Hiring Pages (2026)
- How to Scrape Recruitee Pages for Lead Sourcing (2026)
- Pillar: Mobile Proxies for Recruitment: Scrape Indeed, LinkedIn Glassdoor at Scale
-
How to Scrape Electric Vehicle Charging Station Maps (2026)
EV charging station data is one of the most actively updated datasets on the web, and scraping it cleanly in 2026 requires understanding a mix of public APIs, JavaScript-heavy map renderers, and rate-limited tile servers. Whether you’re building a route planner, a fleet management dashboard, or a competitive analysis tool for charging networks, the core challenge is the same: the data lives inside map widgets that weren’t designed to be parsed.
Where EV Charging Data Actually Lives
Most charging station maps pull from one of three sources: a proprietary API (PlugShare, ChargePoint, Electrify America), an open dataset like the US Department of Energy’s AFDC or Open Charge Map, or a hybrid that combines both with real-time availability overlays.
The open sources are the easiest starting point. Open Charge Map exposes a clean REST API at
api.openchargemap.io/v3/poi/with no authentication required for read-only access. The AFDC (Alternative Fuels Station Locator) from the DOE requires a free API key but returns structured JSON with GPS coordinates, connector types, network operator, and access hours.Proprietary networks are harder. PlugShare, for instance, renders its map via a private GraphQL endpoint that requires session tokens. ChargePoint uses a mix of REST and WebSocket for real-time status. These aren’t documented publicly, which means you’re reverse-engineering from browser DevTools.
Scraping Map Tile Renderers vs. API Endpoints
The technical split here matters a lot for tooling choice:
Source type Tooling Auth complexity Data freshness Open REST API (AFDC, OCM) httpx, requests API key or none Hourly to daily Private GraphQL (PlugShare) Playwright + session High (login + tokens) Near real-time Map tile overlays Playwright + intercept Medium (cookie-based) Real-time Embedded iframe widgets Playwright + frame Low to medium Varies If your target exposes an API, use it. Network interception via Playwright is more brittle and breaks whenever the frontend team ships a new build. For maps that don’t, you intercept XHR/fetch calls while the page renders and capture the JSON payload before it hits the DOM.
This is structurally similar to scraping gas station pricing apps, where the rendered map is just a skin over an internal pricing API. Check out How to Scrape Gas Station Pricing Apps at Scale (2026) for a detailed look at the intercept pattern applied to fuel price tiles.
Pulling Open Charge Map Data with Python
For any project that can tolerate OCM’s coverage gaps, this is the fastest path to production:
import httpx OCM_BASE = "https://api.openchargemap.io/v3/poi/" params = { "output": "json", "countrycode": "US", "maxresults": 500, "compact": True, "verbose": False, "latitude": 37.7749, "longitude": -122.4194, "distance": 50, "distanceunit": "Miles", "connectiontypeid": "33,32", # CCS, CHAdeMO } with httpx.Client(timeout=30) as client: r = client.get(OCM_BASE, params=params) stations = r.json() for s in stations: print(s["AddressInfo"]["Title"], s["AddressInfo"]["Latitude"], s["AddressInfo"]["Longitude"])Key fields to extract:
AddressInfo(location),Connections(connector type + power level),StatusType,OperatorInfo, andUsageCost. TheDataQualityLevelfield (1-5) is useful for filtering out stale community submissions.Pagination uses
offsetandmaxresults. For national coverage, loop in 500-record pages across a bounding box grid, or use thecountrycodefilter with a state-levellatitude/longitudesweep.Handling Anti-Bot Protections on Proprietary Networks
PlugShare and ChargePoint both deploy bot mitigation. PlugShare specifically uses Cloudflare with JS challenge pages. ChargePoint uses a combination of rate limiting and device fingerprinting on their mobile API endpoints.
The practical approach in 2026:
- Use Playwright with a stealth patch (playwright-stealth or rebrowser-patches) to pass JS challenges
- Rotate residential proxies, not datacenter IPs. Cloudflare’s scoring heavily penalizes ASNs associated with hosting providers
- Intercept the XHR call, not the rendered DOM. Right-click the map in DevTools, filter by XHR, trigger a pan or zoom, and watch for the JSON payload
- Respect session tokens. PlugShare tokens expire; refresh them with a re-login flow rather than hammering the same token until it 429s
This is roughly the same anti-bot stack you’d use for time-sensitive retail scraping. The How to Scrape Black Friday Deal Sites in Real-Time (2026) guide covers the real-time intercept and token rotation pattern in depth, which maps cleanly here.
One specific trap: some networks (Blink, EVgo) serve their connector availability data via WebSocket, not REST. You can’t intercept a WebSocket payload with a simple XHR listener. Use Playwright’s
page.on("websocket", ...)handler and parse the binary or JSON frames directly.If you’re building a large-scale competitor intelligence layer across multiple charging networks, the infrastructure overlap with EV vehicle marketplace data is significant. The How to Scrape Cars.com Vehicle Listings and Dealer Data (2026) article covers the pagination and deduplication patterns that apply equally well when you’re normalizing station data across PlugShare, ChargePoint, and AFDC into a single schema.
Structuring and Normalizing the Output
Raw station data across sources is messy. Connector type IDs differ between Open Charge Map (type ID 33 = CCS) and ChargePoint’s internal enum. Power levels are sometimes in kW, sometimes in amps with voltage separate. Network names aren’t consistent.
A minimal normalization schema for a cross-source station record:
station_id(source-prefixed, e.g.,ocm_12345,cp_abc123)name,lat,lng,addressnetwork(normalized string: “ChargePoint”, “PlugShare”, “Tesla”, etc.)connectors(array of{type, power_kw, count, status})access(“public”, “private”, “restricted”)last_verified(timestamp from source or your scrape time)source(“ocm”, “afdc”, “chargepoint_api”, “plugshare_scrape”)
Deduplication is a real problem when you combine sources. The same physical station can appear in OCM, AFDC, and ChargePoint’s own API with slightly different GPS coordinates (within 5-20 meters). Cluster by proximity (< 50m) and connector type overlap before merging. This exact deduplication challenge comes up in affiliate data work too; the How to Scrape Coupon Aggregator Sites for Affiliate Tracking (2026) article covers a similar entity-matching approach for offers that appear across multiple aggregators.
Store raw responses alongside normalized records. EV network data changes fast, connector types get added, power levels get upgraded, and having the raw payload lets you re-parse without re-scraping.
Bottom Line
Start with Open Charge Map and the AFDC API for free, clean coverage of public stations in the US, EU, and UK. Move to browser-based interception only for networks with no public API and only if your use case genuinely requires their proprietary availability data. Use residential proxies and stealth patches for Cloudflare-protected targets, and invest in a normalization layer from day one if you’re combining more than one source. DRT covers the full scraping stack for infrastructure, mobility, and data-heavy verticals, so bookmark this site if EV data is part of a larger pipeline you’re building.
Related guides on dataresearchtools.com
-
How to Scrape Gas Station Pricing Apps at Scale (2026)
Gas station price scraping is one of the more deceptively hard niches in data collection — prices update every few hours, apps use aggressive fingerprinting, and the same station can show a different price depending on your GPS coordinates. If you’re building a fuel price tracker, competitive intelligence tool, or consumer-facing app, here’s how to scrape gas station pricing data at scale in 2026 without getting rate-limited into oblivion.
Why Gas Station Apps Are Harder Than They Look
Apps like GasBuddy, Waze Fuel, and AAA TripTik don’t just serve static HTML. They rely on mobile APIs with JWT tokens that rotate on session start, GPS-bound queries (prices differ by lat/lng radius), and crowdsourced update timestamps. If you try to hit the API with the same IP or device fingerprint twice in a row, you’ll get stale cached data or a 403.
The core challenge is that most gas price APIs are location-parameterized. A request for prices near downtown Houston returns different data than one for the same stations from a San Jose IP. That means your scraper needs to spoof accurate geolocation headers, not just rotate IPs. This is the same pattern you encounter when scraping electric vehicle charging station maps, where charger availability is also GPS-gated.
Reverse Engineering the Mobile API
Start with a rooted Android emulator or a physical device running Charles Proxy or mitmproxy. Intercept traffic from GasBuddy or the AAA app during a normal session. What you’re looking for:
- The base API endpoint (often
api.gasbuddy.com/graphqlor a REST variant) - The
X-GB-Session-Tokenor equivalent header - How latitude/longitude are passed (query param vs. POST body)
- Whether the app pins certificates (if so, use Frida to bypass)
Once you have the raw request shape, replicate it in Python with
httpx. Do not userequestsfor this — async matters when you’re querying thousands of lat/lng grid points.import httpx import asyncio HEADERS = { "User-Agent": "GasBuddy/8.2.1 (Android 13; Pixel 7)", "X-GB-Session-Token": "<rotated_token>", "Accept": "application/json", } async def fetch_prices(client, lat, lng): resp = await client.get( "https://api.gasbuddy.com/v3/stations/near", params={"lat": lat, "lng": lng, "limit": 50, "fuel": 1}, headers=HEADERS, timeout=10.0, ) resp.raise_for_status() return resp.json() async def scrape_grid(grid_points): async with httpx.AsyncClient() as client: tasks = [fetch_prices(client, lat, lng) for lat, lng in grid_points] return await asyncio.gather(*tasks, return_exceptions=True)Token rotation is the hard part. Generate fresh session tokens by replaying the app’s auth flow (device ID + app version fingerprint) on a schedule. Aim for one token per 200-300 requests max.
Proxy Strategy for Location-Accurate Data
Residential mobile proxies are non-negotiable here. Datacenter IPs get blocked on GasBuddy within minutes. You need IPs that geolocate to the metro you’re querying — Houston prices from a Houston IP, not a Frankfurt datacenter.
Provider Type Block Rate Location Accuracy Cost/GB Datacenter Very High Poor $0.50-$1 Residential Static Medium Good $3-$6 Mobile Residential Low Excellent $8-$15 ISP (AS-matched) Medium-Low Good $4-$7 For national coverage across 50 metros, budget for mobile residential proxies with US carrier pools. Rotate per request, not per session — session stickiness actually hurts you here because the same IP across many lat/lng combos looks like a bot. This is a different pattern from coupon or deal scraping: if you’ve scraped coupon aggregator sites for affiliate tracking, you’re used to session stickiness being useful. For geo-parameterized APIs, it’s the opposite.
Handling Rate Limits and Anti-Bot Layers
GasBuddy’s GraphQL endpoint uses query complexity scoring. Requesting 50 stations with full price history in one query triggers a complexity limit (HTTP 429 with a
Retry-Afterheader). Strategies that actually work:- Reduce query depth — fetch station list first, then price details in a second pass
- Jitter your concurrency — don’t fire all 50 async tasks at once; use a semaphore capped at 8-10 concurrent requests
- Rotate device fingerprints — keep a pool of 10-20 distinct
User-Agent+ device ID combinations - Respect
Retry-After— parse it and back off; hammering through it gets your token pool flagged
Some apps layer Cloudflare Turnstile or a lightweight TLS fingerprint check (JA3). If you hit these, you need a browser automation layer (Playwright with a real Chromium build) for the auth/token step only. The pricing API calls themselves can stay in
httpxonce you have a valid token. The same JA3 bypass approach applies when scraping high-frequency commerce data — see the guide on scraping Black Friday deal sites in real-time for a worked example with Playwright + API handoff.Building the Grid and Storage Layer
Coverage is a grid problem. To scrape prices for all US stations, you need to tile the country with overlapping lat/lng query points. A 25-mile radius per query works well for suburban/rural coverage; drop to 5 miles in dense metros like NYC or LA or you’ll miss stations.
Key schema decisions:
- Store
(station_id, price_cents, fuel_grade, observed_at, source_lat, source_lng)— not just the station price - Index on
(station_id, observed_at)for time-series queries - Deduplicate by
(station_id, observed_at::hour)to avoid redundant writes from overlapping grid queries
For write throughput at scale (5,000+ stations, hourly updates), TimescaleDB or ClickHouse outperform Postgres for raw price history. For the station metadata layer (address, brand, amenities), Postgres is fine. Anti-bot complexity scales with query volume in a way that’s similar to what you see in other high-cardinality scraping targets — the Temu anti-bot guide covers the fingerprinting evasion patterns in depth if you need a primer on the underlying mechanics.
Scheduling and Freshness
Gas prices move 3-4 times per day on average, with spikes around refinery news or crude futures swings. Practical update schedule:
- High-volume metros: every 2 hours
- Suburban/rural: every 4-6 hours
- Overnight (1am-5am local): skip or reduce — prices rarely change and traffic is low
Use a task queue (Celery + Redis, or Temporal for more complex retry logic) rather than cron. Grid points should be queued as individual tasks so failures don’t stall a whole region.
Bottom Line
Scraping gas station pricing apps at scale is solvable with mobile residential proxies, token rotation, and a location-aware grid architecture — but don’t underestimate the lat/lng dimension, which breaks naive scrapers immediately. Start with one metro, get the auth flow nailed, then scale out. DRT covers this kind of niche infrastructure scraping regularly; the pattern here (mobile API + geo-gating + high update frequency) shows up across more verticals than you’d expect.
Related guides on dataresearchtools.com
- The base API endpoint (often