Your cart is currently empty!
Category: Uncategorized
-
How to Scrape Personio Career Sites (2026)
Personio career sites are a goldmine for recruiting intelligence and job market analysis — and because Personio powers HR for thousands of European SMBs, scraping their public job boards gives you structured hiring data that isn’t reliably indexed on LinkedIn or Indeed. This guide covers how to scrape Personio career sites in 2026, including the two main URL patterns, anti-bot posture, and a working Python approach.
How Personio Career Pages Are Structured
Personio hosts career pages on two URL patterns:
https://{company}.jobs.personio.de/(German-hosted, common for EU companies)https://{company}.jobs.personio.com/(global variant)
Some companies embed the widget on their own domain via an iframe or JavaScript snippet, but the underlying data still comes from Personio’s API. Each job listing URL follows the pattern
/job/{id}, and the index page loads a full JSON payload on the client side — which is your primary extraction target.The job listing data is injected into the page as a
window.__NUXT__server-side rendered object, or in newer deployments, fetched from a JSON API endpoint athttps://{company}.jobs.personio.com/api/v1/jobs(no auth required). That API endpoint is the clean path — you bypass HTML parsing entirely.Extracting the JSON API Directly
The undocumented but stable endpoint returns a JSON array of all open roles:
import httpx import json COMPANY = "yourcompany" BASE = f"https://{COMPANY}.jobs.personio.com" def fetch_jobs(): r = httpx.get( f"{BASE}/api/v1/jobs", headers={"Accept": "application/json", "Accept-Language": "en"}, timeout=15, ) r.raise_for_status() return r.json() # list of job dicts jobs = fetch_jobs() for job in jobs: print(job["id"], job["name"], job.get("department", {}).get("name"))Each job object contains
id,name,department,office,employment_type,created_at, and ajob_descriptionsarray with HTML content blocks. For full JD text, you’ll need a second call to/job/{id}or parse thejob_descriptionskey already in the response.If the
/api/v1/jobsendpoint returns 404, fall back to scraping the rendered HTML and extracting theJSON blob, then parsing the nested state tree.Anti-Bot Posture and Rate Limits
Personio's job pages are relatively permissive compared to enterprise ATS platforms. There's no Cloudflare challenge on most subdomains, and Akamai is not present. That said, a few behaviors to watch for:
- Rate limiting: Aggressive crawling (more than ~30 req/min) triggers 429s from their CDN. Space requests with a 2-4 second jitter.
- User-Agent checks: The bare
python-httpxUA gets blocked on some subdomains. Use a realistic browser UA string. - Geo-blocking: Some Personio customers restrict their career page to specific regions. If you're scraping a German Mittelstand company from a US IP, you may get a redirect or empty results.
For the geo issue, residential proxies from a German or European IP pool resolve it cleanly. This is the same pattern used when scraping ATS platforms like SmartRecruiters hiring pages -- the target's CDN sees a local visitor rather than a datacenter range.
Issue Symptom Fix 429 Too Many Requests Burst of requests blocked Add jitter (2-4s), reduce concurrency Empty JSON array []response but page shows jobsSwitch Accept-Language header to de404 on /api/v1/jobsOlder Personio tenant Parse __NUXT_DATA__from HTMLGeo-block redirect 302 to /not-availableUse EU residential proxy iframe embed Jobs on company domain, not Personio Trace network tab for Personio API origin Parsing Job Detail Pages
If you need structured job descriptions (not just titles), here's the hierarchy to expect inside
job_descriptions:- Each object has a
namefield (section heading like "Your Role", "Requirements") - The
valuefield contains raw HTML - Nested
job_descriptionsarrays appear for grouped sections
A clean parse using BeautifulSoup:
from bs4 import BeautifulSoup def extract_text_blocks(job: dict) -> dict: sections = {} for block in job.get("job_descriptions", []): heading = block.get("name", "body") html = block.get("value", "") text = BeautifulSoup(html, "html.parser").get_text(separator="\n").strip() sections[heading] = text return sectionsThis is cleaner than scraping the rendered page and avoids the flaky CSS selectors that break when Personio updates their frontend. Similar structured-extraction approaches work well for Recruitee job pages, which also expose a JSON-first data layer.
Scaling Across Multiple Companies
If you're building a broader job market dataset -- tracking hiring velocity, team growth, or competitive intelligence -- you'll need to crawl many Personio tenants, not just one.
The main challenge is discovery. There's no public Personio company directory. Practical sourcing approaches:
- Search
site:jobs.personio.deorsite:jobs.personio.comon Google to surface active tenants - Pull from LinkedIn company pages (many link directly to their Personio career site)
- Use Crunchbase or Apollo to filter EU SMBs, then probe the Personio subdomain pattern
Once you have a list of subdomains, the
/api/v1/jobscall is identical across all tenants. A simple async crawler withhttpx.AsyncClientand a semaphore of 5-10 concurrent workers handles hundreds of companies per hour without triggering rate limits.For pipelines that need to stay current, schedule daily or weekly crawls and diff against your previous snapshot. New
created_attimestamps flag fresh postings; disappearing job IDs signal closed roles. Platforms like Ashby career sites use a similar versioned-listing model, making diff-based freshness detection a reusable pattern across ATS targets.If your use case extends beyond job data into broader HR intelligence -- org structure signals, headcount trends, or location expansion -- consider pairing Personio data with iCIMS career site scrapes to cover the mid-market US segment that Personio doesn't reach.
One note on scope: Personio job data is B2B recruiting intelligence. If your pipeline is pivoting toward real estate or property listings across emerging markets, the extraction patterns here generalize, though the tooling differs -- DRT has separate coverage on scraping Latin American real estate sites like Imovelweb and Mercado Libre for those use cases.
Handling Edge Cases
A few Personio-specific quirks that'll bite you in production:
- Multilingual listings: A job may have separate entries for
deandenlanguage variants. They share the same base ID but differ by locale parameter. Filter by?language=enquery param on the API call. - Department normalization: Department names are free-text set by the employer. "Engineering", "Tech", "R&D", and "Product & Engineering" all need to be mapped in your downstream schema.
- Employment type inconsistency: Some tenants use "Full-time" vs "Vollzeit" vs custom strings. Normalize to a canonical enum before storing.
- Deleted vs. filled roles: Personio removes filled roles from the index immediately. If you need historical data, snapshot on every crawl.
Bottom Line
Personio's
/api/v1/jobsendpoint is the right entry point -- skip the HTML, go straight to the JSON, add a realistic UA and EU residential proxy for geo-sensitive tenants, and run async crawls with conservative concurrency. The data is clean, consistently structured, and requires no authentication. For broader ATS coverage or freshness-detection patterns across European hiring markets, dataresearchtools.com covers the full stack of platforms engineers actually encounter in production pipelines.Related guides on dataresearchtools.com
-
How to Scrape Recruitee Pages for Lead Sourcing (2026)
Recruitee powers career pages for thousands of mid-market companies across Europe and North America, and its consistent URL structure makes it one of the more approachable ATS targets for lead sourcing at scale. If you’re building a list of companies actively hiring in a specific role, location, or tech stack, scraping Recruitee pages gives you a real-time signal that job boards like LinkedIn lag by days.
Understanding Recruitee’s URL and API Structure
Every Recruitee career site follows the same pattern:
https://{company}.recruitee.com/for the public jobs page, andhttps://{company}.recruitee.com/api/offers/for the JSON feed. That API endpoint is the main event — it returns structured job data without any rendering requirement.A typical response from
/api/offers/looks like this:{ "offers": [ { "id": 182934, "title": "Senior Data Engineer", "department": "Engineering", "location": "Amsterdam, Netherlands", "remote": true, "created_at": "2026-04-12T08:00:00Z", "career_url": "https://acme.recruitee.com/o/senior-data-engineer" } ] }No authentication, no token rotation, just a clean GET. For most companies this endpoint returns 200 with
Content-Type: application/json. It doesn’t paginate (all offers come back in one call), which keeps the scraper simple.The harder part is getting the list of company subdomains to query. There’s no public Recruitee directory, so you need to seed your target list from a separate source: Apollo, Crunchbase, or a curated vertical list. This is conceptually similar to the approach covered in How to Scrape Yellow Pages Business Data, where you build a domain list first, then loop your scraper over it.
Building the Scraper
Use
httpxwith async for throughput. Recruitee doesn’t aggressively rate-limit individual company subdomains, but if you’re hitting 1,000+ subdomains in one run, you’ll want concurrency caps and retry logic.import asyncio import httpx async def fetch_offers(client: httpx.AsyncClient, slug: str) -> dict: url = f"https://{slug}.recruitee.com/api/offers/" try: r = await client.get(url, timeout=10) if r.status_code == 200: return {"slug": slug, "offers": r.json().get("offers", [])} except (httpx.TimeoutException, httpx.RequestError): pass return {"slug": slug, "offers": []} async def main(slugs: list[str]): async with httpx.AsyncClient(follow_redirects=True) as client: tasks = [fetch_offers(client, s) for s in slugs] return await asyncio.gather(*tasks)Run this with a semaphore (limit to 20-30 concurrent) and you can process 5,000 companies in under 10 minutes on a standard VPS. The
follow_redirects=Truematters — some companies migrate away from Recruitee and the old subdomain 301s somewhere unhelpful.Useful fields to extract per offer:
title,department,location,remote,created_at,career_url. Thecreated_atfield is the most valuable for freshness filtering — jobs posted in the last 14 days indicate active hiring, which is a strong lead qualifier.Anti-Bot Considerations and Proxy Use
The JSON API endpoint is low-friction for most companies, but if you’re scraping the HTML career pages (to capture structured data not in the API, like required skills parsed from job descriptions), you’ll hit Cloudflare on some subdomains. Recruitee’s default configuration doesn’t block the API path, but aggressive crawling of the HTML listings will get your IP flagged.
For the API-only approach, residential proxies are overkill — a rotating datacenter pool at ~3 req/s per IP is fine. If you’re parsing HTML job descriptions at scale, use residential or mobile IPs and add a randomized delay between 1.5 and 4 seconds. Keep your User-Agent consistent with a recent Chrome build.
Compared to more heavily protected ATS platforms, Recruitee is relatively open:
ATS Platform API Available Cloudflare Present Auth Required JS Rendering Needed Recruitee Yes ( /api/offers/)Sometimes (HTML only) No No (API) Workday No public API Yes Yes Yes SmartRecruiters Partial Yes Sometimes Yes Personio No public API Varies No No Ashby Yes ( /api/job-board/)Minimal No No Workday is the most locked down by far — as covered in How to Scrape Workday Career Sites at Scale (2026), it requires full browser automation and per-tenant URL discovery. Recruitee is closer to Ashby in permissiveness, which is why it’s a good starting point if you’re new to ATS scraping.
Enriching and Qualifying the Lead Data
Raw job posting data alone is a weak lead signal. You need to layer on company-level attributes to prioritize outreach. A useful enrichment stack:
- Resolve the company slug to a domain using the Clearbit or Hunter enrichment API
- Cross-reference against your CRM to filter out existing customers or known churned accounts
- Pull headcount and funding stage from Apollo or Crunchbase to segment by company size
- Score by job recency — offers posted within the last 7 days get the highest priority
- Filter by department if you’re targeting specific buyers (e.g., only “Engineering” or “Data” roles indicate a technical buyer)
What the Recruitee API doesn’t give you is department headcount or seniority distribution across all open roles. For that, you’d need to aggregate across multiple job posts. If a company has 8 open engineering roles across data/ML/backend, that’s a much stronger signal than one generic posting.
This enrichment workflow mirrors what’s needed when targeting other ATS sources. How to Scrape SmartRecruiters Hiring Pages (2026) and How to Scrape Personio Career Sites (2026) cover similar enrichment approaches for those platforms — the enrichment logic is largely portable once you have normalized offer records.
Handling Edge Cases
A few things that will break a naive scraper:
- Subdomain not found (404/NXDOMAIN): Some companies deactivate their Recruitee account but the subdomain persists in your seed list. Catch DNS failures separately from HTTP errors and flag them for removal.
- Empty offers array: A company may have a valid Recruitee account with zero active postings. Log these separately — they’re worth re-checking in 30 days rather than discarding.
- Non-English job descriptions: Recruitee is popular in the Netherlands, Germany, and Poland. If your downstream NLP pipeline assumes English, add a language detection step (langdetect or fasttext) before parsing.
- Custom domains: Some companies configure a custom domain (e.g.,
jobs.acme.com) that proxies to Recruitee. The API path still works:https://jobs.acme.com/api/offers/. Check the page source for the Recruitee widget script to confirm.
How to Scrape Ashby Career Sites for Talent Pipelines (2026) documents a nearly identical custom-domain issue — it’s a common pattern across modern ATS platforms that support white-labeling.
Bottom Line
Recruitee’s
/api/offers/endpoint is the cleanest ATS scraping target available right now — no auth, no JS rendering, structured JSON out of the box. The real work is in building a quality seed list of company subdomains and enriching the output into actionable lead records. Start with a focused vertical (SaaS companies in the Netherlands, for example), validate your pipeline on 500 companies before scaling, and re-run the scrape weekly for fresh hiring signals. DRT covers this class of infrastructure scraping targets in depth — if you’re building a full multi-ATS pipeline, bookmark the full series.Related guides on dataresearchtools.com
-
How to Scrape SmartRecruiters Hiring Pages (2026)
It looks like write permission to
~/Desktop/drt-articles/is being blocked. can you grant access to that path, or let me know an alternative folder to save to?Related guides on dataresearchtools.com
-
How to Scrape Workday Career Sites at Scale (2026)
Workday career sites are some of the most frustrating scrape targets in the job data space — heavily JavaScript-rendered, rate-limited per IP, and protected by Cloudflare or Akamai depending on the employer. If you’re building a job aggregator, a recruiting intelligence tool, or a competitive headcount tracker, you need a reliable pipeline that handles Workday’s quirks without burning through proxies or getting your IP ranges blocked inside 48 hours.
How Workday Serves Job Data
Workday career pages follow a consistent pattern. The public-facing URL is typically
https://. The page shell loads via a React-based SPA, then fetches job listings through a GraphQL-like REST endpoint:.wd1.myworkdayjobs.com/en-US/ /jobs GET https://<company>.wd1.myworkdayjobs.com/wday/cxs/<company>/<tenant>/jobsThe payload is a POST with a JSON body:
{ "limit": 20, "offset": 0, "searchText": "", "locations": [], "categories": [] }This endpoint is unauthenticated for most public career sites, which makes it the cleanest extraction path. Skip Selenium entirely for the initial crawl — hit the API directly with
httpxorrequests, paginate by incrementingoffsetby 20, and parse thejobPostingsarray in the response. You’ll get title, location, requisition ID, and a relative URL per listing.The detail page for each job is a second request:
GET /jobDetails?jobPostingId=. This returns full description HTML in a JSON field. Parse it withBeautifulSouporlxmland you’re done.Anti-Bot Layers and Where They Kick In
The direct API approach works until it doesn’t. Workday deploys different protection stacks depending on the employer’s contract tier:
Protection Layer Trigger Symptoms Cloudflare Bot Management High request velocity from single IP 403 with CF ray header Akamai Bot Manager Headless browser fingerprint Empty response body or redirect loop Workday rate limiting >50 req/min per IP 429 with Retry-AfterheaderTenant-level blocks Repeated scraping of same tenant 503 or silent empty results For Fortune 500 employers — think Salesforce, JPMorgan, or Deloitte — you will hit Cloudflare or Akamai. For mid-market companies, basic IP rotation is usually sufficient.
Rotate residential or mobile proxies, not datacenter IPs. Workday’s bot scoring is sensitive to ASN reputation. A Singapore or US residential pool with 5-10 second request delays per IP handles the majority of mid-market tenants without triggering blocks. Unlike simpler ATS platforms such as Lever and Greenhouse, Workday applies consistent bot scoring across all employer tenants, so you can’t exploit per-tenant gaps.
Scaling Across Thousands of Workday Tenants
The harder engineering problem is discovery: finding all Workday tenants worth scraping. There is no public tenant directory.
Three practical approaches:
- Seed from LinkedIn company pages. Filter companies by ATS using tools like Apify’s LinkedIn Company Scraper or a custom crawler, then check for the
wd1.myworkdayjobs.compattern in theCareerslink. - Use Google dorks:
site:wd1.myworkdayjobs.com -site:myworkday.comreturns indexed tenant subdomains. Export 100-200 at a time, deduplicate, and build your tenant list. - Buy a commercial dataset. Revelio Labs and Coresignal both maintain ATS-tagged company datasets. $500-2000 gets you a CSV with Workday tenant slugs for 15,000+ companies.
Once you have tenants, the crawl architecture matters. A naive sequential crawler will take weeks at scale. Use a job queue (Celery + Redis, or RQ) with per-tenant rate limiting. Set a max concurrency of 1 request per tenant per minute and run 50-100 workers. At that rate, 10,000 tenants with an average of 30 job postings each is a ~6-hour full crawl.
import httpx import time def fetch_jobs(tenant: str, company: str, offset: int = 0) -> dict: url = f"https://{company}.wd1.myworkdayjobs.com/wday/cxs/{company}/{tenant}/jobs" payload = {"limit": 20, "offset": offset, "searchText": "", "locations": [], "categories": []} headers = { "Content-Type": "application/json", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36", } r = httpx.post(url, json=payload, headers=headers, timeout=15) r.raise_for_status() return r.json()For tenants behind Akamai, swap
httpxfor a Playwright or Camoufox session that passes browser fingerprinting. Keep headless sessions warm across multiple requests to the same tenant rather than spinning up a new context per page — cold browser fingerprints score worse than warm ones.Storing and Deduplicating Job Postings
Job postings are volatile. The same requisition ID appears across multiple crawl cycles, and companies close and reopen roles. Your schema needs a few things:
requisition_id+tenant_slugas a composite unique keyfirst_seen_atandlast_seen_attimestamps for freshness trackingis_activeboolean flipped to false when a job disappears from the feed- A
raw_jsoncolumn for the full response payload so you can reparse without re-crawling
PostgreSQL with a partial index on
(tenant_slug, is_active)handles tens of millions of rows without issue. If you’re running this alongside other ATS scrapers — SmartRecruiters or Recruitee for example — normalize job records into a single canonical schema with anats_sourcefield. Cross-ATS analysis gets much easier when the data model is unified from day one.Proxy Selection for Workday at Scale
Not all proxy types perform equally against Workday’s bot stack:
- Residential rotating proxies: best default. US residential pools (Brightdata, Oxylabs, Smartproxy) handle 90% of tenants. Expect $3-8 per GB.
- Mobile proxies: highest trust score, best for Cloudflare-protected Fortune 500 tenants. More expensive at $15-25/GB, but failure rates drop significantly. The same logic applies when scraping boutique recruitment sites that use shared Cloudflare plans.
- Datacenter proxies: avoid entirely for Workday. Block rates exceed 60% even on premium providers.
- ISP/static residential: middle ground. Good for low-volume, high-fidelity scraping of a fixed tenant list.
Proxy rotation strategy matters as much as proxy type. The same principles that apply to review site scraping hold here: use sticky sessions per tenant (not per request), keep session duration under 10 minutes, and retire any IP that returns a 429 or 403 for a minimum of 30 minutes before reassignment.
Key failure signals to handle in your retry logic:
429withRetry-After: back off for the specified duration, then retry with a fresh IP403with CF ray header: rotate IP immediately, add 5s jitter before retry- Empty
jobPostingsarray with HTTP 200: silent block, treat as soft failure, retry after 15 minutes - Connection timeout: infrastructure issue or hard IP block, retire IP for 1 hour
Bottom Line
Workday’s direct JSON API is your fastest path to structured job data — skip the browser automation unless you’re targeting the Cloudflare tier of employers. Pair it with residential or mobile proxy rotation, a per-tenant rate limiter, and a deduplication schema built around
requisition_id. At 50-100 workers, you can maintain a fresh, full-coverage dataset across 10,000+ tenants on a single cloud instance. DRT covers the full ATS scraping stack across all major platforms if you’re building a multi-source job data pipeline.Related guides on dataresearchtools.com
- How to Scrape Boutique Recruitment Site Postings (2026)
- How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
- How to Scrape SmartRecruiters Hiring Pages (2026)
- How to Scrape Recruitee Pages for Lead Sourcing (2026)
- Pillar: How Proxies Help Scrape Reviews at Scale: Yelp, Google, Trustpilot (2026)
- Seed from LinkedIn company pages. Filter companies by ATS using tools like Apify’s LinkedIn Company Scraper or a custom crawler, then check for the
-
How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
Lever and Greenhouse power job listings for thousands of tech companies, and scraping them programmatically is one of the cleanest data collection tasks you can do in 2026 — both platforms expose structured APIs and predictable URL patterns that make bulk extraction far more reliable than scraping legacy ATS systems.
Why Lever and Greenhouse Are Scraper-Friendly
Neither platform is trying to hide its job data. Greenhouse publishes a public JSON board API that requires no authentication for read access. Lever similarly exposes a public posting endpoint per employer. Both are designed this way intentionally: companies want their listings indexed and aggregated.
That said, “scraper-friendly” doesn’t mean “rate-limit-free.” Both platforms throttle aggressive crawlers, and Greenhouse’s newer board configurations increasingly route through Cloudflare. If you’re building a high-volume aggregator, you’ll hit walls that a naive requests loop won’t survive. For context on how other ATS stacks compare, see How to Scrape Workday Career Sites at Scale (2026) — Workday is the opposite of Greenhouse: no public API, full JS rendering required.
Greenhouse API: The Right Way to Pull Job Data
Greenhouse’s Job Board API is the cleanest starting point. Each company has a unique board token, and the base endpoint is:
GET https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=trueThe
content=trueparameter pulls full job descriptions. Without it, you only get metadata.import httpx import time BOARD_TOKENS = ["stripe", "airbnb", "figma", "notion"] def fetch_greenhouse_jobs(token: str) -> list[dict]: url = f"https://boards-api.greenhouse.io/v1/boards/{token}/jobs" r = httpx.get(url, params={"content": "true"}, timeout=15) r.raise_for_status() return r.json().get("jobs", []) all_jobs = [] for token in BOARD_TOKENS: jobs = fetch_greenhouse_jobs(token) all_jobs.extend(jobs) time.sleep(1.2) # stay under rate limits print(f"Collected {len(all_jobs)} listings")Each job record returns
id,title,location,updated_at,content(HTML),departments,offices, andmetadata. The department and office arrays are particularly useful for org-structure analysis or filtering by function.Finding the board token for a given company is the only friction point. Most companies embed it in their careers page URL (
boards.greenhouse.io/{token}) or in the page source as a data attribute. A simple regex against the page source finds it in under a second.Lever API: Postings and Department Filters
Lever’s public posting endpoint follows the same pattern:
GET https://api.lever.co/v0/postings/{company_slug}?mode=jsonThe
mode=jsonparameter is essential — without it, Lever returns HTML. Lever also supports department and team filtering via query params, which makes targeted extraction much cleaner than post-processing a full dump.Useful query parameters:
?department=Engineering— filter by department?team=Backend— filter by team?commitment=Full-time— filter by job type?mode=json&limit=50&offset=0— pagination (default page size is 25)
Lever responses include
id,text(job title),categories(department, team, location, commitment),description(HTML),descriptionPlain,lists(responsibilities and requirements as structured arrays),salaryRange, andapplyUrl. ThesalaryRangefield is populated for US roles when companies opt into transparency.Handling Scale: Board Tokens in Bulk
If you’re building a jobs aggregator covering thousands of companies, the bottleneck is token/slug discovery, not the API calls themselves. A practical pipeline looks like this:
- Seed a company list from a SaaS review aggregator (G2, Capterra). If you want structured data from those platforms, How to Scrape G2.com and Capterra SaaS Reviews Programmatically covers the extraction path in detail.
- For each company, check careers page URLs for Greenhouse or Lever patterns.
- Validate the token/slug returns a 200 before adding it to your active list.
- Run nightly delta syncs using
updated_atfiltering rather than full re-pulls.
Delta syncing matters because both APIs are fast but not free under load. Greenhouse returns
updated_atper job; Lever returnscreatedAtandupdatedAt. A daily sync that only fetches listings updated in the last 24 hours reduces your request volume by 80-90% on a mature dataset.Platform Comparison
Feature Greenhouse Lever Public API Yes, JSON Yes, JSON Auth required No (read) No (read) Salary data Rare US roles, opt-in Department filtering Post-process Native query param Cloudflare protection Increasing Minimal Pagination style Single response Limit/offset HTML job descriptions Yes ( content=true)Yes + plain text Greenhouse is better for companies with complex location/department hierarchies. Lever is cleaner for filtering by commitment type or team before pulling content.
When the API Breaks Down
Both APIs have edge cases that will quietly return garbage without throwing errors:
- Deleted listings return 200: Greenhouse keeps deleted jobs in responses for up to 72 hours with no flag. Check
updated_atrecency and watch for emptydepartmentsarrays as a signal. - Lever pagination silently truncates: If a company has more than 250 listings, Lever’s API stops at 250 without a “next page” indicator in some configurations. Always check if your result count equals a round number and retry with offset.
- Cloudflare 1020 on Greenhouse: This is an access denied, not a rate limit. Rotating residential IPs fixes it. Data center IPs increasingly trigger 1020 on high-traffic boards.
- Board token changes: Companies occasionally reorg and migrate to a new board slug. Build dead-link detection into your pipeline.
For ATS platforms without public APIs, the extraction approach differs substantially. How to Scrape SmartRecruiters Hiring Pages (2026) covers a platform that requires a hybrid API-plus-DOM strategy, and How to Scrape Recruitee Pages for Lead Sourcing (2026) covers another mid-market ATS popular in Europe. If your target companies use smaller regional systems, How to Scrape Boutique Recruitment Site Postings (2026) is the right starting point for less structured targets.
Bottom line
Greenhouse and Lever are the easiest ATS platforms to scrape at scale in 2026 — both expose well-documented public APIs that return structured JSON with no login required. Start with the API before reaching for a browser automation tool. Use delta syncing by
updated_atto keep costs low, rotate residential IPs to handle Cloudflare friction on high-traffic Greenhouse boards, and build dead-slug detection from day one. DRT covers the full ATS ecosystem if you need extraction guides for the other major platforms your targets use.Related guides on dataresearchtools.com
-
How to Scrape Boutique Recruitment Site Postings (2026)
Boutique recruitment sites — niche job boards, regional talent platforms, and vertical-specific hiring portals — are some of the richest, least-contested sources of hiring signal you can scrape in 2026. Unlike LinkedIn or Indeed, they rarely invest in serious bot mitigation, but they also lack the clean APIs that mainstream ATS platforms expose. That gap is exactly why scraping boutique recruitment site postings requires a different playbook from what you’d use on enterprise systems.
What “boutique recruitment site” actually means
The category is broad. It includes:
- Vertical job boards (tech-only boards like Wellfound, finance-specific boards like eFinancialCareers, legal boards like Lawjobs)
- Regional platforms (JobsDB in Southeast Asia, StepStone in Europe, JobStreet across APAC)
- Staffing agency portals that publish live client postings on their own subdomains
- White-label ATS installs that don’t expose a public API (TeamTailor, Breezy HR, Pinpoint)
The staffing agency portals are the most valuable for competitive intelligence: they reveal which companies are hiring before those roles hit aggregators. The tradeoff is that each portal has a custom HTML structure with no predictable schema.
Fingerprinting the stack before writing a scraper
Before touching Playwright or BeautifulSoup, spend ten minutes identifying the underlying tech. Open DevTools, check the Network tab for XHR requests, and look at the page source for giveaway class names or API subdomains.
Common patterns you’ll hit:
Signal Likely stack Best extraction method /api/jobs?page=XHR callsCustom REST API Direct HTTP requests /__api/widgets/endpointsTeamTailor JSON from undocumented widget API data-listing-idattributesCustom CMS HTML parse with CSS selectors jobs.lever.coiframe embedLever See dedicated Lever guide Cloudflare challenge on first load Any stack Residential/mobile proxy + browser render If you see Lever or Greenhouse embeds, you’re better served by the documented approach in How to Scrape Lever and Greenhouse Job Boards Programmatically (2026) rather than treating the host site as a scrape target. Similarly, if the careers page redirects to a
mycompany.workday.comsubdomain, the How to Scrape Workday Career Sites at Scale (2026) guide covers that path specifically.Extraction strategies by site type
Static or server-rendered HTML
Regional job boards (JobStreet, StepStone, and smaller country-level clones) are often server-rendered with paginated listing pages. These are the simplest to scrape: a
requests+lxmlloop with polite delays is enough for low-volume pulls.import httpx from lxml import html import time BASE = "https://example-jobboard.com/jobs" HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)"} def scrape_page(page_num: int) -> list[dict]: resp = httpx.get(f"{BASE}?page={page_num}", headers=HEADERS, timeout=15) tree = html.fromstring(resp.text) jobs = [] for card in tree.cssselect(".job-card"): jobs.append({ "title": card.cssselect(".job-title")[0].text_content().strip(), "company": card.cssselect(".company-name")[0].text_content().strip(), "url": card.cssselect("a")[0].get("href"), }) return jobs for i in range(1, 20): print(scrape_page(i)) time.sleep(2)Adjust the CSS selectors per site. Keep delays above 1.5 seconds per page — boutique boards have thin infrastructure and you’ll trigger rate limits or kill their server faster than a big platform would notice.
JavaScript-rendered pages with a hidden JSON API
TeamTailor and Breezy HR render listings client-side but pull from an internal REST API. The fastest approach is to intercept that API call rather than rendering the DOM.
For TeamTailor specifically, the widget endpoint follows a predictable pattern:
https://career.teamtailor.com/api/widget/v1/jobs?company_id=XXXX. The company ID is in the page source. Hit that endpoint directly and skip the browser entirely.For Breezy HR portals (
company.breezy.hr), the job list loads fromhttps://company.breezy.hr/json. Again, direct HTTP, no rendering needed.Sites that require a browser
If the page loads a Cloudflare managed challenge or uses bot-detection fingerprinting (mouse movement checks, canvas fingerprinting, TLS JA3 matching), you need a real browser context. Playwright with stealth patches is the standard tool here.
Key configuration points:
- Set a realistic viewport (1440×900, not headless defaults)
- Randomize the User-Agent per session using a current Chrome string
- Add random delays between scroll events (50-200ms)
- Rotate IPs between sessions, not between requests
IP rotation matters more than most people expect. Many boutique boards block on subnet-level reputation, not individual IPs. A residential or mobile proxy pool will pass where datacenter IPs fail — mobile IPs in particular score well on the carrier trust signals that Cloudflare uses.
Handling pagination and deduplication
Boutique sites paginate in three ways: query string (
?page=2), cursor-based (?after=TOKEN), and infinite scroll that fires a new XHR on viewport entry.Numbered pagination is easy to loop over. Cursor-based requires extracting the next-page token from each response, usually in a
meta.next_cursororlinks.nextfield. Infinite scroll requires Playwright — listen for the network request the page fires when you scroll to 80% of the page height, then extract the URL pattern and replay it as direct HTTP.For deduplication across runs, hash the canonical job URL. Don’t hash on title plus company because boutique boards repost expired roles constantly. The URL is stable; the surrounding metadata drifts.
Structuring the output for downstream use
Recruitment data has a short shelf life — a posting can close in 48 hours. Design your schema with a
first_seen_atandlast_seen_attimestamp pair so downstream consumers can calculate posting age and detect removals without keeping a separate diff log.Minimum viable schema:
job_id(hashed URL or extracted native ID)title,company,location,posted_atdescription_html(raw, don’t strip yet)source_url,first_seen_at,last_seen_atstatus(active / closed)
Keep the raw HTML in
description_html. Structured extraction (skills, salary, seniority) should be a separate pass — either a regex pipeline or an LLM extraction step — so you can reprocess historical data without re-scraping.If you are scraping a staffing agency portal specifically for lead sourcing rather than job intelligence, the data model in How to Scrape Recruitee Pages for Lead Sourcing (2026) maps the same fields to a CRM-ready format. And if your target has migrated to SmartRecruiters, How to Scrape SmartRecruiters Hiring Pages (2026) covers the platform-specific quirks.
Bottom line
Boutique recruitment sites reward patient, targeted work: identify the stack first, hit the JSON API directly when available, and only bring a browser when you genuinely need one. Rotate mobile proxies for anything behind Cloudflare — datacenter ranges will get you blocked in minutes on sites this small. DRT covers the full recruitment scraping stack from ATS platforms to niche boards, so bookmark the guides linked above if you’re building a broader pipeline.
Related guides on dataresearchtools.com
- How to Scrape Lever and Greenhouse Job Boards Programmatically (2026)
- How to Scrape Workday Career Sites at Scale (2026)
- How to Scrape SmartRecruiters Hiring Pages (2026)
- How to Scrape Recruitee Pages for Lead Sourcing (2026)
- Pillar: Mobile Proxies for Recruitment: Scrape Indeed, LinkedIn Glassdoor at Scale
-
How to Scrape Electric Vehicle Charging Station Maps (2026)
EV charging station data is one of the most actively updated datasets on the web, and scraping it cleanly in 2026 requires understanding a mix of public APIs, JavaScript-heavy map renderers, and rate-limited tile servers. Whether you’re building a route planner, a fleet management dashboard, or a competitive analysis tool for charging networks, the core challenge is the same: the data lives inside map widgets that weren’t designed to be parsed.
Where EV Charging Data Actually Lives
Most charging station maps pull from one of three sources: a proprietary API (PlugShare, ChargePoint, Electrify America), an open dataset like the US Department of Energy’s AFDC or Open Charge Map, or a hybrid that combines both with real-time availability overlays.
The open sources are the easiest starting point. Open Charge Map exposes a clean REST API at
api.openchargemap.io/v3/poi/with no authentication required for read-only access. The AFDC (Alternative Fuels Station Locator) from the DOE requires a free API key but returns structured JSON with GPS coordinates, connector types, network operator, and access hours.Proprietary networks are harder. PlugShare, for instance, renders its map via a private GraphQL endpoint that requires session tokens. ChargePoint uses a mix of REST and WebSocket for real-time status. These aren’t documented publicly, which means you’re reverse-engineering from browser DevTools.
Scraping Map Tile Renderers vs. API Endpoints
The technical split here matters a lot for tooling choice:
Source type Tooling Auth complexity Data freshness Open REST API (AFDC, OCM) httpx, requests API key or none Hourly to daily Private GraphQL (PlugShare) Playwright + session High (login + tokens) Near real-time Map tile overlays Playwright + intercept Medium (cookie-based) Real-time Embedded iframe widgets Playwright + frame Low to medium Varies If your target exposes an API, use it. Network interception via Playwright is more brittle and breaks whenever the frontend team ships a new build. For maps that don’t, you intercept XHR/fetch calls while the page renders and capture the JSON payload before it hits the DOM.
This is structurally similar to scraping gas station pricing apps, where the rendered map is just a skin over an internal pricing API. Check out How to Scrape Gas Station Pricing Apps at Scale (2026) for a detailed look at the intercept pattern applied to fuel price tiles.
Pulling Open Charge Map Data with Python
For any project that can tolerate OCM’s coverage gaps, this is the fastest path to production:
import httpx OCM_BASE = "https://api.openchargemap.io/v3/poi/" params = { "output": "json", "countrycode": "US", "maxresults": 500, "compact": True, "verbose": False, "latitude": 37.7749, "longitude": -122.4194, "distance": 50, "distanceunit": "Miles", "connectiontypeid": "33,32", # CCS, CHAdeMO } with httpx.Client(timeout=30) as client: r = client.get(OCM_BASE, params=params) stations = r.json() for s in stations: print(s["AddressInfo"]["Title"], s["AddressInfo"]["Latitude"], s["AddressInfo"]["Longitude"])Key fields to extract:
AddressInfo(location),Connections(connector type + power level),StatusType,OperatorInfo, andUsageCost. TheDataQualityLevelfield (1-5) is useful for filtering out stale community submissions.Pagination uses
offsetandmaxresults. For national coverage, loop in 500-record pages across a bounding box grid, or use thecountrycodefilter with a state-levellatitude/longitudesweep.Handling Anti-Bot Protections on Proprietary Networks
PlugShare and ChargePoint both deploy bot mitigation. PlugShare specifically uses Cloudflare with JS challenge pages. ChargePoint uses a combination of rate limiting and device fingerprinting on their mobile API endpoints.
The practical approach in 2026:
- Use Playwright with a stealth patch (playwright-stealth or rebrowser-patches) to pass JS challenges
- Rotate residential proxies, not datacenter IPs. Cloudflare’s scoring heavily penalizes ASNs associated with hosting providers
- Intercept the XHR call, not the rendered DOM. Right-click the map in DevTools, filter by XHR, trigger a pan or zoom, and watch for the JSON payload
- Respect session tokens. PlugShare tokens expire; refresh them with a re-login flow rather than hammering the same token until it 429s
This is roughly the same anti-bot stack you’d use for time-sensitive retail scraping. The How to Scrape Black Friday Deal Sites in Real-Time (2026) guide covers the real-time intercept and token rotation pattern in depth, which maps cleanly here.
One specific trap: some networks (Blink, EVgo) serve their connector availability data via WebSocket, not REST. You can’t intercept a WebSocket payload with a simple XHR listener. Use Playwright’s
page.on("websocket", ...)handler and parse the binary or JSON frames directly.If you’re building a large-scale competitor intelligence layer across multiple charging networks, the infrastructure overlap with EV vehicle marketplace data is significant. The How to Scrape Cars.com Vehicle Listings and Dealer Data (2026) article covers the pagination and deduplication patterns that apply equally well when you’re normalizing station data across PlugShare, ChargePoint, and AFDC into a single schema.
Structuring and Normalizing the Output
Raw station data across sources is messy. Connector type IDs differ between Open Charge Map (type ID 33 = CCS) and ChargePoint’s internal enum. Power levels are sometimes in kW, sometimes in amps with voltage separate. Network names aren’t consistent.
A minimal normalization schema for a cross-source station record:
station_id(source-prefixed, e.g.,ocm_12345,cp_abc123)name,lat,lng,addressnetwork(normalized string: “ChargePoint”, “PlugShare”, “Tesla”, etc.)connectors(array of{type, power_kw, count, status})access(“public”, “private”, “restricted”)last_verified(timestamp from source or your scrape time)source(“ocm”, “afdc”, “chargepoint_api”, “plugshare_scrape”)
Deduplication is a real problem when you combine sources. The same physical station can appear in OCM, AFDC, and ChargePoint’s own API with slightly different GPS coordinates (within 5-20 meters). Cluster by proximity (< 50m) and connector type overlap before merging. This exact deduplication challenge comes up in affiliate data work too; the How to Scrape Coupon Aggregator Sites for Affiliate Tracking (2026) article covers a similar entity-matching approach for offers that appear across multiple aggregators.
Store raw responses alongside normalized records. EV network data changes fast, connector types get added, power levels get upgraded, and having the raw payload lets you re-parse without re-scraping.
Bottom Line
Start with Open Charge Map and the AFDC API for free, clean coverage of public stations in the US, EU, and UK. Move to browser-based interception only for networks with no public API and only if your use case genuinely requires their proprietary availability data. Use residential proxies and stealth patches for Cloudflare-protected targets, and invest in a normalization layer from day one if you’re combining more than one source. DRT covers the full scraping stack for infrastructure, mobility, and data-heavy verticals, so bookmark this site if EV data is part of a larger pipeline you’re building.
Related guides on dataresearchtools.com
-
How to Scrape Gas Station Pricing Apps at Scale (2026)
Gas station price scraping is one of the more deceptively hard niches in data collection — prices update every few hours, apps use aggressive fingerprinting, and the same station can show a different price depending on your GPS coordinates. If you’re building a fuel price tracker, competitive intelligence tool, or consumer-facing app, here’s how to scrape gas station pricing data at scale in 2026 without getting rate-limited into oblivion.
Why Gas Station Apps Are Harder Than They Look
Apps like GasBuddy, Waze Fuel, and AAA TripTik don’t just serve static HTML. They rely on mobile APIs with JWT tokens that rotate on session start, GPS-bound queries (prices differ by lat/lng radius), and crowdsourced update timestamps. If you try to hit the API with the same IP or device fingerprint twice in a row, you’ll get stale cached data or a 403.
The core challenge is that most gas price APIs are location-parameterized. A request for prices near downtown Houston returns different data than one for the same stations from a San Jose IP. That means your scraper needs to spoof accurate geolocation headers, not just rotate IPs. This is the same pattern you encounter when scraping electric vehicle charging station maps, where charger availability is also GPS-gated.
Reverse Engineering the Mobile API
Start with a rooted Android emulator or a physical device running Charles Proxy or mitmproxy. Intercept traffic from GasBuddy or the AAA app during a normal session. What you’re looking for:
- The base API endpoint (often
api.gasbuddy.com/graphqlor a REST variant) - The
X-GB-Session-Tokenor equivalent header - How latitude/longitude are passed (query param vs. POST body)
- Whether the app pins certificates (if so, use Frida to bypass)
Once you have the raw request shape, replicate it in Python with
httpx. Do not userequestsfor this — async matters when you’re querying thousands of lat/lng grid points.import httpx import asyncio HEADERS = { "User-Agent": "GasBuddy/8.2.1 (Android 13; Pixel 7)", "X-GB-Session-Token": "<rotated_token>", "Accept": "application/json", } async def fetch_prices(client, lat, lng): resp = await client.get( "https://api.gasbuddy.com/v3/stations/near", params={"lat": lat, "lng": lng, "limit": 50, "fuel": 1}, headers=HEADERS, timeout=10.0, ) resp.raise_for_status() return resp.json() async def scrape_grid(grid_points): async with httpx.AsyncClient() as client: tasks = [fetch_prices(client, lat, lng) for lat, lng in grid_points] return await asyncio.gather(*tasks, return_exceptions=True)Token rotation is the hard part. Generate fresh session tokens by replaying the app’s auth flow (device ID + app version fingerprint) on a schedule. Aim for one token per 200-300 requests max.
Proxy Strategy for Location-Accurate Data
Residential mobile proxies are non-negotiable here. Datacenter IPs get blocked on GasBuddy within minutes. You need IPs that geolocate to the metro you’re querying — Houston prices from a Houston IP, not a Frankfurt datacenter.
Provider Type Block Rate Location Accuracy Cost/GB Datacenter Very High Poor $0.50-$1 Residential Static Medium Good $3-$6 Mobile Residential Low Excellent $8-$15 ISP (AS-matched) Medium-Low Good $4-$7 For national coverage across 50 metros, budget for mobile residential proxies with US carrier pools. Rotate per request, not per session — session stickiness actually hurts you here because the same IP across many lat/lng combos looks like a bot. This is a different pattern from coupon or deal scraping: if you’ve scraped coupon aggregator sites for affiliate tracking, you’re used to session stickiness being useful. For geo-parameterized APIs, it’s the opposite.
Handling Rate Limits and Anti-Bot Layers
GasBuddy’s GraphQL endpoint uses query complexity scoring. Requesting 50 stations with full price history in one query triggers a complexity limit (HTTP 429 with a
Retry-Afterheader). Strategies that actually work:- Reduce query depth — fetch station list first, then price details in a second pass
- Jitter your concurrency — don’t fire all 50 async tasks at once; use a semaphore capped at 8-10 concurrent requests
- Rotate device fingerprints — keep a pool of 10-20 distinct
User-Agent+ device ID combinations - Respect
Retry-After— parse it and back off; hammering through it gets your token pool flagged
Some apps layer Cloudflare Turnstile or a lightweight TLS fingerprint check (JA3). If you hit these, you need a browser automation layer (Playwright with a real Chromium build) for the auth/token step only. The pricing API calls themselves can stay in
httpxonce you have a valid token. The same JA3 bypass approach applies when scraping high-frequency commerce data — see the guide on scraping Black Friday deal sites in real-time for a worked example with Playwright + API handoff.Building the Grid and Storage Layer
Coverage is a grid problem. To scrape prices for all US stations, you need to tile the country with overlapping lat/lng query points. A 25-mile radius per query works well for suburban/rural coverage; drop to 5 miles in dense metros like NYC or LA or you’ll miss stations.
Key schema decisions:
- Store
(station_id, price_cents, fuel_grade, observed_at, source_lat, source_lng)— not just the station price - Index on
(station_id, observed_at)for time-series queries - Deduplicate by
(station_id, observed_at::hour)to avoid redundant writes from overlapping grid queries
For write throughput at scale (5,000+ stations, hourly updates), TimescaleDB or ClickHouse outperform Postgres for raw price history. For the station metadata layer (address, brand, amenities), Postgres is fine. Anti-bot complexity scales with query volume in a way that’s similar to what you see in other high-cardinality scraping targets — the Temu anti-bot guide covers the fingerprinting evasion patterns in depth if you need a primer on the underlying mechanics.
Scheduling and Freshness
Gas prices move 3-4 times per day on average, with spikes around refinery news or crude futures swings. Practical update schedule:
- High-volume metros: every 2 hours
- Suburban/rural: every 4-6 hours
- Overnight (1am-5am local): skip or reduce — prices rarely change and traffic is low
Use a task queue (Celery + Redis, or Temporal for more complex retry logic) rather than cron. Grid points should be queued as individual tasks so failures don’t stall a whole region.
Bottom Line
Scraping gas station pricing apps at scale is solvable with mobile residential proxies, token rotation, and a location-aware grid architecture — but don’t underestimate the lat/lng dimension, which breaks naive scrapers immediately. Start with one metro, get the auth flow nailed, then scale out. DRT covers this kind of niche infrastructure scraping regularly; the pattern here (mobile API + geo-gating + high update frequency) shows up across more verticals than you’d expect.
Related guides on dataresearchtools.com
- The base API endpoint (often
-
How to Scrape Black Friday Deal Sites in Real-Time (2026)
Black Friday deal scraping is one of the hardest real-time data problems in web scraping — sites like Slickdeals, DealNews, and retailer flash-sale pages throw 10x normal traffic at their CDN edge, rotate anti-bot configs mid-event, and serve stale cached HTML to anyone who looks like a crawler. If your pipeline isn’t built for sub-60-second latency with session rotation, you’re collecting yesterday’s data when the deal already expired.
Why Black Friday Sites Are a Different Class of Problem
Standard e-commerce scraping tolerates a 5-10 minute lag. Black Friday does not. A $200 PS5 bundle sells out in under 90 seconds on BestBuy.com during a flash window. Useful price-comparison or affiliate data requires that your collector, parser, and downstream consumer all run within the same 60-second window.
The additional complication is that retailers specifically harden their stacks in October and November. Cloudflare Turnstile, Akamai Bot Manager, and PerimeterX all get fresh rule updates before the sale season. Headless browser fingerprinting that worked fine in August will fail by November 20.
The same session-rotation principles apply when you scrape coupon aggregator sites for affiliate tracking — but the timing pressure on Black Friday is an order of magnitude tighter.
Infrastructure: What You Actually Need
You need three layers that can each scale independently:
- Collector fleet — rotating residential or mobile proxies, ideally with per-request IP rotation
- Parser workers — stateless, containerized, horizontally scalable (Kubernetes or ECS)
- Event bus — Kafka or Redis Streams to decouple collection from processing
For proxy choice, mobile IPs outperform residential on retail sites during peak season. Retailers’ bot rules increasingly flag datacenter and static residential ranges between Nov 25-29.
Proxy Type Avg Block Rate (BF Week) Latency Cost/GB Datacenter 65-80% 30-80ms $0.50-2 Static residential 25-40% 80-150ms $3-8 Rotating residential 15-25% 100-200ms $5-15 Mobile 4G/5G 5-12% 120-250ms $15-40 The cost jump for mobile is real, but for a 4-hour Black Friday window on high-value targets like BestBuy, Walmart, or Amazon, the conversion rate difference justifies it.
Collector Design for Sub-60-Second Freshness
The core pattern is a polling loop with jitter, not a fixed interval. Fixed 30-second intervals create synchronized request spikes that bot detectors flag as non-human.
import asyncio, random, httpx from datetime import datetime async def poll_deal_page(url: str, proxy: str, interval_base: int = 30): async with httpx.AsyncClient(proxies={"https://": proxy}, timeout=15) as client: while True: jitter = random.uniform(0.7, 1.4) await asyncio.sleep(interval_base * jitter) try: r = await client.get(url, headers={"User-Agent": rotate_ua()}) if r.status_code == 200: await publish_to_stream(url, r.text, datetime.utcnow()) except httpx.TimeoutException: await asyncio.sleep(10)Key details: the
rotate_ua()call should pull from a weighted pool of real Chrome user-agents with matchingsec-ch-uaheaders. Mismatched UA/client-hints pairs are a primary signal for Akamai’s detection layer in 2026.For JavaScript-heavy pages (Target, BestBuy), you need Playwright or Camoufox with proper browser fingerprint patching. Playwright-extra with the stealth plugin is still functional but requires the
rebrowser-patchesfork as of mid-2026 to pass Cloudflare’s updated TLS fingerprinting checks.Parsing the Deal Data
Black Friday pages rarely have stable schemas. Retailers restructure their promo layouts between campaigns. Build parsers against JSON-LD structured data where it exists (most major retailers expose
ProductandOfferschema), and fall back to CSS selectors only when needed.Priority extraction fields for a deal record:
- SKU or product ID (stable across page reloads)
- Current price and original/strike-through price
- Stock status string (not just in/out — “only 3 left” is a signal)
- Deal timestamp or “posted X minutes ago” relative time
- Coupon code if surfaced in DOM
Slickdeals and DealNews expose RSS feeds that are dramatically easier to poll than their HTML. RSS gives you deal metadata at 5-minute freshness with zero anti-bot risk. Use the HTML scraper only for the linked retailer page where the actual purchase happens.
This parsing approach mirrors what’s needed for real-time price intelligence in other competitive verticals — the techniques used to scrape gas station pricing apps at scale apply directly here, especially the delta-detection logic to avoid reprocessing unchanged prices.
Handling Anti-Bot at Scale During Peak Hours
Between 12am-6am EST on Black Friday, Cloudflare challenge rates spike significantly on retail domains. Your error handling needs to distinguish between retriable and non-retriable failures:
- 429 Too Many Requests — back off 90-120 seconds, rotate proxy
- 403 Forbidden — rotate proxy + user-agent immediately, don’t retry same session
- 503 / 524 (Cloudflare timeout) — site-side load issue, retry with exponential backoff
- CAPTCHA challenge page — session is burned, discard and rotate
For programmatic CAPTCHA solving, 2captcha and CapSolver both support Turnstile as of 2026, but solve latency averages 8-15 seconds, which destroys real-time freshness on a 60-second poll cycle. The better answer is avoiding Turnstile triggers entirely through better fingerprint hygiene and proxy quality rather than solving reactively.
Geographic targeting matters too. If you’re scraping US Black Friday deals, residential IPs in the same US region as the retailer’s CDN PoP reduce TLS fingerprint anomaly scores. The same regional IP logic comes up when collecting location-specific data like electric vehicle charging station maps, where geo-relevance affects what data is returned.
Storing and Serving Real-Time Deal Data
Raw HTML goes to object storage (S3 or R2) with a timestamp key. Parsed deal records go to a time-series-friendly store — TimescaleDB or ClickHouse both handle the append-heavy, time-ordered write pattern well at deal-scraping volumes.
For a mid-scale operation (500 product pages, 30-second poll cycle), you’re generating roughly 1 million rows per day during BF week. Postgres with a
dealshypertable and a composite index on(sku, scraped_at)handles this cleanly without needing a separate analytics DB.Alert logic should fire on price drops exceeding a threshold, not on every record. A simple Redis sorted set keyed by SKU with the last-seen price lets you compute deltas in O(log n) before writing to your main store.
If you’re building an affiliate or price-comparison product, this pipeline architecture is essentially identical to what you’d deploy for year-round deal tracking — the same patterns used to scrape Latin American real estate sites like Imovelweb and Mercado Libre apply to any high-volume, time-sensitive listing scrape.
Bottom Line
For Black Friday scraping, mobile proxies plus jittered async polling plus JSON-LD-first parsing is the stack that actually works in 2026 — anything cheaper cuts corners that retailers have specifically patched against. Start your infrastructure testing no later than two weeks before the sale, because bot rule configs change in the final days. DRT covers proxy infrastructure and anti-bot bypass year-round, so bookmark the site if real-time data collection is part of your stack.
Related guides on dataresearchtools.com
-
How to Scrape Coupon Aggregator Sites for Affiliate Tracking (2026)
Coupon aggregator sites like RetailMeNot, Honey, and Coupons.com are goldmines for affiliate tracking data — if you can get to it. Scraping coupon aggregator sites requires navigating JavaScript-heavy frontends, session-based rendering, and aggressive bot detection that has gotten sharper in 2026. This guide walks through the architecture, tooling, and affiliate-specific data extraction patterns that actually work.
Why Coupon Sites Are Hard to Scrape
Most coupon aggregators load deal data client-side via XHR or GraphQL calls, not raw HTML. The visible page is a shell; the coupons populate after JavaScript executes. On top of that, affiliate redirect chains (the
/go/,/out/, or/track/URLs) are often encoded, tokenized, or short-lived — they expire within minutes to prevent link-jacking.Bot detection on these sites is also heavier than you’d expect for a content site. RetailMeNot runs Cloudflare with JavaScript challenges. Coupons.com uses a combination of fingerprinting and behavioral scoring. Honey (now PayPal-owned) rate-limits aggressively on repeat category scans. The same scraping infrastructure you’d use for Black Friday deal sites in real-time works here, but you need to tune session rotation more tightly.
Intercept the API, Skip the DOM
Before building a full browser automation pipeline, spend 20 minutes in Chrome DevTools Network tab. Most aggregators expose undocumented JSON APIs their own frontend calls. These are far more stable than HTML selectors and return clean structured data.
On RetailMeNot, filtering by XHR in DevTools reveals calls like:
GET https://www.retailmenot.com/api/v2/retailer/TARGET/offers ?type=coupon&sort=popular&limit=50Replay that with
curlorhttpx, rotate a real browser User-Agent, and you get JSON withcode,expires,affiliate_url, andtracking_networkfields — everything you need for affiliate attribution without ever rendering the page.Not all sites are this cooperative. When the API is gated behind auth tokens embedded in the page, use Playwright to capture the initial page load, extract the token from the DOM or a cookie, then use
httpxfor all subsequent paginated requests. This hybrid approach cuts browser overhead by 70-80% on large category crawls.Handling Affiliate Redirect Chains
The real extraction challenge is tracking affiliate links through redirect chains. A typical coupon aggregator redirect looks like:
/go/retailmenot?merchant=target&code=SAVE20 → impact.com/c/12345?u=https://target.com/checkout → target.com/checkout?coupon=SAVE20&affid=rmnTo map the full chain, you need to follow redirects without JavaScript (most intermediate hops are 301/302) and capture each step:
import httpx def trace_affiliate_chain(url: str) -> list[str]: chain = [] with httpx.Client(follow_redirects=False) as client: while url: r = client.get(url, headers={"User-Agent": "Mozilla/5.0"}) chain.append(url) url = r.headers.get("location") if r.is_redirect else None return chainThis gives you the full hop sequence. Parse each URL for affiliate network identifiers — Impact uses
/c/, CJ usesanrdoezrs.net, Rakuten useslinksynergy.com. Once you’ve catalogued which merchant uses which network, you can detect attribution changes without following the full chain on every scrape cycle.Proxy and Session Strategy
Coupon sites use IP reputation scoring. Residential IPs on a clean rotation outperform datacenter IPs by a wide margin — expect 3-5x fewer CAPTCHAs. Here’s how the main proxy types compare for this use case:
Proxy Type Cost / GB Block Rate (coupon sites) Affiliate Chain Success Datacenter $0.50-1 High (40-60%) Low Residential rotating $5-15 Low (5-15%) High Mobile LTE $15-40 Very low (<5%) Very high ISP (static residential) $2-8 Medium (15-30%) Medium For periodic batch scrapes (daily deal snapshots), residential rotating is cost-effective. For anything that needs to complete a coupon reveal flow — where the site issues a one-time code only on button click — mobile LTE proxies are worth the premium because they carry the highest trust scores.
The same residential-vs-mobile tradeoff applies when you’re scraping location-sensitive pricing. Work I’ve done on gas station pricing apps and EV charging station maps shows the same pattern: the more a site gates data behind geo-trust signals, the more you need carrier-grade IPs.
Data Schema for Affiliate Tracking
Once you’ve extracted the raw fields, structure them for downstream affiliate analysis. The minimum viable schema for coupon tracking:
merchant_id— normalized merchant slug (not the site’s internal ID)coupon_code— raw string, nullable (some deals are auto-apply, no code)affiliate_network— detected from redirect chain (impact, cj, rakuten, awin)affiliate_id— the publisher ID embedded in the redirect URLexpires_at— ISO 8601, null if evergreenscraped_at— timestamp of collectiondeal_type— enum:code,auto,sale,cashbackverified_at— last time the code was confirmed working (via a test request, not live purchase)
Store
verified_atseparately fromscraped_at. Coupon sites don’t remove expired codes immediately — some stay listed for weeks after expiry. If you’re building an affiliate monitoring dashboard, freshness of verification matters more than freshness of scrape.For merchants operating across regions — particularly in Latin America where platforms like Mercado Libre aggregate deals across multiple countries — consider how geographic affiliate IDs vary. The affiliate tracking patterns for those cross-border retail sites are covered in depth in this guide on scraping Latin American real estate and commerce sites.
Scheduling and Change Detection
Coupon inventories turn over fast. A daily full crawl is baseline; for competitive affiliate monitoring you want change detection running every 2-4 hours on high-traffic merchants.
A simple change-detection loop:
- Hash each
(merchant_id, coupon_code, affiliate_id)tuple on each crawl - Compare against the previous snapshot stored in Postgres or Redis
- On hash mismatch, flag the record and queue a redirect-chain trace
- Alert if an affiliate ID changes on a high-volume merchant (indicates partner swap or hijack)
Step 4 is the one most affiliate managers overlook. An affiliate ID change mid-campaign means commissions may be routing to a different publisher — sometimes a hijacker who’s injecting their ID upstream. Automated detection on scraped data catches this faster than any manual audit.
Bottom Line
Scraping coupon aggregators is tractable with the right stack: intercept XHR/GraphQL APIs first, use Playwright only for token extraction, trace full redirect chains with a non-JS HTTP client, and run residential or mobile proxies for anything behind bot detection. The affiliate-specific data — network, publisher ID, expiry, verification timestamp — is where the real signal lives, and it’s what separates a useful dataset from a raw code dump. DRT covers scraping infrastructure across the full retail and pricing data stack, so if you’re building out a larger data collection pipeline, dig into the rest of the site’s coverage for patterns that port directly to coupon use cases.
Related guides on dataresearchtools.com