Category: Uncategorized

Best Proxy Types for Scraping Google Maps and Local Pack (2026)

Scraping Google Maps and local pack results without the right proxy type will get you rate-limited or soft-blocked within minutes — which proxy types work best for scraping Google Maps and local pack results is one of the most common questions we get from data engineers running local SEO pipelines.

Google’s anti-bot stack for Maps is stricter than its main SERP. It ties geolocation signals, ASN reputation, and request cadence together. Residential and mobile IPs pass that check; datacenter IPs usually don’t, at least not at scale.

Why Google Maps Is Harder Than Regular SERP Scraping

Maps requests carry implicit location expectations. When you query https://www.google.com/maps/search/plumbers+near+me, Google expects the IP to be in the same metro area as the search intent. A datacenter IP in Frankfurt hitting a Chicago local pack query raises an immediate signal mismatch.

The local pack (the “3-pack” of business listings that appear above organic results) has the same problem. Because local pack data drives high-value lead generation for competitors, Google applies tighter fingerprinting here than it does for informational queries. If you’re building a maps scraper, read How to Scrape Google Local Pack Results (Maps + Business Data) 2026 first — it covers the full selector and pagination logic before you even think about proxy strategy.

Proxy Type Comparison for Maps Scraping

Here’s how the main proxy categories stack up for this specific target:

Proxy type	Success rate (est.)	Cost per GB	Best use case
Residential rotating	85-93%	$3-$8	General maps + local pack scraping
Mobile (4G/5G)	91-96%	$8-$20	High-value targets, review scraping
ISP/static residential	78-88%	$2-$5	Long sessions, business detail pages
Datacenter rotating	30-55%	$0.5-$1.5	Bulk SERP only, maps not recommended
Datacenter (premium ASN)	55-70%	$1-$3	Low-risk batch jobs at off-peak hours

Mobile proxies consistently outperform everything else on Maps. Google treats mobile IPs as real users on the move — the exact behavioral profile of someone searching for a nearby restaurant or plumber.

Residential Proxies: the Practical Default

For most teams, residential rotating proxies hit the right balance of cost and success rate. Providers like Smartproxy, Oxylabs, and Bright Data let you target by city or zip code, which matters for local pack accuracy. A Chicago query needs a Chicago IP; otherwise the local pack you receive won’t match what a real Chicago user sees.

Key settings that matter:

Session length: use sticky sessions of 3-5 minutes per business page, not per-request rotation
Geo-targeting: city-level, not just country-level — state-level is often too coarse for local pack
Concurrency: keep it under 5 concurrent requests per IP pool to avoid pattern detection

If you’re scraping Yelp or Google reviews alongside Maps, How Proxies Help Scrape Reviews at Scale: Yelp, Google, Trustpilot (2026) covers how the same residential pool can serve both targets without getting burned.

Mobile Proxies: Worth the Price for High-Stakes Jobs

Mobile IPs (real SIM cards on 4G/5G networks) carry the lowest suspicion score on Google’s network. The tradeoff is cost — $8-$20/GB versus $3-$8 for residential. For one-off enrichment jobs or when you’re scraping business data that feeds a live product, the higher success rate justifies it.

For daily monitoring jobs where you’re checking ranking changes in a local pack, mobile is often overkill. That’s a case where ISP proxies (static IPs assigned to real ISPs, not datacenters) are the better call — they’re stable enough for repeated requests to the same URLs. Do Proxies Help Daily Housing Listing Monitoring? Real-World Test ran this exact comparison on a real monitoring pipeline and the ISP proxy numbers held up well for daily cadence.

A Minimal Working Config

This is a Python snippet using requests with a residential proxy endpoint. Adjust the geo params to match your target city:

import requests

proxies = {
    "http": "http://user-city-chicago:pass@gate.smartproxy.com:10000",
    "https": "http://user-city-chicago:pass@gate.smartproxy.com:10000",
}

headers = {
    "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15",
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.google.com/maps/search/plumbers+near+me/@41.8781,-87.6298,13z"
response = requests.get(url, proxies=proxies, headers=headers, timeout=15)
print(response.status_code, len(response.text))

Using a mobile user agent alongside a residential IP improves pass-through rates — the combination looks like a real phone user on home wifi. Don’t use a Chrome desktop UA with a mobile IP or vice versa; the mismatch is a detectable signal.

For Google Shopping scraping (a different but related target with similar bot defenses), Scraping Google Shopping with sh-dgr__content Selector (2026 Guide) has a comparable setup you can adapt.

What to Avoid

A few things that cause unnecessary failures:

Using datacenter IPs for maps detail pages — the /maps/place/ URLs have heavier bot scoring than the search entry point
Rotating IP on every request during a multi-page session — Google tracks session continuity
Ignoring 429 responses — back off for 60-120 seconds minimum, not 5 seconds
Scraping without a realistic Accept-Language header — it’s a low-cost signal Google reads

The same discipline applies when scraping review platforms. Scraping Airbnb Reviews with data-review-id Selector (2026 Guide) shows how header hygiene and session management reduce block rates on Airbnb, and the same principles carry directly to Maps.

Bottom Line

For Google Maps and local pack scraping in 2026, start with residential rotating proxies at city-level geo-targeting — they cover 80% of use cases at a reasonable cost. Upgrade to mobile proxies for high-value or high-frequency jobs where a 5-10% bump in success rate matters. Avoid datacenter IPs on Maps entirely unless you’re doing low-volume testing. DRT covers proxy strategy and scraping infrastructure regularly; the pillar guide linked above goes deeper on the full Maps scraping pipeline if you’re building this from scratch.

Related guides on dataresearchtools.com

May 6, 2026

NewsAPI.org Free Tier Limits 2026: Quotas, Pricing, Alternatives

—

if you’ve hit a 426 Upgrade Required error from NewsAPI.org, you’ve run into the newsapi.org free tier limits 2026 wall — and you’re not alone. the free developer plan is one of the most-used entry points for news data pipelines, but its constraints are narrow enough to catch engineers off guard mid-project. this article breaks down exactly what you get, what you don’t, and what to do when the free plan stops being enough.

what the free developer plan actually gives you

NewsAPI.org’s free tier is officially called the Developer plan. as of 2026, the core quotas are:

100 requests per day (hard cap, resets at midnight UTC)
1 month of historical data (articles older than 30 days are not returned)
no commercial use — the license explicitly prohibits production apps
no HTTPS endpoints on the free tier (some integrations break silently)
results capped at 100 articles per request, max 1 page of results per query

the 100-requests-per-day limit sounds workable for a side project, but if you’re polling multiple topics, languages, or sources, it disappears fast. a single keyword monitor running every 15 minutes burns all 96 daily slots in one day.

for a full breakdown of what changes at each paid tier, the NewsAPI Pricing 2026: Plans, Per-Call Cost, Best Alternatives overview covers per-call economics and where the price/quota curve stops making sense.

the historical data wall is the real killer

the 30-day lookback restriction is the constraint that bites hardest in real projects. sentiment analysis, trend detection, and media monitoring pipelines almost always need at least 90 days of context. on the free plan, you’re locked to a rolling 30-day window.

paid tiers extend this:

plan	daily requests	historical data	commercial use
developer (free)	100	30 days	no
business	250,000	1 year	yes
enterprise	custom	full archive	yes

the jump from free to business is not incremental — there’s no mid-tier plan at $30/month. if you need more than 100 req/day or more than 30 days of history, you’re looking at the business plan, which starts at several hundred dollars per month depending on usage. that pricing gap is what drives most developers toward alternatives.

the NewsAPI Developer Plan 2026: Pricing, Features, Limits Explained article goes deeper on what “commercial use” means in practice and how the upgrade path is structured.

common errors and what they mean

when you hit quota walls or misconfigure your request, NewsAPI returns structured JSON errors. the ones you’ll see most often:

{
  "status": "error",
  "code": "rateLimited",
  "message": "You have made too many requests recently."
}

error codes to know:

rateLimited — you’ve exceeded requests per day or per second (free plan has a 1 req/sec burst limit too)
maximumResultsReached — you’re trying to paginate past page 1 on the free plan
sourcesTooMany — free plan limits source filtering to 20 sources per query
parameterInvalid — often triggered when passing from dates older than 30 days on the free plan
apiKeyDisabled — account suspended, usually for ToS violations (commercial use on a free key)

the parameterInvalid error on date ranges is particularly frustrating because the API doesn’t tell you the date was out of range — it just rejects the request, and developers often waste time debugging the wrong parameter.

how to stretch the free plan further

if you’re doing genuine development work (not production scraping), a few patterns help you stay under 100 req/day without restructuring your pipeline:

batch your queries — instead of polling every keyword separately, use NewsAPI’s q parameter with OR operators: q=bitcoin OR ethereum OR crypto
cache responses locally — store results in SQLite or a flat JSON file and query the cache instead of re-hitting the API for the same date range
use pageSize=100 — you get one page, so max it out to 100 articles per call

import requests

API_KEY = "your_key_here"
params = {
    "q": "bitcoin OR ethereum OR DeFi",
    "language": "en",
    "pageSize": 100,
    "apiKey": API_KEY
}
response = requests.get("https://newsapi.org/v2/everything", params=params)
data = response.json()
print(f"fetched {len(data.get('articles', []))} articles")

this approach can stretch a 100-req budget surprisingly far for narrow topics. it won’t help with the 30-day history wall, but it keeps you off the upgrade path longer for monitoring use cases.

alternatives worth considering in 2026

the honest picture: NewsAPI’s free plan is best suited for prototyping, not for anything running in CI or powering a dashboard. when you outgrow it, here are the real alternatives:

provider	free tier	historical data	notes
NewsAPI.org (paid)	n/a	1 year (business)	large jump from free to paid
GDELT Project	unlimited	full archive (2015+)	raw, noisy, requires cleaning
The Guardian API	12 req/sec, unlimited	full archive	high quality, limited sources
Mediastack	500 req/month	none on free	clean API, 7,500+ sources
Bing News Search API	1,000 req/month	limited	good freshness, Microsoft pricing
Common Crawl NEWS	unlimited (bulk)	full archive	requires S3 access + preprocessing

GDELT is the most powerful free option but requires serious data engineering — you’re pulling from BigQuery or flat files, not a clean REST endpoint. the Guardian API is underrated for English-language political and economic news. Mediastack fits the “small project” slot that NewsAPI’s free plan used to own, but with better terms for lightweight commercial use.

bottom line

the newsapi.org free tier is genuinely useful for rapid prototyping and student projects, but the 100 req/day cap and 30-day history window make it unsuitable for anything running in production or requiring trend context. if you need more than one month of lookback or plan to use the data commercially, budget for the business plan or switch to GDELT or the Guardian API before you build your pipeline around a limit you’ll hit in week two. DRT covers this category of news data tooling in depth — bookmark the site if you’re evaluating providers before committing to a stack.

Related guides on dataresearchtools.com

May 6, 2026

Scraping Airbnb Reviews with data-review-id Selector (2026 Guide)

Airbnb’s review section is one of the richest datasets in short-term rental research, but it sits behind a JavaScript-rendered page that blocks naive scrapers within minutes. The key to reliable extraction in 2026 is the data-review-id attribute, a stable HTML hook that Airbnb uses to identify each review card regardless of CSS class churn.

Why data-review-id Is the Right Selector

Airbnb’s frontend has been rebuilt several times. Class names like _1gjypya rotate with deploys, but the data-review-id attribute is tied to the underlying data model and has stayed consistent through multiple redesigns. Selecting on [data-review-id] anchors your parser to structure, not style.

Each review card looks roughly like this in the DOM:

<div data-review-id="1102847563982741504">
  <span data-testid="review-author">María G.</span>
  <span data-testid="review-date">April 2026</span>
  <div data-testid="review-body">
    Absolutely loved the place. Clean, quiet, and ...
  </div>
</div>

Your selector chain in Python with Playwright or Puppeteer:

reviews = page.query_selector_all('[data-review-id]')
for r in reviews:
    review_id = r.get_attribute('data-review-id')
    author = r.query_selector('[data-testid="review-author"]').inner_text()
    body = r.query_selector('[data-testid="review-body"]').inner_text()
    print(review_id, author, body)

The data-review-id value is the canonical review identifier you can use for deduplication and delta updates. Store it as a primary key from day one.

Rendering and Pagination Challenges

Airbnb loads reviews via GraphQL calls, and the review section does not appear in raw HTML responses. You need a headless browser or a tool that replays the underlying API. There are two practical approaches in 2026:

Headless browser (Playwright/Puppeteer): Accurate but slow. One listing with 50 reviews takes 8-15 seconds to fully render at 4G-equivalent bandwidth. Pagination requires clicking the “Show more reviews” button or intercepting the GraphQL call and replaying it with incremented cursors.

GraphQL endpoint replay: Faster and more scalable. Use browser devtools to capture the PdpReviews query, then replay it directly with requests or httpx. Paginate by incrementing the offset variable. This cuts render time to under 1 second per page but requires session cookies and is more brittle to schema changes.

For large-scale collection across thousands of listings, endpoint replay wins on cost and speed. For small-scale or one-off pulls, Playwright with [data-review-id] is simpler to maintain.

Proxy Setup and Anti-Bot Avoidance

Airbnb runs Akamai Bot Manager and applies rate limits aggressively by IP. Without proxies, you will get 403s or silent rate limiting after 10-20 requests from a single residential IP.

Residential proxies with sticky sessions are the standard choice here. Datacenter IPs are flagged immediately on Airbnb; mobile proxies work but are expensive for high-volume jobs. For a comparison of IP types across review-heavy targets, the guide on How Proxies Help Scrape Reviews at Scale: Yelp, Google, Trustpilot (2026) covers the tradeoffs in detail.

proxy type	block rate (Airbnb)	cost per GB	best for
datacenter	very high	$0.50-1	not recommended
residential rotating	low	$3-8	bulk listing sweeps
residential sticky	very low	$5-10	session-bound scraping
mobile (4G/5G)	minimal	$15-30	high-value targets only

Sticky sessions matter here because Airbnb uses cookie-based session fingerprinting. Rotating your IP mid-session resets the fingerprint and triggers re-verification. Keep one IP for the full duration of a listing scrape, then rotate to a new one for the next listing.

The proxy rotation logic that works for Airbnb reviews is similar to what you would use for local business data. The article on Scraping Google Maps Data with Proxies: Business Listings and Reviews (2026) covers the session management pattern in depth and is a useful reference if you are building a unified review pipeline across platforms.

Rate Limiting and Request Pacing

Airbnb’s rate limits are not published, but empirical testing in early 2026 suggests:

Safe pace: 1 listing per 4-8 seconds per session
Soft limit trigger: ~50 requests per hour from one IP
Hard block: typically at 80-120 requests per hour

Practical pacing rules:

Randomize delay between requests using a uniform distribution (e.g., random.uniform(3, 7) seconds).
Use one sticky proxy session per listing, not per request.
Rotate user-agent strings from a realistic browser pool.
Respect HTTP 429 responses by backing off for 60-120 seconds before retrying.
Cap concurrent sessions at 5-10 to avoid subnet-level detection.

If you are monitoring listing availability in addition to reviews, the proxy discipline is the same. The writeup on Do Proxies Help Daily Housing Listing Monitoring? Real-World Test has real latency and block-rate numbers from a sustained housing data pull that applies directly here.

Parsing and Storing Review Data

Once you are reliably pulling review cards, structure the output around the data-review-id as your canonical identifier. A minimal schema:

{
  "review_id": "1102847563982741504",  # from data-review-id
  "listing_id": "12345678",
  "author": "María G.",
  "date": "2026-04",
  "rating": 5,
  "body": "Absolutely loved the place...",
  "language": "en",
  "scraped_at": "2026-05-06T14:22:00Z"
}

A few extraction notes:

Ratings are rendered as SVG stars, not a numeric attribute. Count filled star elements or capture the aria-label text (e.g., “5 out of 5 stars”).
Review dates are relative strings like “3 weeks ago” on first load. If you need exact dates, the GraphQL response includes ISO timestamps; prefer the API replay approach for time-sensitive datasets.
Language detection matters if you are doing sentiment analysis. langdetect or lingua handles this well in a post-processing step.

The same selector-anchoring strategy used here generalises across structured retail data. Scraping Google Shopping with sh-dgr__content Selector (2026 Guide) applies the same data-attribute anchoring pattern to product grids, and the two approaches can share the same proxy and session management layer.

For geographic spread in your dataset, residential proxies in the same country as the listing produce the lowest block rates. Airbnb serves localized content and applies stricter bot checks to cross-border traffic. If you are collecting UK listings from a US IP, expect more verification friction. The detailed breakdown of IP type by geography in Best Proxy Types for Scraping Google Maps and Local Pack (2026) maps out which proxy types win by region.

Bottom Line

Use [data-review-id] as your stable anchor, sticky residential proxies for session continuity, and GraphQL replay for anything over a few hundred listings. Playwright with data-attribute selectors is the right choice for smaller jobs where maintainability matters more than throughput. DRT covers this category of scrape-target infrastructure regularly, including selector stability, proxy pairing, and anti-bot patterns across review platforms.

Related guides on dataresearchtools.com

May 6, 2026

Amazon Seller Account Isolation 2026: Which Browser Tool Is Safest

Amazon seller account isolation is the single most common reason multi-account operators get suspended, and in 2026 it comes down almost entirely to which browser environment tool you choose. Amazon’s bot detection stack has matured significantly: it now fingerprints canvas noise, WebGL renderer strings, AudioContext output, font enumeration, and navigator attributes all in a single page load. One sloppy fingerprint across two seller accounts is enough for a linked-account flag.

Why Standard Browsers Fail for Multi-Account Management

Chrome profiles and Firefox containers share the underlying browser binary, which means their hardware fingerprints leak through shared GPU signatures and system fonts. Amazon’s ThreatMetrix integration (rebranded under LexisNexis Risk Solutions) cross-references these signals against behavioral biometrics: scroll velocity, click timing, and typing cadence.

If you’re running more than one seller account from the same machine without proper isolation, the risk isn’t theoretical. For context on how platform-level bot detection works at this depth, the breakdown in Facebook Ads Manager Bot Detection: Bypass Tactics for Automation (2026) applies almost identically to Amazon Seller Central — both platforms use fingerprint graph correlation to link accounts, not just IP matching.

The Four Anti-Detect Browsers Worth Using in 2026

The market has consolidated. Four tools dominate serious Amazon sellers: Multilogin X, AdsPower, Dolphin Anty, and Incogniton. Here’s how they compare on the dimensions that matter:

Tool	Fingerprint Engine	Profile Isolation	Team Seats	Price/mo (starter)	Amazon-specific notes
Multilogin X	Mimic (Chromium) + Stealthfox (FF)	Full OS-level	Yes	~$99	Most mature; passes most fingerprint tests
AdsPower	SunBrowser + FlowerBrowser	Full	Yes	$9	Cheaper; weaker WebGL spoofing
Dolphin Anty	Chromium-based	Full	Yes	$89	Good for bulk; API-first
Incogniton	Chromium-based	Full	Limited	$29	Best budget option; limited API

Multilogin X is the default recommendation for high-stakes Amazon accounts. Its Mimic engine randomizes canvas hash, WebGL vendor/renderer, and AudioContext fingerprint per profile independently — not just at session start, but on each render call. AdsPower works at lower account counts but its WebGL spoofing has known gaps that ThreatMetrix flags on certain GPU profiles.

Proxy Pairing: This Is Where Most Operators Get it Wrong

An anti-detect browser profile means nothing if two profiles share an IP. The rule is simple: one residential or mobile IP per seller account, never recycled between profiles in the same 30-day window.

For Amazon specifically, mobile proxies (4G/LTE) outperform residential proxies because the IP rotation pattern mimics real consumer behavior. A static residential IP that never changes looks more suspicious to Amazon than a mobile IP that rotates every session. For a detailed breakdown of proxy-to-account ratios, How Many Proxies Do You Need for Multi-Account Management (2026) covers the math in depth.

The same IP hygiene logic applies across platforms. Best OnlyFans Proxies 2026: Residential, Mobile, and Account Safety shows how platform detection models penalize shared IPs even when fingerprints are clean — the principle is identical for Seller Central.

Recommended proxy config per Multilogin X profile:

{
  "proxy_type": "socks5",
  "proxy_host": "gate.provider.io",
  "proxy_port": 10000,
  "proxy_login": "user-session-ACC001",
  "proxy_password": "pass",
  "rotate_on_startup": false,
  "sticky_session": true
}

Set sticky_session: true. Amazon’s login flow makes multiple requests during auth, and if your proxy rotates mid-session the IP change triggers a re-verification challenge.

Fingerprint Testing Before You Go Live

Before logging into any seller account from a new profile, run it through these checks:

Open browserleaks.com from the profile and verify canvas hash is unique across profiles
Check WebGL renderer — it should NOT show your real GPU (e.g., “NVIDIA GeForce RTX 4090” is a red flag)
Run coveryourtracks.eff.org and confirm the profile is not uniquely identifiable
Verify timezone matches the proxy’s geolocation (a US proxy with Asia/Shanghai timezone is a hard fail)
Check navigator.language and navigator.languages match the proxy country

Common fingerprint mismatches that cause Amazon flags:

Screen resolution set to 1920×1080 but devicePixelRatio of 2.0 (contradicts non-retina display claim)
Font list containing system fonts from a different OS than the spoofed user agent
WebRTC leaking the real local IP when STUN is enabled

Disable WebRTC in every anti-detect profile. Multilogin X does this by default; in AdsPower you must set it manually under “Advanced” in profile settings.

Account Warm-Up Protocol

New seller accounts need behavioral warm-up before listing products or running ads. Amazon’s risk model scores account age against activity velocity.

Follow this sequence:

Days 1-3: Log in once daily, browse the Seller Central dashboard, update one setting
Days 4-7: Add bank account and address, browse catalog pages
Days 8-14: List one product, respond to any verification prompts
Day 15+: Begin normal operations

Do not skip steps or compress the timeline. Accounts that jump straight to high-volume listing activity from day one get flagged at a disproportionate rate, regardless of fingerprint quality. The same warm-up discipline applies to proxy account safety, as covered in OnlyFans Proxy Guide 2026: Setup, Risks, and Provider Picks for a different platform but with identical behavioral logic.

For a complete end-to-end workflow covering browser configuration, proxy assignment, and account structure for Amazon specifically, the Anti-Detect Browser for Amazon Selling: Multi-Account Guide 2026 pillar covers every layer in detail.

Bottom Line

For Amazon seller account isolation in 2026, use Multilogin X paired with dedicated mobile proxies — one IP per account, sticky sessions, WebRTC disabled, and timezone matching proxy geolocation. AdsPower works at low scale if budget is the constraint, but do not cut corners on proxy quality regardless of which tool you use. DRT continues to track browser tool updates and proxy provider changes as Amazon’s detection stack evolves, so check back when major browser engine versions ship.

Related guides on dataresearchtools.com

May 6, 2026

Scraping Google Shopping with sh-dgr__content Selector (2026 Guide)

Google Shopping’s sh-dgr__content selector is the anchor point for every product card in the results grid, and if you’re building a price monitor in 2026, it’s the first CSS class you need to understand. Google has shuffled its Shopping HTML structure several times over the past two years, but this class has remained stable enough to be a reliable extraction target — as long as you know which child selectors to reach for and when to rotate your IPs.

What `sh-dgr__content` Actually Is

Each product tile in a Google Shopping results page sits inside a div.sh-dgr__content wrapper. Within that wrapper, the child class you’ll spend most of your time with is a8pemb, which Google uses for the clickable product link and title anchor. The combination of sh-dgr__content and a8pemb gives you a reliable two-step selector that survives most minor DOM tweaks.

The HTML structure, simplified, looks like this:

<div class="sh-dgr__content">
  <a class="a8pemb" href="/shopping/product/...">
    <h4 class="translate-content">Blue Mechanical Keyboard</h4>
  </a>
  <div class="a8Pemb-price">$49.99</div>
  <span class="E5ocAb">4.3 ★ (212)</span>
</div>

Note the case sensitivity: a8pemb on the anchor and a8Pemb-price (capital P) on the price container. Mixing these up is the single most common reason scrapers return empty price fields.

Extracting Products with Python and BeautifulSoup

For a straightforward batch scrape, BeautifulSoup handles the parsing cleanly. Playwright or Puppeteer are better choices when Google serves a JS-rendered grid, but for cached SERP HTML fetched via a proxy API, this is enough:

from bs4 import BeautifulSoup

def parse_shopping_cards(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    results = []
    for card in soup.select("div.sh-dgr__content"):
        title_el = card.select_one("a.a8pemb h4")
        price_el = card.select_one("div.a8Pemb-price")
        rating_el = card.select_one("span.E5ocAb")
        results.append({
            "title": title_el.get_text(strip=True) if title_el else None,
            "price": price_el.get_text(strip=True) if price_el else None,
            "rating": rating_el.get_text(strip=True) if rating_el else None,
            "link": card.select_one("a.a8pemb")["href"] if card.select_one("a.a8pemb") else None,
        })
    return results

Run this against a live fetch and you’ll typically get 20 to 30 product records per page. If len(results) == 0, you hit a CAPTCHA wall or a bot-detection interstitial — not a selector miss. Check the raw HTML first before blaming the parser.

For broader context on selector-based scraping across Google properties, the full breakdown in How to Scrape Google Shopping Results for Price Monitoring covers pagination, URL parameter control, and currency normalization in depth.

Handling Bot Detection and CAPTCHAs

Google Shopping is one of the harder Google surfaces to scrape at volume. It uses a layered detection stack: user-agent fingerprinting, TLS fingerprint checks, behavioral scoring, and IP reputation. Residential rotating proxies are non-negotiable above roughly 500 requests per day. Datacenter IPs get flagged within minutes on Shopping — Google appears to be more aggressive here than on web search.

The same infrastructure logic applies when scraping other Google surfaces. Best Proxy Types for Scraping Google Maps and Local Pack (2026) walks through the proxy tier tradeoffs in detail, and the conclusions carry over directly to Shopping.

Recommended proxy and rendering combinations by volume:

Daily Request Volume	Proxy Type	Rendering
Under 200	Shared datacenter	requests + lxml
200 – 2,000	Residential rotating	requests + lxml
2,000 – 20,000	Residential rotating (sticky)	Playwright headless
20,000+	ISP proxies or mobile	Playwright + stealth plugin

At the 20k+ tier, also add request delays with jitter (1.5 to 4 seconds between requests per proxy thread) and rotate Accept-Language headers to match your target geo.

Structuring a Price Monitoring Pipeline

For ongoing monitoring rather than a one-shot scrape, you need a schedule, a delta detector, and a storage layer. Here’s the minimal pipeline shape that holds up in production:

Fetch layer — Playwright headless with a residential proxy pool. Rotate IPs per request, not per session.
Parse layer — BeautifulSoup on the raw HTML using the sh-dgr__content / a8pemb selector pair above.
Storage layer — Postgres or BigQuery. Store raw HTML alongside parsed fields so you can re-parse when Google changes the DOM.
Delta detection — Compare current price to previous snapshot. Alert on changes over a configurable threshold (e.g., ±5%).
Retry layer — On CAPTCHA or empty parse, backoff and retry from a different IP. Log failure reason, not just failure count.

Storing raw HTML is the step most people skip and later regret. DOM changes are inevitable, and having the source lets you backfill without re-fetching.

The same pipeline logic — storing raw HTML, delta detection, retry handling — applies outside Shopping. Do Proxies Help Daily Housing Listing Monitoring? Real-World Test documents what breaks in production when you skip these layers on a high-frequency scrape, and the failure modes are nearly identical.

Common Errors and What They Mean

sh-dgr__content returns 0 results: you have a CAPTCHA page, a “did you mean” redirect, or a consent interstitial. Print soup.title.text to confirm.
a8pemb link exists but href is relative (starts with /shopping/): normal. Prepend https://www.google.com before storing.
Price field is None for some cards: some listings are price-range or “check site” placements. These have a different price container class. Don’t error out — just log as null.
Title returns garbled text: Google wraps titles in a translate-content class that can include hidden spans for translation fallback. Use .get_text(strip=True) and strip non-printable characters.

For comparison, structured data selectors on other review platforms behave differently but the error pattern taxonomy is similar — Scraping Airbnb Reviews with data-review-id Selector (2026 Guide) covers the same “selector returns empty, why?” debugging workflow applied to a different target.

If you’re scaling to review aggregation across multiple platforms alongside Shopping data, How Proxies Help Scrape Reviews at Scale: Yelp, Google, Trustpilot (2026) has the proxy pool sizing math worth reading before you provision infrastructure.

Bottom Line

Target div.sh-dgr__content as your container and a.a8pemb as your product link selector — that combination is the most stable extraction point on Google Shopping in 2026. Use residential rotating proxies from the start, store raw HTML alongside parsed fields, and build retry logic that distinguishes between a selector miss and a CAPTCHA wall. DRT will keep tracking selector stability as Google rolls out Shopping UI updates through the year.

Related guides on dataresearchtools.com

May 6, 2026

Best News APIs Compared: 12 Options for Developers in 2026

whether you are building a news aggregator, monitoring brand mentions, feeding data into an AI model, or tracking industry trends, a news API saves you from scraping hundreds of news sites yourself. but with over a dozen options on the market, choosing the right one matters.

looking for premium 4G/5G IPs? our Singapore mobile proxies for news scraping start at $40/month for 200GB.

this comparison covers the 12 most popular news APIs available in 2026. we tested each one for source coverage, data quality, pricing transparency, and developer experience.

Quick Comparison Table

API	Free Tier	Starting Price	Sources	Historical Data	Best For
NewsAPI.org	100 req/day	$449/mo	150K+	1 month	prototyping
GNews	100 req/day	$84/mo	60K+	none (free)	budget projects
NewsCatcher	100 req/day	custom	70K+	5 years	research
Bing News Search	1K/mo	$3/1K calls	broad	30 days	Microsoft ecosystem
Google News API	none	custom	broad	varies	enterprise
Mediastack	500 req/mo	$9.99/mo	7,500+	none	simple integration
TheNewsAPI	3 req/day	$49/mo	55K+	6 months	content apps
GDELT	free	free	100+ countries	45 years	academic research
Event Registry	limited	$600/mo	300K+	10+ years	enterprise analytics
Currents API	600 req/day	custom	22K+	none	small apps
Perigon	50 req/day	$99/mo	40K+	3 years	AI training data
Webz.io (formerly Webhose)	limited	custom	2M+	10+ years	massive scale

1. NewsAPI.org

best for: quick prototyping and hobby projects

NewsAPI.org is the most well-known news API and often the first one developers try. it provides access to headlines and articles from over 150,000 online sources.

strengths: – simple REST API with excellent documentation – fast response times under 200ms – supports searching by keyword, source, language, and country – provides article metadata including author, published date, and image URL

weaknesses: – free tier is extremely limited (100 requests/day, no commercial use) – paid plans start at $449/month, which is steep for small projects – historical data limited to 1 month on most plans – no full article text on free tier (only titles and descriptions)

example request:

import requests

API_KEY = "your_newsapi_key"

response = requests.get(
    "https://newsapi.org/v2/everything",
    params={
        "q": "web scraping",
        "language": "en",
        "sortBy": "publishedAt",
        "pageSize": 10,
        "apiKey": API_KEY,
    },
)

data = response.json()
for article in data["articles"]:
    print(f"{article['title']}")
    print(f"  source: {article['source']['name']}")
    print(f"  published: {article['publishedAt']}")
    print()

pricing: free tier (100 req/day dev only), Business $449/mo, Enterprise custom

2. GNews

best for: budget-conscious developers

GNews offers a clean, straightforward API at a fraction of the cost of NewsAPI.org. it pulls from Google News and provides access to over 60,000 sources.

strengths: – cheapest paid option starting at $84/month – free tier is generous for testing (100 requests/day) – simple query syntax – multilingual support (38 languages)

weaknesses: – smaller source pool than some competitors – no historical archive on free tier – limited filtering options compared to premium APIs – rate limiting can be aggressive on lower tiers

example request:

response = requests.get(
    "https://gnews.io/api/v4/search",
    params={
        "q": "proxy industry",
        "lang": "en",
        "max": 10,
        "token": "your_gnews_key",
    },
)

for article in response.json()["articles"]:
    print(f"{article['title']} - {article['source']['name']}")

pricing: free (100 req/day), Basic $84/mo, Pro $279/mo

3. NewsCatcher

best for: research and analytics

NewsCatcher differentiates itself with deep search capabilities and long historical archives. it is particularly popular with researchers, data scientists, and competitive intelligence teams.

strengths: – 5-year historical archive – NLP-powered features like topic classification, entity extraction, and sentiment analysis – 70,000+ sources with strong international coverage – clustering of related articles

weaknesses: – no published pricing (sales-driven) – free tier is limited to 100 requests per day – response times can be slower than simpler APIs due to NLP processing – steeper learning curve for advanced features

example request:

headers = {"x-api-key": "your_newscatcher_key"}

response = requests.get(
    "https://v3-api.newscatcherapi.com/api/search",
    headers=headers,
    params={
        "q": "artificial intelligence data collection",
        "lang": "en",
        "from_": "2025-01-01",
        "to_": "2026-03-01",
        "page_size": 10,
    },
)

data = response.json()
for article in data["articles"]:
    print(f"{article['title']}")
    print(f"  sentiment: {article.get('sentiment', 'N/A')}")
    print(f"  topic: {article.get('topic', 'N/A')}")

pricing: free tier available, paid plans require contacting sales

4. Bing News Search API

best for: Microsoft ecosystem integration

part of Microsoft’s Azure Cognitive Services, the Bing News Search API provides access to Bing’s news index with enterprise-grade reliability.

strengths: – backed by Microsoft’s infrastructure – excellent for trending topics and breaking news – supports category-based browsing – integrates well with other Azure services

weaknesses: – results are biased toward English and US sources – requires an Azure subscription – limited historical data (30 days) – pricing based on transactions makes costs unpredictable at scale

example request:

headers = {"Ocp-Apim-Subscription-Key": "your_bing_key"}

response = requests.get(
    "https://api.bing.microsoft.com/v7.0/news/search",
    headers=headers,
    params={
        "q": "data privacy regulation",
        "count": 10,
        "mkt": "en-US",
        "freshness": "Week",
    },
)

for article in response.json()["value"]:
    print(f"{article['name']}")
    print(f"  provider: {article['provider'][0]['name']}")

pricing: free (1K transactions/mo), S1 $3/1K transactions

5. Mediastack

best for: simple integration without complexity

Mediastack provides a no-frills news API that is easy to set up and affordable. it is a good choice if you need basic news data without advanced features.

strengths: – starts at $9.99/month, making it one of the cheapest options – live news data from 7,500+ sources in 50+ countries – supports 13 languages – straightforward REST API

weaknesses: – no NLP features (sentiment, entity extraction) – limited source pool compared to premium APIs – no historical archive on lower tiers – HTTPS only available on paid plans

example request:

response = requests.get(
    "http://api.mediastack.com/v1/news",
    params={
        "access_key": "your_mediastack_key",
        "keywords": "web scraping",
        "languages": "en",
        "limit": 10,
    },
)

for article in response.json()["data"]:
    print(f"{article['title']} ({article['source']})")

pricing: free (500 req/mo), Basic $9.99/mo, Standard $49.99/mo

6. TheNewsAPI

best for: content applications

TheNewsAPI focuses on providing clean, well-structured article data suitable for content applications. it deduplicates content and provides good categorization.

strengths: – strong deduplication removes repeat stories – category-based browsing – sentiment analysis included – 6-month historical archive

weaknesses: – very limited free tier (3 requests/day) – paid plans start at $49/month – source pool is smaller than top-tier options

pricing: free (3 req/day), Basic $49/mo, Pro $149/mo

7. GDELT

best for: academic research and global event monitoring

GDELT (Global Database of Events, Language, and Tone) is a free, open dataset that monitors news from every country. it is not a traditional API but a massive data platform.

strengths: – completely free – covers 100+ countries and dozens of languages – historical data going back to 1979 – real-time monitoring with 15-minute update cycles – includes geolocation, sentiment, themes, and entity data

weaknesses: – steep learning curve – raw data requires significant processing – API rate limits can be restrictive – documentation is scattered and sometimes outdated

example request:

# GDELT DOC API
response = requests.get(
    "https://api.gdeltproject.org/api/v2/doc/doc",
    params={
        "query": "proxy server",
        "mode": "ArtList",
        "maxrecords": 10,
        "format": "json",
    },
)

for article in response.json().get("articles", []):
    print(f"{article['title']}")
    print(f"  tone: {article.get('tone', 'N/A')}")
    print(f"  domain: {article['domain']}")

pricing: free

8. Event Registry

best for: enterprise-grade news analytics

Event Registry aggregates news from over 300,000 sources and clusters articles into events. it is designed for enterprise analytics use cases.

strengths: – largest source pool (300K+ sources) – event-based clustering groups related articles – 10+ years of historical data – advanced analytics including topic trends, entity tracking, and media monitoring

weaknesses: – expensive (starting at $600/month) – complex API with many parameters – overkill for simple news integration

pricing: starts at $600/mo

9. Currents API

best for: small applications and side projects

Currents API provides a simple, generous free tier that works well for small applications.

strengths: – generous free tier (600 requests/day) – 22,000+ sources – simple API design – no credit card required for free tier

weaknesses: – limited documentation – no historical archive – fewer features than premium APIs – data quality can be inconsistent

pricing: free (600 req/day), paid plans available

10. Perigon

best for: AI and ML training data

Perigon is designed specifically for feeding news data into AI models. it provides clean, structured data with rich metadata.

strengths: – designed for AI/ML pipelines – 3-year historical archive – entity extraction and topic classification built in – content clustering and deduplication – structured JSON output optimized for data processing

weaknesses: – relatively new entrant – smaller source pool than the largest competitors – free tier limited to 50 requests/day

example request:

headers = {"x-api-key": "your_perigon_key"}

response = requests.get(
    "https://api.goperigon.com/v1/all",
    headers=headers,
    params={
        "q": "machine learning proxy",
        "from": "2025-06-01",
        "size": 10,
        "showReprints": "false",
    },
)

for article in response.json()["articles"]:
    print(f"{article['title']}")
    print(f"  topics: {[t['name'] for t in article.get('topics', [])]}")

pricing: free (50 req/day), Starter $99/mo, Growth $499/mo

11. Webz.io

best for: massive scale data collection

Webz.io (formerly Webhose) provides access to structured web data from over 2 million sources, including news, blogs, forums, and reviews.

strengths: – enormous source pool (2M+ sources) – 10+ years of historical data – covers news, blogs, forums, and dark web – high throughput for bulk data extraction – real-time streaming option

weaknesses: – expensive (enterprise pricing) – complex pricing model – overkill for simple news needs

pricing: custom enterprise pricing

12. Google News API (Custom Search)

best for: Google News results in your application

Google does not offer a dedicated news API, but you can use the Custom Search JSON API configured for news to get Google News results programmatically.

strengths: – Google’s news ranking quality – broad source coverage – works with Google Cloud billing

weaknesses: – limited to 100 queries/day on free tier – $5 per 1K queries after that – not a true news API (it is a search API) – limited metadata compared to dedicated news APIs

pricing: free (100 queries/day), $5/1K queries

How to Choose

By Budget

free: GDELT, Currents API
under $50/mo: Mediastack, TheNewsAPI
$50-200/mo: GNews, Perigon
$200-500/mo: NewsAPI.org
$500+: Event Registry, NewsCatcher, Webz.io

By Use Case

news aggregator app: NewsAPI.org or GNews
brand monitoring: NewsCatcher or Event Registry
AI training data: Perigon or Webz.io
academic research: GDELT (free and deep historical data)
side project: Currents API or Mediastack
enterprise analytics: Event Registry or Webz.io

By Technical Requirements

best documentation: NewsAPI.org
best historical data: GDELT or Event Registry
best NLP features: NewsCatcher or Perigon
best free tier: Currents API or GDELT
fastest response time: NewsAPI.org or Bing News

Building a News Pipeline with Proxies

if none of these APIs fully meet your needs, you can build your own news collection pipeline. use proxies to scrape RSS feeds and news sites directly:

import feedparser
import requests
from datetime import datetime


def collect_news_from_rss(feeds, proxy_url=None):
    """collect news from RSS feeds with proxy support."""

    proxies = {}
    if proxy_url:
        proxies = {"http": proxy_url, "https": proxy_url}

    articles = []

    for feed_url in feeds:
        try:
            response = requests.get(
                feed_url, proxies=proxies, timeout=15
            )
            feed = feedparser.parse(response.content)

            for entry in feed.entries:
                articles.append({
                    "title": entry.get("title", ""),
                    "url": entry.get("link", ""),
                    "published": entry.get("published", ""),
                    "summary": entry.get("summary", ""),
                    "source": feed.feed.get("title", feed_url),
                })
        except Exception as e:
            print(f"error fetching {feed_url}: {e}")

    return articles


# example: collect from major tech news RSS feeds
tech_feeds = [
    "https://techcrunch.com/feed/",
    "https://feeds.arstechnica.com/arstechnica/index",
    "https://www.theverge.com/rss/index.xml",
    "https://feeds.feedburner.com/venturebeat/SZYF",
]

articles = collect_news_from_rss(
    tech_feeds,
    proxy_url="http://user:pass@proxy.provider.com:8080",
)

Conclusion

the best news API depends on your specific requirements. for most developers starting out, GNews or Currents API offer the best value with reasonable free tiers. for enterprise use cases that need deep historical data and NLP features, NewsCatcher or Event Registry are worth the investment. for AI and ML applications, Perigon is purpose-built for that workflow.

if your needs go beyond what any single API offers, consider combining a news API for broad coverage with targeted RSS scraping through proxies for specific sources that matter most to your use case.

April 4, 2026

Category: Uncategorized

Best Proxy Types for Scraping Google Maps and Local Pack (2026)

Why Google Maps Is Harder Than Regular SERP Scraping

Proxy Type Comparison for Maps Scraping

Residential Proxies: the Practical Default

Mobile Proxies: Worth the Price for High-Stakes Jobs

A Minimal Working Config

What to Avoid

Bottom Line

Related guides on dataresearchtools.com

NewsAPI.org Free Tier Limits 2026: Quotas, Pricing, Alternatives

what the free developer plan actually gives you

the historical data wall is the real killer

common errors and what they mean

how to stretch the free plan further

alternatives worth considering in 2026

bottom line

Related guides on dataresearchtools.com

Scraping Airbnb Reviews with data-review-id Selector (2026 Guide)

Why data-review-id Is the Right Selector

Rendering and Pagination Challenges

Proxy Setup and Anti-Bot Avoidance

Rate Limiting and Request Pacing

Parsing and Storing Review Data

Bottom Line

Related guides on dataresearchtools.com

Amazon Seller Account Isolation 2026: Which Browser Tool Is Safest

Why Standard Browsers Fail for Multi-Account Management

The Four Anti-Detect Browsers Worth Using in 2026

Proxy Pairing: This Is Where Most Operators Get it Wrong

Fingerprint Testing Before You Go Live

Account Warm-Up Protocol

Bottom Line

Related guides on dataresearchtools.com

Scraping Google Shopping with sh-dgr__content Selector (2026 Guide)

What sh-dgr__content Actually Is

Extracting Products with Python and BeautifulSoup

Handling Bot Detection and CAPTCHAs

Structuring a Price Monitoring Pipeline

Common Errors and What They Mean

Bottom Line

Related guides on dataresearchtools.com

Best News APIs Compared: 12 Options for Developers in 2026

Best News APIs Compared: 12 Options for Developers in 2026

Quick Comparison Table

1. NewsAPI.org

2. GNews

3. NewsCatcher

4. Bing News Search API

5. Mediastack

6. TheNewsAPI

7. GDELT

8. Event Registry

9. Currents API

10. Perigon

11. Webz.io

12. Google News API (Custom Search)

How to Choose

By Budget

By Use Case

By Technical Requirements

Building a News Pipeline with Proxies

Conclusion

What `sh-dgr__content` Actually Is