news data feeds dozens of use cases: sentiment analysis, event detection, content aggregation, financial signal extraction, and competitive monitoring. the right API depends on volume requirements, geo coverage, and budget.

1. newsapi.org

the most developer-friendly entry point. free tier covers 100 requests/day. paid plans start at $449/month. covers 80,000+ sources in 50 languages.

import requests
API_KEY = 'your_newsapi_key'
params = {'q': 'web scraping', 'language': 'en', 'sortBy': 'publishedAt', 'apiKey': API_KEY}
r = requests.get('https://newsapi.org/v2/everything', params=params)
for a in r.json().get('articles', []):
    print(a['title'], '|', a['publishedAt'])

2. newscatcher api

a news data API focused on clean, enriched, ready-to-use news data. the Newscatcher API automatically clusters similar articles to cut duplicate noise, then layers entity resolution and industry-standard tagging on top of advanced source, country, language, and topic filtering — a strong fit for media monitoring, sentiment analysis, and financial signal extraction.

3. the guardian api

completely free for non-commercial use with 500 requests/day. covers all Guardian content back to 1999.

params = {'q': 'artificial intelligence', 'api-key': 'your_key', 'show-fields': 'body', 'page-size': 10}
r = requests.get('https://content.guardianapis.com/search', params=params)
for item in r.json()['response']['results']:
    print(item['webTitle'], '|', item['webPublicationDate'])

4. gdelt project

GDELT is a free, massive database of news events updated every 15 minutes. covers 100+ languages from 1979 to present. free to query via BigQuery. use BigQuery’s SQL interface to filter by actor, event type, country, and date range.

5. mediastack

affordable pricing starting at $9.99/month for 10,000 requests. covers 7,500+ sources in 50+ countries. good for startups needing more volume than free tiers allow.

6. currents api

free tier includes 250 requests/day. sources span 168 countries in 20+ languages. useful for independent developers building news apps on zero budget.

when to scrape instead of using an api

APIs have limitations: they do not cover every site, they impose rate limits, and they charge per request at scale. direct scraping via RSS is often cheaper and faster for specific sources.

import feedparser
feed = feedparser.parse('https://feeds.bbci.co.uk/news/technology/rss.xml')
for entry in feed.entries[:5]:
    print(entry.title, '|', entry.link)

RSS feeds are structured, machine-readable, and updated frequently. most major publishers still maintain them. see our guide on what is web scraping for broader context.

summary comparison

for scraping at volume, route requests through a rotating proxy. see our comparison of SOCKS5 vs HTTP proxy and what is a proxy server.

sources and further reading

• NewsAPI documentation
• The Guardian open platform docs
• GDELT dataset documentation

related guides

• what is web scraping
• SOCKS5 vs HTTP proxy
• what is a proxy server

why use a data marketplace

scraping everything yourself costs time, infrastructure, and anti-bot bypass work. for common datasets (company firmographics, consumer demographics, financial data), buying from a marketplace is often cheaper than building the pipeline. on the sell side, marketplaces give scrapers a distribution channel without building a customer acquisition machine.

commercial data marketplaces

Snowflake Data Marketplace

the largest B2B data marketplace by revenue. strength is the zero-copy sharing model: data stays in Snowflake, you query it directly in your own account without ETL. 2,000+ listings across financial, demographic, weather, and alternative data. requires a Snowflake account (minimum $25/month on pay-as-you-go). best for: enterprise teams already in the Snowflake ecosystem.

AWS Data Exchange

Amazon’s data marketplace, integrated into S3 and AWS services. 3,500+ datasets. strong in financial data, healthcare, and satellite imagery. delivery is S3-based; you subscribe and data lands in your bucket. pricing ranges from free to $50,000+/month for premium financial feeds. best for: teams already on AWS who need automated data delivery into their pipelines.

Databricks Marketplace

newer than Snowflake but growing fast. similar zero-copy model for Delta Lake tables. strong in ML training datasets and AI-specific data products. if you are using Databricks for ML pipelines, check here before scraping training data yourself.

Datarade

a data marketplace aggregator that lists datasets from 1,000+ providers and lets you compare them. not a data host itself. search by data type, geography, update frequency, and delivery format. strong in B2B contact data, location data, and web data.

free and open datasets

Hugging Face Datasets

the default destination for ML training data in 2025-2026. 100,000+ datasets, free to download. strongest in NLP, computer vision, and tabular ML:

from datasets import load_dataset
ds = load_dataset("wikipedia", "20220301.en", split="train[:1%]")
print(ds[0]["text"][:500])

Kaggle Datasets

130,000+ user-contributed datasets. best for: historical financial data, sports statistics, public health records, and competition datasets. API access via the kaggle CLI makes bulk downloading easy.

data.gov and equivalents

US federal datasets: 300,000+ datasets across all agencies. equivalent platforms: data.gov.uk (UK), data.gov.sg (Singapore), data.europa.eu (EU). strong in economic statistics, health data, geospatial data, and census data.

World Bank Open Data

macroeconomic and development indicators for 217 countries, 1960-present. the API is clean and well-documented. Python wrapper: wbgapi. essential for any economic research or content involving global statistics.

alternative data sources

Nasdaq Data Link (formerly Quandl)

financial and alternative data. free tier includes most economic data; premium tiers ($50-500+/month) cover equity fundamentals, options data, and alternative signals. useful for: backtesting, financial research, investment content.

Common Crawl

the largest freely available web crawl. monthly snapshots, 250-300TB of compressed data per crawl. hosted on S3 via AWS Open Data. process it with Athena, Spark, or the cdx-toolkit Python library for targeted queries. use this if you need historical web content without scraping it yourself.

selling your data

if you have built a proprietary dataset, the fastest path to revenue is Datarade (for B2B data buyers), Gumroad or Lemon Squeezy (for one-time CSV sales), or building a direct API product. niche datasets (e.g., weekly Amazon pricing data for a specific product category, daily SERP rankings for an industry) are more valuable than broad commodity datasets. see how to monetize web scraping for the full playbook.

sources and further reading

• Datarade data marketplace directory
• Hugging Face Datasets hub
• Common Crawl: open web crawl data

related guides

• how to monetize web scraping
• what is web scraping
• 25+ web scraping project ideas
• web scraping for data journalism

how to pick the right project

the best scraping projects have three things: a data source that updates regularly, a use case that justifies the infrastructure cost, and a clear output format (CSV, API, dashboard, or product). one-shot scrapes of static data are not worth engineering. aim for pipelines that run on a schedule and compound over time.

the projects below are grouped by difficulty. beginner means you can do it with requests and BeautifulSoup on a single IP. intermediate requires rotating proxies or browser automation. advanced requires custom anti-bot bypass, distributed infrastructure, or real-time processing.

beginner projects

1. job listing aggregator

scrape LinkedIn, Indeed, and Glassdoor for specific job titles across multiple cities. store in SQLite, detect new listings, send email alerts. stack: requests, BeautifulSoup, SQLite, smtplib. LinkedIn requires proxy rotation even at beginner scale.

2. e-commerce price tracker

track prices for a list of Amazon ASINs or Shopify product URLs. store daily snapshots, alert on drops. stack: requests, lxml, PostgreSQL. Amazon needs curl-cffi at minimum; Shopify stores are usually open.

3. real estate listing monitor

scrape Zillow, Realtor.com, or local MLS aggregators for new listings matching criteria. Zillow has heavy bot protection; start with smaller regional sites. output: Telegram notifications per new match.

4. twitter/x keyword monitor

use the Nitter frontend to scrape tweets without API costs. track brand mentions, competitor names, or industry keywords. stack: requests, Nitter, JSON storage.

5. news sentiment tracker

scrape Google News for a topic using tbm=nws, run headlines through a sentiment model (VADER or GPT-4o-mini), chart sentiment over time. useful for financial research and brand monitoring.

intermediate projects

6. serp rank tracker

track Google positions for a list of keywords and URLs. requires proxy rotation to avoid IP blocks. store daily snapshots, detect rank changes, generate weekly reports. see our web scraping fundamentals guide for the technical foundation.

7. domain expiry monitor

scrape WHOIS data for a list of domains, alert when expiry is within 30 days. resell as a service to agencies. stack: python-whois, CRON, Telegram bot.

8. linkedin company data scraper

scrape employee count, job openings, and follower growth for a list of companies. LinkedIn has aggressive bot detection; requires Playwright and residential proxies. output: weekly CSV for sales intelligence.

9. product review aggregator

pull reviews from Amazon, Trustpilot, G2, and Capterra for a product category. run through LLM to extract common complaints and feature requests. useful for competitive intelligence and product research.

10. stock news pipeline

scrape Yahoo Finance, MarketWatch, and Seeking Alpha for ticker-specific news. correlate news volume with price movement. requires fast scraping (sub-5 minute latency for breaking news). stack: async aiohttp, Redis queue, TimescaleDB.

11. hotel rate monitor

track Booking.com and Expedia rates for specific properties over a 90-day forward window. rates change dynamically; scrape 2x per day. requires residential proxies. output: price matrix by check-in date.

12. github trend monitor

scrape GitHub trending (no auth required), track stars-per-hour for new repos, alert on repos breaking 100 stars in first 24 hours. clean data source, no bot protection. great for finding emerging tools early.

advanced projects

13. amazon asin bulk scraper

scrape product details, BSR rank, pricing history, and review count for 10,000+ ASINs. Amazon uses Imperva; you need the full bypass stack. output: product intelligence database for private label sellers.

14. court records monitor

scrape PACER (federal) or state court systems for new filings on specific companies or individuals. legal intelligence service. some courts require browser automation; others have open XML feeds.

15. flight price matrix

scrape Google Flights, Kayak, or Skyscanner for origin-destination pairs across 90 days. each site requires browser automation. output: fare prediction model or cheap-flight alert service.

16. patent monitor

scrape USPTO and Google Patents for new filings by competitor companies. track patent activity as a signal for R&D direction. clean data source with reasonable rate limits.

17. social media follower tracker

track follower counts and engagement rates for competitor accounts across Instagram, TikTok, and YouTube. each requires different bypass approaches. output: weekly competitive benchmarking report.

18. real-time sports odds aggregator

scrape odds from 20+ bookmakers, compute arbitrage opportunities in real-time. latency matters here; you need async scraping with sub-second refresh cycles. stack: aiohttp, Redis, FastAPI dashboard.

19. influencer database builder

scrape Instagram and TikTok for creators in a niche by hashtag. extract follower count, engagement rate, contact info from bio. sell as a SaaS tool to agencies. requires aggressive proxy rotation and browser automation.

20. glassdoor salary database

scrape Glassdoor salary reports by role, company, and location. Glassdoor requires login for most data; needs account pooling and residential proxies. output: compensation benchmarking dataset.

monetization angles

21-25. data-as-a-service projects

any of the above can become a data product. the playbook: scrape the data, clean it, expose it via a simple API with Stripe-gated access. projects 6 (SERP tracking), 9 (review aggregation), 11 (hotel rates), and 15 (flight prices) have proven markets with existing paid competitors.

for distribution, the fastest path to revenue is selling CSV exports to small operators rather than building a full SaaS. list on Gumroad or Lemon Squeezy, generate fresh exports weekly, update the listing. no infra required beyond a cron job and file storage.

tech stack by difficulty

beginner: requests, BeautifulSoup4, pandas, SQLite, CRON. intermediate: curl-cffi, playwright, PostgreSQL, Redis, Celery. advanced: distributed Playwright workers, residential and mobile proxy pools, Kafka or Redis Streams for real-time, ClickHouse for analytics. see SOCKS5 vs HTTP proxy for routing considerations at scale.

sources and further reading

• Scrapy framework documentation
• Playwright for Python
• Real Python: web scraping practical introduction

related guides

• what is web scraping
• SOCKS5 vs HTTP proxy
• how to monetize web scraping
• web scraping with Python requests

the base url structure

every Google search starts at https://www.google.com/search. the query string carries all the search configuration. the minimum required parameter is q (the query). everything else is optional but powerful.

for scraping purposes, always target google.com with explicit gl and hl parameters rather than country-specific domains. the gl parameter gives you cleaner, more predictable results and avoids regional redirect chains.

core parameters

q: query

the search query. URL-encode it. spaces become + or %20. use urllib.parse.quote_plus() in Python. advanced operators (site:, intitle:, filetype:) go inside q.

num: results per page

valid values: 10 (default), 20, 30, 50, 100. setting num=100 gives you the full first page in one request, which is essential for efficient scraping. SERP quality degrades beyond position 30; positions 31-100 are often thin or duplicate content.

start: pagination offset

zero-indexed. start=0 is page 1, start=10 is page 2 (with default num=10). combine with num: num=100&start=0 fetches 100 results in one shot.

gl: geolocation country

two-letter country code. gl=us, gl=gb, gl=sg. for rank tracking, always fix gl so results are consistent across requests.

hl: interface language

hl=en forces English interface. if you omit this, Google infers language from IP location, which breaks scraping consistency when you rotate proxies across regions.

lr: language restrict

restricts results to pages in a specific language. format: lr=lang_en, lr=lang_zh-TW. different from hl: hl controls the UI language, lr controls the language of pages returned.

date and freshness parameters

tbs: time-based search

values: tbs=qdr:h (past hour), tbs=qdr:d (past 24 hours), tbs=qdr:w (past week), tbs=qdr:m (past month), tbs=qdr:y (past year), tbs=cdr:1,cd_min:1/1/2025,cd_max:12/31/2025 (custom date range). for news monitoring pipelines, tbs=qdr:h pairs with tbm=nws for news-specific results.

result type parameters

tbm: type of search

• tbm=nws – Google News
• tbm=isch – Google Images
• tbm=vid – Google Videos
• tbm=shop – Google Shopping
• tbm=bks – Google Books

output and format parameters

filter

filter=0 disables duplicate filtering and the omitted similar results cluster. always set this when scraping for comprehensive results. filter=1 (default) silently drops results Google considers duplicates.

nfpr

nfpr=1 disables automatic query corrections, which is critical for rank tracking exact queries.

a working python scraping example

import urllib.parse
from curl_cffi import requests as cffi_requests

def google_serp(query, num=10, gl="us", tbs=None):
    params = {"q": query, "num": str(num), "gl": gl, "hl": "en", "filter": "0", "nfpr": "1"}
    if tbs:
        params["tbs"] = tbs
    url = "https://www.google.com/search?" + urllib.parse.urlencode(params)
    s = cffi_requests.Session(impersonate="chrome120")
    return s.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36", "Accept-Language": "en-US,en;q=0.9"}).text

html = google_serp("web scraping python", num=100, gl="us", tbs="qdr:m")

parameters to avoid

some parameters seen in older guides no longer work or actively trigger bot detection. pws=0 (disable personalization) was removed in 2021. as_sitesearch works but is slower than using site: inside q. complete=0 has no effect on server-side responses.

sources and further reading

• Google Custom Search API reference
• SerpApi Google Search parameters reference
• Moz: Google search parameters guide

related guides

• what is web scraping
• SOCKS5 vs HTTP proxy
• rotating proxies guide
• web scraping with Python requests