DataResearchTools

Category: Web Scraping Guides

  • Google Search URL Parameters: Complete 2026 Reference

    TL;DR
    Google’s search URL accepts dozens of undocumented parameters that control results, date filters, language, location, and output format. this is the working reference for 2026, sourced from reverse-engineering and official documentation.

    the base url structure

    every Google search starts at https://www.google.com/search. the query string carries all the search configuration. the minimum required parameter is q (the query). everything else is optional but powerful.

    for scraping purposes, always target google.com with explicit gl and hl parameters rather than country-specific domains. the gl parameter gives you cleaner, more predictable results and avoids regional redirect chains.

    core parameters

    q: query

    the search query. URL-encode it. spaces become + or %20. use urllib.parse.quote_plus() in Python. advanced operators (site:, intitle:, filetype:) go inside q.

    num: results per page

    valid values: 10 (default), 20, 30, 50, 100. setting num=100 gives you the full first page in one request, which is essential for efficient scraping. SERP quality degrades beyond position 30; positions 31-100 are often thin or duplicate content.

    start: pagination offset

    zero-indexed. start=0 is page 1, start=10 is page 2 (with default num=10). combine with num: num=100&start=0 fetches 100 results in one shot.

    gl: geolocation country

    two-letter country code. gl=us, gl=gb, gl=sg. for rank tracking, always fix gl so results are consistent across requests.

    hl: interface language

    hl=en forces English interface. if you omit this, Google infers language from IP location, which breaks scraping consistency when you rotate proxies across regions.

    lr: language restrict

    restricts results to pages in a specific language. format: lr=lang_en, lr=lang_zh-TW. different from hl: hl controls the UI language, lr controls the language of pages returned.

    date and freshness parameters

    tbs: time-based search

    values: tbs=qdr:h (past hour), tbs=qdr:d (past 24 hours), tbs=qdr:w (past week), tbs=qdr:m (past month), tbs=qdr:y (past year), tbs=cdr:1,cd_min:1/1/2025,cd_max:12/31/2025 (custom date range). for news monitoring pipelines, tbs=qdr:h pairs with tbm=nws for news-specific results.

    result type parameters

    tbm: type of search

    • tbm=nws – Google News
    • tbm=isch – Google Images
    • tbm=vid – Google Videos
    • tbm=shop – Google Shopping
    • tbm=bks – Google Books

    output and format parameters

    filter

    filter=0 disables duplicate filtering and the omitted similar results cluster. always set this when scraping for comprehensive results. filter=1 (default) silently drops results Google considers duplicates.

    nfpr

    nfpr=1 disables automatic query corrections, which is critical for rank tracking exact queries.

    a working python scraping example

    import urllib.parse
from curl_cffi import requests as cffi_requests

def google_serp(query, num=10, gl="us", tbs=None):
    params = {"q": query, "num": str(num), "gl": gl, "hl": "en", "filter": "0", "nfpr": "1"}
    if tbs:
        params["tbs"] = tbs
    url = "https://www.google.com/search?" + urllib.parse.urlencode(params)
    s = cffi_requests.Session(impersonate="chrome120")
    return s.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36", "Accept-Language": "en-US,en;q=0.9"}).text

html = google_serp("web scraping python", num=100, gl="us", tbs="qdr:m")

    parameters to avoid

    some parameters seen in older guides no longer work or actively trigger bot detection. pws=0 (disable personalization) was removed in 2021. as_sitesearch works but is slower than using site: inside q. complete=0 has no effect on server-side responses.

    sources and further reading

    related guides

  • How to Scrape Yahoo Finance Stock Data in 2026

    How to Scrape Yahoo Finance Stock Data in 2026

    Yahoo Finance is one of the most widely used financial data platforms, providing free access to stock prices, historical data, financial statements, analyst estimates, and market news for thousands of publicly traded companies worldwide. For quantitative traders, financial analysts, investment researchers, and fintech developers, scraping Yahoo Finance provides comprehensive market data at no cost.

    looking for premium 4G/5G IPs? our Singapore mobile proxies for finance scraping start at $40/month for 200GB.

    This guide covers how to extract Yahoo Finance data using Python with the yfinance library and custom scraping approaches.

    What Data Can You Extract?

    Yahoo Finance provides extensive financial data:

    • Stock prices (real-time quotes, historical OHLCV data)
    • Financial statements (income statement, balance sheet, cash flow)
    • Company information (sector, industry, employees, description)
    • Analyst recommendations and price targets
    • Earnings data (EPS, revenue, earnings dates)
    • Dividend history and yield
    • Options chain data
    • Market indices and ETF data
    • Financial news and articles

    Example JSON Output

    {
  "ticker": "AAPL",
  "company_name": "Apple Inc.",
  "current_price": 245.67,
  "market_cap": 3890000000000,
  "pe_ratio": 32.5,
  "dividend_yield": 0.0044,
  "52_week_high": 260.10,
  "52_week_low": 164.08,
  "earnings_date": "2026-04-28",
  "analyst_target_price": 270.00,
  "recommendation": "Buy"
}

    Prerequisites

    pip install yfinance requests beautifulsoup4 pandas

    Method 1: Using yfinance (Recommended)

    The yfinance library is the most popular and reliable way to access Yahoo Finance data.

    import yfinance as yf
import pandas as pd
import json
from datetime import datetime, timedelta

class YahooFinanceScraper:
    def __init__(self):
        pass

    def get_stock_info(self, ticker):
        """Get comprehensive stock information."""
        stock = yf.Ticker(ticker)
        info = stock.info

        return {
            "ticker": ticker,
            "name": info.get("longName"),
            "sector": info.get("sector"),
            "industry": info.get("industry"),
            "current_price": info.get("currentPrice"),
            "market_cap": info.get("marketCap"),
            "pe_ratio": info.get("trailingPE"),
            "forward_pe": info.get("forwardPE"),
            "dividend_yield": info.get("dividendYield"),
            "52_week_high": info.get("fiftyTwoWeekHigh"),
            "52_week_low": info.get("fiftyTwoWeekLow"),
            "volume": info.get("volume"),
            "avg_volume": info.get("averageVolume"),
            "beta": info.get("beta"),
            "earnings_date": str(info.get("earningsDate")),
            "target_mean_price": info.get("targetMeanPrice"),
            "recommendation": info.get("recommendationKey"),
            "total_revenue": info.get("totalRevenue"),
            "net_income": info.get("netIncomeToCommon"),
            "employees": info.get("fullTimeEmployees"),
        }

    def get_historical_data(self, ticker, period="1y", interval="1d"):
        """Get historical price data."""
        stock = yf.Ticker(ticker)
        hist = stock.history(period=period, interval=interval)

        return hist.reset_index().to_dict(orient="records")

    def get_financials(self, ticker):
        """Get financial statements."""
        stock = yf.Ticker(ticker)

        return {
            "income_statement": stock.financials.to_dict() if not stock.financials.empty else {},
            "balance_sheet": stock.balance_sheet.to_dict() if not stock.balance_sheet.empty else {},
            "cash_flow": stock.cashflow.to_dict() if not stock.cashflow.empty else {},
        }

    def get_analyst_recommendations(self, ticker):
        """Get analyst recommendations."""
        stock = yf.Ticker(ticker)
        recs = stock.recommendations

        if recs is not None and not recs.empty:
            return recs.tail(20).reset_index().to_dict(orient="records")
        return []

    def get_options_chain(self, ticker, expiration_date=None):
        """Get options chain data."""
        stock = yf.Ticker(ticker)

        if expiration_date:
            opts = stock.option_chain(expiration_date)
        else:
            expirations = stock.options
            if expirations:
                opts = stock.option_chain(expirations[0])
            else:
                return None

        return {
            "calls": opts.calls.to_dict(orient="records"),
            "puts": opts.puts.to_dict(orient="records"),
        }

    def get_multiple_stocks(self, tickers, period="1mo"):
        """Get data for multiple stocks at once."""
        data = yf.download(tickers, period=period, group_by="ticker")
        return data

    def get_earnings_history(self, ticker):
        """Get historical earnings data."""
        stock = yf.Ticker(ticker)
        earnings = stock.earnings_history

        if earnings is not None and not earnings.empty:
            return earnings.to_dict(orient="records")
        return []

    def screen_stocks(self, tickers, min_market_cap=None, max_pe=None, min_dividend=None):
        """Simple stock screener."""
        results = []

        for ticker in tickers:
            try:
                info = self.get_stock_info(ticker)
                passed = True

                if min_market_cap and (info.get("market_cap") or 0) < min_market_cap:
                    passed = False
                if max_pe and (info.get("pe_ratio") or float('inf')) > max_pe:
                    passed = False
                if min_dividend and (info.get("dividend_yield") or 0) < min_dividend:
                    passed = False

                if passed:
                    results.append(info)

            except Exception as e:
                print(f"Error processing {ticker}: {e}")

        return results


# Usage
scraper = YahooFinanceScraper()

# Get stock info
aapl = scraper.get_stock_info("AAPL")
print(json.dumps(aapl, indent=2, default=str))

# Get historical data
hist = scraper.get_historical_data("AAPL", period="6mo")
print(f"Historical data points: {len(hist)}")

# Get financials
financials = scraper.get_financials("AAPL")
print(f"Income statement columns: {len(financials['income_statement'])}")

# Get analyst recommendations
recs = scraper.get_analyst_recommendations("AAPL")
print(f"Analyst recommendations: {len(recs)}")

# Simple screen
tech_stocks = ["AAPL", "MSFT", "GOOGL", "META", "NVDA"]
screened = scraper.screen_stocks(tech_stocks, min_market_cap=1e12)
print(f"Stocks passing screen: {len(screened)}")

    Method 2: Direct Web Scraping

    For data not available through yfinance:

    import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
import json

class YahooFinanceWebScraper:
    def __init__(self, proxy_url=None):
        self.session = requests.Session()
        self.ua = UserAgent()
        self.proxy_url = proxy_url

    def _get_headers(self):
        return {
            "User-Agent": self.ua.random,
            "Accept": "text/html,application/xhtml+xml",
        }

    def _get_proxies(self):
        if self.proxy_url:
            return {"http": self.proxy_url, "https": self.proxy_url}
        return None

    def get_trending_tickers(self):
        """Scrape trending tickers from Yahoo Finance."""
        url = "https://finance.yahoo.com/trending-tickers"
        try:
            response = self.session.get(url, headers=self._get_headers(), proxies=self._get_proxies(), timeout=30)
            response.raise_for_status()
            soup = BeautifulSoup(response.text, "lxml")

            tickers = []
            rows = soup.select("table tbody tr")
            for row in rows:
                cells = row.select("td")
                if len(cells) >= 4:
                    tickers.append({
                        "symbol": cells[0].get_text(strip=True),
                        "name": cells[1].get_text(strip=True),
                        "price": cells[2].get_text(strip=True),
                        "change": cells[3].get_text(strip=True),
                    })
            return tickers
        except Exception as e:
            print(f"Error: {e}")
            return []

    def get_news(self, ticker):
        """Scrape news articles for a ticker."""
        url = f"https://finance.yahoo.com/quote/{ticker}/news"
        try:
            response = self.session.get(url, headers=self._get_headers(), proxies=self._get_proxies(), timeout=30)
            response.raise_for_status()
            soup = BeautifulSoup(response.text, "lxml")

            articles = []
            news_items = soup.select("li[class*='stream-item'], div[class*='news-stream'] li")
            for item in news_items:
                title = item.select_one("h3, a")
                link = item.select_one("a[href]")
                articles.append({
                    "title": title.get_text(strip=True) if title else None,
                    "url": link["href"] if link else None,
                })
            return articles[:20]
        except Exception as e:
            print(f"Error: {e}")
            return []


# Usage
web_scraper = YahooFinanceWebScraper(proxy_url="http://user:pass@proxy:port")
trending = web_scraper.get_trending_tickers()
print(json.dumps(trending[:5], indent=2))

    Proxy Recommendations

    Proxy Type Necessity Best For
    None yfinance library Standard data access
    Datacenter Optional High-frequency data pulls
    Residential Optional Web scraping at scale

    The yfinance library typically doesn’t require proxies. For high-frequency data access or web scraping, residential proxies can help avoid rate limits.

    Legal Considerations

    1. Terms of Service: Yahoo Finance’s ToS restrict automated data collection beyond their API.
    2. Data Redistribution: Redistribution of financial data may violate exchange agreements.
    3. Real-Time Data: Real-time quotes may have licensing requirements.
    4. Commercial Use: Consult legal counsel for commercial financial data products.

    See our web scraping compliance guide for details.

    Frequently Asked Questions

    Is the yfinance library official?

    No. yfinance is an unofficial library that accesses Yahoo Finance data. It’s the most widely used method for accessing Yahoo Finance data programmatically but is not endorsed by Yahoo.

    How often can I pull data with yfinance?

    yfinance has no strict rate limits, but excessive requests may result in temporary blocks. For real-time data, limit pulls to once per minute. For historical data, batch your requests.

    Can I get real-time stock prices?

    yfinance provides near-real-time prices (15-20 minute delay for US markets). For true real-time data, consider paid data providers or broker APIs.

    What are alternatives to Yahoo Finance for financial data?

    Alpha Vantage (free API), IEX Cloud, Polygon.io, and Finnhub are popular alternatives with their own APIs and pricing tiers.

    Conclusion

    Yahoo Finance is one of the most accessible sources for financial market data. The yfinance library handles most data needs without proxies or complex scraping. For supplementary data like news and trending tickers, web scraping with proxies provides additional coverage.

    For more financial data guides, visit our web scraping proxy guide and proxy provider comparisons.

    Related Reading

  • How to Scrape SEC EDGAR Filings Data in 2026

    How to Scrape SEC EDGAR Filings Data in 2026

    SEC EDGAR (Electronic Data Gathering, Analysis, and Retrieval) is the U.S. Securities and Exchange Commission’s free database of corporate filings, containing over 21 million filings from public companies. For financial analysts, compliance professionals, investment researchers, and fintech developers, EDGAR provides the most authoritative source of public company financial data in the United States.

    looking for premium 4G/5G IPs? our Singapore mobile proxies for scraping start at $40/month for 200GB.

    Unlike most scraping targets, SEC EDGAR is explicitly designed for public data access and provides a well-documented API, making it one of the most scraper-friendly data sources available.

    What Data Can You Extract?

    SEC EDGAR contains comprehensive regulatory filings:

    • Annual reports (10-K) and quarterly reports (10-Q)
    • Current reports (8-K) for material events
    • Insider trading (Form 3, 4, 5)
    • Proxy statements (DEF 14A)
    • Registration statements (S-1 for IPOs)
    • XBRL financial data (structured financial statements)
    • Company information (CIK, SIC codes, addresses)
    • Filing history and amendments

    Example JSON Output

    {
  "company": {
    "cik": "0000320193",
    "name": "Apple Inc.",
    "ticker": "AAPL",
    "sic": "3571",
    "state": "CA"
  },
  "recent_filing": {
    "form_type": "10-K",
    "filing_date": "2025-11-01",
    "accession_number": "0000320193-25-000123",
    "primary_document": "aapl-20250927.htm",
    "url": "https://www.sec.gov/Archives/edgar/data/320193/..."
  }
}

    Prerequisites

    pip install requests sec-edgar-downloader beautifulsoup4 pandas

    Method 1: Using SEC EDGAR API (Recommended)

    SEC provides a free, public API (EDGAR Full-Text Search and company data APIs).

    import requests
import json
import time

class SECEdgarScraper:
    def __init__(self, user_agent="YourName your@email.com"):
        self.session = requests.Session()
        self.base_url = "https://efts.sec.gov/LATEST"
        self.data_url = "https://data.sec.gov"
        self.headers = {
            "User-Agent": user_agent,
            "Accept": "application/json",
        }

    def search_companies(self, query):
        """Search for companies by name or ticker."""
        url = f"{self.data_url}/submissions/CIK{query.zfill(10)}.json"

        try:
            response = self.session.get(url, headers=self.headers, timeout=30)
            if response.status_code == 200:
                return response.json()
        except Exception:
            pass

        # Fallback: full-text search
        url = f"{self.base_url}/search-index?q={query}&dateRange=custom&startdt=2024-01-01&enddt=2026-12-31"
        try:
            response = self.session.get(url, headers=self.headers, timeout=30)
            response.raise_for_status()
            return response.json()
        except Exception as e:
            print(f"Error: {e}")
            return None

    def get_company_filings(self, cik, form_type=None):
        """Get filings for a company by CIK number."""
        cik_padded = str(cik).zfill(10)
        url = f"{self.data_url}/submissions/CIK{cik_padded}.json"

        try:
            response = self.session.get(url, headers=self.headers, timeout=30)
            response.raise_for_status()
            data = response.json()

            filings = data.get("filings", {}).get("recent", {})
            results = []

            forms = filings.get("form", [])
            dates = filings.get("filingDate", [])
            accessions = filings.get("accessionNumber", [])
            documents = filings.get("primaryDocument", [])

            for i in range(len(forms)):
                if form_type and forms[i] != form_type:
                    continue

                accession_clean = accessions[i].replace("-", "")
                results.append({
                    "form_type": forms[i],
                    "filing_date": dates[i],
                    "accession_number": accessions[i],
                    "primary_document": documents[i],
                    "url": f"https://www.sec.gov/Archives/edgar/data/{cik}/{accession_clean}/{documents[i]}",
                })

            return {
                "company_name": data.get("name"),
                "cik": cik,
                "ticker": data.get("tickers", [""])[0] if data.get("tickers") else None,
                "filings": results,
            }

        except requests.RequestException as e:
            print(f"Error: {e}")
            return None

    def get_xbrl_data(self, cik, taxonomy="us-gaap", tag="Revenue"):
        """Get structured XBRL financial data."""
        cik_padded = str(cik).zfill(10)
        url = f"{self.data_url}/api/xbrl/companyfacts/CIK{cik_padded}.json"

        try:
            response = self.session.get(url, headers=self.headers, timeout=30)
            response.raise_for_status()
            data = response.json()

            facts = data.get("facts", {}).get(taxonomy, {}).get(tag, {})
            units = facts.get("units", {})

            results = []
            for unit_type, values in units.items():
                for v in values:
                    results.append({
                        "value": v.get("val"),
                        "unit": unit_type,
                        "period_end": v.get("end"),
                        "period_start": v.get("start"),
                        "form": v.get("form"),
                        "filing_date": v.get("filed"),
                    })

            return results

        except Exception as e:
            print(f"Error: {e}")
            return []

    def get_insider_trading(self, cik):
        """Get insider trading filings (Form 4)."""
        return self.get_company_filings(cik, form_type="4")

    def search_filings(self, query, form_type=None, date_from=None, date_to=None):
        """Full-text search across all filings."""
        params = {"q": query, "from": 0, "size": 50}
        if form_type:
            params["forms"] = form_type
        if date_from:
            params["startdt"] = date_from
        if date_to:
            params["enddt"] = date_to

        try:
            response = self.session.get(
                f"{self.base_url}/search-index",
                params=params, headers=self.headers, timeout=30
            )
            response.raise_for_status()
            return response.json()
        except Exception as e:
            print(f"Error: {e}")
            return None


# Usage
scraper = SECEdgarScraper(user_agent="DataResearch admin@dataresearchtools.com")

# Get Apple filings
apple = scraper.get_company_filings(320193, form_type="10-K")
print(f"Company: {apple['company_name']}")
print(f"10-K filings: {len(apple['filings'])}")

# Get revenue data
revenue = scraper.get_xbrl_data(320193, tag="Revenues")
print(f"Revenue data points: {len(revenue)}")
for r in revenue[-4:]:
    print(f"  {r['period_end']}: ${r['value']:,.0f}")

# Get insider trading
insider = scraper.get_insider_trading(320193)
print(f"Form 4 filings: {len(insider['filings'])}")

    SEC EDGAR Access Rules

    SEC EDGAR has specific access requirements:

    • User-Agent: Must include your name and email address
    • Rate Limit: Maximum 10 requests per second
    • No Authentication: All data is freely accessible
    • robots.txt: Allows broad scraping with reasonable rate limits
    # Required User-Agent format
headers = {
    "User-Agent": "CompanyName admin@company.com"
}

    Proxy Recommendations

    Proxy Type Necessity Best For
    None Sufficient Standard use
    Datacenter Optional High-volume batch jobs

    SEC EDGAR is designed for public access. Proxies are rarely needed. Just respect the 10 requests/second rate limit.

    Legal Considerations

    1. Public Data: SEC filings are public records. No restrictions on accessing or using the data.
    2. Fair Access: SEC requests that users limit to 10 requests per second for fair access.
    3. Attribution: While not legally required, citing SEC as the data source is best practice.
    4. Redistribution: No restrictions on redistributing SEC filing data.

    Frequently Asked Questions

    Is SEC EDGAR data free?

    Yes. All SEC EDGAR data is freely available to the public. No API key, registration, or authentication is required.

    How do I find a company’s CIK number?

    Search by company name or ticker at https://www.sec.gov/cgi-bin/browse-edgar?company=&CIK=AAPL. The CIK for Apple is 0000320193.

    Can I download full financial statements?

    Yes. Use the XBRL API for structured financial data, or download complete filing documents (HTML, XML) from the filing URLs.

    How quickly are new filings available?

    SEC filings typically appear on EDGAR within minutes of submission. Real-time filing notifications are available via the SEC’s RSS feeds.

    Conclusion

    SEC EDGAR is the gold standard for public company data access — free, well-documented, and explicitly designed for programmatic access. The API provides structured data for filings, financial statements, and company information without any anti-bot protections. Focus on the XBRL API for structured financial data and the full-text search for research queries.

    For more financial data guides, visit our web scraping proxy guide and proxy provider comparisons.

    Related Reading