How to Scrape Indeed Job Listings with Proxies in 2026

Indeed remains the world’s largest job aggregator, hosting millions of job listings across dozens of countries. Recruiters, HR tech companies, job market analysts, and researchers frequently need structured access to Indeed’s job data. However, Indeed has invested heavily in anti-scraping technology, making proxies essential for any meaningful data extraction.

This guide covers how to scrape Indeed job listings using Python with proxy rotation, including geo-targeted approaches for location-specific job data.

Why Scrape Indeed?

Indeed’s data powers a wide range of business applications:

Job market analytics — Track hiring trends, salary benchmarks, and demand by role
Competitive intelligence — Monitor competitor hiring patterns and team expansions
Recruitment automation — Aggregate listings for job boards or applicant tracking systems
Salary research — Build compensation databases across industries and locations
Economic research — Analyze employment trends as economic indicators
Lead generation — Identify companies that are hiring and may need related services

Indeed’s Protection Measures

Indeed uses multi-layered anti-bot defenses:

IP-based rate limiting — Strict request limits per IP address with progressive blocking
CAPTCHA challenges — Google reCAPTCHA triggered after suspicious browsing patterns
JavaScript rendering — Key content loaded dynamically via JavaScript
Session fingerprinting — Tracks browser characteristics across requests
Behavioral analysis — Detects non-human browsing patterns (linear navigation, consistent timing)
Honeypot links — Hidden links designed to trap automated crawlers
Request header validation — Rejects requests missing standard browser headers

Data Points to Extract

A typical Indeed job listing scrape targets:

Data Point	Source	Notes
Job title	Listing card / detail page	Primary searchable field
Company name	Listing card	Employer identity
Location	Listing card	City, state, remote status
Salary	Listing card (when available)	Range or exact, Indeed-estimated
Job description	Detail page	Full text of posting
Date posted	Listing card	Relative or absolute
Job type	Tags	Full-time, part-time, contract
Benefits	Detail page	Insurance, PTO, etc.
Rating	Company rating badge	Employer star rating
Apply link	Detail page	Indeed apply or external URL

Setting Up Your Environment

pip install requests beautifulsoup4 lxml fake-useragent

Python Code: Scraping Indeed with Proxies

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
import json
import time
import random
import logging
from urllib.parse import urlencode

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class IndeedScraper:
    def __init__(self, proxy_list: list):
        self.proxy_list = proxy_list
        self.ua = UserAgent()
        self.base_url = "https://www.indeed.com"
        self.jobs = []

    def get_proxy(self) -> dict:
        proxy = random.choice(self.proxy_list)
        return {"http": f"http://{proxy}", "https": f"http://{proxy}"}

    def get_headers(self) -> dict:
        return {
            "User-Agent": self.ua.random,
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
            "Accept-Encoding": "gzip, deflate, br",
            "Connection": "keep-alive",
            "Upgrade-Insecure-Requests": "1"
        }

    def search_jobs(self, query: str, location: str, max_pages: int = 10):
        """Search Indeed for jobs matching query and location."""
        for page in range(max_pages):
            start = page * 10
            params = {
                "q": query,
                "l": location,
                "start": start
            }
            url = f"{self.base_url}/jobs?{urlencode(params)}"
            logger.info(f"Scraping page {page + 1}: {url}")

            try:
                response = requests.get(
                    url,
                    headers=self.get_headers(),
                    proxies=self.get_proxy(),
                    timeout=30
                )

                if response.status_code == 200:
                    new_jobs = self.parse_search_results(response.text)
                    if not new_jobs:
                        logger.info("No more results found")
                        break
                    self.jobs.extend(new_jobs)
                    logger.info(f"Found {len(new_jobs)} jobs on page {page + 1}")
                elif response.status_code == 403:
                    logger.warning("Blocked -- rotating proxy and waiting")
                    time.sleep(random.uniform(10, 20))
                    continue
                else:
                    logger.error(f"Status {response.status_code}")

            except requests.exceptions.RequestException as e:
                logger.error(f"Request failed: {e}")

            time.sleep(random.uniform(3, 7))

    def parse_search_results(self, html: str) -> list:
        """Parse job listings from search results page."""
        soup = BeautifulSoup(html, "lxml")
        jobs = []

        # Indeed uses data-jk attribute for job cards
        job_cards = soup.select("[class*='job_seen_beacon'], [class*='jobsearch-ResultsList'] > li")

        for card in job_cards:
            job = {}

            # Job title
            title_el = card.select_one("h2 a, [class*='jobTitle'] a")
            if title_el:
                job["title"] = title_el.get_text(strip=True)
                job["url"] = self.base_url + title_el.get("href", "")
                # Extract job key from URL
                href = title_el.get("href", "")
                if "jk=" in href:
                    job["job_key"] = href.split("jk=")[1].split("&")[0]

            # Company name
            company_el = card.select_one("[class*='company'], [data-testid='company-name']")
            if company_el:
                job["company"] = company_el.get_text(strip=True)

            # Location
            location_el = card.select_one("[class*='location'], [data-testid='text-location']")
            if location_el:
                job["location"] = location_el.get_text(strip=True)

            # Salary
            salary_el = card.select_one("[class*='salary'], [class*='estimated-salary']")
            if salary_el:
                job["salary"] = salary_el.get_text(strip=True)

            # Snippet / description preview
            snippet_el = card.select_one("[class*='snippet'], [class*='job-snippet']")
            if snippet_el:
                job["snippet"] = snippet_el.get_text(strip=True)

            # Date posted
            date_el = card.select_one("[class*='date'], .date")
            if date_el:
                job["date_posted"] = date_el.get_text(strip=True)

            if job.get("title"):
                jobs.append(job)

        return jobs

    def scrape_job_detail(self, job_url: str) -> dict:
        """Scrape full job description from detail page."""
        try:
            response = requests.get(
                job_url,
                headers=self.get_headers(),
                proxies=self.get_proxy(),
                timeout=30
            )

            if response.status_code != 200:
                return {}

            soup = BeautifulSoup(response.text, "lxml")
            detail = {}

            # Full description
            desc_el = soup.select_one("#jobDescriptionText, [class*='jobsearch-JobComponent-description']")
            if desc_el:
                detail["description"] = desc_el.get_text(strip=True)

            # Benefits
            benefits = []
            benefit_els = soup.select("[class*='benefit'], [class*='Benefits'] li")
            for b in benefit_els:
                benefits.append(b.get_text(strip=True))
            detail["benefits"] = benefits

            # Job type
            type_el = soup.select_one("[class*='jobsearch-JobMetadataHeader']")
            if type_el:
                detail["job_type"] = type_el.get_text(strip=True)

            return detail

        except requests.exceptions.RequestException as e:
            logger.error(f"Detail scrape failed: {e}")
            return {}


# Usage
if __name__ == "__main__":
    proxies = [
        "user:pass@us-residential1.proxy.com:8080",
        "user:pass@us-residential2.proxy.com:8080",
        "user:pass@us-residential3.proxy.com:8080",
    ]

    scraper = IndeedScraper(proxy_list=proxies)
    scraper.search_jobs(
        query="software engineer",
        location="San Francisco, CA",
        max_pages=5
    )

    # Get details for first 5 jobs
    for job in scraper.jobs[:5]:
        if "url" in job:
            detail = scraper.scrape_job_detail(job["url"])
            job.update(detail)
            time.sleep(random.uniform(3, 6))

    print(f"Total jobs scraped: {len(scraper.jobs)}")
    with open("indeed_jobs.json", "w") as f:
        json.dump(scraper.jobs, f, indent=2)

Geo-Targeted Scraping for Location-Specific Jobs

One of Indeed’s most valuable features is location-based job search. To accurately scrape jobs for a specific location, you need proxies from that geographic area:

US jobs — Use US residential proxies, ideally from the target state
UK jobs — Use indeed.co.uk with UK proxies
Canada — Use ca.indeed.com with Canadian proxies
Indeed country variants — Indeed operates localized sites (indeed.de, indeed.fr, etc.)

# Geo-targeted scraping example
INDEED_DOMAINS = {
    "us": "https://www.indeed.com",
    "uk": "https://www.indeed.co.uk",
    "ca": "https://ca.indeed.com",
    "au": "https://au.indeed.com",
    "de": "https://de.indeed.com",
    "fr": "https://www.indeed.fr",
    "in": "https://www.indeed.co.in",
}

def scrape_by_country(country_code: str, query: str, location: str):
    """Scrape Indeed for a specific country."""
    domain = INDEED_DOMAINS.get(country_code, INDEED_DOMAINS["us"])
    # Use proxies from the matching country
    # This ensures Indeed shows local results and pricing
    scraper = IndeedScraper(proxy_list=get_proxies_for_country(country_code))
    scraper.base_url = domain
    scraper.search_jobs(query, location, max_pages=10)
    return scraper.jobs

Use our IP lookup tool to verify your proxy’s geographic location before targeting country-specific Indeed sites.

API Alternatives vs Scraping

Before building a scraper, consider Indeed’s official options:

Indeed Publisher API — Provides job search results for approved publishers. Requires application and approval. Limited data fields compared to scraping.
Indeed Apply API — For integrating Indeed’s apply flow into external platforms.
Third-party job APIs — Services like Adzuna, The Muse, or JSearch aggregate job data from multiple sources including Indeed.

The official APIs have limitations: restricted fields, rate limits, and approval requirements. Scraping provides access to the full data set but comes with technical and legal challenges.

Recommended Proxy Type

For Indeed scraping:

Residential rotating proxies — Best overall choice. Rotate every 1-3 requests for search pages.
Sticky sessions (5-10 minutes) — Use for scraping job detail pages where session continuity matters.
Geo-targeted — Essential for location-specific job data. Match proxy location to target job market.
Datacenter proxies — Not recommended. Indeed blocks datacenter IP ranges aggressively.

Calculate your expected costs with our proxy cost calculator.

Troubleshooting

Problem: Search returns zero results despite valid queries

Indeed may be serving a CAPTCHA page. Check the response HTML for CAPTCHA markers.
Verify your proxy location matches the Indeed domain you are targeting.
Try adding more realistic headers including Referer and Sec-Fetch headers.

Problem: Job detail pages return 403 Forbidden

Rotate to a fresh proxy IP before each detail page request.
Add a delay of 5-10 seconds between detail page requests.
Include a Referer header pointing to the Indeed search results page.

Problem: Salary data is missing from most listings

This is normal. Only about 30-40% of Indeed listings include salary data. Indeed sometimes estimates salary ranges, but these appear differently in the HTML.
Look for Indeed’s “Estimated salary” badges which are distinct from employer-posted salaries.

Problem: Getting redirected to different country versions

Use geo-targeted proxies matching the Indeed domain you want.
Set explicit Accept-Language headers for the target locale.
Access the country-specific domain directly rather than relying on redirects.

Legal and Ethical Considerations

Indeed scraping carries notable legal considerations:

hiQ v. LinkedIn precedent — The 2022 ruling established that scraping publicly available data is not a CFAA violation. However, this does not override ToS or other legal frameworks.
Indeed’s Terms of Service — Explicitly prohibit scraping. Indeed has pursued legal action against scrapers in the past.
Personal data — Job listings may not contain personal data, but reviewer and salary report data could implicate privacy laws (GDPR, CCPA).
Rate limiting — Overwhelming Indeed’s servers could constitute a denial-of-service attack. Always implement respectful delays.
Data usage — Republishing scraped job listings may infringe on Indeed’s database rights and the original employers’ content.

Consult with a legal professional before scraping Indeed at commercial scale.

Conclusion

Scraping Indeed requires residential proxies with geo-targeting capabilities, realistic browser emulation, and patient rate limiting. The Python code above provides a solid foundation for extracting job listings and detail data. Start with small-scale tests, verify your data quality, and scale up gradually while monitoring block rates. For production use, consider combining scraping with Indeed’s official APIs to reduce your reliance on scraping alone.