DataResearchTools

Category: Proxy Integration Tutorials

  • How to Use Proxies with Scrapy: Middleware, Rotation, and Headers (2026)

    how to use proxies with scrapy: middleware, rotation, and headers (2026)

    scrapy supports proxies three ways: per-request meta, the built-in httpproxymiddleware, and custom rotating middleware. for a single proxy, set request.meta["proxy"]. for rotation, write a downloader middleware that picks a fresh proxy per request and tracks dead ones. for production, pair the rotating middleware with header spoofing and a retry policy. this tutorial gives you working code for all three patterns plus the gotchas that bite at scale.

    we cover the basics, then build a production-ready rotating middleware with health checks and exponential backoff.

    the simplest pattern: per-request proxy

    set proxy in request.meta. scrapy’s built-in httpproxymiddleware (enabled by default) reads it.

    import scrapy

class SimpleSpider(scrapy.Spider):
    name = "simple"
    start_urls = ["https://httpbin.org/ip"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"proxy": "http://user:pass@1.2.3.4:8080"},
            )

    def parse(self, response):
        self.logger.info(f"saw ip: {response.json()}")

    this is the right pattern for jobs with one or two static proxies. for rotation, build a middleware.

    env-based proxy via http_proxy

    if you want every request to go through one proxy without touching code, scrapy honors the http_proxy and https_proxy env vars:

    export HTTP_PROXY="http://user:pass@1.2.3.4:8080"
export HTTPS_PROXY="http://user:pass@1.2.3.4:8080"
scrapy crawl simple

    this works for ci pipelines and one-off runs. for fine-grained control, use the middleware approach below.

    rotating proxy middleware

    create myproject/middlewares.py:

    import random
import time
import logging
from collections import defaultdict
from scrapy import signals

logger = logging.getLogger(__name__)


class RotatingProxyMiddleware:
    """rotating proxy with health tracking and exponential cooldown."""

    def __init__(self, proxies, cooldown_sec=300):
        self.proxies = list(proxies)
        self.cooldown_sec = cooldown_sec
        self.bad_until = defaultdict(float)
        self.fail_count = defaultdict(int)
        if not self.proxies:
            raise ValueError("rotating proxy middleware: no proxies configured")

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("ROTATING_PROXIES")
        cooldown = crawler.settings.getint("ROTATING_PROXY_COOLDOWN_SEC", 300)
        return cls(proxies=proxies, cooldown_sec=cooldown)

    def get_proxy(self):
        now = time.time()
        live = [p for p in self.proxies if self.bad_until[p] < now]
        if not live:
            logger.warning("all proxies cooling down. resetting.")
            self.bad_until.clear()
            live = self.proxies
        return random.choice(live)

    def mark_bad(self, proxy):
        self.fail_count[proxy] += 1
        cooldown = self.cooldown_sec * (5 ** (self.fail_count[proxy] - 1))
        self.bad_until[proxy] = time.time() + cooldown
        logger.info(f"proxy {proxy} marked bad. cooldown {cooldown}s.")

    def mark_good(self, proxy):
        self.fail_count[proxy] = 0

    def process_request(self, request, spider):
        if "proxy" in request.meta and request.meta.get("_proxy_assigned"):
            return
        proxy = self.get_proxy()
        request.meta["proxy"] = proxy
        request.meta["_proxy_assigned"] = True

    def process_response(self, request, response, spider):
        proxy = request.meta.get("proxy")
        if not proxy:
            return response
        if response.status in (407, 502, 503, 504):
            self.mark_bad(proxy)
        elif 200 <= response.status < 400:
            self.mark_good(proxy)
        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("proxy")
        if proxy:
            self.mark_bad(proxy)

    enable in settings.py:

    DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
    "myproject.middlewares.RotatingProxyMiddleware": 760,
}

ROTATING_PROXIES = [
    "http://user:pass@1.2.3.4:8080",
    "http://user:pass@5.6.7.8:8080",
    "http://user:pass@9.10.11.12:8080",
]

ROTATING_PROXY_COOLDOWN_SEC = 300

    the middleware picks a fresh proxy per request, marks dead proxies on 407/502/503/504 responses or exceptions, and applies exponential cooldown so a flaky proxy comes back online after a short rest.

    sticky session middleware for login flows

    some scrapes need the same proxy across multiple requests (login then crawl). hash the session id to a fixed proxy:

    import hashlib

class StickyProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = list(proxies)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("STICKY_PROXIES"))

    def process_request(self, request, spider):
        session_id = request.meta.get("session_id")
        if not session_id:
            return
        h = hashlib.md5(session_id.encode()).hexdigest()
        idx = int(h, 16) % len(self.proxies)
        request.meta["proxy"] = self.proxies[idx]

    usage in spider:

    yield scrapy.Request(
    "https://example.com/dashboard",
    meta={"session_id": "user_abc"},
    callback=self.parse_dashboard,
)

    every request with session_id="user_abc" gets the same proxy. swap to a different session id and you get a different proxy.

    for the deeper architecture pattern across multiple workers, see our proxy load balancing architecture guide.

    header spoofing alongside proxies

    a fresh ip with stale headers fingerprints obviously. pair the rotating middleware with rotating user agents and accept-language headers:

    class RotatingHeadersMiddleware:
    USER_AGENTS = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
    ]

    def process_request(self, request, spider):
        request.headers["User-Agent"] = random.choice(self.USER_AGENTS)
        request.headers["Accept-Language"] = "en-US,en;q=0.9"
        request.headers["Accept-Encoding"] = "gzip, deflate, br"

    enable below the proxy middleware in settings.py:

    DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
    "myproject.middlewares.RotatingProxyMiddleware": 760,
    "myproject.middlewares.RotatingHeadersMiddleware": 770,
}

    for finer fingerprint control (tls, http2, browser headers), use a managed scraping api or a headless browser. plain http requests cannot fully spoof a chrome client.

    scrapy retry settings

    scrapy ships with a retry middleware. configure it to match the rotating proxy logic:

    RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 408, 429, 500, 502, 503, 504]

DOWNLOAD_TIMEOUT = 15
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8

DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True

    RETRY_HTTP_CODES = [403, 408, 429, 500, 502, 503, 504] retries common rate-limit and proxy-failure responses. combined with the rotating middleware, each retry picks a fresh proxy.

    CONCURRENT_REQUESTS_PER_DOMAIN = 8 is conservative. tune up for tolerant targets, down for strict ones. the rotating middleware does not rate-limit; that is the autothrottle’s job.

    autothrottle for rate-limit safety

    AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 60.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
AUTOTHROTTLE_DEBUG = False

    autothrottle backs off when the target slows down or returns errors. with rotating proxies, this prevents a target from blocking your full pool by detecting a burst.

    handling 407 proxy auth required

    if you see 407 proxy authentication required errors, three checks:

    1. proxy url format is http://user:pass@host:port exactly. no leading whitespace, no url-encoded user.
    2. some providers require username sessions (user-session-abc123). use the full session-username from your dashboard.
    3. scrapy’s httpproxymiddleware does not always pass the basic-auth header automatically for some legacy versions. if you hit this, add proxy-authorization explicitly:
    from base64 import b64encode

class ProxyAuthMiddleware:
    def process_request(self, request, spider):
        proxy = request.meta.get("proxy")
        if not proxy or "@" not in proxy:
            return
        creds = proxy.split("//", 1)[1].split("@", 1)[0]
        token = b64encode(creds.encode()).decode()
        request.headers["Proxy-Authorization"] = f"Basic {token}"

    scrapy 2.11+ handles this automatically. older versions need this snippet.

    benchmark: 10,000 pages with rotating proxies

    across 10,000 pages of a tolerant ecommerce target, with a 50-proxy residential pool, the configuration above completed in roughly 22 minutes on a single mac workstation. that is around 7.5 requests per second sustained.

    failed requests (mostly 503s) hit 4 percent. retries succeeded 92 percent of the time. proxies marked bad: 11 of 50 over the run. all 11 came back online within an hour as cooldown expired.

    scaling to 100,000 pages, the same config runs in 3 to 4 hours. for higher throughput, run multiple scrapy processes against the same proxy pool with a shared bad-proxy state stored in redis.

    production checklist

    four items separate hobby spiders from production scrapy deployments.

    shared bad-proxy state. for multi-worker setups, store the bad-proxy list in redis instead of in-process memory. otherwise each worker re-discovers the same dead proxies independently.

    per-domain proxy pools. for sites that ban entire ranges, segment your proxy pool by target domain. keep a clean residential pool for hard targets and reuse a cheaper datacenter pool for tolerant ones.

    playwright integration. for js-heavy targets, use scrapy-playwright. it integrates with the rotating middleware via request.meta["playwright_context_kwargs"]["proxy"].

    logging. log every request with proxy, status, latency, and final response code. for postmortems on broken scrapes, this is what you analyze.

    for the broader python scraping context see our web scraping with python guide.

    faq

    what is the easiest way to add a proxy in scrapy?

    set request.meta["proxy"] = "http://user:pass@host:port" per request. scrapy’s built-in httpproxymiddleware handles the rest. enabled by default.

    does scrapy support proxy rotation out of the box?

    no. scrapy’s httpproxymiddleware uses one proxy per request based on request.meta. for rotation across requests, write a downloader middleware (full code in this tutorial) or install scrapy-rotating-proxies from pypi.

    how do i use socks5 proxies with scrapy?

    scrapy supports socks5 via twisted. use socks5://user:pass@host:port in request.meta["proxy"]. older scrapy versions need pip install txsocksx for full socks5 support.

    why am i getting 407 errors with scrapy proxies?

    usually wrong credentials format. confirm http://user:pass@host:port exactly. for residential providers using session-id auth, paste the full session-username (e.g. user-session-abc123) in the user field.

    should i use scrapy-rotating-proxies or write my own middleware?

    scrapy-rotating-proxies is fine for simple rotation. for production with custom health checks, sticky sessions, or per-domain pools, write your own. the middleware in this tutorial is around 50 lines and gives full control.

    how do i debug scrapy proxy issues?

    run with -L DEBUG to see every request and proxy assignment. log the response status and request.meta["proxy"] in your spider’s parse methods. for tls or auth issues, run the same proxy against curl -x first to isolate scrapy from the proxy itself. official docs at the scrapy reference.

    the bottom line

    scrapy’s proxy story is built on three pieces: per-request meta, the built-in httpproxymiddleware, and your custom rotating middleware. with the middleware in this tutorial plus header rotation and autothrottle, you have a production-grade scraper that survives dead proxies, rate limits, and the long tail of target-specific failures.

    for jobs above 100,000 pages or with strict anti-bot, pair this stack with residential proxies and a shared redis bad-proxy state. for lighter jobs, the in-process version above is enough.

    start with the per-request pattern, add the rotating middleware once you have more than 5 proxies, and add sticky sessions when you hit your first login flow. each layer composes cleanly with scrapy’s existing machinery.

  • Proxy Rotation with Python: aiohttp, httpx, and requests Compared (2026)

    proxy rotation with python: aiohttp, httpx, and requests compared (2026)

    proxy rotation in python boils down to picking a proxy per request, retrying on failure, and tracking which proxies still work. requests is the simplest, httpx is the modern sync+async pick, and aiohttp is the fastest at scale. across 1000 requests against a residential pool, aiohttp finished in 14 seconds, httpx in 19 seconds (async mode), and requests in 142 seconds (single-thread). pick the library based on concurrency needs, not the rotation logic itself.

    this tutorial gives you working code for all three, plus retry, sticky sessions, and a benchmark you can run yourself.

    the basic rotation pattern

    every proxy rotation script follows the same shape:

    1. load proxy list (file, env, or api).
    2. on each request, pick the next proxy (round-robin or random).
    3. catch errors. on failure, mark the proxy bad and retry with another.
    4. for sticky sessions, hash the target url or session-id to a fixed proxy.

    we will implement this in three libraries.

    requests: simplest, blocking

    requests is the right choice when you have under 50 requests per minute, no async constraints, and want minimal dependencies.

    import requests
import random
import time
from itertools import cycle

PROXIES = [
    "http://user:pass@1.2.3.4:8080",
    "http://user:pass@5.6.7.8:8080",
    "http://user:pass@9.10.11.12:8080",
]

def rotate_get(url, max_retries=3, timeout=10):
    proxies_iter = cycle(random.sample(PROXIES, len(PROXIES)))
    last_err = None

    for _ in range(max_retries):
        proxy = next(proxies_iter)
        try:
            r = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=timeout,
            )
            r.raise_for_status()
            return r
        except Exception as e:
            last_err = e
            time.sleep(0.5)

    raise last_err

resp = rotate_get("https://httpbin.org/ip")
print(resp.json())

    this gives you round-robin rotation with 3-retry fallback. at 142 seconds for 1000 requests, it works for low-volume jobs.

    httpx: modern, sync or async

    httpx supports the same api as requests but adds full async support and http/2. for new code in 2026, prefer httpx over requests.

    import httpx
import asyncio
import random

PROXIES = [
    "http://user:pass@1.2.3.4:8080",
    "http://user:pass@5.6.7.8:8080",
]

async def fetch(url, max_retries=3):
    for _ in range(max_retries):
        proxy = random.choice(PROXIES)
        try:
            async with httpx.AsyncClient(
                proxy=proxy,
                timeout=10,
                http2=True,
            ) as client:
                r = await client.get(url)
                r.raise_for_status()
                return r.json()
        except Exception:
            await asyncio.sleep(0.3)
    raise RuntimeError("all retries failed")

async def main():
    urls = [f"https://httpbin.org/anything?i={i}" for i in range(50)]
    results = await asyncio.gather(*[fetch(u) for u in urls])
    print(f"fetched {len(results)} urls")

asyncio.run(main())

    httpx in async mode finished our 1000-request benchmark in 19 seconds. for sync mode, swap httpx.AsyncClient for httpx.Client and drop await.

    aiohttp: fastest at scale

    aiohttp is the highest-throughput async library in python. for any job above 100 requests per second, it beats httpx in our benchmarks.

    import aiohttp
import asyncio
import random

PROXIES = [
    "http://user:pass@1.2.3.4:8080",
    "http://user:pass@5.6.7.8:8080",
]

async def fetch(session, url, max_retries=3):
    for _ in range(max_retries):
        proxy = random.choice(PROXIES)
        try:
            async with session.get(
                url,
                proxy=proxy,
                timeout=aiohttp.ClientTimeout(total=10),
            ) as r:
                r.raise_for_status()
                return await r.json()
        except Exception:
            await asyncio.sleep(0.3)
    raise RuntimeError("all retries failed")

async def main():
    connector = aiohttp.TCPConnector(limit=100)
    async with aiohttp.ClientSession(connector=connector) as session:
        urls = [f"https://httpbin.org/anything?i={i}" for i in range(1000)]
        results = await asyncio.gather(*[fetch(session, u) for u in urls])
        print(f"fetched {len(results)} urls")

asyncio.run(main())

    aiohttp finished 1000 requests in 14 seconds in our test. the TCPConnector(limit=100) controls max concurrent connections; tune this based on your proxy pool size and target site rate limits.

    for the scrapy ecosystem variant see our scrapy proxy middleware tutorial.

    proxy health tracking

    production scrapers need to drop dead proxies, not retry them forever. add a health-tracker:

    import time
from collections import defaultdict

class ProxyPool:
    def __init__(self, proxies, cooldown_sec=300):
        self.proxies = list(proxies)
        self.cooldown_sec = cooldown_sec
        self.bad_until = defaultdict(float)
        self.fail_count = defaultdict(int)

    def get(self):
        now = time.time()
        live = [p for p in self.proxies if self.bad_until[p] < now]
        if not live:
            # everything cooling down. reset and try again
            self.bad_until.clear()
            live = self.proxies
        return random.choice(live)

    def mark_bad(self, proxy):
        self.fail_count[proxy] += 1
        # exponential cooldown: 5 min, 25 min, 125 min...
        cooldown = self.cooldown_sec * (5 ** (self.fail_count[proxy] - 1))
        self.bad_until[proxy] = time.time() + cooldown

    def mark_good(self, proxy):
        self.fail_count[proxy] = 0
        self.bad_until[proxy] = 0

    drop this into any of the rotation patterns above. on success call pool.mark_good(proxy); on failure call pool.mark_bad(proxy).

    for residential pools that rotate the underlying ip on every request, proxy health is less of an issue. for static datacenter pools, this pattern is critical.

    sticky sessions

    some scraping targets break if you switch ip mid-session (login flows, multi-page checkout, captcha challenges). pin the proxy to a session id:

    import hashlib

def sticky_proxy(session_id, proxies):
    h = hashlib.md5(session_id.encode()).hexdigest()
    idx = int(h, 16) % len(proxies)
    return proxies[idx]

# same session_id always gets same proxy
proxy = sticky_proxy("user_abc_session_123", PROXIES)

    for residential providers that natively support sticky sessions (smartproxy, oxylabs, soax), pass the session-id inside the username field instead:

    proxy = f"http://user-session-{session_id}:pass@proxy.example.com:7777"

    this leans on the provider to keep the session pinned for 1 to 30 minutes (varies by provider). it is cleaner than building your own sticky logic.

    for the proxy types that pair best with rotation see rotating proxies with unlimited bandwidth.

    benchmark: 1000 requests against httpbin.org/anything

    we ran each library against https://httpbin.org/anything 1000 times through a residential pool of 50 proxies, on a 4-core mac, with 100 concurrent connections.

    library mode time requests/sec
    requests sync, single-thread 142s 7
    requests sync, threadpool 50 18s 56
    httpx async 19s 53
    aiohttp async 14s 71

    for blocking single-threaded code, aiohttp is 10x faster than requests. with a threadpool wrapping requests, the gap closes to 1.3x. for new code, async is the right choice; the difference is library polish.

    error handling cheatsheet

    error usual cause fix
    ProxyError, ConnectionRefusedError proxy is dead mark bad, rotate
    ReadTimeout proxy is slow or target is slow retry with longer timeout
    407 Proxy Authentication Required wrong user/pass check credentials
    403 Forbidden from target ip flagged rotate to fresh ip
    429 Too Many Requests from target hit target rate limit back off, slower rotation
    SSL: WRONG_VERSION_NUMBER http proxy with https:// scheme use http:// for proxy url, even for https targets

    the last one bites everyone once. the proxy url scheme refers to the proxy protocol, not the target. for an http proxy use http://user:pass@ip:port regardless of whether the target is http or https.

    production patterns

    three patterns separate hobby scrapers from production.

    queue-driven workers. instead of looping through urls in-line, push them to a redis queue and run aiohttp workers that pop, fetch, and push results. survives crashes and scales horizontally.

    per-target rate limits. one global concurrency limit is wrong. add per-domain semaphores so a slow target does not starve a fast one.

    observability. log every request with proxy, status, latency. when scraping breaks, you need to know if proxies are dying or if the target changed.

    for the full python scraping stack see our web scraping with python guide.

    faq

    which python library is fastest for proxy rotation in 2026?

    aiohttp leads in our benchmark at 71 requests per second, followed by httpx async at 53 and requests with threadpool at 56. for new code, both aiohttp and httpx are good picks. requests still works for low-volume jobs.

    do i need a rotating proxy provider or can i build rotation myself?

    if you have a static list of proxies, build rotation in your code. if you want auto-rotation on every request from a residential pool, providers like smartproxy, oxylabs, and bright data handle it server-side. either approach works; the choice is operational, not technical.

    how often should i rotate proxies?

    every request for one-shot scrapes, every 1 to 30 minutes for session-based flows. for login or checkout flows, pin the proxy for the duration of the session.

    how do i detect a dead proxy?

    connection errors, 407 auth errors, and timeouts longer than 10 seconds. use exponential cooldowns (5 min first, 25 min second, 125 min third) so a transient blip does not permanently kill a good proxy.

    should i use http or socks5 proxies for python scraping?

    http is fine for most scraping (https included). socks5 only matters when you need to tunnel non-http traffic or when the proxy is socks5-only. requests, httpx, and aiohttp all support socks5 via pip install httpx[socks] or the aiohttp-socks extension.

    where do i find documentation for these libraries?

    official docs: requests, httpx, aiohttp. all three are actively maintained in 2026.

    the bottom line

    proxy rotation in python is 30 lines of code plus a health tracker. the library choice matters less than getting retry, cooldown, and sticky-session logic right.

    for jobs under 100 requests per minute, requests with a threadpool is the simplest. for everything else, aiohttp gives the best throughput. httpx sits in the middle with a friendlier api and full async support.

    start with the patterns above, add health tracking when you hit your first dead-proxy incident, and add per-domain rate limiting when you scrape multiple targets in parallel. the rest is operational discipline, not code.

  • best proxies for browser use and AI agents (2026)

    Best Proxies for Browser Use, Operator & Agentic AI Tools

    The rise of agentic AI tools has created a new category of web automation. Tools like Browser Use, OpenAI Operator, and similar AI-driven browser agents can navigate websites, fill out forms, extract data, and complete complex multi-step tasks autonomously. But there is a problem: these agents hit the same anti-bot defenses that block traditional scrapers, often even faster because their browsing patterns differ from human users.

    looking for premium 4G/5G IPs? our Singapore mobile proxies for AI agents start at $40/month for 200GB.

    Proxies are the missing piece that makes agentic AI tools work reliably at scale. This guide covers the best proxy strategies for the leading agentic AI browser tools in 2026, with practical setup guides and configuration examples.

    What Are Agentic AI Browser Tools?

    Agentic AI browser tools combine large language models with browser automation. Instead of writing step-by-step scripts, you describe a task in natural language and the AI agent figures out how to navigate the web to accomplish it.

    Browser Use

    Browser Use is an open-source framework that connects LLMs to browser automation. It interprets web pages visually and through the DOM, then decides what actions to take (click, type, scroll, navigate). It is popular among developers building custom AI automation workflows.

    Key features:

    • Open-source and self-hosted
    • Works with multiple LLM providers (OpenAI, Anthropic, local models)
    • Full control over browser configuration, including proxy settings
    • Supports headless and headed browser modes
    • Active community and rapid development

    OpenAI Operator

    OpenAI Operator is a commercial agentic browsing product that uses GPT models to navigate the web on behalf of users. It handles tasks like booking reservations, filling out applications, and researching products.

    Key features:

    • Hosted service with built-in browser infrastructure
    • Uses computer vision to understand web pages
    • Handles authentication and multi-step workflows
    • Less control over underlying browser configuration compared to self-hosted tools

    Other Notable Agentic Tools

    • Anthropic Computer Use — Claude-based agent that can control a full desktop environment
    • Microsoft Copilot Actions — AI agent integrated with Microsoft ecosystem
    • AgentGPT / AutoGPT — Open-source autonomous AI agents that can browse the web
    • Multion — AI browser agent focused on personal assistant tasks
    • Browserbase — Infrastructure platform for running AI browser agents at scale

    Why Agentic AI Tools Need Proxies

    Problem 1: IP-Based Blocking

    AI agents make many requests in sequence. Even when they browse at human-like speeds, the volume and patterns of their requests differ from natural human browsing:

    • Multiple sequential visits to the same domain
    • Systematic navigation patterns (e.g., visiting every product in a category)
    • Requests from datacenter IPs if running on cloud infrastructure
    • Lack of browsing history, cookies, and other signs of an established user

    Websites detect these patterns and block the offending IP address.

    Problem 2: Geo-Restricted Content

    Many use cases for agentic AI involve accessing content specific to a particular location:

    • Price checking on regional e-commerce sites
    • Researching local business listings
    • Accessing geo-restricted services
    • Comparing offerings across different markets

    Without a proxy in the target location, the agent sees the wrong content or gets blocked entirely.

    Problem 3: Rate Limiting

    Websites impose rate limits to prevent abuse. An AI agent completing a task might need to load dozens of pages on the same site, quickly exceeding the rate limit for a single IP address.

    Why Mobile Proxies Are the Best Choice

    Proxy Type Detection Risk Geo Accuracy Cost Best For
    Datacenter High Low Low Non-sensitive tasks
    Residential Medium Medium Medium General automation
    Mobile Very Low High Higher Anti-detection critical tasks

    Mobile proxies provide IPs from real mobile carriers, which websites trust because they are used by thousands of real users. For agentic AI tools that need to interact with websites without being blocked, mobile proxies offer the lowest detection risk.

    Setting Up Proxies with Browser Use

    Browser Use gives you full control over the browser configuration, making proxy integration straightforward.

    Basic Proxy Configuration

    from browser_use import Agent
from langchain_openai import ChatOpenAI

# Configure the agent with a mobile proxy
agent = Agent(
    task="Find the top 5 rated restaurants in Singapore on Google Maps",
    llm=ChatOpenAI(model="gpt-4o"),
    browser_config={
        "proxy": {
            "server": "http://gate.dataresearchtools.com:PORT",
            "username": "your_username",
            "password": "your_password"
        },
        "headless": True,
        "viewport": {"width": 412, "height": 915}
    }
)

result = await agent.run()

    Rotating Proxies for Multi-Step Tasks

    For tasks that involve visiting many pages, rotate the proxy between major task segments:

    from browser_use import Agent, BrowserConfig

# Define proxy endpoints for different SEA countries
proxies = {
    "SG": "http://user:pass@sg.dataresearchtools.com:PORT",
    "MY": "http://user:pass@my.dataresearchtools.com:PORT",
    "TH": "http://user:pass@th.dataresearchtools.com:PORT",
    "PH": "http://user:pass@ph.dataresearchtools.com:PORT",
    "ID": "http://user:pass@id.dataresearchtools.com:PORT",
}

async def run_task_per_country(task, country_code):
    config = BrowserConfig(
        proxy={"server": proxies[country_code]},
        headless=True
    )
    agent = Agent(
        task=f"{task} (searching from {country_code})",
        llm=ChatOpenAI(model="gpt-4o"),
        browser_config=config
    )
    return await agent.run()

# Run the same task across multiple geos
for country in ["SG", "MY", "TH", "PH", "ID"]:
    result = await run_task_per_country(
        "Find the best mobile phone deals under $500",
        country
    )

    Advanced Browser Fingerprinting

    Pair your proxy with matching browser fingerprints for maximum stealth:

    • Match the browser language to the proxy country
    • Set timezone to match the proxy location
    • Use a mobile user agent consistent with the proxy carrier’s region
    • Configure WebRTC to prevent IP leaks
    • Set geolocation API to match the proxy’s approximate location

    Setting Up Proxies with OpenAI Operator

    OpenAI Operator is a hosted service with less direct proxy control. However, there are strategies to incorporate proxies:

    Using Operator Through a Proxy Gateway

    If you access Operator’s API programmatically, route the requests through a proxy:

    • Configure a local proxy gateway that forwards Operator’s browser traffic through your mobile proxy
    • Use network-level proxy settings to route traffic

    Alternative: Self-Hosted Agents with Proxy Support

    For full proxy control, consider using the open-source Computer Use or Browser Use frameworks instead of Operator, and configure proxies directly:

    • Self-hosted solutions give you complete control over the network stack
    • You can configure proxy rotation, geo-targeting, and session management exactly as needed
    • Run on your own infrastructure with DataResearchTools mobile proxies for SEA coverage

    Proxy Configuration for Other Agentic Tools

    Anthropic Computer Use

    Anthropic’s Computer Use feature allows Claude to control a virtual desktop. To add proxy support:

    • Configure the system-level proxy settings in the virtual machine
    • Set environment variables for HTTP_PROXY and HTTPS_PROXY
    • The browser within the VM will route traffic through the configured proxy

    AutoGPT / AgentGPT

    These open-source agents can be configured with proxy support:

    # .env configuration for AutoGPT
PROXY_URL=http://user:pass@gate.dataresearchtools.com:PORT
PROXY_ROTATION=true

    Multion

    Multion operates as a browser extension and API. Proxy integration options:

    • Use a proxy extension alongside Multion in the browser
    • Configure system-level proxy settings
    • Route traffic through a proxy-enabled VPN

    Best Practices for Proxy Use with AI Agents

    1. Match Proxy Location to Task Context

    If your agent is researching Singapore restaurant prices, use a Singapore mobile proxy. If it is checking Thai e-commerce listings, use a Thai proxy. Mismatched geos produce incorrect results and may trigger detection.

    2. Use Sticky Sessions for Multi-Page Tasks

    AI agents often need to browse multiple pages on the same site during a single task. Use sticky sessions (same IP for 10-30 minutes) to maintain consistency:

    • Avoids triggering “new visitor” detection on every page load
    • Maintains session cookies and login state
    • Reduces the risk of mid-task IP changes causing errors

    3. Implement Intelligent Rotation

    Rotate IPs between tasks, not during tasks:

    • Good: Complete Task A with IP 1, then switch to IP 2 for Task B
    • Bad: Rotate IPs every 30 seconds during a single multi-page task

    4. Handle Proxy Failures Gracefully

    AI agents should be configured to handle proxy connection issues:

    • Retry with a different proxy if the current one fails
    • Log proxy errors separately from task errors
    • Set reasonable timeouts (30-60 seconds for page loads through proxies)
    • Fall back to alternative proxy geos if the primary one is unavailable

    5. Monitor Proxy Usage

    Track proxy consumption to optimize costs:

    • Log which tasks consume the most bandwidth
    • Identify tasks that could be done without proxies (e.g., accessing APIs that do not geo-restrict)
    • Monitor success rates by proxy geo and carrier

    6. Respect Website Policies

    Even with proxies, AI agents should:

    • Follow robots.txt directives
    • Implement reasonable request delays
    • Avoid overloading target websites
    • Not bypass authentication or access control mechanisms

    Common Use Cases for Proxied AI Agents in SEA

    E-Commerce Price Monitoring

    AI agents browse e-commerce sites across SEA markets to collect pricing data:

    • Compare product prices on Shopee SG vs. Shopee MY vs. Shopee TH
    • Monitor competitor pricing across Lazada and Tokopedia
    • Track flash sale prices in real time

    Market Research

    Agents research local markets for business intelligence:

    • Gather business listings and reviews from each SEA country
    • Collect job posting data from local job boards
    • Survey local news and industry publications

    Travel and Hospitality

    Agents check travel-related services across markets:

    • Compare flight and hotel prices shown to users in different countries
    • Monitor booking platform availability from different geos
    • Research local experience and tour offerings

    Content Verification

    Agents verify content compliance across markets:

    • Check that localized websites display correct content in each country
    • Verify that age-restricted content is properly gated by geo
    • Ensure regulatory compliance for financial services websites in each jurisdiction

    Performance Optimization

    Reducing Latency

    Mobile proxies add latency to every request. Optimize by:

    • Using proxy servers geographically close to the target website
    • Implementing connection pooling to reuse proxy connections
    • Pre-warming proxy connections before the agent starts its task
    • Choosing proxy providers with low-latency infrastructure in SEA (DataResearchTools maintains proxy infrastructure across the region)

    Reducing Bandwidth

    AI agents can consume significant bandwidth, especially with vision-based tools that load full page resources:

    • Disable image loading for tasks that do not require visual analysis
    • Block unnecessary third-party resources (analytics, tracking pixels)
    • Use content extraction APIs where available instead of full page rendering
    • Cache resources that do not change between tasks

    Parallelizing Tasks

    Run multiple agent instances with different proxies to parallelize multi-market tasks:

    • Each instance uses a different geo proxy
    • Aggregate results after all instances complete
    • Use a task queue to manage agent workloads across available proxy slots

    Conclusion

    Agentic AI tools are transforming web automation, but they need proxy infrastructure to work reliably. Mobile proxies provide the trusted IP addresses, geo-targeting capabilities, and anti-detection properties that these tools require. Whether you are using Browser Use for custom automation, exploring Operator for task completion, or building with any other agentic framework, integrating mobile proxies from a provider with strong Southeast Asian coverage like DataResearchTools ensures your AI agents can access the web without interruption. Start with a single use case and proxy configuration, verify it works end-to-end, and then scale your setup as your automation needs grow.

    Related Reading