Scrapy middleware order: how the downloader stack actually runs

Why this trips people up in production

Most Scrapy bugs that look like “the proxy isn’t rotating” or “retries aren’t working” are order bugs. Someone wrote a custom middleware, dropped it into DOWNLOADER_MIDDLEWARES with a number that felt reasonable, and never checked where that number put it relative to the built-ins that were already doing work. The spider runs fine for an hour, then a batch of responses comes back with a stale proxy header, or a retry fires without ever touching the rotation logic, and nobody can explain why from reading the spider code alone. You have to read the middleware stack, in order, to see it.

This is worth understanding properly because middleware order is the actual execution model. Get it backwards and you’ll ship a pipeline that looks correct in a two-minute test run and quietly degrades over a long crawl.

Two separate stacks

Scrapy has two middleware systems that people conflate constantly: downloader middlewares and spider middlewares. They’re configured in different settings (DOWNLOADER_MIDDLEWARES and SPIDER_MIDDLEWARES) and they sit in different places in the request/response lifecycle.

Downloader middlewares wrap the actual HTTP call. This is where proxy assignment, header injection, retries, redirects, and compression live. If you’re building proxy infrastructure or rotation logic, this is the stack you care about.

Spider middlewares wrap the input to and output from your spider’s parse callbacks. This is where things like OffsiteMiddleware (dropping requests outside allowed_domains) and DepthMiddleware live. Less relevant to proxy work, but the same ordering rules apply, and we’ll come back to it briefly.

The priority number, and why the direction flips

Every entry in DOWNLOADER_MIDDLEWARES is a class mapped to an integer:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyRotationMiddleware": 350,
    "myproject.middlewares.RetryCounterMiddleware": 590,
}

Lower numbers sit closer to the engine, higher numbers sit closer to the downloader (the component that actually opens the socket). That’s the part most people remember. What trips people up is that the direction of travel isn’t the same for requests and responses.

For process_request, Scrapy walks the stack from low number to high number, on the way out to the downloader.

For process_response and process_exception, Scrapy walks it in reverse, from high number to low, on the way back in.

So a middleware at 350 sees a request before a middleware at 590 does. But on the response side, the 590 middleware sees the response first, and the 350 middleware sees it last. Picture it as a request passing through a stack of layers to reach the downloader, then the response climbing back out through the same layers in reverse. It’s symmetric, not sequential.

This matters immediately once you have more than one middleware doing something to the same request or response. If your proxy assignment middleware sits at a lower number than your retry middleware, the proxy gets attached before the retry decision is made on the way out, which is what you want. But on the way back, the retry middleware (higher number) sees the response first and decides whether to retry before the proxy middleware ever sees it. If your retry logic needs to know which proxy just failed so it can mark it dead or pick a different one next time, and that bookkeeping lives in the proxy middleware instead of the retry middleware, the order means it never gets the chance.

Where Scrapy’s own middlewares sit

Scrapy ships a default DOWNLOADER_MIDDLEWARES_BASE that’s already populated, and your custom entries get merged into it. The stock order (numbers as shipped, roughly, since they do shift slightly between versions) runs something like:

• RobotsTxtMiddleware, 100
• HttpAuthMiddleware, 300
• DownloadTimeoutMiddleware, 350
• DefaultHeadersMiddleware, 400
• UserAgentMiddleware, 500
• RetryMiddleware, 550
• AjaxCrawlMiddleware, 560
• MetaRefreshMiddleware, 580
• HttpCompressionMiddleware, 590
• RedirectMiddleware, 600
• CookiesMiddleware, 700
• HttpProxyMiddleware, 750
• HttpCacheMiddleware, 900

Two things stand out if you’re building proxy or rotation logic on top of this.

First, HttpProxyMiddleware is deliberately near the end of the outbound chain, at 750. That’s late. It reads request.meta['proxy'] (or the http_proxy/https_proxy environment variables) and sets the actual proxy on the request right before it hits the downloader. If your custom proxy-selection middleware runs after 750, it’s too late, the request has already been dispatched with whatever HttpProxyMiddleware picked. Custom proxy middlewares almost always need a number lower than 750 so they can set request.meta['proxy'] before HttpProxyMiddleware reads it.

Second, RetryMiddleware sits at 550, well before RedirectMiddleware at 600 and HttpProxyMiddleware at 750. On the response side, because of the reversal, that means RetryMiddleware (550) sees the response after HttpProxyMiddleware (750) and RedirectMiddleware (600) have already had their turn. Practically: redirects get followed and proxy-level response handling happens before Scrapy decides whether to retry. If you want a custom middleware to inspect a response and force a retry based on something proxy-specific, like a captive-portal page a bad exit node serves instead of the real site, that middleware needs to sit at a number lower than 550, so it runs its process_response after RetryMiddleware‘s and can override the outcome, or higher than 750 if you want first look at the raw response before anything else touches it. Which one you want depends on whether you’re trying to catch the failure before Scrapy’s retry logic fires, or reacting to it after.

Custom exception handling is where this bites hardest

A common pattern for people running rotating proxy pools is a middleware that catches connection failures, marks the proxy that failed, and re-queues the request through process_exception. This only works if the middleware is positioned so it actually gets the exception. process_exception follows the same high-to-low order as process_response. If a middleware closer to the downloader (higher number) already handled the exception and returned a Response or a new Request instead of letting it propagate, your lower-numbered middleware never sees it at all. That’s the stack working as designed, on a request you didn’t position correctly.

The fix is always the same: decide what your middleware needs to see and when, and pick its number based on that, not based on what number “felt free.” Read through the base middleware list, find the two Scrapy middlewares immediately above and below where you want to sit, and pick a number between them.

Spider middleware order, briefly

The spider middleware stack (SPIDER_MIDDLEWARES) follows the same reversal pattern. process_spider_input runs high to low as a response goes toward your parse callback, process_spider_output runs low to high as items and requests come back out. DepthMiddleware and OffsiteMiddleware sit in the defaults here, and if you’re adding something that filters or reshapes items before they hit your pipelines, the same “pick a number relative to what’s already there” logic applies.

A quick way to check your actual order

Don’t trust your own mental model of the merged stack, verify it. Scrapy will print the resolved middleware order at startup if you bump the log level, or you can inspect it directly:

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerRunner

settings = get_project_settings()
runner = CrawlerRunner(settings)

or more simply, run the spider with -s LOG_LEVEL=DEBUG and look for the “Enabled downloader middlewares” line in the startup log. It lists the final merged order, which is what actually runs, not what you assume from reading your settings file in isolation. When a pipeline that touches proxies, retries, and custom headers stops behaving the way you expect, that log line is the first thing to check, before you start adding print statements to the middleware itself.

If you’re building or debugging a scraping pipeline that leans on proxy rotation, Data Research Tools covers the infrastructure side of this in more depth, from middleware design to how proxy pools actually get run in production.

Get new guides and videos first — join the Telegram channel.