How to Use Proxies with Scrapy: Rotation and Retry Tutorial
TL;DR
Scrapy routes a request through a proxy when the request carries meta["proxy"], while environment variables can provide defaults for HTTP and HTTPS traffic.
A rotating gateway keeps spider code small; custom downloader middleware is useful when the application must choose, score, or retire individual proxy endpoints.
Proxy rotation does not replace request pacing or semantic validation: a 200 response can still be a block page, login wall, or incomplete result.
Retries should be bounded and should select a different endpoint only for retryable transport or target responses, with credentials loaded from environment variables.
Introduction: Why Scrapy proxy rotation belongs in downloader middleware
Scrapy sends requests through a downloader middleware chain, so proxy selection is easiest to reason about at the same boundary as timeouts, retries, and response handling. The built-in HttpProxyMiddleware reads http_proxy, https_proxy, and request.meta["proxy"]; a per-request value takes precedence over the environment. This tutorial uses an authorized, bounded request against a public test endpoint and keeps all credentials as placeholders.
For a provider-backed gateway, the provider can rotate the exit IP while Scrapy always connects to one configured endpoint. For a local pool, the spider or middleware chooses a proxy URL for each request. The second design gives more control but also creates health state, retry state, and secret-handling work.
The right Scrapy proxy model depends on whether rotation belongs to the provider or to your application.
Model
What Scrapy receives
Best fit
Main trade-off
Rotating gateway
One proxy URL
Most spiders and simple geo/session needs
Less application-level control
Explicit proxy list
A different URL in request metadata
Small, testable pools with custom selection
You own health and cooldown logic
Third-party rotating middleware
A configured pool plus middleware state
Teams that want health checks built in
Version and configuration coupling
The safest default is to start with a gateway, test it against the target you are authorized to access, and add application-side selection only when the workflow needs per-endpoint decisions.
Configure a single Scrapy proxy
Scrapy can use a single authenticated proxy through request.meta["proxy"] or the process environment.
Method 1: Set a proxy per request
This approach makes the routing decision visible at the request site:
import os
import scrapy
classIpCheckSpider(scrapy.Spider): name ="ip_check" start_urls =["https://httpbin.org/ip"]defstart_requests(self): proxy = os.environ["SCRAPY_PROXY_URL"]for url in self.start_urls:yield scrapy.Request(url, meta={"proxy": proxy})defparse(self, response):yield{"url": response.url,"body": response.text}
Use a value such as http://USERNAME:PASSWORD@HOST:PORT in SCRAPY_PROXY_URL; do not commit the real value. Scrapy's built-in middleware recognizes this metadata key, but support still depends on the selected download handler and proxy scheme.
Method 2: Set environment defaults
Environment variables are convenient when every outbound request in a run should use the same route:
Keep no_proxy for hosts that must remain direct, such as local services. A request-level meta["proxy"] overrides these defaults and ignores no_proxy for that request.
Nstdata proxy documentation is the place to confirm the current endpoint, protocol, session, and location settings for your account. Nstdata Residential Lite Proxies can be evaluated when the target needs rotating residential traffic for public data collection; verify the current channel details before using them.
Nstdata Residential Lite Proxies fit a cost-focused test when you need a managed residential route and do not want to maintain a local endpoint list. The product page should be treated as the source of current gateway and package details, not this article.
Nstdata also provides other proxy types when the target, session length, or workload changes the selection criteria.
The code marks a proxy unavailable after a downloader exception. It does not decide that every 403 or 200 is a proxy failure; that decision belongs in response validation, where you can distinguish a target's policy response from a malformed page.
Add bounded retries without looping forever
Retry logic should track attempts per request and stop when the spider has exhausted its retry budget.
In production, prefer Scrapy's configured retry middleware for standard HTTP retry behavior and use custom code only for a documented target-specific condition. Validate the response body as well as its status: a block template can return 200, while a transient upstream error can return a non-HTML body. Add logging for URL, status, retry count, and a redacted proxy identifier; never log the proxy password.
Troubleshoot the common failure modes
Most Scrapy proxy failures come from a mismatch between proxy selection, middleware order, and response interpretation.
The proxy is ignored: confirm the metadata key is exactly proxy, the value includes a scheme, and HttpProxyMiddleware remains enabled.
Authentication fails: URL-encode special characters in credentials, keep the scheme and port correct, and test the same endpoint outside Scrapy.
HTTPS requests fail: use the proxy scheme and download handler combination supported by your Scrapy version.
The same bad endpoint repeats: remove or cool down the failed endpoint before rescheduling, and cap retries per request.
A response is 200 but unusable: check required selectors, content type, canonical URL, and a small semantic marker before accepting the item.
The crawl overwhelms the target: reduce concurrency, obey robots and terms where applicable, and use DOWNLOAD_DELAY or AutoThrottle alongside proxy rotation.
Scrapy's middleware documentation explains that request middleware runs in increasing order and response middleware runs in decreasing order, so order is part of the behavior rather than cosmetic configuration.
Conclusion
Scrapy proxy rotation works when the route decision, health state, and retry budget are explicit. Start with a single gateway for a small authorized test, move to custom middleware only when you need local endpoint selection, and validate accepted content instead of treating an HTTP status as success. When a workflow needs managed geo or session controls, another relevant Nstdata capability to evaluate is its Proxy Manager, provided the current product surface matches the operational requirements.
Q: Can Scrapy use a proxy without custom middleware?
Yes. Set request.meta["proxy"] or the relevant environment variables and keep Scrapy's built-in HttpProxyMiddleware enabled.
Q: How do I rotate proxies in Scrapy?
Use a provider-managed rotating gateway or a downloader middleware that selects a proxy per request and retires failed endpoints.
Q: Should a retry always use a new proxy?
No. Change the route when the failure indicates a transport or endpoint problem; preserve the route when the workflow requires a sticky session or the failure is unrelated to the proxy.
Q: Why can a 200 response still be a failed scrape?
A 200 response can contain a block page or incomplete markup, so validate content type, required fields, and a semantic marker before yielding an item.
Q: Can I use Scrapy proxy rotation for any website?
Use it only for public or otherwise authorized access, follow the target's terms and applicable law, and keep request volume bounded.
Marcus Chen
Sep. 9th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
