Scrapy’s built-in HttpProxyMiddleware reads Request.meta["proxy"] or standard proxy environment variables; a custom middleware is only needed for selection policy or telemetry.
Set the proxy URL on the request before the built-in middleware handles it. A per-request meta["proxy"] overrides environment proxies and ignores no_proxy.
Rotate only among configured, authorized proxy routes and keep a stable route for stateful request chains.
Scrapy’s RetryMiddleware handles temporary errors separately from proxy selection. Do not retry 407 authentication failures blindly.
Nstdata supplies the generated proxy Host, Port, Username, Password, and session parameters; Scrapy does not generate those values.
What Do You Need Before Setting Up Scrapy Proxy Middleware?
You need a Scrapy project, a permitted target, and a working proxy URL from your provider. For Nstdata, create a Channel, enable Proxy Generation, select product and routing parameters, and copy the generated URL from Channel details. Keep the URL in a secret store. The Nstdata Proxy guide lists gateway Host, Port, Channel ID, password, country, session duration, and session ID. Do not substitute a guessed gateway or hard-code a real Channel password in settings.py.
Install Scrapy in a virtual environment with python -m pip install scrapy, then create a project with scrapy startproject proxy_demo. Put the examples below in the project’s existing settings and spider files. These snippets are documentation-backed but not a live Nstdata handshake: no Channel credential is available in this article.
Experience Nstproxy Proxy - Start Your Free Trial Today
How Does Scrapy Choose a Proxy for a Request?
Scrapy chooses a proxy through the request’s proxy meta key or through proxy environment variables handled by its built-in HttpProxyMiddleware. The Scrapy downloader middleware reference says a per-request value takes precedence over http_proxy and https_proxy, and it also ignores no_proxy. That precedence can surprise teams whose runtime has a corporate proxy configured.
A proxy URL usually starts with http:// even for an HTTPS destination. The scheme identifies the proxy connection, not the target. Scrapy’s built-in download handlers do not all support SOCKS; the current documentation names HttpxDownloadHandler as supporting SOCKS and notes a limitation for H2DownloadHandler. Use the protocol and handler combination supported by your deployment.
Method 1: How Do You Set One Proxy on a Request?
Set one proxy in the spider when the job has a fixed route and no selection logic.
Step 1: Load the URL from the runtime
Inject PROXY_URL through a secret manager. It should contain the generated protocol, Host, Port, Username, and Password. Do not print it.
Step 2: Attach it to the request
import os
import scrapy
classIpCheckSpider(scrapy.Spider): name ="ip_check"defstart_requests(self):yield scrapy.Request("https://api.ipapi.is/", meta={"proxy": os.environ["PROXY_URL"]}, callback=self.parse, dont_filter=True,)defparse(self, response): self.logger.info("status=%s url=%s", response.status, response.url)yield{"status": response.status,"body_sample": response.text[:200]}
Nstdata’s documentation uses the IP-check endpoint to inspect the exit. The sample body can reveal location, so handle it as diagnostic data and remove it from production logs if not needed.
Step 3: Verify the route
Run scrapy crawl ip_check -O ip-check.json in your authorized environment. Inspect the observed exit IP and region. Confirm that the request used the intended Channel without exposing the credential.
Method 2: How Do You Configure a Project-Wide Proxy?
A project-wide environment proxy works when every request should use the same route and there is no per-request exception.
Step 1: Supply the environment variable
Set https_proxy and, if required, http_proxy to the generated proxy URL through the job’s secret injection. Avoid a committed .env file containing credentials.
Step 2: Leave the built-in middleware enabled
Scrapy enables HttpProxyMiddleware by default under its standard settings. Do not replace DOWNLOADER_MIDDLEWARES_BASE; custom entries belong in DOWNLOADER_MIDDLEWARES. The official middleware reference documents the activation and ordering model.
Step 3: Check for per-request overrides
A spider request with meta["proxy"] overrides the environment route. Record which policy is active in your job configuration. If a job mixes proxied and direct requests, explicit per-request configuration is easier to audit.
Method 3: How Do You Rotate Proxies in Custom Middleware?
A custom downloader middleware can select from a bounded list of already generated, approved proxy URLs. It should not invent provider usernames or create unlimited sessions.
Step 1: Store approved routes as secrets
Inject a JSON array in PROXY_URLS_JSON. Each element should be a complete generated URL for a permitted route. In production, prefer a secret store over a plain environment dump.
Step 2: Add a selection middleware
import itertools
import json
import os
classProxySelectionMiddleware:def__init__(self): routes = json.loads(os.environ["PROXY_URLS_JSON"])ifnot routes:raise ValueError("PROXY_URLS_JSON must contain a route") self.routes = itertools.cycle(routes)defprocess_request(self, request, spider):if request.meta.get("keep_proxy"):returnNone request.meta["proxy"]=next(self.routes)returnNone
Configure it before Scrapy’s built-in proxy middleware:
The middleware selects a route, while Scrapy’s built-in handler performs the proxy connection. This code is illustrative and requires an actual project, routes, and authorized target to execute.
Step 3: Preserve stateful sessions and inspect retries
For a request chain that must retain the same exit, pass meta={"proxy": existing_url, "keep_proxy": True} to follow-up requests. Confirm the provider’s sticky-session policy separately. Scrapy’s RetryMiddleware documentation lists temporary response codes and a finite default retry budget; a retry may re-enter your selection middleware. Decide whether a retry should keep or change the route, then test that behavior with a controlled fixture. Never use rotation to disregard a target’s access policy.
Why Does a Scrapy Proxy Fail or Rotate Unexpectedly?
A Scrapy proxy can fail because of incorrect gateway credentials, middleware precedence, unsupported protocols, or target responses. A proxy 407 points to authentication or Channel state. A connection error suggests Host, Port, egress firewall, or protocol mismatch. A target 403 or 429 calls for source-policy and rate review. A 200 with a challenge page is a content failure, not a successful extraction. The Nstdata troubleshooting guide covers 407 and connectivity.
If rotation appears unexpected, log a redacted route ID and session ID for each request. Check whether process_request runs again on retries, whether a follow-up request lost keep_proxy, and whether the provider’s sticky duration expired. Scrapy statistics should distinguish attempted requests, proxy failures, target errors, and accepted items. Store accepted items with source URL and fetch time so an IP change does not hide duplicate collection.
Which Setup Should You Use?
Use per-request meta["proxy"] for explicit routing, environment variables for a uniform job, and a custom middleware when you need a tested selection policy. Start with one IP-check request, then a small authorized spider. Keep retry and content validation separate from route assignment.
Nstdata Proxy provides the network route; Nstdata Proxy Manager can centralize pools and diagnostics for multiple jobs. If you need managed site discovery and page conversion rather than a self-managed Scrapy pipeline, Nstdata Crawl is a separate Web Crawling API for LLM-ready data.
