How to Use a Proxy with Python Requests (Tutorial)
TL;DR
Python Requests accepts a proxies mapping with separate http and https keys; the proxy URL scheme describes the proxy connection, not the destination.
Use a Session with trust_env = False when you need to prevent ambient proxy environment variables from changing the route.
Proxy credentials belong in the proxy URL or a secret-backed configuration, not in requests.get(auth=...), which authenticates to the destination.
Set connect and read timeouts, call raise_for_status(), and validate the returned content. A successful proxy connection is not a valid scrape.
Nstdata’s generated Proxy URL supplies the actual Host, Port, Username, Password, and product parameters; do not guess gateway values.
What Do You Need Before Using a Proxy with Python Requests?
You need Python, Requests, a permitted test URL, and an active proxy configuration containing Protocol, Host, Port, Username, and Password. For Nstdata, create a Channel, enable Proxy Generation, configure its parameters, and copy the generated Proxy URL from the official Channel workflow. The Nstdata proxy guide documents the gateway username’s Channel, proxy type, country, session duration, and session ID components. Keep the password in a secret manager and never commit a full proxy URL.
Install Requests in a virtual environment with python -m pip install requests. If your actual proxy is SOCKS5, install python -m pip install 'requests[socks]'; the distinguishes client-side DNS from proxy-side DNS. Choose the scheme the provider documents. The examples below assume an HTTP proxy URL copied into by your runtime’s secret injection. They cannot be executed against Nstdata without your credential, so the output checks are shown as instructions rather than fabricated results.
How Does Requests Route HTTP and HTTPS Through a Proxy?
Requests routes a destination through the proxy selected by the http or https mapping key. For an HTTPS destination, the value may still begin with http:// when the gateway is an HTTP proxy. The client first connects to the gateway, then establishes the destination tunnel. A 407 response is proxy authentication failure; a destination 401 usually concerns the target account.
Requests can also inherit HTTP_PROXY, HTTPS_PROXY, and NO_PROXY from the environment. Its own documentation warns that a Session.proxies mapping may be overwritten by environment settings. Explicit per-request proxies or trust_env = False make the routing decision easier to reproduce. A corporate proxy or VPN can still affect network behavior outside Python, so verify the observed exit IP.
Method 1: How Do You Set a Proxy for One Request?
A per-request mapping is the clearest choice for a single authorized check because the route is visible at the call site.
Step 1: Load the generated URL from a secret
Read PROXY_URL at runtime. It should have the documented protocol and generated Host, Port, Username, and Password. The environment variable is an example transport; a managed secret store is preferable for production.
Step 2: Pass both destination schemes
import os
import requests
proxy_url = os.environ["PROXY_URL"]# e.g. http://<user>:<password>@<host>:<port>proxies ={"http": proxy_url,"https": proxy_url}response = requests.get("https://api.ipapi.is/", proxies=proxies, timeout=(5,20),)response.raise_for_status()print(response.text[:300])# Inspect the returned exit IP and location.
The target is the IP-check endpoint used in Nstdata’s documentation. Replace it with a URL you are permitted to access after the route is verified.
Step 3: Validate the exit and content
Confirm that the observed exit IP and location match the job’s requirement. For a content page, check the expected title or fields, not only HTTP 200. Stop if the response is a challenge, wrong locale, or unexpected redirect.
Method 2: How Do You Reuse a Proxy Across a Session?
A Requests Session reuses connection state for related calls; it does not guarantee that the provider will preserve one exit IP unless the proxy session policy does too.
Step 1: Create a session without ambient proxies
Set trust_env = False if the job must use only the supplied proxy. This also disables environment-based configuration such as .netrc; use explicit target authentication if needed.
The example URLs are placeholders; use approved endpoints. Passing proxies explicitly on each call avoids the documented environment-precedence surprise.
Step 3: Check whether continuity is required
If these calls form one stateful workflow, configure a supported sticky session in the Nstdata generated username and verify the exit IP. A Requests Session alone only manages client-side connection and cookies.
Method 3: How Do You Rotate Between Independent Requests?
Rotation is appropriate for independent, permitted requests when a fresh exit is required by the job design. With Nstdata, use the generated proxy format and a documented new session ID, or select the provider’s rotation option; do not invent a gateway URL.
Step 1: Define a bounded workload
Choose a small set of authorized URLs, a rate ceiling, and an acceptance check. Separate independent requests from multi-step flows that need a sticky exit.
Step 2: Generate the proper proxy configuration
Use Channel → Proxy Generation for the intended country and session duration. If your application constructs usernames programmatically, follow the exact documented parameter order and URL-encode credentials; provider-generated URLs are less error-prone.
Step 3: Record and validate each result
Log a redacted session identifier, destination, HTTP status, elapsed time, and accepted/rejected result. Do not log passwords or full proxy URLs. If many requests fail, stop and diagnose rather than increasing rotation frequency.
Proxy failures should be separated from target and content failures. A 407 suggests bad proxy credentials, an unapproved source IP if IP allowlisting is used, or an inactive Channel. A connect timeout suggests the wrong Host or Port, blocked egress, or a gateway issue. A target 403 or 429 requires reviewing the source’s policy and request rate. A 200 response with the wrong page is a content-validation failure.
The Nstdata troubleshooting guide covers 407 and connectivity. Keep retries bounded and inspect the response body before treating any request as accepted. For more routing control across applications, Nstdata Proxy Manager is a separate operations layer; it does not change Requests’ API.
What Should You Do Next?
Start with Method 1 against an IP-check endpoint, then move to a small authorized page sample. Use Method 2 for related requests and Method 3 only when independent requests justify rotation. Track cost per accepted page, not only proxy connection success. If the job is site discovery and Markdown conversion rather than a custom Requests client, Nstdata Crawl is a separate Web Crawling API for LLM-ready data.
