Use requests and Beautiful Soup when a permitted page returns the required data in its initial HTML response.
Use a browser-rendering or managed Crawl API when JavaScript creates the content after the initial response; parsing an empty HTML shell will not fix a rendering problem.
Set timeouts, identify your client, pace requests by source, and treat 429 and other retryable errors explicitly.
Store source URL, fetch time, response status, parser version, and validation result with every extracted record.
Nstdata Crawl supplies site discovery, recursive crawl, JavaScript rendering, multiple output types, and built-in proxy routing; Python remains responsible for downstream parsing and quality checks.
What Do You Need Before Scraping a Website with Python?
You need Python 3, permission for the intended source and use, and a way to inspect the page's actual response. Create a virtual environment and install the two packages used in the static examples: The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
Experience Nstproxy Crawl - Start Your Free Trial Today
source
.venv/bin/activate
python
-m
pip
install
requests beautifulsoup4
On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. Use your browser's network panel to check whether the data appears in the document response, a later API response, or a rendered DOM. Check the source's terms and applicable robots rules before collecting data. The Robots Exclusion Protocol describes crawler access rules; it is not a content license.
If you use Nstdata, you also need an account and an API key. The documented header is x-api-key. The managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol values.
A scraper has four stages: request, parse, validate, and persist. The request stage obtains a response or rendered page. The parser maps page elements to fields. Validation rejects missing or implausible output. Persistence stores the accepted record and enough provenance to debug or delete it. Mixing all four in one expression makes failures difficult to locate.
HTML is not a stable database schema. A CSS selector can break when a site redesigns a card or inserts a consent screen. Give each parser a version and test it against representative page templates. For repeated runs, save a small sample of raw HTML or rendered output so you can distinguish a source change from a code regression.
The Requests quickstart documents response handling and request options used in the static method.
How Do You Scrape a Static Page with Requests and Beautiful Soup?
Use this method when the needed content is present in the initial HTML. The example uses example.com only as a connectivity and parsing demonstration; replace it with an approved source and selectors you have inspected.
Method 1: Fetch and Parse One HTML Page
Step 1. Fetch the page with a timeout and an honest User-Agent.
Step 2. Parse a field that exists in the returned document.
soup = BeautifulSoup(response.text,"html.parser")heading = soup.select_one("h1")if heading isNone:raise ValueError("Expected h1 was not found")record ={"source_url": response.url,"title": heading.get_text(" ", strip=True)}print(record)
Step 3. Save the result with a fetch timestamp and parser version. In production, also save the response status and a content hash. raise_for_status() catches HTTP errors, but it cannot tell whether a 200 response is a login page or a consent wall.
Limit:requests does not execute page JavaScript. An empty selector on a JavaScript page may mean the content never appeared in the HTTP response.
How Do You Extract Repeated Records Without Mixing Fields?
Scope each field lookup to one container element. Selecting all titles and all prices independently can pair the wrong values when cards have missing elements.
Method 2: Parse a List of Cards
Step 1. Inspect an approved page and identify a stable container selector plus the title, price, and link selectors within it. Replace the example selectors below with those observed on that page.
Step 2. Iterate containers and reject incomplete records.
from urllib.parse import urljoin
defparse_cards(html:str, page_url:str)->list[dict[str,str]]: soup = BeautifulSoup(html,"html.parser") items =[]for card in soup.select("article.product-card"): title = card.select_one(".product-title") price = card.select_one(".price") link = card.select_one("a[href]")ifnot(title and price and link):continue items.append({"title": title.get_text(" ", strip=True),"price_text": price.get_text(" ", strip=True),"url": urljoin(page_url, link["href"]),})return items
Step 3. Compare a sample of records with the page. Preserve price_text until you know the site's locale and currency; converting a displayed value to a number too early can lose meaning. Add a stable product identifier if the source exposes one.
Limit: These selectors are illustrative, not selectors for example.com or a promised schema on every store.
How Do You Scrape JavaScript Pages or an Entire Site?
Use a rendering and crawl layer when the initial HTML omits the content or when you need discovery across linked pages. Nstdata Crawl can render JavaScript and discover pages recursively. Its Crawl API documentation shows a single-page sync endpoint and a site-job endpoint. The response is provider-specific; inspect it before mapping to your application schema.
Step 1. Set your API key in the shell. Never hard-code it in a script or commit it to a repository.
exportNSTDATA_API_KEY="your-api-key"
Step 2. Submit a single approved page and inspect the JSON response. The endpoint and header follow the current Nstdata documentation; review the exact request body fields in your account's live docs before running.
Step 3. For a site crawl, submit a documented /api/v1/crawl job with bounded maxDepth, maxPages, includeUrls, and excludeUrls; then poll its status and retrieve pages. Validate each page before inserting it into your dataset. Large outputs may be referenced by a storage token instead of appearing inline.
Limit: Nstdata is an acquisition and conversion service, not an arbitrary business-schema extractor. Your Python code still decides which fields constitute an accepted record. The key is a prerequisite for a live call; the example above is not a claimed live test.
Why Does a Python Scraper Return Empty or Incorrect Data?
Symptom
Likely cause
Next check
Selector returns None
Wrong selector or JavaScript-rendered content
Inspect response.text and the browser network panel
200 with irrelevant text
Consent, login, or error page
Check title and required page elements
429 or repeated timeouts
Request rate or source overload
Slow down, honor Retry-After, and review allowed access
Duplicate records
Same page via query variants or repeated discovery
Define a canonical URL and source ID
Broken characters
Incorrect response encoding
Inspect headers, bytes, and document charset
Do not work around access controls to make a scraper pass. If a source requires authentication or restricts automated access, use a permitted API, obtain permission, or select a different source.
How Do You Make the Output Reliable Enough for an AI Product?
Record source_url, fetched_at, parser_version, and a content hash. Validate required fields, language, page type, and obvious error pages. Keep raw and processed artifacts separately so parser fixes can be replayed. For RAG, attach source URL and document version to every chunk and test that answers cite the current record. For model training, add rights and source provenance to the dataset manifest.
The Python urllib.robotparser documentation describes one way to read robots rules, but policy decisions and permission still belong to the application. Once one-page quality is stable, add bounded concurrency and durable job state; the companion asyncio guide covers that step.
How Do You Test a Python Scraper Before Scheduling It?
Build fixtures around page types, not just URLs. Include a normal page, a missing field, a redirect, an HTTP error, a consent or login page if one can appear, and a JavaScript shell when relevant. Save the source HTML in a controlled test location and assert that the parser either returns a valid record or an explicit rejection reason. A test that only checks whether requests.get() returned 200 will miss the failures most likely to contaminate a dataset.
Separate fetch tests from parser tests. A parser fixture should be replayable without network access so a selector change can be diagnosed quickly. A small live smoke test can confirm that the current source still matches the fixture. When a site changes, update the fixture and parser together after reviewing the new page; do not silently weaken validation until all pages pass.
For production runs, log counts by source and page type: submitted, fetched, rejected, accepted, retried, and terminal failures. Review a sample of accepted and rejected output after every deployment. A high acceptance rate is not automatically good if the validator accepts cookie banners as articles.
How Do You Choose Between Local Python and a Managed Crawl API?
Choose local fetching for stable static pages when your team can own request pacing, retries, extraction, and operations. Choose managed rendering or Crawl when the page depends on JavaScript, when discovery across a site is needed, or when browser and network maintenance would distract from the product schema. Test both paths on the same authorized sample and compare accepted records, diagnostic effort, and end-to-end lag.
In either case, the Python application should define the final data contract. For an AI knowledge base, that contract may be document title, canonical URL, Markdown body, heading path, and source timestamp. For commerce data, it may be product ID, variant, currency, displayed price, availability, and evidence URL. A generic page-to-JSON output is only a starting point if those business fields have not been validated.
Start with one template, prove the data is correct, then add pages and concurrency. If a source offers a documented feed or API that fits the use, evaluate it before parsing HTML. It may provide more stable identifiers and change signals than a website, while still requiring license and usage review.
Q: Is Beautiful Soup enough for JavaScript-rendered pages?
No. Beautiful Soup parses HTML it receives; it does not execute JavaScript. Use a rendering path if the required content is absent from the initial response.
Q: Does Nstdata return my exact product schema?
No. Nstdata returns page content and supported artifacts. Your application maps and validates the fields needed for its product.
Q: What should I store with each extracted row?
Store the source URL, acquisition time, parser version, validation status, and a content hash at minimum.
This Python tutorial moves from one static page to validated records and managed rendering. It includes runnable examples, failure diagnosis, source controls, and an AI-ready data contract.
Marcus Chen
Oct. 29th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
