How to Scrape Websites to Markdown: A Practical Guide
TL;DR
Scraping websites to Markdown requires bounded discovery, correct page acquisition, main-content selection, structural conversion, and validation.
Start with a reviewed URL list; a sitemap or link crawler should expand scope only under explicit host, path, and page limits.
Preserve code, headings, links, tables, and source metadata before chunking the document for search or RAG.
The Python example below handles one static page and resolves relative links; it is a baseline, not a universal article extractor.
Nstdata Crawl is a managed acquisition alternative for broader page and site workflows, while downstream fidelity checks remain necessary.
What a reliable website-to-Markdown pipeline produces
A reliable website-to-Markdown pipeline produces a faithful document plus enough metadata to trace that document to a source observation. A text file containing flattened page content is not automatically suitable for retrieval. Navigation can overwhelm the useful paragraphs, a removed sidebar can erase a warning, and a broken relative link can make the resulting citation unusable.
Define the consumer first. A documentation index needs heading paths and code examples. A research corpus needs attributable claims and links. A product-analysis workflow may require structured fields alongside Markdown. The URL-to-Markdown overview explains the format's role; the implementation below focuses on retaining meaning rather than maximizing character output.
The complete sequence is approved URLs → fetch → classify → select content → convert → validate → store → refresh. Every arrow is a failure boundary. Nstdata Crawl can support acquisition and transformation, but the application must still decide whether the resulting document meets its contract.
Prerequisites and acceptance criteria
The local method requires Python, an HTTP client, an HTML parser, and a Markdown converter. Use public pages you are authorized to process and inspect their structure before expanding the job. The example uses the public demonstration page at example.com and does not require a paid-service credential.
Experience Nstproxy Crawl - Start Your Free Trial Today
Choose representative pages with paragraphs, inline links, a table, code, and a meaningful note. Acceptance should require preservation of important headings, the opening and closing content, key citations, and indispensable code. A minimum character count catches obvious emptiness; it does not prove correctness.
For a site-wide job, define allowed hosts, path prefixes, relevant query parameters, maximum pages, and revisit frequency. Exclude account, cart, search, and duplicate filter paths unless they are explicitly required. The bounded website-crawling guide provides the discovery companion to this transformation workflow.
Detailed Tutorial
The tutorial begins with an inspectable single-page converter and then adds the controls needed for multiple pages. Keep the single-page acceptance sample available as the collection grows.
Method 1: Convert static HTML locally
Step 1: Install the parsing dependencies
Install the three libraries in a project-specific virtual environment.
The HTTPX quickstart is the reference for request and response handling. The markdownify project documents conversion behavior and options. Pin versions after validating your sample rather than assuming a future converter release preserves every output detail.
Step 2: Fetch and convert one approved page
The following function checks the scheme, sets a timeout, validates the response type, removes common non-content elements inside the selected container, and resolves link destinations. Save it as page_to_markdown.py and execute it with Python.
from datetime import datetime, timezone
from hashlib import sha256
from urllib.parse import urljoin, urlparse
import httpx
from bs4 import BeautifulSoup
from markdownify import markdownify
defpage_to_markdown(url:str)->dict:if urlparse(url).scheme notin{"http","https"}:raise ValueError("Expected an approved HTTP(S) page URL")with httpx.Client( timeout=20, follow_redirects=True, headers={"User-Agent":"MarkdownResearchExample/1.0"},)as client: response = client.get(url) response.raise_for_status()if"text/html"notin response.headers.get("content-type","").lower():raise ValueError("Response is not HTML") soup = BeautifulSoup(response.text,"html.parser") title = soup.title.get_text(" ", strip=True)if soup.title else"" root = soup.select_one("main")or soup.select_one("article")or soup.body
if root isNone:raise ValueError("No candidate content container") base_url =str(response.url) base_tag = soup.find("base", href=True)if base_tag: candidate = urljoin(base_url, base_tag["href"])if urlparse(candidate).scheme in{"http","https"}: base_url = candidate
for node in root.select("script, style, nav, footer"): node.decompose()for node in root.select("a[href], img[src]"): attribute ="href"if node.name =="a"else"src" node[attribute]= urljoin(base_url, node[attribute]) content = markdownify(str(root), heading_style="ATX").strip()ifnot content:raise ValueError("Empty Markdown output")return{"source_url": url,"final_url":str(response.url),"title": title,"fetched_at": datetime.now(timezone.utc).isoformat(),"representation":"source_html","extractor_version":"example-v1","content_hash": sha256(content.encode("utf-8")).hexdigest(),"markdown": content,}if __name__ =="__main__": result = page_to_markdown("https://example.com/")print(result["title"])print(result["markdown"])
This baseline requests a controlled public demonstration URL. It is not a public URL-fetching endpoint: accepting arbitrary user input would require network restrictions, host and redirect validation, and protection against internal-network requests. For a controlled project, provide an approved list rather than exposing this function directly to strangers.
Step 3: Replace the broad selector with a reviewed rule
The main → article → body fallback is a candidate selection strategy, not a claim that the first matching element is always the right document. Inspect your source and choose a stable wrapper when one exists. A full body can include recommendations, cookie text, and duplicated article previews.
The code deliberately retains aside elements. On technical sites, an aside may contain a warning. Removing all asides creates a cleaner-looking file while changing the meaning of the instructions. Likewise, meaningful image alt text or links may deserve preservation even when decorative images can be omitted.
Check conversion on a page with fenced code and a complex table. If a table loses its relationships, preserve a richer representation or add a reviewed source-specific transformation. Do not label every syntactically valid Markdown file accepted.
Step 4: Store provenance alongside the document
Write the Markdown and metadata atomically or store them as one versioned record. Keep submitted and final URL, observation time, representation, extraction version, and content hash. A stable storage identity should not depend only on an easily colliding filename derived from a long URL.
The hash describes the converted content, not the raw response. Changing the parser can change the hash without a source edit. Preserve the extraction version so refresh logic can distinguish those events. Store a permitted source artifact when the task's retention policy supports replaying corrections.
Method 2: Use managed page and site acquisition
Managed acquisition is appropriate when the project must process multiple page types or maintain a bounded site collection without owning all fetching and rendering components. Nstdata Crawl is a candidate for that role. Start in the current product or Playground surface and test the exact pages and formats required. Use the generated current example rather than copying an unverified API field from a third-party article. A managed path changes acquisition ownership, while content classification and acceptance remain application responsibilities.
Set page-count, depth, and inclusion or exclusion boundaries before submitting a site job. A documentation entry page can lead to archives, localized copies, examples, and external references. Only admit the source set the knowledge base is meant to cover.
The current Nstdata Crawl product surface documents site collection and page-output options. Test Markdown on representative pages, and retain an alternate artifact where it helps diagnose missing content.
Step 2: Keep acquisition controls explicit
Use rendering when source HTML lacks the required text, not automatically for every page. Compare the initial response with the visible content before selecting the path. For rendered pages, completion should depend on a content-specific signal rather than an arbitrary delay.
Record the job configuration and the page-level outcome. A job being accepted is not the same as all pages being valid. Keep failed, restricted, unavailable, and accepted pages distinguishable.
Step 3: Evaluate the complete operating cost
Consult the current Crawl pricing model, including any separately charged proxy usage, rather than deriving cost from a copied headline rate. Count pages that pass the document acceptance criteria. Add storage, downstream processing, and review effort when comparing with local extraction.
Paid Nstdata requests require an account and credential. This article reports the local Python path, not an authenticated managed-service benchmark. Generate and execute the current account-specific example before relying on the managed route in production.
From a few documents to a maintained corpus
A maintained corpus needs a refresh and removal policy in addition to initial conversion. Start from a reviewed URL list, then add sitemap discovery or bounded traversal when the use case requires it. Make writes idempotent, track page identity across redirects, and separate fetch retries from content-quality failures.
A changing hash can reflect a substantive edit or a rotating widget. Compare main-content differences before reindexing. Keep localized pages and product variants distinct when their content matters. Avoid indiscriminate query-string removal, which can erase meaningful source differences.
For retrieval, validate the full document before splitting it. The RAG ingestion guide explains the downstream role, while the chunking strategies overview helps preserve heading context. Chunks should inherit URL, title, heading path, source version, and observation time.
Troubleshooting and responsible source handling
Troubleshooting starts by classifying the received page. A nearly empty output may come from an application shell, access restriction, redirect, or genuinely short document. A long output may still be navigation rather than content. Inspect the response and selected container before changing the converter.
Repeated introductory text across many files suggests boilerplate leakage. Missing closing paragraphs suggest an incomplete wrapper or late-loaded section. Broken links suggest incorrect base resolution. Incorrect code indentation requires comparison with the source rather than a character-count threshold.
Use authorized sources, respect applicable access instructions and terms, and retain only what the purpose requires. The Robots Exclusion Protocol defines a machine-readable access signal; it does not settle every right to retain, redistribute, or train on content. Propagate relevant corrections and removals into stored documents, indexes, and caches.
Conclusion
Scraping websites to Markdown is a source-fidelity task before it is a formatting task. Begin with one approved page, establish acceptance criteria, and preserve provenance before expanding discovery or indexing. Compare local parsing with Nstdata Crawl on the same representative documents. For an existing multi-source acquisition system, Nstdata Proxy Manager is an additional option when centralized proxy operations become a distinct requirement.
Evaluate the Markdown acquisition step
Test Nstdata Crawl on a bounded source set and compare important headings, links, code, and page-level failures.
One page request converts one response. A site-wide workflow also needs discovery, per-page processing, bounded scope, storage, and completion tracking.
Q: Does every page require JavaScript rendering?
No. Inspect the initial response first and render only when the required content is absent and an authorized rendering path is appropriate.
Q: Is the Python example a universal article extractor?
No. Its content selector is a baseline. Review source-specific wrappers and important page structures before increasing volume.
Q: Why preserve metadata outside the Markdown?
Metadata identifies the source observation and extraction version, enabling citations, refresh decisions, debugging, and replay without cluttering document text.
Q: Is Markdown always the best RAG format?
No. Markdown is convenient when it preserves the needed structure. Complex tables, diagrams, or layout-sensitive documents may need a richer representation.
Q: Can the local example run without a paid API key?
Yes. The static-page example uses local libraries and a public demonstration page. Managed-service requests need the appropriate account and credential.
Lena Zhou
Oct. 13th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
