How to Scrape a Website to Markdown: A Practical Guide
TL;DR
Converting a site to Markdown is an extraction pipeline, not a tag-stripping operation: identify the canonical pages, fetch the right representation, isolate main content, convert structure, and validate output.
Start with a bounded URL list or sitemap, honor access rules, and avoid treating navigation, cookie banners, and recommendations as article text.
Preserve headings, links, code blocks, tables, canonical URL, and observation time; these are often more valuable to a retrieval system than decorative HTML.
A small Python pipeline can handle static pages with httpx, Beautiful Soup, and markdownify. Render JavaScript only when the target content is absent from the initial response.
Measure content fidelity on sampled pages before indexing a whole site; clean-looking Markdown can still omit essential facts or include the wrong page.
What does it mean to scrape a website to Markdown?
Scraping a website to Markdown means collecting authorized pages and converting their meaningful content into a text format that preserves document structure. The result should still show title hierarchy, paragraphs, links, lists, code, and tables. A file full of flattened text is easier to produce but harder to cite, chunk, or review.
The work has two separate parts. Discovery determines which URLs belong in scope. Extraction determines which part of each fetched page is the actual content. A conversion library can handle HTML-to-Markdown syntax, but it cannot know whether a newsletter modal is part of the article or whether a product variant was selected correctly. The website URL discovery guide covers the first problem; this tutorial concentrates on the second.
For large or JavaScript-heavy sites, Nstdata Crawl is an acquisition option to evaluate on representative public pages. Whether you use a managed service or local Python, the output still needs scope controls, provenance, and human review.
Begin with an explicit list of URLs you are allowed to process. A sitemap is a convenient starting point, but filter it by hostname and path prefix. Exclude login, cart, search, and duplicate parameterized URLs unless they are part of the task. Google's sitemap overview describes sitemaps as discovery hints, not guarantees that listed pages are valid or complete.
Define the desired Markdown record before coding. A useful minimum is source_url, final_url, title, fetched_at, content_hash, and markdown. If a page can vary by locale, account, or product option, capture that context too. Do not make a filename from a raw URL alone; long paths, query strings, and collisions make that unreliable. Use a stable slug plus a hash or database key.
Set a quality target: for example, the Markdown must retain every H2, the main code examples, and the outbound citations while excluding menus and footer navigation. That target can be checked on a sample. A high character count is not a quality metric if most characters are boilerplate.
Step 2: Fetch a page with bounded requests
The following example uses Python 3, httpx, Beautiful Soup, and markdownify. Install them with python -m pip install httpx beautifulsoup4 markdownify. The code requests a single public URL, sets a timeout, checks the content type, and converts the page's main element when available. The fallback to article or body is intentionally conservative only for a demonstration; inspect your target site's layout before running at scale.
from datetime import datetime, timezone
from hashlib import sha256
from urllib.parse import urlparse
import httpx
from bs4 import BeautifulSoup
from markdownify import markdownify as to_markdown
defpage_to_markdown(url:str)->dict[str,str]: parsed = urlparse(url)if parsed.scheme notin{"http","https"}:raise ValueError("Only HTTP(S) URLs are allowed")with httpx.Client( timeout=15.0, follow_redirects=True, headers={"User-Agent":"ContentResearchExample/1.0"},)as client: response = client.get(url) response.raise_for_status() content_type = response.headers.get("content-type","").lower()if"text/html"notin content_type:raise ValueError(f"Expected HTML, got {content_type}") soup = BeautifulSoup(response.text,"html.parser")for node in soup.select("script, style, nav, footer, aside, form"): node.decompose() main = soup.select_one("main")or soup.select_one("article")or soup.body
if main isNone:raise ValueError("No content container found") title = soup.title.get_text(" ", strip=True)if soup.title else"Untitled" markdown = to_markdown(str(main), heading_style="ATX").strip()iflen(markdown)<40:raise ValueError("Extracted content is unexpectedly short")return{"source_url": url,"final_url":str(response.url),"title": title,"fetched_at": datetime.now(timezone.utc).isoformat(),"content_hash": sha256(markdown.encode()).hexdigest(),"markdown": markdown,}if __name__ =="__main__": record = page_to_markdown("https://example.com/")print(record["title"])print(record["markdown"][:300])
This example intentionally does not recurse or bypass access controls. The len guard catches an obviously empty result, not a wrong-content result. Test on pages with tables, code blocks, and inline links before trusting the converter. The HTML specification's main element definition is a useful semantic starting point, but many sites do not use it consistently.
Step 3: Improve main-content selection
The broad selector in the example is only a baseline. A production extractor should use a source-specific rule when the site has a stable article wrapper, and fall back to a readability-style extractor when the layout varies. Compare the source page and Markdown side by side. Look for missing introductions, duplicate headings, inline footnotes, related-post cards, and content injected below the article.
Do not remove every aside blindly when an editorial aside contains a warning or definition. Similarly, deleting all tables or images may remove the only representation of a specification. Replace meaningful images with alt text or a link when your downstream use needs them. For code samples, verify that indentation and fences survive conversion. For links, prefer absolute URLs and keep link text descriptive; otherwise citations in a retrieval answer may lead nowhere.
The HTML extraction methods guide compares parsing approaches. For a retrieval corpus, preserving structure generally matters more than maximizing plain-text compression. Chunk only after you have inspected a faithful document, and keep heading paths with each chunk.
If the initial HTTP response contains only an application shell, the Python example cannot extract the article because the text arrives later through client-side requests. Confirm this by viewing the response body and the browser's rendered DOM. Then choose an authorized source API, a browser renderer, or a managed rendering service. Do not render every page by default: it adds latency, cost, and failure points without helping static pages.
The JavaScript-rendered Python guide explains this diagnosis in more detail. When using a renderer, wait for a content-specific signal rather than a fixed sleep. A page can be visually complete while a table or citation list is still loading. Record whether the result came from source HTML or rendered DOM because the same URL may produce different content in each path.
If access requires authentication, use only accounts and content you are explicitly authorized to process. Do not turn a browser session into an implicit license for harvesting private pages. For public pages, check robots.txt and site terms; RFC 9309 defines the robots protocol, not the full set of legal rights to reuse content.
Step 5: Validate and store the Markdown
Create a test sample that includes the site's hardest page types: short pages, long articles, tables, code, nested lists, and pages with many recommendations. For each sample, compare major headings, first and last paragraphs, a known fact, links, and any critical code block. Mark a page failed when important content is missing even if the Markdown is syntactically valid.
Deduplicate by canonical URL and content hash, but keep an observation history. A hash change can mean a meaningful edit or merely a rotating recommendation card. Compare the main-content diff before re-indexing. Store the raw or rendered source under a retention policy so extraction bugs can be investigated. If the content contains personal data, keep only what the use case needs and propagate deletion to copies and indexes.
For a retrieval workflow, store metadata alongside each Markdown file or chunk: URL, title, heading path, fetched time, source version, and access rights. The LlamaIndex web scraping and RAG guide offers a broader ingestion context. Do not label a retrieved answer βliveβ merely because the Markdown file was created with a web scraper once.
Common mistakes and how to diagnose them
If every file starts with the same navigation text, the content selector is too broad. If files are nearly empty, inspect whether the page is JavaScript-rendered, blocked, redirected, or simply not an article. If links become relative, resolve them against the final fetched URL before conversion. If a table collapses into one paragraph, choose a converter or post-processing step that preserves table semantics and verify the result manually.
If duplicate pages appear, normalize only parameters that are actually irrelevant; do not merge product variants or languages. If headings are missing, check whether they were rendered as styled div elements rather than semantic headings. A reliable pipeline should report these as page-type exceptions rather than silently outputting poor Markdown.
Conclusion
Good website-to-Markdown extraction is a controlled transformation with a clear source set, a faithful main-content selector, structural conversion, and sampled verification. Start with a few representative public pages, compare source and output, then expand the URL scope. Preserve provenance so every passage in a downstream search or RAG system can be traced to a page and observation time.
Test a page-to-Markdown workflow
If local parsing does not cover your authorized page types, evaluate Nstdata Crawl on a small set with known expected headings and links. Keep the same validation and metadata contract regardless of the acquisition tool.
Q: Can I convert an entire site to Markdown with one request?
No. One request converts one response. A site-wide workflow also needs bounded URL discovery, per-page fetching, deduplication, scheduling, and validation.
Q: Should Markdown include images?
Markdown should retain meaningful images as links or alt text when they convey information needed downstream. Decorative images can usually be omitted, but charts and diagrams may require separate extraction or description.
Q: Is Markdown better than HTML for RAG?
Markdown is often easier to inspect and chunk, but fidelity matters more than format. If conversion drops tables, citations, or code, the original HTML or a richer representation may be better.
Q: How do I know whether a page needs JavaScript rendering?
Compare the initial HTTP response with the rendered page. If the required content is missing from the response but present after client-side execution, use an authorized rendering or API path and test for content-specific completion.
Lena Zhou
Oct. 13th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more β without managing crawling infrastructure.
