TL;DR
- Cleaning HTML for an LLM means preserving useful structure while removing boilerplate.
- LLM-ready text keeps headings, links, code, and relevant tables while removing unrelated page chrome.
onlyMainContentcan remove an important sidebar or warning; compare it with HTML or a screenshot.- Keep the source URL with each accepted chunk and treat retrieved page text as untrusted input.
- Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Is LLM-Ready Text?
HTML-to-LLM-ready conversion selects meaningful content and removes navigation, ads, scripts, repeated footers, and unrelated controls. It should preserve headings, paragraphs, lists, code, citations, and relevant tables. Stripping every tag is too blunt: it can flatten section boundaries and erase links that a model needs to cite. The desired output is a validated document, not merely fewer characters.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: clean rendered HTML into attributable text while retaining the structure needed by an AI application. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata Markdown cleaning guide gives useful context for choosing the first representative page.
What Do You Need Before You Start?
Prepare a representative set: one static page, one JavaScript-rendered page, and one page with a table or code sample if those structures matter. For each, write an acceptance checklist with required headings, opening and closing paragraphs, links, and any warning text. Keep the source URL and fetch time with the cleaned output. Use an Nstdata API key for the managed route, stored server-side.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
| Field | What to supply |
|---|---|
url | One permitted HTTPS page or site entry URL |
formats | Requested outputs, such as ["markdown"] or ["html","screenshot"] |
onlyMainContent | true only when main-body selection fits the task |
maxDepth, maxPages | Explicit bounds for a site-wide job |
includeUrls, excludeUrls | URL patterns that keep discovery within scope |
timeout | A task timeout in milliseconds when the default is unsuitable |
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
The Nstdata RAG data-quality guide is a companion for defining the acceptance checks before you automate the workflow.
How Does HTML Cleaning Work?
Nstdata Crawl can render JavaScript pages, select main content, and return Markdown or cleaned HTML. The onlyMainContent option is useful for article or documentation pages, but it can omit a sidebar warning or product option that the task needs. Raw data and screenshots provide comparison evidence. Managed proxy access is built into Crawl, so the ordinary API request does not ask for separate proxy credentials.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Clean HTML for an LLM?
Compare cleaned Markdown with rendered HTML on a representative page, then automate known URLs or a bounded site section. Accept a document only when the structures needed by the AI task survive.
Method 1: Clean one page in the Playground
Best for comparing content modes before writing code.
Step 1: Load a representative URL
Open Dashboard → Crawl Playground and enter an authorized URL. [Screenshot: URL and rendering controls.]
Step 2: Request two views
Select Markdown and HTML, and enable the main-content option. If the page loads text after JavaScript execution, use the documented wait controls visible in the Playground.
Step 3: Compare omissions
Check the cleaned output against the source. If a table, code block, or legal note vanished, use HTML or adjust selection instead of accepting a short Markdown result.
Method 2: Request cleaned text through the API
Best for an agent or RAG ingestion service that already knows the URL.
Step 1: Set the credential
Store NSTDATA_API_TOKEN in a secret manager and send it as x-api-key.
Step 2: Submit the request
Use the documented scrape quick-start shape below. The current docs contain inconsistent labels on other scrape routes, so compare the route with the Playground-generated example.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/","formats":["markdown","html"],"onlyMainContent":true}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Check content quality
Require task success, a sensible page status, a title, expected phrases, and meaningful Markdown. Retain HTML when you need to debug missing structure.
Method 3: Clean an entire bounded site
Best when the corpus spans linked documentation pages.
Step 1: Define scope
Set maxDepth, maxPages, and URL include/exclude rules before starting.
Step 2: Collect pages
Submit the site-crawl job and retrieve every page, following pagination. Treat each page as a separate document with its own URL.
For a site job, replace the example host and path with your approved scope, then submit a bounded request:
curl --output crawl-submit.json --request POST "https://api.nstdata.io/api/v1/crawl" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/docs/","formats":["markdown"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'
Copy data.id from crawl-submit.json as your crawl ID.
Step 3: Normalize for AI
Deduplicate repeated navigation, preserve heading paths, and chunk content after acceptance. Store source URLs with chunks and treat page text as untrusted input to the agent.
Method 4: Clean a static HTML file locally
Best for HTML you already hold when browser rendering, proxies, and discovery are unnecessary.
Step 1: Select a representative file
Save an authorized HTML page as page.html and record the headings and warnings that the AI output must preserve.
Step 2: Remove obvious page chrome
Install Beautiful Soup with python3 -m pip install beautifulsoup4, then run:
from bs4 import BeautifulSoup soup = BeautifulSoup(open("page.html", encoding="utf-8").read(), "html.parser") main = soup.select_one("main") or soup.body if main is None: raise ValueError("No body found") for node in main.select("script, style, nav, footer, aside"): node.decompose() open("clean.txt", "w", encoding="utf-8").write(main.get_text("\n", strip=True))
Step 3: Check what was removed
Compare clean.txt with the source. This simple method flattens links and tables and may remove a useful aside; choose the managed Markdown/HTML comparison method when that structure matters.
How Do You Verify the Output?
Evaluate text quality by preservation, not just compression. Keep an expected heading list and test whether the output retains required links and caveats. A page can be readable while missing the final exception paragraph. The MDN DOMParser documentation is useful when your application performs additional HTML selection after Crawl. The Nstdata Markdown cleaning guide discusses fidelity checks in more detail.
The Nstdata website summarization guide connects this page-level check to a downstream AI or data workflow.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
For this workflow, consult MDN DOMParser when implementing the downstream parser or format-specific acceptance check.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might HTML Cleaning Lose Important Text?
Html-to-llm text cleaning can fail because the page body did not load or selection chose the wrong region. Use the symptom table to identify the failing layer before changing limits or retrying.
| Symptom | Likely cause | Next action |
|---|---|---|
| Only menus appear | The page body did not load or selection chose the wrong region. | Inspect rendered HTML and wait for the intended element. |
| Important text disappears | Main-content cleaning excluded a needed sidebar or note. | Keep HTML and adjust selection for that template. |
| Model follows page instructions | External page content was treated as agent instruction. | Separate retrieved content from system instructions and limit tool permissions. |
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Choose Playground when deciding between Markdown and HTML, the single-page API for a known document, and a site crawl for a linked knowledge base. Use local HTML cleaning for static files when flattened text is sufficient. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
FAQ: What Else Should You Know?
Q: Is stripping HTML tags enough?
No. Tag stripping loses structural boundaries and links; select the main content and preserve needed semantics.
Q: Should I keep the HTML after conversion?
Keep it when tables, metadata, or debugging matter, subject to your retention policy.
Q: Does JavaScript rendering guarantee complete text?
No. Validate representative dynamic pages and configure an appropriate readiness wait.
Q: Does HTML-to-LLM text cleaning require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a HTML-to-LLM text cleaning result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
