TL;DR
- Clean article extraction requires the body, title, source URL, and visible caveats to survive.
- Article extraction must retain the headline, body boundary, citations, and disclosures.
- A clean-looking result can still omit a final paragraph or include related-story cards.
- Compare one representative article per template before running a broader collection.
- Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Is Article Text Extraction?
Article extraction separates a page’s main story from menus, recommendation cards, ads, repeated footers, and unrelated widgets. A useful record includes the headline, body, URL, and any publication details that are actually present. It should also preserve quotations, list structure, and linked citations. The words “any web page” describe the input shape, not a guarantee that every site is accessible or that every layout can be parsed automatically.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: extract a readable article body from a known URL while preserving its source and boundaries. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata URL-to-Markdown guide gives useful context for choosing the first representative page.
What Do You Need Before You Start?
Choose an article you may process, preferably with a headline, byline, subheadings, links, and a closing paragraph. Write down the expected headline, first and last sentences, and any disclosure you must retain. Decide whether your downstream product needs full text, a short excerpt, or only a summary. Obtain an Nstdata API key if you use the managed route and store it server-side.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
| Field | What to supply |
|---|---|
url | One permitted HTTPS page or site entry URL |
formats | Requested outputs, such as ["markdown"] or ["html","screenshot"] |
onlyMainContent | true only when main-body selection fits the task |
maxDepth, maxPages | Explicit bounds for a site-wide job |
includeUrls, excludeUrls | URL patterns that keep discovery within scope |
timeout | A task timeout in milliseconds when the default is unsuitable |
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
The Nstdata Markdown cleaning guide is a companion for defining the acceptance checks before you automate the workflow.
How Does an Article Extraction API Find the Main Body?
An article extractor identifies the main body after a page is retrieved and rendered, then removes surrounding navigation and recommendations. Nstdata Crawl can render JavaScript content and provides onlyMainContent to reduce boilerplate. It returns Markdown and page metadata, with HTML and screenshots available for investigation. Article extraction still needs a page-level acceptance check because an important disclosure or final paragraph may sit outside the selected main region. Managed proxy routing is built into the service; a standard request has no separate proxy Host, Port, Username, or Password fields.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Extract Clean Article Text from a Web Page?
Use a representative article to define body boundaries, then choose managed single-page retrieval or a local parser for HTML already in your control. Validate each template before scaling.
Method 1: Check the article in Crawl Playground
Best for deciding whether the automatic main-content selection is accurate.
Step 1: Open a sample article
Sign in and open Dashboard → Crawl Playground. Paste the authorized article URL. [Screenshot: article URL and output controls.]
Step 2: Request text and evidence
Choose Markdown and HTML, enable the main-content option, and run the scrape. For client-rendered news sites, use the documented readiness controls shown in the current interface.
Step 3: Compare the boundary
Check that the extracted headline, first and last paragraphs, captions, citations, and disclosures match the source. If a key part is missing, use HTML to inspect the selection before processing more articles.
Method 2: Call the article extraction API route
Best for known URLs in an application or agent workflow.
Step 1: Keep the API key private
Store NSTDATA_API_TOKEN in your server environment and send it as x-api-key. Do not send it from browser JavaScript.
Step 2: Submit the article URL
Use the documented single-page quick-start request below with Markdown and onlyMainContent. The docs show conflicting labels on other scrape routes, so confirm the current generated route in the Playground.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/article","formats":["markdown","html"],"onlyMainContent":true}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Validate the result
Inspect nested success and page status, title, and Markdown. Reject a result that contains only navigation, a consent wall, or a partial story. Save URL and retrieval time with the accepted text.
Method 3: Use local extraction for HTML you already hold
Best for trusted, already-rendered HTML when a managed fetch is unnecessary.
Step 1: Select a local parser
Mozilla Readability is an established article-content parser for DOM input; install it from its official package source if your application uses JavaScript.
Step 2: Run it on the captured DOM
Provide the actual rendered document, not only the initial static shell. Keep the original HTML alongside the extracted result for debugging.
Step 3: Check template coverage
Test several article layouts and compare opening, closing, captions, and links. Local extraction does not itself solve access, JavaScript loading, or proxy routing.
How Do You Verify the Output?
Accept an article only when its headline, first paragraph, last paragraph, and at least one distinctive phrase match the source. A character count catches an obvious empty result but cannot prove that a disclosure survived. The Mozilla Readability repository documents a local parser option for pages whose HTML is already available. The Nstdata article-to-Markdown guide explains how cleaned text supports downstream AI use.
The Nstdata website summarization guide connects this page-level check to a downstream AI or data workflow.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
For this workflow, consult Mozilla Readability when implementing the downstream parser or format-specific acceptance check.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might Article Extraction Miss Content?
Article text extraction can fail because the article body loaded later or selection stopped too early. Use the symptom table to identify the failing layer before changing limits or retrying.
| Symptom | Likely cause | Next action |
|---|---|---|
| Only a headline appears | The article body loaded later or selection stopped too early. | Inspect rendered HTML and wait for a body element. |
| Recommendations appear as body text | The page template includes adjacent cards in the selected region. | Review the HTML boundary and add a template-specific filter. |
| Disclosure or final paragraph is missing | Main-content selection was too aggressive. | Compare the page ending and retain the required section manually or through a revised selector. |
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Choose Playground for boundary review, the managed API for a known article URL, or a local parser when you already possess HTML. Use the local parser only after testing its behavior on each article template. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
FAQ: What Else Should You Know?
Q: Can an article extractor handle every web page?
No. It works best on article-like layouts; access limits, dynamic loading, and unusual templates require checks or custom rules.
Q: Should I preserve the original HTML?
Preserve it when your retention policy allows and when you need to investigate omissions or citations.
Q: Is the article title enough to prove success?
No. Verify the opening, closing, and any required caveats against the source.
Q: Does article text extraction require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a article text extraction result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
