TL;DR
- A news crawler discovers permitted article URLs, retrieves them, extracts editorial content and metadata, and keeps a versioned record of changes.
- Start discovery with publisher feeds, section links, or sitemaps; do not assume that a homepage or one news sitemap covers the archive.
- Separate publication time, modification time, crawl time, and event time. They answer different freshness questions.
- Accept an article only after checking canonical URL, headline, publisher, language, body completeness, and rights for the intended use.
- Nstdata Crawl is a Web Crawling API for LLM-ready data that can handle bounded discovery, JavaScript rendering, Markdown or HTML output, proxy routing, and task status; your application still manages licenses, deduplication, and editorial semantics.
What Is a News Crawler?
A news crawler is a system that repeatedly discovers and retrieves news articles from approved publishers, then turns them into records for search, monitoring, or AI retrieval. Unlike a one-time scraper, it must handle new stories, corrections, updated headlines, moved URLs, withdrawn pages, and publication schedules. The output is a maintained corpus, not a pile of HTML files.
The first design decision is access rights. A publisher's RSS feed or sitemap is a discovery mechanism, not automatically a license to store full text, redistribute it, or use it for model training. Review publisher terms and any syndication agreement before retention or downstream AI use. The Robots Exclusion Protocol defines crawler access instructions, but its rules are not an authorization system. A defensible workflow tracks the policy and license that applies to each source.
Why Does News Data Need a Different Pipeline?
News data needs a different pipeline because timeliness and revision history are part of the record. A story can be published at 09:00, corrected at 09:20, and discovered by your system at 10:00. If the database stores only “timestamp,” a downstream analyst cannot tell whether the article was fresh when the alert fired. Keep datePublished, dateModified, firstSeenAt, and fetchedAt separately.
Google's news-sitemap guidance says publishers should add fresh article URLs and retain news-specific sitemap metadata only for recent articles. It is useful for discovery, but it is not a complete historical archive. Google also recommends distinct, permanent article URLs and visible article links in its publisher technical guidance. These are good reasons to combine multiple permitted discovery channels and canonicalize before storage.
For an AI product, the same story may appear as an article, syndicated copy, live blog entry, and follow-up. URL deduplication is necessary but insufficient. Use publisher, canonical URL, headline, publication time, and a content fingerprint to group versions or related copies. Do not merge a correction into an unrelated story simply because the headline is similar.
How Should a Production News Crawler Be Architected?
A production news crawler should use a source registry, bounded discovery, an acquisition queue, extraction, validation, versioning, and delivery. Each stage should emit counts so an operator can distinguish “no new stories” from “discovery broke.”
| Stage | Main decision | Evidence to retain |
|---|---|---|
| Source registry | Which publishers and paths are approved? | License, robots policy, locale, allowed fields |
| Discovery | Which URLs are new or changed? | Feed or sitemap URL, discovery time |
| Scheduling | Which pages are due for fetch or revisit? | Priority, retry budget, previous status |
| Acquisition | Was the article page actually rendered? | HTTP/page status, final URL, artifact, error |
| Extraction | Are headline, body, dates, and author present? | Field provenance and parser version |
| Acceptance | Is it a full, licensed article rather than a teaser? | Quality checks and rejection reason |
| Delivery | Should search, analytics, or RAG update? | Stable ID, version, content hash |
An illustrative accepted record might be:
{ "publisher": "Example News", "canonical_url": "https://news.example.org/technology/story-42", "headline": "Example headline", "language": "en", "date_published": "2026-11-06T09:00:00Z", "date_modified": "2026-11-06T09:20:00Z", "first_seen_at": "2026-11-06T09:05:00Z", "fetched_at": "2026-11-06T09:25:00Z", "content_hash": "illustrative-hash", "rights_profile": "licensed_for_internal_search", "quality_status": "accepted" }
The record is illustrative, not a live feed result. A publisher's article schema can help extraction: Google's Article structured-data guidance covers headline, dates, images, and author. Still compare markup with visible page content. A page may have stale structured data, an embedded teaser, or a timestamp rendered in a different timezone. For syndication or enterprise exchange, the IPTC NewsML-G2 guidelines provide a stronger metadata model where the publisher supplies it.
How Do You Decide When to Revisit an Article?
Revisit an article according to its publisher, age, update signals, and business value rather than polling every page at a fixed interval. New stories may need faster checks while a live event runs; older articles may only need a revisit when a feed announces an update or a downstream user requests confirmation. Respect each source's permitted rate and licensing terms.
Use conditional requests or a content hash when the source supports them. If a page changes, preserve the previous version and record the changed fields. A headline correction should update search metadata; a body correction may require new RAG chunks and embeddings. A removed article needs a deletion or tombstone policy, especially where licensing or rights change. A crawler that only appends records will quietly serve stale material.
Freshness metrics should reflect the product. “Median crawl delay” is useful for alerts, while “percent of cited answers based on current article version” matters for RAG. Measure source-specific lag, not one global average that hides a failing publisher.
Where Does Nstdata Crawl Fit a News Workflow?
Nstdata Crawl can serve as the collection and cleaning layer between an approved article URL and a news system. The current Crawl API documentation describes single-page tasks, bounded site crawling, JavaScript rendering, task polling, and Markdown, HTML, raw data, links, screenshot, and PDF outputs. Its proxy routing is built into the workflow, so a client does not need to configure an additional proxy gateway. The application still owns feed ingestion, editorial identity, source rights, classification, and update semantics.
- Bounded discovery: Use a publisher section as a seed only within approved include and exclude paths, page caps, and depth limits.
- Rendered extraction: Escalate to browser rendering when article text is absent from initial HTML; do not render every URL by default.
- LLM-ready output: Markdown can feed a licensed RAG pipeline after source metadata, attribution, and rights are attached. HTML and raw data help diagnose missing bylines or date fields.
- Task diagnostics: Poll completed, pending, and failed counts, and inspect per-page success. A queued request is not an accepted news article.
- Cost boundary: Crawl is billed by URL according to the current plan, and proxy traffic may be charged separately. Measure cost per accepted, fresh article.
This illustrative request uses a publisher domain that the operator controls or has licensed. It shows the documented site-crawl path and scope controls; it was not run against a live publisher because no API key or publisher agreement was supplied.
curl -X POST "https://api.nstdata.io/api/v1/crawl" \ -H "x-api-key: $NSTDATA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://news.example.org/technology/", "formats": ["markdown", "html"], "maxDepth": 2, "maxPages": 30, "includeUrls": ["*news.example.org/technology/*"], "excludeUrls": ["*news.example.org/account/*"], "ignoreQuery": true, "onlyMainContent": true, "timeout": 60000 }'
Read GET /api/v1/crawl/status/{crawlId} for job progress and GET /api/v1/crawl/pages/{crawlId} for page records. Validate the returned article fields and rights before storing the Markdown. Large artifacts may be returned as reference tokens that require the documented storage read endpoint. The Crawl pricing page is the current source for URL and proxy billing models.
Test Nstdata Crawl on an approved news source →
What Failures Should a News Team Monitor?
A news team should monitor discovery, content, rights, and freshness failures separately. Discovery failures include an empty feed, moved section URL, malformed sitemap, or duplicate query variants. Content failures include a 200 response containing only a cookie wall, a teaser instead of the body, a wrong-language edition, or a stale cached version. Rights failures include a source whose license expired or whose permitted retention changed. Freshness failures occur when the crawler runs but misses a correction.
Define acceptance checks before scaling: nonempty headline and body, plausible published date, canonical URL on the expected host, language match, permitted rights profile, and a sample-based body-completeness test. Reject pages with challenge markers or boilerplate-only Markdown. Keep a small set of known articles to catch parser or rendering regressions. Make retries finite, honor publisher rate limits, and send repeatedly failing URLs to a review queue rather than silently dropping them.
When Is a News Crawler Worth Building?
A news crawler is worth building when a team has permission to use the sources and a product need for monitored freshness, version history, and cited retrieval. Start with one publisher and a small approved section, prove the acceptance rules, then expand the source registry. A broad crawl without rights or quality controls is a liability and a poor AI dataset.
Nstdata Crawl can remove browser, proxy, and page-format plumbing from that approved pipeline. The news system must still decide what a story is, when it changed, how long it may be retained, and how downstream answers cite it. For broader AI ingestion patterns, see Crawl for RAG knowledge bases.
FAQ
Q: Is a news sitemap a complete archive?
No. News sitemaps focus on recent articles; use approved archives or feeds for historical coverage.
Q: Can a crawler store full text from every public publisher?
No. Public access does not itself grant rights to retain, redistribute, or train on full text; check each publisher's terms and agreements.
Q: Which timestamp should an alert use?
An alert should distinguish publication and modification times from when the crawler first observed and fetched the article.
Q: Does HTTP 200 mean the article is usable?
No. A 200 response may contain a teaser, challenge, cookie wall, or incomplete rendered page.
Q: Where does Nstdata Crawl stop?
Nstdata Crawl collects and converts page content, while the application defines licenses, deduplication, versioning, semantic validation, and downstream delivery.
