Training data and RAG data are different products: training changes model weights, while RAG retrieves external evidence at request time.
AI teams source data from licensed collections, first-party content, public datasets, partner feeds, and permitted web pages; each source needs a recorded use basis.
A dataset manifest should preserve source, acquisition time, rights or policy status, transformation history, and deletion route.
Nstdata Crawl can acquire and convert permitted web pages into Markdown, HTML, raw data, links, screenshots, and PDF; it does not grant training rights or decide dataset suitability.
Quality, provenance, and refresh requirements should be defined before increasing collection volume.
Where Do AI Companies Get Training and RAG Data?
AI companies combine sources according to the task. First-party documentation and customer-approved knowledge bases often support RAG. Training projects may use licensed corpora, public datasets with explicit terms, partner data, human-created examples, and permitted web material. Synthetic examples can supplement coverage, but their generation method and source dependencies still need documentation. A public URL is a location, not proof that its contents may be used for every purpose. The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
The distinction between training and retrieval affects architecture. In RAG, an application can often replace a document and rebuild an index without retraining the model. In training, removing one source may require dataset reconstruction and, depending on the model and commitments, further remediation. The Data Provenance Initiative documents how licensing and origin information can be lost as datasets are repackaged. Keep that information at ingestion rather than trying to recover it afterward.
Use a source register before building crawlers. Each entry should name an owner, URL or dataset identifier, collection method, intended use, applicable terms, jurisdictional review, personal-data risk, update pattern, and contact for correction or removal. Do not collapse “publicly viewable” into “approved for training.” The same page may be appropriate for a search snippet under one arrangement and unsuitable for training under another.
Source
Typical strength
Main review question
First-party docs
Clear ownership and update path
Which versions are live?
Licensed or partner feed
Explicit delivery contract
Does the license cover the model use and retention?
Public dataset
Reproducible collection
Are component licenses and personal data documented?
Permitted web pages
Broad and current coverage
Are access, reuse, quality, and deletion rules satisfied?
Synthetic examples
Targeted edge cases
Can the generation process be traced and evaluated?
How Does the Collection Pipeline Differ for Training and RAG?
For RAG, design for short freshness lag, stable source URLs, document versioning, chunk provenance, and selective re-indexing. A useful record contains source_url, fetched_at, content_hash, heading path, chunk ID, and index version. A failed refresh should not overwrite a good document with an error page. Evaluate retrieval and answer citation, not only ingestion yield.
For training, design for dataset releases. Record source group, license or policy decision, collection time, filtering rules, deduplication method, language and domain distribution, and benchmark contamination controls. Hold out evaluation data before training. A model trained on a larger crawl can perform worse if duplicate templates, spam, or private information dominate. The AWS Responsible AI Lens discusses provenance and separating evaluation material from training inputs.
Nstdata Crawl is an acquisition and conversion layer for a reviewed set of sources. It can start from one URL, discover linked pages recursively within maxDepth, maxPages, include, and exclude rules, render JavaScript, and return page content or visual artifacts through the Crawl API. Built-in proxy routing means a standard managed request does not need separate proxy Host, Port, Username, Password, or Protocol fields. A large artifact may be returned as a reference token, which must be read through the documented storage endpoint.
Use these outputs as candidate evidence, not as automatically approved training material. The application should reject wrong page states, map source policy to each record, filter sensitive content under its own rules, and retain a correction path. Cloud Browser is a separate fit when the task requires continuing interaction with a page rather than turning pages into a dataset.
What Should Be Verified Before a Dataset Release?
Sample records by source, template, locale, and time period. Check accepted pages against the rendered source, inspect duplicates and near-duplicates, and count rejected records by reason. Verify that rights or policy metadata survives transformations. For RAG, test questions that require recently changed pages and confirm the answer cites the current version. For training, report source distribution and known exclusions with the dataset release.
When a source changes its terms, exposes personal data, or requests removal, your source register should identify every affected object and downstream index. This is why provenance is an operational requirement rather than a paragraph in a model card. The CNIL AI development guidance gives concrete safeguards for personal-data collection in its jurisdiction; involve counsel for the actual sources and uses.
How Do You Decide Whether a Source Belongs in RAG or Training?
Ask how frequently the facts change, whether the content can be corrected, and what rights cover the intended use. A frequently updated API reference is usually better kept in a versioned retrieval index than baked into model weights. A licensed corpus intended to teach style or task behavior may be appropriate for a training pipeline if its license and privacy review support that use. The same source can feed both systems only after the team separately approves each purpose.
Also consider the failure mode. In RAG, a bad document can often be removed from the index and replaced. An answer can cite a source URL and document version, making correction visible. In training, problematic examples can influence model behavior in ways that are harder to isolate. That increases the value of dataset release gates, lineage, and holdout evaluation. Neither path is automatically safer; both need source decisions and quality testing.
For customer content, confirm that contracts cover the exact AI feature and data movement. Do not assume permission for training from permission to store documents for search. For public datasets, inspect component-level provenance rather than relying only on a collection-level label. If a source's permissions cannot be established, exclude it until the question is resolved.
What Should an AI Data Source Manifest Contain?
Make the manifest machine-readable. A source entry should include source_id, owner, origin URL or dataset identifier, collection method, allowed uses, geographic scope where relevant, review decision and date, retention rule, refresh frequency, and removal contact. Each acquisition run should reference the manifest version. This prevents a later change in policy from silently altering the meaning of historical records.
At the record level, store source_id, source_url, fetched_at, content_hash, raw artifact pointer, transformation version, and accepted or rejected status. At the derived level, keep pointers from chunks, training examples, and embeddings back to the accepted record. The manifest then supports a query such as “find every training example derived from source X after date Y.” If the system cannot answer it, rights and deletion reviews will depend on manual guesswork.
Record uncertainty explicitly. A page with missing license information should not inherit the status of a neighboring page by default. A crawler can collect a page and still place it in a quarantine state for review. Use separate states for acquired, approved_for_rag, approved_for_training, and rejected rather than one boolean called usable.
How Can You Test Source Quality Before Increasing Volume?
Select a sample across sources and page types, then compare the collected record with the live source. Inspect language, encoding, boilerplate, duplicate templates, navigation text, personal data, and missing tables or code blocks. For RAG, build questions that require exact citations and recent changes. For training, examine distributions, near duplicates, benchmark overlap, and sensitive-content filters. Track rejection rates by source, not only an aggregate score.
Make the go/no-go criterion specific. For example, require that sampled documentation pages preserve headings and code examples, that every accepted record has a source ID and policy decision, and that a removal test finds all derived chunks. This is more actionable than “high-quality data.” When the sample fails, repair the acquisition or transformation step before collecting another million pages.
Add an adversarial sample of pages that should not pass: consent walls, empty JavaScript shells, duplicate search results, user profiles with personal details, and pages outside the approved path. A source pipeline that accepts these pages will look productive on a throughput dashboard while degrading retrieval or training quality. Keep the rejected sample in regression tests.
No. RAG retrieves external documents at inference time; training incorporates examples into a model-building process. Their correction paths differ.
Q: Can a crawler decide whether content is licensed for training?
No. The team must review source permissions and intended use separately from retrieval success.
Q: What is the minimum provenance record?
Keep source identity, acquisition time, rights or policy decision, transformation version, and a way to locate or remove derived objects.
This Python tutorial moves from one static page to validated records and managed rendering. It includes runnable examples, failure diagnosis, source controls, and an AI-ready data contract.
Marcus Chen
Oct. 29th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
