TL;DR
- URL discovery should combine sitemap parsing with bounded link crawling; either method alone misses important pages.
- Canonicalize early, keep discovery provenance, and reject logout, search, calendar, parameter traps, and out-of-scope hosts.
- A URL map is an inventory, not proof that every page is useful or successfully retrievable.
- Nstdata Crawl can discover internal pages under explicit depth, page, include, exclude, and query rules.
Why URL Discovery Is Harder Than Reading sitemap.xml
A sitemap crawler can start with XML, but many sites have stale sitemap entries, orphan pages, JavaScript navigation, alternate-language URLs, and query combinations that create infinite crawl space.
Mental model: A sitemap is the site's declared inventory; a crawl is the inventory you can actually observe.
The Sitemaps protocol provides a standard URL feed, but it supplements rather than replaces link discovery.
When Teams Need a URL Map
Common uses include migration inventories, SEO audits, documentation ingestion, broken-link analysis, change monitoring, and determining which pages should enter a RAG corpus. Each use needs different filters; βall URLsβ is rarely the correct production target.
A Safe Map Workflow
Step 1: Define the boundary
Set the allowed host, path prefixes, maximum pages, maximum depth, and handling of query strings. Exclude login, logout, cart, account, site search, calendars, faceted navigation, and large files unless explicitly required.
Step 2: Parse declared sitemaps
Read robots.txt for Sitemap: directives, then process sitemap indexes and URL sets. Store lastmod as a hint, not guaranteed truth.
