How to Scrape and Parse Sitemaps for Full-Site Crawling

TL;DR

• A sitemap is a discovery input, not proof that every listed page is accessible or worth crawling.
• A sitemap index points to other sitemaps; a URL set lists candidate pages.
• Sitemap inclusion does not establish access rights, freshness, or complete site coverage.
• Compare a filtered sitemap manifest with a bounded recursive Crawl result to find gaps.
• Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.

What Is a Sitemap and Why Parse It?

A sitemap is an XML document listing page URLs, or an index listing other sitemap documents. Parsing it gives you candidates for a crawl plan. It does not prove that every URL is current, canonical, internally linked, or appropriate for your use. Full-site coverage may also require link discovery because some pages are absent from the sitemap. Combine both inputs under host, path, depth, and page limits.

Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: discover authorized URLs from sitemaps and turn a reviewed subset into a bounded site crawl. The Nstdata full-site crawling guide explains the site-level option.

Try Nstdata for Free →

For the narrower task covered here, the Nstdata website-page discovery guide gives useful context for choosing the first representative page.

What Do You Need Before You Start?

Start with a site you own or may crawl. Check /robots.txt for sitemap declarations and access rules, then inspect the sitemap index or URL set. Record the allowed host, path prefixes, file types, language, and maximum page count. Exclude account, cart, search, calendar, and duplicate query routes unless needed. Keep a manifest of candidate, accepted, rejected, and actually crawled URLs.

The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:

The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.

Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.

The Nstdata full-site crawl guide is a companion for defining the acceptance checks before you automate the workflow.

How Do Sitemap Indexes and URL Sets Work?

A urlset contains page <loc> entries. A sitemapindex contains child-sitemap <loc> entries that may be XML or compressed files. Nstdata Crawl can discover URLs from site structures and follow internal links recursively within configured maxDepth, maxPages, include, and exclude rules. JavaScript rendering helps on dynamic pages, and proxy routing is built into the managed service. The API does not remove the need to review candidate URLs and crawl rights.

For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.

Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.

How Do You Parse a Sitemap and Crawl the Site?

Parse and filter sitemap candidates first, use bounded Crawl discovery to find linked pages, and maintain a manifest for repeatable coverage checks. A sitemap and a recursive crawl answer related but different discovery questions.

Method 1: Inspect the sitemap before crawling

Best for a small site or one-time discovery review.

Step 1: Find sitemap declarations

Open the site’s /robots.txt and look for Sitemap: lines. If none are declared, test only the site’s documented sitemap location; do not assume every site uses /sitemap.xml.

Step 2: Classify the XML

Determine whether the root is sitemapindex or urlset. Follow child sitemap URLs only within the approved host and avoid an unbounded recursive fetch. [Screenshot: XML root and <loc> entries.]

Use a bounded parser rather than copying <loc> values by hand. This standard-library example accepts a sitemap URL, follows at most three sitemap-index levels, stays on the original host, and prints page URLs:

from urllib.request import urlopen
from urllib.parse import urlparse
from xml.etree import ElementTree as ET
import gzip, sys

start = sys.argv[1]
host = urlparse(start).hostname
seen = set()
def visit(url, depth=0):
    if depth > 3 or url in seen or urlparse(url).hostname != host:
        return
    seen.add(url)
    with urlopen(url, timeout=20) as response:
        raw = response.read(10_000_001)
    if len(raw) > 10_000_000:
        raise ValueError("Sitemap exceeds the review size cap")
    if url.endswith(".gz"):
        raw = gzip.decompress(raw)
    root = ET.fromstring(raw)
    ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("s:sitemap/s:loc", ns):
            visit(loc.text.strip(), depth + 1)
    elif root.tag.endswith("urlset"):
        for loc in root.findall("s:url/s:loc", ns):
            if urlparse(loc.text.strip()).hostname == host:
                print(loc.text.strip())
    else:
        raise ValueError("Unexpected sitemap root")
visit(start)

Save it as parse_sitemap.py and run python3 parse_sitemap.py https://example.com/sitemap.xml > urls.txt. Review and deduplicate urls.txt before crawling; the script does not grant permission to collect listed pages.

Step 3: Review candidates

Normalize and deduplicate URLs, then filter by approved host, path, language, and content type. A listed URL is a candidate, not an accepted page.

Method 2: Use Nstdata Crawl for bounded discovery

Best when link discovery should complement the sitemap.

Step 1: Store the API key

Keep NSTDATA_API_TOKEN server-side and send it as x-api-key.

Step 2: Submit a scoped job

Use POST /api/v1/crawl with a start URL, maxDepth, maxPages, include/exclude URL rules, and required formats. The example below follows the official site-crawl request shape.

Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.

curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl" \
  --header "x-api-key: ${NSTDATA_API_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com/docs/","formats":["markdown"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'

The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.

Step 3: Compare coverage

Poll the crawl ID, retrieve pages, and compare returned URLs with your sitemap manifest. Investigate missing high-value pages and unexpected paths rather than assuming discovery was complete.

Method 3: Maintain a repeatable URL manifest

Best for recurring documentation or catalog refreshes.

Step 1: Snapshot candidates

Save normalized URL, source sitemap, and discovery time for each approved candidate.

Step 2: Track crawl outcomes

Record fetched status, content hash, and rejection reason per URL. Keep sitemap changes separate from content changes.

Step 3: Refresh selectively

Revisit changed or high-value pages at a cadence permitted by the target. Avoid recrawling unchanged large sections merely because a sitemap timestamp changed.

How Do You Verify the Output?

A sound crawl manifest distinguishes listed URLs, discovered URLs, fetched URLs, and accepted content. Those sets can differ. The Sitemaps protocol defines sitemapindex, urlset, and <loc> elements, while Google Search Central notes that a sitemap helps discovery without guaranteeing a crawl. The Nstdata guide to finding website pages covers complementary discovery methods.

The Nstdata RAG knowledge-base guide connects this page-level check to a downstream AI or data workflow.

Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.

For this workflow, consult Sitemaps protocol when implementing the downstream parser or format-specific acceptance check.

When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.

Try Nstdata for Free →

Why Might Sitemap Discovery Miss Pages?

Sitemap-based crawling can fail because the root file is a sitemap index. Use the symptom table to identify the failing layer before changing limits or retrying.

Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.

Which Method Is the Best Fit?

Parse the sitemap for a candidate manifest, use Nstdata Crawl for bounded linked-page discovery, and maintain a manifest when coverage must be refreshed. Compare discovered and accepted URLs after each run. The Nstdata site-crawling overview gives more detail on the site route.

What Should You Do After the First Successful Run?

Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.

Try Nstdata for Free →

FAQ: What Else Should You Know?

Q: Does a sitemap contain every page?

No. Sites can omit pages, retain stale URLs, or list URLs outside the section you need. Compare with internal-link discovery.

Q: Can Nstdata Crawl use a sitemap and follow links?

Its product page describes sitemap and link discovery; bound the job with depth, page count, and URL rules.

Q: Is a URL in a sitemap permission to crawl it?

No. Review the target’s rules and your authority to use the content before collection.

Q: Does sitemap-based crawling require a separate proxy configuration with Nstdata Crawl?

A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.

Q: Does HTTP 200 prove a sitemap-based crawling result is usable?

No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.