How to Export Crawler Results to an Excel Spreadsheet
TL;DR
Exporting crawler results to an Excel spreadsheet requires a normalized row schema before file conversion.
Choose XLSX when you need explicit text types, multiple sheets, filters, and predictable spreadsheet behavior.
Use CSV for simple interchange, but control encoding, delimiters, import types, and formula interpretation.
Preserve source URLs, observation times, task outcomes, and missing values instead of exporting only extracted text.
Retrieve every result batch before declaring an export complete, and reconcile submitted, accepted, and rejected records.
Nstdata Crawl supplies LLM-ready page outputs; XLSX generation belongs in your export layer unless the current product explicitly provides that format.
What Does Exporting Crawler Results to Excel Involve?
Exporting crawler results to Excel involves converting accepted crawl records into a stable table and writing that table to a spreadsheet-compatible format. A crawl response may contain page text, links, metadata, task information, or references to larger artifacts. Those elements need a deliberate row model before they become a useful workbook.
For example, a page inventory can use one row per page. A product dataset may use one row per product observation. A link audit needs one row per source-to-destination edge. Placing all three into one ambiguous table creates repeated values and misleading totals.
Nstdata Crawl can supply the approved page-collection layer. This tutorial treats Excel generation as a downstream transformation and does not invent an Nstdata xlsx output format or a nonexistent export endpoint.
Should You Export CSV or XLSX?
Choose CSV for simple system-to-system tabular exchange and XLSX when the consumer needs explicit cell types and workbook features. A file's extension is not the only difference: the importer's interpretation of each cell affects identifiers, dates, and untrusted source text.
Experience Nstproxy Crawl - Start Your Free Trial Today
Requirement
CSV
XLSX
Plain-text interoperability
Convenient
Requires a workbook reader
Explicit text versus numeric cells
Importer decides
Writer can set the cell type
Multiple worksheets
Separate files
Supported in one workbook
Filters and frozen header
Consumer must add them
Writer can include them
Large text and row counts
Consumer-dependent limits
Excel's documented limits still apply
Untrusted webpage text
Requires import precautions
Can be written explicitly as text
Consult Excel's specifications and limits before choosing a workbook as the primary destination for a large crawl. A spreadsheet is often an excellent review artifact while a database or object store remains the system of record.
What Do You Need Before Exporting a Crawl?
You need a completed collection manifest, all result batches, a normalized schema, and an output policy before writing the file. Decide which records are accepted and how failures will appear. A workbook containing only successful rows can hide a major collection problem unless the export also reports what was excluded.
Column
Type
Meaning
source_url
Text
Original approved page identity
title
Text
Accepted page title
observed_at
Text timestamp
UTC collection time in a documented representation
http_status
Optional number
Target status when actually available
accepted
Boolean or explicit text
Application acceptance decision
rejection_reason
Text
Reason a record did not pass validation
These are application-normalized columns, not a claim that every provider returns the same field names. Map the current API response into this contract and preserve the original task ID in your source manifest. Do not label an API gateway's successful response code as the target page's status.
Keep long Markdown documents outside the workbook when they exceed its practical cell limits. Store a stable artifact reference or controlled file path alongside the page row. A silent truncation that removes the evidence passage can make an apparently complete export unusable.
How Does the Export Pipeline Work?
The export pipeline reads all available page results, resolves required artifacts, normalizes rows, validates them, and writes a bounded output. Each stage should preserve counts so a reviewer can reconcile the finished workbook with the source job.
Use bounded recovery policies to resume result retrieval without repeating the entire source job. For an asynchronous crawl, save the job ID before waiting. Check the documented terminal state and then follow result pagination until its end condition is reached. A single result page or cursor batch is not the entire job. If retrieval fails midway, retain the checkpoint and mark the export incomplete rather than producing an apparently final spreadsheet.
Use batch crawling controls to keep collection size intentional. The export should record whether the crawl hit a configured limit, since a bounded completed job can still represent only part of the larger site.
How Do You Export the Results Step by Step?
Export the results by first proving the normalized input, then choosing the file format that fits the consumer. The following methods cover managed collection, a review workbook, and plain-text interchange.
Method 1: Collect and normalize with Nstdata Crawl
Step 1: Request the representations you need
Use the Crawl Playground to configure an approved page or bounded site scope. Choose Markdown for text consumption, HTML for parsing, and links for discovery or graph analysis. Avoid requesting large visual artifacts for every page when the spreadsheet only needs a title and URL.
Step 2: Establish the API connection
Install nstdata-ai-crawl and set NSTDATA_API_TOKEN plus APPROVED_PAGE_URL. The example requires your own credential and an authorized target; this article does not claim an authenticated Nstdata run with a customer token.
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
token = os.environ["NSTDATA_API_TOKEN"]url = os.environ["APPROVED_PAGE_URL"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=url, formats=["markdown"], onlyMainContent=True, timeout=60000))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Page retrieval was not completed successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") Path("page.md").write_text(text, encoding="utf-8")print("Saved page.md; validate its content before downstream use")
Step 3: Write normalized JSONL rows
Map accepted results into one object per line in records.jsonl. Keep original response evidence separately, and include rejected records if the workbook is intended to diagnose the full run. For a site job, retrieve every documented result batch before finalizing the row count.
Method 2: Write a typed XLSX review file
Step 1: Install the workbook writer
Install XlsxWriter in your project environment. The example uses explicit string cells for externally sourced text and disables automatic URL and formula conversion. These choices make the workbook's behavior more predictable when titles or other source text contain special characters.
Step 2: Save the export program
Save this as export_xlsx.py. Its input is the normalized application schema above, not an unmodified vendor response. The code was exercised with a synthetic dataset containing Unicode, a missing status, and formula-like text; that check validates export handling rather than a live collection claim.
import json
import xlsxwriter
columns =["source_url","title","observed_at","http_status","accepted","rejection_reason"]workbook = xlsxwriter.Workbook("crawl-review.xlsx",{"strings_to_formulas":False,"strings_to_urls":False,"constant_memory":True})sheet = workbook.add_worksheet("Pages")header = workbook.add_format({"bold":True,"bg_color":"#E8EDFF"})sheet.write_row(0,0, columns, header)count =0try:withopen("records.jsonl", encoding="utf-8")as source:for row_number, line inenumerate(source, start=1):if row_number >=1_048_576:raise ValueError("Split the export before Excel's row limit") record = json.loads(line)ifnot record.get("source_url"):raise ValueError("Missing source_url")for column_number, name inenumerate(columns): value = record.get(name)if value isNone: sheet.write_blank(row_number, column_number,None)elif name =="http_status"andtype(value)isint: sheet.write_number(row_number, column_number, value)else: text =str(value)iflen(text)>32_767:raise ValueError("Move oversized content to an artifact") sheet.write_string(row_number, column_number, text) count +=1 sheet.freeze_panes(1,0) sheet.autofilter(0,0, count,len(columns)-1) sheet.set_column(0,0,54) sheet.set_column(1,2,32) sheet.set_column(3,5,22)finally: workbook.close()print("exported rows:", count)
Step 3: Open and reconcile the workbook
Run python export_xlsx.py, then open the workbook and compare its data-row count with the normalized input. Check Unicode text, source URLs, blank values, and formula-like source strings. Publish or distribute the file only after the process exits successfully; an interrupted run can leave a partial artifact.
The XlsxWriter worksheet interface documents explicit string and numeric writing. Keeping source text as text avoids having a scraped title unexpectedly become an executable spreadsheet formula.
Method 3: Use CSV for a simple interchange table
Step 1: Choose the import contract
Agree on UTF-8 encoding, a delimiter, column order, and timestamp representation with the receiving system. The Python CSV interface handles quoting and newlines, but it does not control how another application interprets a cell.
Step 2: Write a separate presentation export
If the CSV will be opened interactively, apply a documented policy for formula-like text or choose the typed XLSX method instead. Preserve an unchanged machine-readable dataset separately when a presentation export adds escaping or prefixes to protect spreadsheet consumers.
Step 3: Import identifier columns as text
Use Excel's text/CSV import controls to verify encoding and assign text types to identifiers before loading the table. Do not rely on double-clicking a CSV when leading zeros, long IDs, or date-like strings must remain unchanged.
[Screenshot: Excel's CSV import preview showing UTF-8 encoding, the selected delimiter, and identifier columns configured as text.]
Where Does Nstdata Crawl Fit in the Export Architecture?
Nstdata Crawl fits before the export writer as a Web Crawling API for LLM-ready data. It can discover and recursively collect approved pages within limits, render JavaScript where needed, and return reusable representations. Your application then decides which values become rows, which artifacts remain outside the workbook, and which records are accepted. Built-in proxy access can simplify the standard collection request. A spreadsheet export still needs its own type, safety, and completeness rules.
Multiple output roles: Markdown supports text workflows, HTML supports extraction, and links support discovery. JSON, PDF, and visual outputs address other delivery and review needs; verify the current API representation rather than assuming a native XLSX option.
Managed proxy configuration: The standard managed path can use Nstdata's proxy option without a separate gateway and credential setup. Account for proxy usage in the current billing model.
Task and artifact handling: Save job IDs and resolve the returned large-artifact references you need. Keep the workbook linked to a manifest that records accepted, rejected, and missing results.
Why Do Crawl Exports Lose Data or Open Incorrectly?
Crawl exports lose data or open incorrectly when the writer assumes a flat response, misses result batches, or leaves interpretation entirely to the spreadsheet application. Validate both the dataset and the file.
Symptom
Likely cause
Repair
Only the first batch appears
Result pagination was ignored
Follow the documented continuation contract
IDs lose leading zeros
Automatic numeric conversion
Write or import identifier cells as text
Text becomes a formula
Untrusted values interpreted by the consumer
Use explicit text cells or a documented safe-import policy
Long descriptions disappear
Cell-size limit or silent truncation
Store artifacts outside the table and keep references
Row count looks correct but data is duplicated
Repeated cursor page or weak identity
Reconcile unique record keys, not only total rows
Use structured extraction validation before file generation. For long-running exports, checkpoint the source position and write the final filename only after validation succeeds.
How Should You Deliver the Finished Spreadsheet?
Deliver the spreadsheet with a clear scope, collection date, data dictionary, and reconciliation counts. Explain whether blank cells represent missing source values, inapplicable fields, or rejected extraction. Keep detailed diagnostics in an appropriate review table or manifest so the main worksheet remains usable.
Start with a small normalized sample and prove that identifiers, Unicode, missing values, and untrusted text survive the round trip. Nstdata's alternate HTML output can help investigate disputed fields before they enter the final workbook.
No. XLSX is a structured workbook format. Normalize the source records and use an actual workbook writer or a documented native export.
Q: Does Nstdata Crawl provide a native XLSX response?
This guide does not assume a native XLSX response. It uses Nstdata for collection and a separate application export layer for workbook generation.
Q: Why do I need a normalized schema?
A normalized schema defines what one row means and how each field should be interpreted. It prevents provider-specific envelopes and nested arrays from becoming ambiguous spreadsheet cells.
Q: Is CSV quoting enough to prevent spreadsheet formulas?
No. CSV quoting controls file syntax, while formula interpretation belongs to the receiving application. Use a documented import policy or explicit XLSX text cells.
Q: What proves the export is complete?
A complete export reconciles the intended job scope, all retrieved result batches, accepted and rejected records, unique identities, and the finished file's row count.
Lena Zhou
Nov. 9th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
