How to Migrate from Firecrawl to Nstdata: A Developer Guide
TL;DR
A Firecrawl-to-Nstdata move is an adapter migration, not a simple base-URL swap.
Map start, poll, collect, normalize, and error paths separately; keep your downstream document schema stable.
The public Nstdata quickstart and route sections differ on synchronous scrape paths, so confirm your account's live API reference before wiring production requests.
A small happy-path integration may take 30 minutes; full migration time depends on your crawl filters, error handling, and acceptance tests.
Firecrawl's Crawl API guide describes POST /v2/crawl, a job ID, polling through GET /v2/crawl/{jobId}, and a paginated next link for completed results. Nstdata Crawl documents a site-crawl submission route, status lookup, page listing, and separate storage reads for artifact references. That means the application boundary should be an internal crawl_site() adapter returning your own page records. Keep RAG chunking and vector storage unaware of vendor response shapes.
Pipeline step
Firecrawl public contract
Nstdata public contract
Adapter action
Start crawl
POST /v2/crawl
POST /api/v1/crawl in current docs
Translate URL, path, and limit options
Poll job
GET /v2/crawl/{jobId}
GET /api/v1/crawl/status/{crawlId}
Normalize state and terminal errors
Collect pages
Crawl result pages and next
GET /api/v1/crawl/pages/{crawlId}
Paginate until complete
Read content
Inline page content
Artifact reference such as markdownRef
Resolve storage reference when required
Log failures
Crawl errors plus page status
Job and page statuses
Retain original vendor error fields
The Scrapy overview is useful when you own discovery during a transition. The Nstdata Crawl API documentation is the authority for account-specific request and response fields. Its quickstart and route sections currently show inconsistent scrape endpoint labels. This tutorial intentionally avoids inventing a production-ready request body or pretending a single route works for every account. Copy the verified example from your live reference, then run the normalization test below.
Before: keep Firecrawl behind an adapter
The example below is illustrative and needs your Firecrawl key and installed SDK. It shows the seam to preserve, rather than a claim that this environment executed a paid crawl.
from firecrawl import Firecrawl
client = Firecrawl()result = client.crawl("https://example.com/docs", limit=50)
Avoid having downstream code reach into Firecrawl-specific fields all over the application. The documented .crawl() convenience call can hide polling and pagination. Inspect its current SDK return type before committing a replacement implementation. If your pipeline uses JSON extraction, screenshots, or crawling filters, record each option explicitly. Those have different cost and schema implications, described in Firecrawl's credit pricing.
After: normalize results before changing the vendor
The following Python code is offline-testable and uses standard library only. Feed it page dictionaries from an authenticated Nstdata crawl after verifying the current response structure. It avoids binding the rest of your RAG pipeline to a vendor field name.
from dataclasses import dataclass
from typing import Mapping, Any
@dataclass(frozen=True)classDocument: url:str markdown:str status:int|None source_id:str|Nonedefnormalize_nstdata_page(page: Mapping[str, Any], markdown:str)-> Document: url =str(page.get("url")or page.get("sourceUrl")or"")ifnot url.startswith(("http://","https://")):raise ValueError("Missing absolute source URL") body = markdown.strip()ifnot body:raise ValueError(f"Empty Markdown for {url}") raw_status = page.get("statusCode") status =int(raw_status)if raw_status isnotNoneelseNonereturn Document(url, body, status,str(page.get("id"))if page.get("id")elseNone)
Here markdown is the retrieved artifact text; a markdownRef token is not the body itself. The Nstdata subscription documentation also makes proxy traffic a separate cost consideration. Keep a per-page ledger with input URL, canonical URL, status, artifact reference, retrieval time, and billable units.
A migration script outline you can adapt
Use this sequence in a one-off script: load a fixed URL manifest; submit a bounded Nstdata crawl; poll with backoff until terminal; enumerate every result page; fetch referenced Markdown; call normalize_nstdata_page; compare accepted URLs with a Firecrawl baseline; write a JSON Lines file. Make the script resumable from crawlId, so a timeout does not start a duplicate paid crawl. Limit concurrency and record request IDs to diagnose partial failures. Do not silently drop 403, 404, empty, or duplicate pages.
A passing dry run should prove that the same downstream chunker accepts both adapters. It should also find removed URLs, duplicate canonicals, changes in tables or code fences, and differences in headings. The RAG knowledge-base guide, chunking strategies, and crawl success-rate method provide checks for this handoff. The test should compare accepted documents and answer quality, not byte-for-byte Markdown equality.
What usually takes longer than 30 minutes?
Authentication setup, route verification, result pagination, artifact storage reads, changed path filters, retry policy, and monitoring take more time than replacing one SDK call. Allow a 30-minute proof of concept for a small site; budget separately for production parity. Keep the Firecrawl path behind a feature flag until the Nstdata adapter passes your corpus and cost thresholds. The switch is complete when scheduled recrawls, deletion, and error alerts work, not when the first page returns.
Experience Nstdata — Start Your Free Trial Today
Run a bounded comparison on a site you may crawl and inspect the actual page artifacts before moving scheduled jobs.
