Mastering the Firecrawl Scrape Endpoint for Production Use
TL;DR
Firecrawl's current single-page API is POST https://api.firecrawl.dev/v2/scrape, authenticated with a Bearer API key.
The formats array determines whether the response contains Markdown, HTML, links, screenshots, structured JSON, or other supported artifacts.
Mastering the Firecrawl scrape endpoint means validating the response envelope and page semantics, not merely receiving HTTP 200.
Use onlyMainContent, cache controls, timeout, location, and limited actions deliberately; complex interaction belongs in Firecrawl's Interact endpoint.
Compare managed scraping APIs on usable-page accuracy, diagnostics, latency, and billing model against your own target set.
What the Firecrawl Scrape Endpoint Does
The Firecrawl scrape endpoint turns one known URL into one or more requested page representations. Firecrawl handles fetching and browser rendering on its infrastructure, then returns artifacts selected through formats. It is the right Firecrawl operation when you already know the page URL; site discovery belongs to a crawl operation, while multi-step browser behavior increasingly belongs to Interact.
Firecrawl's current scrape endpoint reference documents url as required and Bearer authentication as mandatory. The broader Firecrawl v2 introduction confirms the https://api.firecrawl.dev base URL and conventional HTTP status handling.
A managed endpoint removes browser, proxy, and rendering operations from your application, but it does not know what makes a record correct for your business. The application still needs acceptance rules, stable identity, retention policy, and retry boundaries. The same division applies when evaluating or an in-house browser fleet.
The smallest request contains url; production requests usually add only the controls that affect the desired output.
Field
What it changes
Decision rule
url
Target page
Use a public or authorized HTTP(S) URL
formats
Returned artifacts
Request only outputs the consumer uses
onlyMainContent
Boilerplate reduction
Enable for article-like text; test on app pages
waitFor
Additional page delay
Use only when a known element or request needs time
timeout
Maximum processing window
Keep bounded; retry async or redesign slow work
location
Geographic/language context
Use when localized output is part of acceptance
storeInCache
Whether Firecrawl may cache the page
Disable when retention or freshness requires it
actions
Simple page actions before capture
Keep deterministic; use Interact for complex flows
The current reference lists a default timeout of 60 seconds and an allowed range from 1,000 to 300,000 milliseconds. These limits are change-sensitive, so confirm them before publishing client-side validation. Firecrawl also documents cache-only and reduced-retention controls; treat them as data-governance choices, not performance toggles.
Authenticate Without Leaking the API Key
Firecrawl expects Authorization: Bearer <token>. Put the token in a secret manager or environment variable and never write it into source code, logs, screenshots, or a committed .env file.
The examples below use $FIRECRAWL_API_KEY. They are schema-verified against current first-party documentation but cannot be executed here without a user-owned Firecrawl credential. Run them against an authorized test URL in your environment and capture the response before adopting the schema.
--fail-with-body preserves the server's error body while returning a failing shell status for HTTP errors. Do not assume that the command's success proves content correctness.
Step 2: Validate the response envelope
A successful Firecrawl response includes a top-level success indicator and a data object. Check both before reading the artifact:
const response =awaitfetch('https://api.firecrawl.dev/v2/scrape',{method:'POST',headers:{Authorization:`Bearer ${process.env.FIRECRAWL_API_KEY}`,'Content-Type':'application/json'},body:JSON.stringify({url:'https://example.com/',formats:['markdown'],onlyMainContent:true})});const payload =await response.json();if(!response.ok|| payload.success!==true){thrownewError(payload.error??`Firecrawl failed with ${response.status}`);}const markdown = payload.data?.markdown;if(typeof markdown !=='string'|| markdown.trim().length<80){thrownewError('Firecrawl returned no acceptable Markdown');}
An acceptance check should also look for target-specific markers: a title, product identifier, date, table header, or other field that proves the intended page was returned. This catches consent pages, soft errors, and content that rendered but did not reach the required state.
Test a Production Scraping Alternative
Compare Firecrawl with Nstdata Crawl on your real URLs, formats, and acceptance checks.
Structured extraction works better when the schema describes only necessary fields and their types. Require a stable source identifier when the page provides one. Avoid asking a model to infer values that do not exist in the page.
{"url":"https://example.com/product/123","formats":[{"type":"json","prompt":"Extract the visible product record. Return null for absent optional fields.","schema":{"type":"object","properties":{"name":{"type":"string"},"sku":{"type":["string","null"]},"availability":{"type":["string","null"]}},"required":["name"]}}]}
Step 2: Validate semantics after schema validation
Schema compliance proves shape, not truth. Reject a generic name, normalize whitespace, map availability to an approved vocabulary, and compare the source URL or SKU with the job input. Store the raw artifact or a content hash when audit requirements permit so an operator can explain how the accepted record was produced.
Method 3: Capture a Screenshot or HTML for Diagnosis
Markdown is efficient for downstream text use, but it can hide why an extraction failed. Request a screenshot when visual state matters and HTML when DOM structure matters. Do not request large artifacts on every recurring job unless they have an explicit debugging, compliance, or archival purpose.
The Firecrawl endpoint supports actions such as waits, clicks, typing, scrolling, screenshots, and JavaScript execution. The current documentation recommends the separate Interact endpoint for complex interactions. Keep actions short and deterministic; authenticated workflows require explicit permission and careful secret handling.
Cache, Freshness, and Data Retention
Cache behavior changes both freshness and data handling. A cached response can reduce latency, but it may be unacceptable for inventory, policy, or monitoring jobs that require a current observation. Conversely, storeInCache: false can be appropriate where the page should not be retained by the provider.
Record the requested freshness policy with every job. If a workflow compares changes, persist the collection time and content hash; do not treat provider cache age as the source page's publication time. Firecrawl's official article on using the scrape API covers formats and examples, but production acceptance remains application-specific.
Error Handling, Rate Limits, and Retries
Retry only failures likely to be transient. Firecrawl documents 429 for rate or concurrency limits; honor any retry guidance, cap attempts, and add exponential backoff with jitter. Retry selected 5xx failures, network interruptions, and timeouts, but do not repeatedly retry invalid URLs, authentication failures, or schema errors.
Make downstream writes idempotent. A job key can combine normalized URL, requested format set, freshness window, and extraction-schema version. Log the job key, HTTP status, Firecrawl request or scrape identifier when returned, elapsed time, artifact sizes, and validation result. Never log the Bearer token or sensitive request headers.
If a team later moves from managed fetching to direct proxy routing, review how rotating proxy sessions affect retries and page consistency before changing the collector.
The public Firecrawl OpenAPI repository is useful for detecting schema changes, but verify the deployed v2 documentation before generating clients because repository and hosted API revisions can differ.
When Firecrawl Scrape Is Not the Right Operation
Use /scrape for one known page. Use Firecrawl crawl when you need bounded discovery across internal links, batch capabilities for a known list of many URLs, and Interact when the workflow needs sustained browser state or several complex actions. An official data API remains preferable when it exposes the required records with clear permission and stable identifiers.
For provider selection, compare the complete operational result. Nstdata Crawl can be tested against the same URL set and acceptance harness. Nstdata Crawl is positioned for page scraping and bounded site crawling with multiple artifacts and task operations; suitability depends on the exact rendering, geographic, diagnostic, and storage requirements.
Page and site workflows: synchronous or asynchronous page scraping sits beside bounded crawl submission and polling.
Artifact choices: Markdown, HTML, raw data, links, screenshots, and PDFs serve different consumers and debugging needs.
Task visibility: task IDs and status checks support slow pages and repeatable operations.
Selection boundary: teams must still benchmark usable-page rate, latency, completeness, and cost per accepted record.
Conclusion: Treat Scrape as One Stage of a Data Contract
Mastering Firecrawl's scrape endpoint requires more than choosing formats. A reliable integration protects the API key, bounds time and actions, validates the response envelope, applies target-specific acceptance tests, and stores records idempotently.
Start with five representative authorized URLs: one static page, one JavaScript page, one redirect, one expected failure, and one locale-sensitive page. Measure usable output rather than HTTP success. If geographic routing and centralized proxy operations later become the bottleneck, evaluate Nstdata Proxy Manager as the related network-control layer.
The current Firecrawl v2 single-page endpoint is POST https://api.firecrawl.dev/v2/scrape. Send a Bearer API key and a JSON body containing at least url.
Q: What is the difference between Firecrawl scrape and crawl?
Firecrawl scrape processes one known URL, while crawl discovers and processes multiple pages from a starting URL within configured boundaries. Choose based on whether URL discovery is part of the job.
Q: Which Firecrawl format should I request?
Request Markdown for text and LLM ingestion, HTML for DOM-aware processing, JSON for a defined record, and screenshots for visual evidence. Request only artifacts that a downstream consumer or diagnostic process uses.
Q: Does an HTTP 200 mean the Firecrawl scrape succeeded?
HTTP 200 does not by itself prove that the intended page data is usable. Check Firecrawl's success field, required artifact, page metadata, and target-specific content markers.
Q: How should I handle Firecrawl 429 errors?
Handle Firecrawl 429 responses with bounded exponential backoff and jitter, honor server retry guidance when supplied, and reduce submission rate or concurrency. Keep writes idempotent so a retry cannot duplicate records.
Q: Can Firecrawl scrape pages that require interaction?
Firecrawl supports simple actions in scrape requests, but its current documentation directs complex browser interactions to the Interact endpoint. Use authenticated interaction only with explicit authorization and secure cookie handling.
Q: Is Firecrawl priced per request?
Firecrawl uses a credit-based service model whose consumption varies by operation and format. Check the current official pricing and billing documentation rather than embedding a changing numeric price in application logic.
Marcus Chen
Sep. 9th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
