Crawlee for Python: A Practical Guide to Web Scraping Workflows
TL;DR
Crawlee for Python is a crawler workflow framework. It manages requests, queues, handlers, and datasets around your extraction code; it is more than a parser or a browser driver.
Start with an HTTP crawler for static or server-rendered pages. Choose BeautifulSoupCrawler or ParselCrawler; move to PlaywrightCrawler only when the content really requires browser rendering.
The current official Python docs are version 1.11 and require Python 3.10 or newer. Pin the version you validate, especially when upgrading storage or HTTP-client behavior.
Use a page cap and an allowlisted domain in your first run. Crawlee deduplicates queued URLs, but you still need to define scope, extract only authorized data, and measure completeness.
Crawlee for Python is an open-source framework for building web crawlers and data extraction workflows. A crawler starts with URLs, sends each request through a queue, invokes a handler for each page, and stores extracted records in a dataset. That gives a project structure for crawl orchestration that a raw requests.get() call does not provide. If you prefer a managed service for permitted URL and site collection, Nstdata Crawl is another option; it does not replace Crawlee’s custom handlers or application-specific API logic.
The practical distinction is:
Requests + Beautiful Soup: you control each request and parse the HTML. Good for a small, fixed task.
Crawlee: you define how to process each request while the framework manages a crawl queue, duplicate requests, handlers, and storage.
Playwright: you automate a browser. It can render JavaScript, but it does not by itself supply a complete crawler workflow.
The current Crawlee Python quick start lists three crawler classes: BeautifulSoupCrawler, ParselCrawler, and PlaywrightCrawler. The HTTP crawlers do not render JavaScript; the Playwright crawler controls a headless browser.
Crawlee for Python vs Scrapy vs Playwright
Tool
Main role
Request queue and crawl workflow
JavaScript rendering
Data storage
Good fit
Crawlee for Python
Python crawling framework
Built-in request queue, handlers, routing, and storage
Yes, through PlaywrightCrawler
Dataset and key-value storage
Python teams that want a managed crawl loop and an HTTP-to-browser choice
Scrapy
Full Python crawling framework
Built-in scheduler, downloader, middleware, and pipelines
Not by itself; add a browser integration when required
Item pipelines and feeds
Larger, highly customized crawling projects with an established Scrapy stack
Playwright
Browser automation library
No general-purpose crawl queue or dataset by itself
Yes
Your application code
Browser tests and pages whose authorized content must be rendered or interacted with
Requests + Beautiful Soup
HTTP client plus HTML parser
No crawl queue by itself
No
Your application code
A few known URLs or a simple API/page parser
A common mistake is to compare Crawlee and Playwright as if they solve the same layer. Crawlee can use Playwright as its browser engine; Playwright alone leaves URL discovery, deduplication, job limits, output, and restart behavior to your application. See the Scrapy spider documentation and Playwright for Python documentation for the boundaries of those tools.
Crawler: chooses an HTTP or browser-backed implementation and runs the job.
Request / RequestQueue: represents pages to visit and keeps the work list. Crawlee deduplicates requests using a URL-derived unique key by default.
Router and handler: decide how to process a request. A single default handler is enough for a first scraper; labels and route handlers help separate listing pages from detail pages later.
Dataset: stores structured records. Key-value storage is useful for artifacts such as files, screenshots, or state.
HTTP Crawlers vs Browser Crawlers
Use an HTTP crawler when the server returns the data in the HTML response. It avoids the overhead of starting a browser and is usually simpler to deploy. Use PlaywrightCrawler when a page needs client-side rendering to expose the content you are permitted to collect.
Before moving to a browser, inspect the page’s initial response and check whether its data is available in HTML or a documented API. Browser rendering adds CPU and memory, browser binaries, longer page loads, and more points of failure. It is not a way to bypass access controls or CAPTCHA challenges.
Request Queues, Routing, and Datasets
A queue is useful when pages link to more pages, or when a job begins with a known list but discovers additional work as it runs. The request handler can extract a record and enqueue the next allowed links. The queue handles URL deduplication; your code still has to define which pages are in scope and what qualifies as a valid record.
For an e-commerce catalog, a listing handler can extract product cards and enqueue the single next-page link. A separate detail handler can process product URLs if the project needs more fields than the listing provides. Keep list-page and detail-page schemas explicit so an empty listing record is not mistaken for a product.
Prerequisites and Installation
Crawlee for Python currently requires Python 3.10 or newer. The quick-start installation is:
For a browser-backed crawler, install the Playwright browser dependencies as documented:
playwright install
Pin the dependency in a lock file for reproducible deployments. The current docs are version 1.11; the Crawlee 1.0 upgrade guide describes changes to storage and the default HTTP client, so do not assume an older code sample behaves identically after a major upgrade.
Step 1: Build Your First Crawlee Python Scraper
This example uses Books to Scrape, a demonstration catalog. It reads product cards from the listing page, follows only the “next page” link, caps the crawl at three requests, and exports the default dataset to CSV. Run examples against a site intended for testing or one you are authorized to crawl.
Configure the Crawler and Start URLs
Save the following as books_catalog.py:
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
asyncdefmain()->None: crawler = BeautifulSoupCrawler(max_requests_per_crawl=3)@crawler.router.default_handlerasyncdefhandle_page(context: BeautifulSoupCrawlingContext)->None: soup = context.soup
for card in soup.select("article.product_pod"): title_link = card.select_one("h3 a") price = card.select_one(".price_color") availability = card.select_one(".availability")if title_link isNone:continueawait context.push_data({"source_url": context.request.url,"title": title_link.get("title")or title_link.get_text(strip=True),"product_url": title_link.get("href"),"price_text": price.get_text(strip=True)if price elseNone,"availability_text": availability.get_text(" ", strip=True)if availability
elseNone,})# Follow the listing's next-page link only; stay on the same hostname.await context.enqueue_links(selector="li.next a")await crawler.run(["https://books.toscrape.com/"])await crawler.export_data(path="books.csv")if __name__ =="__main__": asyncio.run(main())
Then run it:
python books_catalog.py
The request cap intentionally keeps the sample small. For a real project, set a page or request budget that matches the scope you have permission to collect, and record why each request was accepted.
Extract Data and Save Results
Each call to context.push_data() adds one record to the default dataset. The export step writes that dataset to a CSV file. A representative row has this shape:
{"source_url":"https://books.toscrape.com/","title":"A Light in the Attic","product_url":"catalogue/a-light-in-the-attic_1000/index.html","price_text":"£51.77","availability_text":"In stock"}
The exact current values can change on a live site; the snippet shows the fields, not a promised captured result. Preserve the source URL with every record. For production, normalize relative links with urljoin(), store a crawl timestamp, and validate that each product has the fields your downstream job requires.
Step 2: Crawl Multiple Pages with Request Queues
The example already uses Crawlee’s implicit request queue: crawler.run() adds the start URL, and the handler enqueues the next page. When a site has many known URLs, use crawler.run(urls) or the queue APIs to seed the job. Crawlee’s official introduction explains how request queues and link discovery work.
Discover and Enqueue Links
By default, context.enqueue_links() stays on the same hostname; a selector narrows which anchors are considered. For documentation sites, a crawl may use a selector such as a.sidebar-link and a page cap, then apply include/exclude patterns for sections.
Normalize URLs before storing them. Also decide whether query strings identify distinct pages: filters and tracking parameters can create many URLs that show the same content. Queue deduplication helps with exact URL variants, but it cannot infer that two semantically different URLs contain the same article.
Use Routers for Different Page Types
When listing and detail pages need different selectors, assign labels when you enqueue URLs and register separate route handlers. This keeps extraction logic readable:
Listing handler: collect cards, then enqueue detail links.
Detail handler: extract the fields that exist only on a product page.
Error handler: record failed URL, status, and a short error category.
Do not put all page types into one giant conditional once a crawler grows. Separate handlers make it easier to test a page fixture against the correct schema.
Step 3: Scrape JavaScript-Rendered Websites
Switch to PlaywrightCrawler when the permitted page content is missing from the HTTP response and appears only after client-side rendering. The official quick start shows the same basic handler interface with a Playwright context:
This illustrates the browser crawler API; it does not promise that a dynamic target is accessible. If your own application renders after a known selector appears, wait for that selector in a site-specific handler and set a timeout. Avoid a fixed long sleep when an explicit readiness condition is available. Do not use browser automation to defeat a challenge or access restriction.
Step 4: Clean and Validate the Extracted Data
A crawler can finish successfully while producing incomplete records. Add checks that fail the job or route records to a review dataset when required fields are missing:
Require a stable identifier or source URL.
Reject a product record without a title; keep price nullable if the page does not provide one.
Normalize whitespace, currency, and relative URLs in a separate transformation step.
Deduplicate on a business key such as the canonical product URL, not only on title.
Save raw or representative HTML fixtures for parser regression tests.
Crawlee’s storage guide documents datasets and key-value storage. Keep large artifacts out of a relational row; store a reference or object key instead.
Step 5: Benchmark Throughput and Reliability
Do not estimate production throughput from a three-page tutorial run. Benchmark the same authorized URL set on the same machine and record:
Measurement
Why it matters
Valid records / requested pages
Separates extraction quality from HTTP success
Median and p95 page latency
Captures slow pages without hiding the tail
Requests per minute
Measures throughput under the chosen concurrency
CPU and peak memory
Browser crawlers can have a different resource profile
Retry and failure counts
Reveals fragile targets or configuration problems
Output bytes and storage growth
Matters for long-running or artifact-heavy jobs
Run a small pilot, inspect samples manually, then increase volume gradually. Set a maximum request count, per-domain concurrency, and job timeout before moving to scheduled execution.
Step 6: Deploy a Crawlee Python Job
For a small scheduled HTTP crawl, package the Python environment and storage output, then run the script from a container or job scheduler. A browser-backed run also needs compatible browser binaries and system dependencies; include them in the image and test the exact deployment base image.
Plan for process restarts and idempotency. Give each run an ID, preserve source URLs and timestamps, and make downstream writes safe to repeat. If a crawl must resume from a persisted queue after a crash, test that behavior with the storage configuration you intend to deploy rather than assuming a local run proves cloud persistence.
When to Use Crawlee for Python vs a Managed Crawl API
Crawlee is a strong choice when you need custom extraction logic, want control over scheduling and storage, and can operate the crawler yourself. A managed crawl API can reduce the work of browser infrastructure and return normalized page content, but you still own crawl policy, data validation, and downstream processing.
Nstdata Crawl describes URL or site-level crawling with configurable page limits and outputs such as Markdown and HTML. It is useful when the application needs page content rather than a custom crawler codebase. It is not the right tool for every task: use Crawlee when custom control or self-hosting is the requirement; use a managed service when reducing browser and crawl-infrastructure operations is more valuable. Related Nstdata guides cover JavaScript-rendered pages in Python, headless scraping for dynamic websites, and automated data collection.
Conclusion
Crawlee for Python gives a Python project a reusable crawl loop: requests enter a queue, handlers extract records, and a dataset stores output. Begin with an HTTP crawler, make the allowed URL scope explicit, and add Playwright only when rendering is required. A page cap and data-quality checks are more useful than a large crawl that quietly stores incomplete rows.
Experience Nstdata — Start Your Free Trial Today
If you prefer a managed service for permitted URL and site collection, review Nstdata Crawl’s documented inputs and outputs before a proof of concept.
Crawlee is open source and its documentation describes it as free. You still pay for the compute, browser runtime, storage, and infrastructure that you operate.
Q: Can Crawlee render JavaScript?
Yes. Crawlee for Python includes PlaywrightCrawler for browser-backed pages. BeautifulSoupCrawler and ParselCrawler use HTTP responses and do not render JavaScript.
Q: Is Crawlee better than Scrapy?
It depends on the project. Crawlee offers an integrated queue, handlers, and storage workflow with an HTTP-to-Playwright crawler choice; Scrapy is a mature framework with its own scheduler, middleware, and pipelines. Compare the integrations your team already operates.
Q: Does Crawlee automatically avoid duplicate URLs?
Its request queue deduplicates requests using a URL-derived unique key by default. You should still canonicalize URLs and deduplicate records by the business identifier your application needs.
Q: How do I export Crawlee data?
The quick-start pattern stores records in a dataset under local storage. The crawler API also provides an export method, such as crawler.export_data(path="books.csv"), for formats supported by the installed version.
Ivy Lin
Oct. 19th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
