Robots.txt for Web Scraping: A Practical Compliance Guide
TL;DR
robots.txt communicates crawler access preferences for an origin; an allowed path does not grant a license or access to private data.
Match the crawler's declared product token and the exact URL path using a parser whose behavior you have tested.
Treat a robots retrieval failure, an unavailable policy decision, and a target-page failure as separate operational states.
Crawl-delay is an extension, not a directive standardized by the Robots Exclusion Protocol.
Check policies before enqueueing work, then apply independent page, depth, concurrency, and retention limits.
Nstdata Crawl can supply managed collection and LLM-ready outputs after your application has established the permitted scope.
What Does robots.txt Mean for a Web Scraping Project?
A robots.txt file tells cooperating crawlers which URL paths a website operator wants them to access. The file belongs at the root of an origin, such as the robots.txt path on the same scheme and host as the target. A policy for one host should not be silently reused for another host. This robots txt for web scraping guide treats the file as an input to your collection policy, with an explicit decision before a URL enters the queue.
The Robots Exclusion Protocol states that these rules are not a form of access authorization. An Allow directive does not settle copyright, contract, privacy, or permitted reuse. Conversely, robots.txt is not a secure place to hide confidential paths: private resources need actual access controls.
This distinction remains relevant when you use Nstdata Crawl. Moving retrieval to a managed API changes who operates the browser and network infrastructure. Your team must still decide which origins, paths, fields, and uses are approved. Keep that decision separate from whether a service can technically fetch a page.
Which Rules Should You Understand Before You Start?
Understand User-agent groups, Allow and Disallow path rules, and the difference between standardized matching and vendor extensions before implementing a policy check.
Experience Nstproxy Crawl - Start Your Free Trial Today
Item
Practical meaning
Operational consequence
User-agent
Crawler product token to which a group applies
Keep the declared identity consistent with the policy check
Disallow
Paths the group should not crawl
Exclude matching URLs before collection
Allow
Paths the group may crawl despite a broader restriction
Apply the parser's documented precedence rules
Wildcard and end anchor
Matching syntax defined by the protocol
Test representative encoded URLs and query strings
Crawl-delay
Common extension outside RFC 9309
Check support in the chosen parser and apply your own scheduler
Sitemap
Discovery hint, rather than permission
Review every discovered URL against scope and policy
The protocol resolves overlapping path matches by specificity, including its treatment of equivalent Allow and Disallow matches. Do not implement the full algorithm by searching for a substring or taking whichever line appears first. Encoding, group selection, and matching rules make that shortcut unreliable.
Google's robots.txt interpretation guide documents Google's behavior. Use it to understand Google-specific processing, including unsupported directives; do not assume another crawler reproduces every Google behavior. Likewise, a meta robots tag controls other indexing-related behavior and does not replace a crawl-access check.
What Do You Need Before Checking a Site?
Prepare an approved origin, the crawler product token, candidate URLs, a parser, and a policy for unavailable robots files before making target-page requests. This policy should name an owner who can resolve uncertainty instead of letting each worker improvise.
For the Python demonstration, install a supported Python runtime and save the script below as robots_probe.py; it uses the standard library. The script downloads only Google's public robots file and reports two policy decisions. It does not request either tested page. Network access and a working TLS trust store are required; no Nstdata account is needed for this method.
For a production manifest, record origin, candidate URL, declared crawler identity, policy retrieval time, policy content hash, parser version, decision, and decision reason. Keep those fields alongside the source inventory for site crawling. Do not store authentication headers in this manifest.
How Do You Check robots.txt Before Collecting Pages?
Check robots.txt manually for a small review, use a tested parser for a URL manifest, or enable framework-level enforcement for a crawler. These methods can complement one another; none makes a separately unauthorized collection acceptable.
Method 1: Review a small scope manually
Step 1: Retrieve the origin's robots file
Open the origin's /robots.txt location and record the response and retrieval time. Use the policy file on the relevant origin rather than an unrelated parent domain.
Step 2: Identify the applicable crawler group
Locate the group matching your declared crawler product token and review its path rules. If the interpretation is uncertain, keep the URL outside the approved queue until the uncertainty is resolved.
Step 3: Record the decision for representative URLs
Evaluate a permitted-looking page, an excluded path, a query-string variant, and any redirected destination. Save the reasons with your manifest. A manual review is useful for a pilot, but it becomes difficult to maintain when origins and policies change frequently.
Method 2: Add a Python preflight check
Step 1: Save the policy-check script
The following demonstration uses urllib.robotparser, whose documented methods expose checks such as can_fetch. Parser behavior is part of the implementation contract: test it against the directives your sources use before treating it as a complete RFC-conformance solution.
Run python robots_probe.py. The tested public file denied both /search and / for the demonstration identity. The output is a policy evaluation at retrieval time, not evidence that a target page was collected or that a project has reuse rights.
Step 3: Apply the decision before queueing
In your application, admit only approved URLs whose policy decision passes. Route an exception, an oversized robots file, a redirect requiring review, or an unknown decision to a review queue. This deliberately conservative application policy is distinct from the protocol's own rules for particular HTTP and network outcomes.
Method 3: Enforce the rule in a crawler framework
Step 1: Enable the framework setting
For Scrapy, set ROBOTSTXT_OBEY = True in the project settings. Confirm the effective settings in the running project rather than assuming a template or deployment inherited the intended value. The Scrapy robots setting documents the switch and related parser configuration.
Step 2: Set explicit scheduling bounds
Configure a deliberate download delay and concurrency ceiling for the approved workload. Respect supported source-specific limits, and avoid assuming that a missing Crawl-delay means unlimited request volume is appropriate.
Step 3: Test an excluded candidate
Use a controlled test origin to verify that an excluded candidate never reaches the target download stage. A log message saying that middleware loaded is weaker evidence than an observed exclusion. Framework settings do not remove the need to review redirects, offsite requests, and application-created requests that may follow different code paths.
How Can Nstdata Crawl Fit a Policy-Controlled Workflow?
Nstdata Crawl fits after your policy service has approved the collection scope. Nstdata Crawl is a Web Crawling API for LLM-ready data that can convert approved website sections into reusable page outputs. Site discovery and recursive crawling reduce the need to maintain your own link frontier. JavaScript rendering supports pages whose useful content arrives after the initial HTML. The application remains responsible for scope approval, output acceptance, and downstream reuse.
Bounded discovery: Set page and depth limits and verify discovered URLs against your approved inventory. Do not assume a filter alone demonstrates enforcement of every source policy.
Outputs for different checks: Markdown serves RAG and agents, HTML supports parsing, links support discovery, and PDF or screenshots support review. Raw data provides diagnostic bytes; a JSON response envelope does not guarantee business-record correctness.
Built-in proxy routing: Standard managed calls can use Nstdata's built-in proxies without separate Host, Port, Username, Password, or Protocol fields. Proxy routing does not change permissions, and proxy traffic has a separate dimension in the Crawl billing model.
To connect the workflow, obtain an API key through Dashboard → Crawl Playground, install nstdata-ai-crawl under Python 3.10 or later, and set NSTDATA_API_TOKEN, APPROVED_PAGE_URL, and optionally EXPECTED_TEXT. The following credential-dependent example requires your own key and an approved target; it is not an authenticated collection result from this tutorial.
[Screenshot: Nstdata Crawl Playground with the approved target and requested format; keep the API key hidden.]
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
with NstDataClient(os.environ["NSTDATA_API_TOKEN"], max_response_bytes=1_048_576)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=os.environ["APPROVED_PAGE_URL"], formats=["markdown"], onlyMainContent=True, timeout=60000,))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Collection did not complete successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") expected = os.environ.get("EXPECTED_TEXT")if expected and expected notin text:raise RuntimeError("Required source text is missing") Path("accepted-page.md").write_text(text, encoding="utf-8")print("Saved accepted-page.md; validate facts before ingestion")
Start with one approved page and inspect the saved Markdown against the source. For a subsequent site crawl, set maxPages, maxDepth, include/exclude rules, and a reviewable stopping condition. Review batch task handling before expanding the job. Do not claim that submitting one permitted entry URL automatically permits every page discovered beneath it.
Why Might a Policy Check Produce the Wrong Decision?
A policy check can produce the wrong decision when the implementation uses the wrong origin, crawler identity, parser behavior, or policy snapshot. Diagnose the policy stage independently of target retrieval.
Symptom
Likely cause
Corrective action
Allowed in review, denied in the crawler
Different product token or parser
Compare the exact identity and effective parser
Works for a host but fails after redirect
Policy reused across origins
Review the destination origin separately
Decision changes without a code deployment
Source policy changed
Compare policy hashes and retrieval times
Worker proceeds after network failure
Exception converted into an allow decision
Preserve an explicit unknown/review state
Completed job includes excluded URLs
Scope enforcement happened too late
Check the enqueue path and returned URL inventory
Keep previous accepted data until replacement data passes review. If a source changes policy, pause affected work and resolve the new scope before refreshing the dataset. Production retry guidance is useful for transient retrieval failures, but retries should not turn an intentional policy denial into repeated target requests.
What Should You Implement First?
Implement a policy manifest and a tested URL-admission check before building a large crawler. Verify a small set of allowed, denied, redirected, and unavailable-policy cases, then connect only approved work to the collection layer. Nstdata's alternate HTML and visual outputs can help diagnose collection quality once the policy boundary is established.
Q: Does robots.txt grant permission to scrape a website?
No. robots.txt communicates crawler preferences; authorization, terms, privacy, copyright, and reuse still require separate review.
Q: Is Crawl-delay part of RFC 9309?
No. Crawl-delay is an extension, and support depends on the crawler or parser. Keep your own scheduling limits explicit.
Q: Does a sitemap override Disallow rules?
No. A sitemap suggests URLs for discovery; each candidate still needs scope and policy evaluation.
Q: Should a network failure silently allow collection?
No. Preserve the failure as a distinct state and apply your documented policy. The demonstration stops for review rather than inventing an allow decision.
Q: Does a proxy change the policy decision?
No. A different network route does not create permission or remove your source-review obligations.
Q: Can a Python parser be used without a crawler framework?
Yes. A preflight policy check can run before any collector, provided its matching behavior and integration are tested for your workload.
Ivy Lin
Nov. 12th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
