Playwright is useful when an approved page needs JavaScript rendering or interaction; start with the source's documented API or static HTML when those already contain the data.
Use a browser context, a specific locator, a navigation timeout, and a semantic acceptance check. A loaded document is not necessarily a complete data record.
Extract each item inside its own container, preserve the source URL and fetch time, and reject missing fields rather than inventing defaults.
Bound pagination and browser concurrency; capture failed-page evidence when policy permits so markup changes are diagnosable.
Nstdata Crawl can supply rendered Markdown, HTML, links, screenshots, and PDF with built-in proxy routing, while Playwright remains useful for custom interactions.
What Can Playwright Do in a Web Scraping Workflow?
Playwright controls a browser so a permitted scraper can inspect the same rendered page state that a user sees. It can navigate, wait for meaningful elements, click, scroll, read DOM content, and observe network responses. This is useful for client-rendered catalogs or interfaces whose content appears after an action. It does not decide whether a source may be accessed, whether a page is the right entity, or whether extracted data is legally usable.
Nstdata Crawl can handle the rendering and page conversion stage when a custom Playwright session is unnecessary.
The Playwright Python locator guide describes role, text, label, and CSS locators. The navigation guide explains why a page's load event is not a universal signal that application data is ready. Use a page-specific content marker and inspect the result before extraction.
What Do You Need Before Starting?
Use a supported Python version, a reviewed public or authorized page, and a small expected-output fixture. Install the Playwright library and Chromium browser binary according to the :
Write down the page URL, expected heading or item marker, stable item identifier, fields, and a maximum number of pages. Use developer tools to confirm whether the required data is in the initial HTML, a documented API, or the rendered DOM. [Screenshot: Network panel and Elements panel showing the required data after the page has rendered.] If the source restricts automation, use an approved API or request permission instead of tuning the browser to evade controls.
How Do You Fetch and Validate Your First Page?
Method 1: Render One Approved Page
Step 1. Save this example as first_page.py. example.com is a stable documentation domain for the browser handshake; replace it only after inspecting an approved source.
Step 2. Navigate, wait for a heading, and reject an unexpected result:
from playwright.sync_api import sync_playwright
with sync_playwright()as playwright: browser = playwright.chromium.launch(headless=True)try: context = browser.new_context() page = context.new_page() response = page.goto("https://example.com/", wait_until="domcontentloaded", timeout=30000)if response isNoneornot response.ok:raise RuntimeError(f"Navigation failed: {response.status if response else'no response'}") heading = page.get_by_role("heading", level=1) heading.wait_for(state="visible", timeout=10000) title = heading.inner_text().strip()ifnot title:raise ValueError("Expected heading is empty")print({"source_url": page.url,"title": title}) context.close()finally: browser.close()
Step 3. On the real target, replace the heading check with an expected article or product marker and save a rejected reason when it is absent. response.ok only checks HTTP status; a consent screen can still pass it.
How Do You Extract Repeated Records Correctly?
Method 2: Scope Fields to One Card
Step 1. Identify one stable item container and the title, price, and link selectors inside it. Avoid selecting all titles and all prices independently: missing values can shift their positions.
Step 2. Iterate containers using a locator. The selectors below are illustrative and must be replaced with those observed on the approved page:
cards = page.locator("article.product-card")records =[]for i inrange(cards.count()): card = cards.nth(i) title = card.locator(".product-title").inner_text().strip() price_text = card.locator(".price").inner_text().strip() href = card.locator("a[href]").first.get_attribute("href")ifnot title ornot price_text ornot href:continue records.append({"title": title,"price_text": price_text,"href": href})
Step 3. Resolve each href against page.url, add a stable item ID and fetch time, and compare a sample with the visible page. Keep displayed currency and locale with price text; do not convert £12.50 and $12.50 into the same value. If cards.count() is zero, inspect the raw page and screenshot before changing selectors.
How Do You Handle a Load More Button?
Method 3: Advance One State at a Time
Step 1. Confirm that a visible button loads more items and identify a stable item ID in each new card. [Screenshot: Load More button and the item count before clicking.]
Step 2. Record current IDs, click the button once, then wait for an ID or count change with a bounded timeout. Set a maximum number of clicks, for example ten in a pilot. Do not use a fixed sleep as the sole completion condition.
Step 3. Store only new IDs and stop on a disabled or absent button, repeated IDs, or the page budget. A virtualized list may remove earlier DOM nodes, so maintain your own seen_ids set rather than treating current DOM count as total coverage.
Check the first response, browser console, network requests, screenshot, and DOM after the content marker should appear. If the initial HTML lacks the data but the browser shows it, rendering is required. If the browser itself shows a login or challenge page, stop and review access. If one field is missing from otherwise valid cards, inspect that page template and add an explicit rejected reason. A zero-row CSV is not a successful scrape merely because the script exited with code zero.
In a recurring job, preserve raw or rendered evidence where retention rules allow, plus source_url, fetched_at, parser_version, and quality_status. Reprocess saved artifacts after selector changes. Keep a last-good version so a failed refresh cannot replace a correct record with an empty page. The JavaScript-rendering guide covers browser resource management at larger scale.
How Can Nstdata Help When Browser Maintenance Becomes Repetitive?
Nstdata Crawl can acquire and convert approved pages without your team operating the browser fleet for every job. It supports JavaScript rendering, bounded site discovery, and an API for single-page or site jobs. Playwright remains the better fit for an interaction that needs custom clicks, session state, or DOM decisions beyond the documented Crawl configuration.
LLM-ready formats: Markdown is useful for RAG and AI-agent context; cleaned HTML retains structure for custom selectors; raw data helps investigate conversion; links support discovery; screenshot and PDF preserve visual state. The Crawl API guide describes these outputs and reference tokens for large artifacts.
Built-in proxies: A standard managed request inherits proxy routing with no separate proxy Host, Port, Username, Password, or Protocol configuration. This reduces deployment work for permitted collection, but it does not remove source rate or access restrictions.
Scope and jobs:maxDepth, maxPages, includeUrls, and excludeUrls bound recursive discovery. Poll status and retrieve pages before claiming coverage.
How Do You Connect Nstdata Crawl to a Playwright Pipeline?
Step 1. In Dashboard → Crawl Playground, test an approved page and obtain an API key. Keep it in NSTDATA_API_TOKEN on the server.
Step 2. Use Crawl for pages needing acquisition and conversion, while retaining Playwright for the interactions you actually need:
curl--request POST 'https://api.nstdata.io/api/v1/crawl/scrape/submit-sync'\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header'Content-Type: application/json'\--data'{"url":"https://example.com/","formats":["markdown","html","screenshot"],"onlyMainContent":true}'
Step 3. Inspect nested task success and page status, retrieve referenced artifacts if returned, then run the same acceptance checks as your Playwright path. Do not merge an incomplete Crawl result with a valid Playwright record. The example follows documented fields but needs an account key for a live test.
Start with one approved page and a known expected heading. Then compare local Playwright and Crawl on the same data contract before increasing volume. For continuous, custom browser sessions, Nstdata Cloud Browser is a separate product to evaluate.
How Do You Turn the Example into a Repeatable Job?
Put approved URLs in a queue with a unique canonical key, source ID, attempt count, and next eligible fetch time. A worker should lease one URL, open a context, navigate, validate, write a raw artifact if permitted, then write the accepted or rejected record. Acknowledge the task only after the result is durable. If the worker crashes, an expired lease allows the URL to be retried without silently losing it.
Version the extraction logic. Store parser_version and a content hash with each accepted record so you can tell a page change from a code change. A new selector version should be tested against saved normal and failure fixtures before deployment. If a refresh returns a consent screen or empty shell, keep the last-good record and mark it stale instead of overwriting it.
Monitor browser resource use and source behavior separately. Track open contexts, memory, navigation timeouts, 429 responses, accepted-record yield, and queue age by source. A high request count with low accepted yield is a signal to pause and inspect a sample, not to increase concurrency. For multi-page sites, define a page budget and checkpoint discovered links or item IDs so a restart can resume predictably.
