Handling Pagination and Infinite Scroll in Scrapers
TL;DR
Pagination can be a next-page link, numbered URL, cursor API, Load More control, or scroll-triggered request; inspect the actual data source before coding a loop.
Stop on a stable terminal signal: no next link or cursor, an unchanged item set, a page limit, or an explicit source end. Do not use while True without bounds.
Deduplicate by a stable item ID or canonical URL; repeated pages and reordered results are normal in changing catalogs.
Playwright can scroll or click when a permitted site genuinely needs browser interaction. A documented source API is usually easier to checkpoint.
Nstdata Crawl discovers linked pages within maxDepth, maxPages, include, and exclude rules, but a site crawl does not guarantee every scroll-only item.
Why Does Pagination Break Otherwise Working Scrapers?
Pagination breaks scrapers when the script treats the first response as the whole collection or guesses how more results are loaded. A site may place <a rel="next"> in HTML, construct numbered URLs, return a cursor from a JSON endpoint, or append cards after a scroll. These are different contracts. Inspect the browser's network panel and document the source's permitted access method before implementing one.
For multi-page collection, Nstdata Crawl can discover and recursively fetch permitted URLs, while the source-specific continuation rule still needs validation.
The result order can change while a crawl runs. New items may move older ones to the next page, causing duplicates or omissions. Use a stable item key and a crawl window or source snapshot when the API supports one. A page number is a navigation instruction, not an identity for the items it contained.
What Do You Need Before Starting?
Prepare an approved source, a known first page, the item container and stable item ID, the observed next-page mechanism, and an explicit maximum page or item budget. For Python link-based examples, install and Beautiful Soup. For a browser-only workflow, install Playwright and its browser binary. Keep credentials out of source code and respect the site's documented terms and rate limits.
Experience Nstproxy Crawl - Start Your Free Trial Today
requests
Record page_url, fetched_at, next_url or cursor, item IDs, and a fingerprint of the item set for each step. That ledger makes a stalled or repeated page obvious and supports resuming after a crash. A source-provided cursor should be persisted exactly as returned; do not invent a new cursor by incrementing a page number.
How Do You Follow a Next-Page Link?
Use this method when the returned HTML includes a next link. The code is a template for an approved source; selectors must be confirmed on the live page.
Method 1: Follow HTML Pagination
Step 1. Identify the link that points to the next page and a stable item selector. Prefer rel="next" when the site provides it, but verify that it points to the intended list.
Step 2. Fetch one page at a time, resolve relative URLs, and stop on a missing or repeated next URL:
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
defpages(start_url, max_pages=10): seen_urls =set() url = start_url
with requests.Session()as session:for _ inrange(max_pages):if url in seen_urls:raise RuntimeError(f"Pagination loop at {url}") seen_urls.add(url) response = session.get(url, timeout=(5,20)) response.raise_for_status() soup = BeautifulSoup(response.text,"html.parser")yield response.url, soup
link = soup.select_one('a[rel="next"][href]')if link isNone:return url = urljoin(response.url, link["href"])
Step 3. Extract and deduplicate item IDs from each yielded page. Save the last completed page and discovered next URL before continuing. Do not infer that a page with fewer cards is necessarily the last one; use the site's actual terminal signal.
How Do You Handle a Cursor or Numbered API?
Use the source's documented API when it is available and approved. A cursor is an opaque continuation token; pass it back exactly as received. Check whether the API returns an explicit next_cursor, has_more, or next URL. Verify that a null or absent token really means completion in that API. For numbered pages, use the site's observed parameter and cap the page number.
Method 2: Checkpoint Cursor Pagination
Step 1. Save the endpoint, allowed parameters, and an example response schema from the source documentation or network panel.
Step 2. Fetch one page, validate its item array, write accepted records idempotently, then persist the returned cursor. A write must finish before the cursor is advanced; otherwise a crash can skip a page.
Step 3. Stop when the API reports no continuation, or when the returned cursor and item IDs repeat. Rate-limit requests and treat 429 as a backoff signal. A cursor may expire or depend on a session; your recovery plan should state whether to restart from a known snapshot.
This method is usually easier to monitor than browser scrolling because the continuation state is explicit. The Scrapy spider documentation shows how parsers can yield follow-up requests and items as separate outcomes.
How Do You Scrape Infinite Scroll or Load More?
Use browser interaction only when the content is truly loaded by the page and the source permits that workflow. The Playwright actions guide recommends scrolling a known element into view when that action triggers additional content. Avoid blind scrollTo(0, document.body.scrollHeight) loops: a site may append ads, use a scrollable container, or never expose a terminal height.
Method 3: Advance a Browser List with a Bounded Loop
Step 1. Confirm the list container, item ID attribute, and either a Load More button or a sentinel near the list bottom. Use a locatable element, not an arbitrary sleep, as the readiness condition. [Screenshot: List container and Load More control or bottom sentinel in the rendered page.]
Step 2. On each iteration, collect visible item IDs, perform one click or scroll action, then wait for a new ID or a documented end state. Set a maximum iteration count and a timeout. If the ID set does not grow, stop and diagnose the page rather than clicking indefinitely.
Step 3. Save newly accepted items and the observed list state. Some virtualized lists remove old DOM nodes while scrolling; maintain your own seen_ids set instead of treating the current DOM count as total coverage.
The Playwright locator documentation supports role- and text-based locators and built-in waiting. For a permitted source, a stable button role and accessible name are generally easier to maintain than a generated CSS class. If the site exposes the same items through an official feed, prefer that source and retain the browser path only for verification.
How Can Nstdata Help Collect Multi-Page Sites?
Nstdata Crawl is a Web Crawling API for LLM-ready data that can turn a reviewed site root into a bounded set of page records. It discovers links recursively, handles JavaScript rendering, and returns formats suited to downstream parsing or AI ingestion. It is appropriate for linked documentation sections, category pages, and content inventories. For a list that exists only behind custom scroll state, test one representative page and inspect output before assuming complete coverage.
LLM-ready outputs: Markdown gives RAG and agent workflows cleaner text with headings and links; HTML and raw data preserve structure for item extraction; links expose discovered navigation; screenshots and PDF help verify the rendered state. These six categories are listed in the Crawl API documentation.
Built-in proxies: A standard Crawl job inherits Nstdata's managed routing. You provide x-api-key and job parameters, with no separate proxy Host, Port, Username, Password, or Protocol fields. This simplifies repeated acquisition but is not permission to exceed a source's allowed rate.
Scope controls:maxDepth, maxPages, includeUrls, and excludeUrls keep linked discovery bounded. Your application still reconciles discovered URLs with the expected inventory and checks for missing scroll-only items.
How Do You Add Nstdata Crawl to a Pagination Workflow?
Step 1. In Dashboard β Crawl Playground, test the first approved list page and obtain an API key. Decide whether the list exposes linked detail pages; if not, validate the scroll behavior separately.
Step 2. Submit a bounded site job. Replace the sample domain and patterns with the approved source scope:
curl--request POST 'https://api.nstdata.io/api/v1/crawl'\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header'Content-Type: application/json'\--data'{"url":"https://example.com/docs/","formats":["markdown","links"],"maxDepth":2,"maxPages":50,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/account/*"]}'
Step 3. Save the returned crawl ID, poll GET /api/v1/crawl/status/{crawlId}, then page through GET /api/v1/crawl/pages/{crawlId} using its returned cursor. Compare discovered URLs and accepted item IDs with the source's expected list. The documented API requires a live key; this is a configuration example, not a claimed live crawl.
For high-change lists, checkpoint item IDs and revisit policy matter as much as the pagination mechanism. The website page-discovery guide covers ways to seed a URL inventory, while the automated collection guide discusses recurring jobs. Nstdata Cloud Browser is the relevant separate product when an approved workflow requires continuing page interaction.
How Do You Confirm That a Multi-Page Crawl Is Complete?
Compare the accepted item count and stable IDs with the source's displayed total or documented API count when available. A matching count alone is insufficient if duplicate IDs replaced missing ones. Review first, middle, and last pages; confirm the terminal next link or cursor; and sample items added while the crawl was running. For changing lists, record the crawl window and accept that a snapshot may require a second reconciliation pass. Report coverage as a measured set, not as an unqualified βall pagesβ claim.
For a bounded crawl design, the headless browser guide adds readiness and checkpoint considerations.
