Python Pagination: How to Collect Data Across Paginated Pages
TL;DR
Pagination is a contract between your client and the data source. Inspect the response for a page number, offset, cursor, or next link before writing the loop.
For limit/offset APIs, advance by the number of records actually returned and stop at the reported total or an empty page. Add a maximum-page guard and deduplicate by a stable ID.
Cursor APIs are opaque by design. Send the exact next cursor returned by the server; do not manufacture page numbers from it.
This tutorial uses DummyJSON as a demo API. It is useful for learning pagination mechanics, not as a production business dataset or guaranteed availability contract.
Python pagination is the logic that requests a large result set in smaller pages. APIs and websites use pagination because returning every record in one response can be slow, memory-heavy, or exceed server limits.
A scraper that fetches only the first response can look healthy while silently missing most of the dataset. The correct loop depends on the source’s pagination style: page numbers, offset and limit, cursor tokens, a next-page URL, or content loaded as the user scrolls. Always read the provider’s API documentation and inspect a sample response before choosing a loop. For website-level URL discovery and page extraction, Nstdata Crawl is separate from application-specific cursor or offset pagination.
Common Pagination Patterns
Pattern
Request example
How to know when to stop
Common failure
Page number
?page=3
Empty page, no next-page field, or known total
Assuming every endpoint starts at page 1
Offset + limit
?skip=60&limit=30
Returned total or no records
Incrementing by the wrong page size
Cursor / continuation token
?cursor=opaque-value
No next cursor
Treating the cursor as a number or changing it
HTML next link
An anchor labelled Next
No usable next link
Building the next URL by string concatenation
Infinite scroll
Browser fetches more after scrolling
Site-specific end marker or exhausted documented API
Scrolling forever or calling undocumented endpoints without permission
The parameter names are examples, not universal API standards. Some APIs return a full next URL; others return a cursor inside a nested object. Use the names and stopping condition that the source documents.
Build a Limit/Offset Pagination Script in Python
DummyJSON’s products endpoint documentation defines limit and skip parameters and returns the total result count. The example below requests bounded pages, validates the response shape, deduplicates by product ID, and writes CSV.
import csv
import time
from typing import Any
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
API_URL ="https://dummyjson.com/products"PAGE_SIZE =30MAX_PAGES =100defmake_session()-> requests.Session: retry = Retry( total=3, connect=3, read=2, status=3, backoff_factor=0.5, status_forcelist=(429,500,502,503,504), allowed_methods=frozenset({"GET"}), respect_retry_after_header=True,) session = requests.Session() session.mount("https://", HTTPAdapter(max_retries=retry)) session.headers.update({"User-Agent":"pagination-demo/1.0"})return session
deffetch_all_products()->list[dict[str, Any]]: session = make_session() products:list[dict[str, Any]]=[] seen_ids:set[int]=set() skip =0 total:int|None=Nonefor page_number inrange(1, MAX_PAGES +1): response = session.get( API_URL, params={"limit": PAGE_SIZE,"skip": skip}, timeout=(5,20),) response.raise_for_status() payload = response.json() batch = payload.get("products") page_total = payload.get("total")ifnotisinstance(batch,list)ornotisinstance(page_total,int):raise ValueError("Unexpected API response: expected products[] and integer total") total = page_total
ifnot batch:breakfor item in batch: item_id = item.get("id")ifnotisinstance(item_id,int):continueif item_id notin seen_ids: seen_ids.add(item_id) products.append(item)# Advance by the number returned, not the requested limit. skip +=len(batch)if skip >= total:break time.sleep(0.1)else:raise RuntimeError(f"Stopped at the safety limit of {MAX_PAGES} pages")if total isnotNoneandlen(products)< total:print(f"Warning: received {len(products)} unique products; API reported {total}")return products
defmain()->None: products = fetch_all_products() fields =["id","title","category","price"]withopen("products.csv","w", newline="", encoding="utf-8")as output: writer = csv.DictWriter(output, fieldnames=fields, extrasaction="ignore") writer.writeheader() writer.writerows(products)print(f"Saved {len(products)} unique rows to products.csv")if __name__ =="__main__": main()
This example uses a retry policy for transient HTTP failures and honors Retry-After when the server provides it. It does not retry every error: a 400 or 401 usually requires fixing the request or credentials, not repeating it.
Why the Loop Advances by the Returned Count
The API may return fewer records than requested at the end of the collection or when it enforces a maximum page size. Updating skip by len(batch) keeps the next request aligned with the response. The loop ends when the offset reaches total, or earlier if the API returns an empty batch.
The maximum-page guard protects against a broken total field or an endpoint that keeps returning the same page. If the warning appears, compare IDs and offsets in the saved logs before increasing the cap. Duplicate IDs can mean the source changed during collection or that the pagination parameters were not applied as expected.
How to Handle Page-Number APIs
For a page-number API, begin at the page defined by its docs, then stop on the source’s documented terminal condition. If the response includes a total-page count, use it. If it includes a next-page URL, follow that URL. If the API returns a short final page but no total, stop when the batch is empty or shorter than the requested size only if that rule is documented.
Avoid assumptions such as “page 1 always exists” or “the last page is the first empty page.” Some APIs return an explicit has_more flag. Prefer that over inferring completion from the number of rows.
The field names above are illustrative. In a real integration, copy the exact response field and request parameter from the API docs. The loop is conceptually:
Make the initial request without a cursor.
Process and persist the returned items.
Read the next cursor from the response.
Stop if the provider returns no next cursor.
Otherwise send that cursor unchanged with the next request.
Persist the cursor along with the job state if you need to resume after a crash. Make record writes idempotent so a page retried after a timeout does not duplicate downstream rows.
How to Follow HTML Pagination Links
When the source is a permitted HTML site rather than an API, use Requests to fetch the page, parse it with Beautiful Soup, and resolve relative URLs against the current response URL. Do not append a page number unless the website’s URL scheme is stable and documented.
from urllib.parse import urljoin
from bs4 import BeautifulSoup
import requests
defget_next_url(page_url:str)->str|None: response = requests.get(page_url, timeout=(5,20)) response.raise_for_status() soup = BeautifulSoup(response.text,"html.parser") link = soup.select_one("a.next")if link isNone:returnNone href = link.get("href")return urljoin(response.url, href)if href elseNone
Beautiful Soup’s official documentation explains how selectors and parsed attributes work. Add a seen-URL set and a maximum-page limit around this helper. If the next link points outside your approved domain scope, stop and review it rather than following it automatically.
Infinite Scroll and JavaScript-Paginated Pages
Infinite scroll is a presentation behavior, not a universal pagination protocol. First check whether the site documents an API or exposes a permitted next cursor in its own interface. If you own the application, use the same API contract as the frontend. If you do not, obtain permission before automating the browser or calling an endpoint that is not intended for your use.
For an authorized page that requires JavaScript, Playwright for Python can wait for a known element and perform a bounded number of scroll actions. Define a maximum item count or a stable end marker; a loop that scrolls until “nothing changes” can hang on a page with periodic content or a transient network error.
Prevent Missing Records and Pagination Loops
A reliable collection job needs more than a loop condition:
Stable identity: deduplicate records by an ID or canonical URL.
Progress tracking: log page number, offset/cursor, returned count, and cumulative count.
Bounded work: set maximum pages, elapsed time, and request rate.
Schema checks: verify the collection field and pagination metadata exist.
Retry discipline: retry transient timeouts and server errors with backoff; stop on authorization or validation errors.
Consistency checks: compare unique records received with the reported total when the source provides one.
Resume state: store the last successful page or cursor if a job can span restarts.
Offset pagination can still skip or repeat records when the underlying dataset changes during a long run. If the source supports a snapshot token or stable cursor, prefer it for mutable data. Otherwise, record collection time and use a later reconciliation pass.
When to Use a Crawling Framework Instead of Manual Pagination
A short offset/limit loop is often enough for one API. Use a framework such as Crawlee for Python when you need persistent queues, page discovery, multiple page handlers, retries across a large set of URLs, or reusable storage. Keep the concepts separate: API pagination retrieves pages in one data endpoint; site crawling discovers and visits linked URLs.
Nstdata Crawl is aimed at URL and website-level collection, including bounded page/depth discovery and normalized page outputs. It does not automatically know an arbitrary application’s business-specific cursor parameter. For page parsing and dynamic content, see Nstdata’s guides to extracting HTML text in Python, JavaScript-rendered pages, and automated data collection.
Python pagination is dependable when you follow the source’s actual contract, make every page bounded, and verify that the collected IDs and totals make sense. Start with a small run and save the raw response and page state. Use a cursor exactly as returned, resolve HTML next links with urljoin(), and switch to a crawler framework only when the workflow requires more than sequential API requests.
Keep Site Crawling Separate from API Pagination
Nstdata Crawl can help with permitted website URL discovery and page extraction. Continue to follow each API’s own page, offset, or cursor contract in application code.
Read the source’s pagination fields, loop using its page number, offset, cursor, or next URL, and stop on the documented terminal condition. Add a maximum-page limit and deduplicate records by a stable key.
Q: How do I handle pagination with requests?
Use a requests.Session, set timeouts, validate each response, and update the pagination parameter from the response contract. Persist progress if the collection needs to resume.
Q: How do I scrape infinite scroll?
For a site you own or are authorized to access, use its documented API or a browser automation flow with a clear end condition and item cap. Do not automate around access restrictions.
Q: How do I stop a Python pagination loop?
Stop when the API says there are no more pages, the cursor is absent, the next link is missing, the offset reaches the total, or the response is empty—according to that API’s documented behavior. Keep a hard page cap as a safety guard.
Q: How do I scrape cursor pagination?
Send the opaque cursor returned by one response in the next request and stop when the API returns no next cursor. Do not increment or decode the cursor unless the provider documents that behavior.
Ivy Lin
Oct. 19th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
