A price-watching agent is a small pipeline: read approved product URLs, parse a specific price field, store observations, compare with history, and alert on meaningful changes.
Nstdata Crawl handles page retrieval and can return Markdown or HTML. Your application must implement product matching, price parsing, scheduling, persistence, and notifications.
Reject missing prices, currency changes, and mismatched variants before comparing values. Keep the source URL and timestamp with every observation.
The runnable example uses a site-specific text marker and SQLite so its limitations are visible; adapt the parser to each target site's actual markup.
Suppose a retail team wants to know when a public product page changes price. A cron job that downloads HTML can work on a static page, but a production monitor also needs rendering, extraction checks, history, deduplication, and an alert policy. Nstdata Crawl can supply the page content; the monitoring decisions belong in the application. This tutorial uses illustrative product data and makes no claim about a live competitor price.
A price-watching agent turns observations into a small decision: “Did the same product, in the same currency and variant, move enough to notify a reviewer?” It differs from a basic scraping script by preserving state, applying validation rules, and explaining its alerts. It cannot infer a competitor's commercial intent from a price change. A lower displayed value could be a promotion, an alternate variant, a region-specific offer, or a parsing error.
For each product, define a stable product_id, expected name or SKU, source URL, currency, and check interval. Do not silently merge product variants. If the target site exposes a licensed feed or official API that supplies the data you need, evaluate that path first.
How a Price-Watching Agent Works
The pipeline is product URLs → Crawl API → price extraction → validation → SQLite history → change detection → alert. The crawler reads a page. The parser identifies the intended price. The database preserves observations. The comparison step applies business thresholds. A scheduler runs the workflow, and a notification service sends alerts. Each stage should have its own error status.
Crawl API and Web Data Collection
For a known public product URL, request main-content Markdown. If the target price is embedded in a structured element or a visual component, request HTML or a screenshot for diagnosis. Nstdata's Crawl documentation describes Markdown, HTML, screenshots, rendering, and page waits. Fetching a page does not mean the correct price field was extracted; build a separate parser and validate its output.
Record the observed amount as a decimal, not a floating-point number. Keep currency, product identity, URL, and UTC observation time. A product page can show list price, sale price, subscription price, unit price, and variant prices simultaneously. Select the field you intend to monitor and document that choice. Store failed extraction attempts too, so a missing price does not masquerade as “no change.”
Agent Decisions and Alerts
Calculate absolute and percentage change against the last valid observation of the same product and currency. Add a minimum threshold and suppress duplicate alerts for the same observation. Human review is useful for large changes or when a parser has recently changed. A language model may write a readable alert summary, but should not decide the numeric change from unverified text.
Step-by-Step: Build a Price-Watching Agent with Python
Step 1: Set Up the Project and Product URL List
Use Python 3.10+ and pip install nstdata-ai-crawl. Set NSTDATA_API_TOKEN as an environment variable. Replace https://shop.example.com/products/widget with a product page you are permitted to monitor. The sample's Price: USD 129.00 marker is a test fixture, not a universal rule for stores. The runtime needs an available Nstdata account; the live call was not executed here.
Step 2: Fetch Product Pages with Nstdata Crawl
The code requests Markdown with onlyMainContent=True and checks for an empty result. JavaScript rendering can help pages whose product details are populated client-side, but you still need to verify that the chosen price element appeared in the returned content.
Example: Crawl a Product Page
fetch_markdown() below is the SDK boundary. It returns text to a parser; it does not claim that the API has already identified the product price.
Step 3: Extract and Normalize Prices
The parser matches an explicit marker and expected currency. For a real target site, prefer a site-specific HTML selector or documented structured data field, backed by fixtures for sale prices, unavailable products, and multiple variants. Never take the first currency-like number on a page and call it “the price.”
Step 4: Store Price History in SQLite
Store both valid and rejected observations. SQLite is appropriate for a single-process tutorial and modest workloads; a larger monitor may need a transactional server database and a worker queue. The table includes product_id, observed_price, currency, observed_at, source_url, and extraction_status.
Step 5: Detect Meaningful Price Changes
For old price P₀ and new price P₁, absolute change is P₁ − P₀; percentage change is (P₁ − P₀) / P₀ × 100. Do not calculate a percent change when the baseline is zero. Compare only values with the same currency and product variant. The example alerts at an absolute change of at least 5 currency units and a relative change of at least 3%; adjust both thresholds for your business.
Step 6: Schedule Checks and Generate Alerts
Run the script from cron, a worker scheduler, or your orchestration system. This example prints an alert that a surrounding service can deliver. Scheduling and notification are not Crawl API features. Log crawl errors separately and retry transient failures with a bounded backoff.
import os
import re
import sqlite3
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
DB_PATH ="prices.sqlite3"PRODUCTS =[{"product_id":"widget-standard","product_name":"Widget Standard","source_url":"https://shop.example.com/products/widget","currency":"USD",}]deffetch_markdown(client: NstDataClient, url:str)->str: result = client.submit_scrape_task_sync( ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], onlyMainContent=True, timeout=60000,)) body = result.data.get_markdown()ifnot body ornot body.strip():raise ValueError("No usable product page content")return body
defparse_price(markdown:str, currency:str)-> Decimal:# Replace this target-specific marker with a tested parser for your site. pattern =rf"(?m)^Price:\s*{re.escape(currency)}\s+([0-9]+(?:\.[0-9]{{2}})?)\s*$" matches = re.findall(pattern, markdown)iflen(matches)!=1:raise ValueError(f"Expected one {currency} price marker, found {len(matches)}")try: amount = Decimal(matches[0])except InvalidOperation as exc:raise ValueError("Invalid decimal price")from exc
if amount <=0:raise ValueError("Price must be positive")return amount
definit_db(db: sqlite3.Connection)->None: db.execute("""
CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY,
product_id TEXT NOT NULL,
observed_price TEXT,
currency TEXT NOT NULL,
observed_at TEXT NOT NULL,
source_url TEXT NOT NULL,
extraction_status TEXT NOT NULL
)
""") db.commit()defprevious_price(db: sqlite3.Connection, product_id:str, currency:str): row = db.execute("""SELECT observed_price FROM observations
WHERE product_id = ? AND currency = ? AND extraction_status = 'ok'
ORDER BY id DESC LIMIT 1""",(product_id, currency),).fetchone()return Decimal(row[0])if row elseNonedefsave(db, product, observed_at, status, price=None): db.execute("""INSERT INTO observations
(product_id, observed_price, currency, observed_at, source_url, extraction_status)
VALUES (?, ?, ?, ?, ?, ?)""",( product["product_id"],str(price)if price isnotNoneelseNone, product["currency"], observed_at, product["source_url"], status,),) db.commit()defmain()->None:with sqlite3.connect(DB_PATH)as db: init_db(db)with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client:for product in PRODUCTS: observed_at = datetime.now(timezone.utc).isoformat()try: markdown = fetch_markdown(client, product["source_url"]) price = parse_price(markdown, product["currency"]) baseline = previous_price( db, product["product_id"], product["currency"]) save(db, product, observed_at,"ok", price)if baseline isNone:print(f"Baseline recorded: {product['product_id']}{price}")continue delta = price - baseline
pct =abs(delta / baseline)*100if baseline else Decimal("0")ifabs(delta)>= Decimal("5")and pct >= Decimal("3"):print(f"ALERT {product['product_id']}: {baseline} -> {price} "f"{product['currency']} ({pct:.2f}%); "f"source={product['source_url']} at={observed_at}")except Exception as exc: save(db, product, observed_at,"error")print(f"ERROR {product['product_id']}: {exc}")if __name__ =="__main__": main()
For a local parser check before any network call, parse_price("Price: USD 129.00", "USD") should return Decimal("129.00"). It should reject "List price: USD 129.00" and a page with two Price: markers. A production parser needs additional fixtures for discounts, currency symbols, locale-specific separators, and out-of-stock text. A crontab entry such as 0 */6 * * * /path/to/python /path/to/price_watch.py >> /path/to/price_watch.log 2>&1 runs every six hours; set the frequency to suit your rights, budget, and freshness needs.
Why Use Nstdata Crawl for Price Monitoring?
Nstdata is useful as the collection layer when approved public product pages need rendering, repeated retrieval, and consistent output. Its documented single-page API, Markdown and HTML output, screenshots, synchronous and asynchronous tasks, and proxy location options can reduce the page-access code you maintain. Those capabilities do not extract a price reliably across every store. The parser, database, scheduler, and alert delivery remain your responsibility. Nstdata's automated data collection guide provides related operational context.
The separation lets you swap a page parser without changing your collection client. Markdown is convenient when the price is visible as text; HTML is preferable when you need an exact structured-data or DOM field. A screenshot can help diagnose visual mismatches. Proxy region settings may matter when the displayed currency or offer varies by location, but geographic variation must be captured as part of the observation key rather than treated as a price change.
Python SDK Integration Example
fetch_markdown() above is the small SDK adapter. For a new merchant, first collect a handful of permitted sample pages and write site-specific extraction tests. Only then turn on scheduled checks. Avoid promoting a “page fetched” metric into a “price successfully observed” metric.
Testing, Costs, and Common Problems
Track Crawl request cost, separately billed proxy traffic where used, database and scheduler cost, and model tokens if an LLM writes summaries. The useful denominator is valid, correctly matched price observations, not raw requests. If retries rise or extraction fails after a layout change, cost per valid observation climbs even if the request unit price stays fixed.
Common Errors and Troubleshooting
For blank content, inspect the page's rendered state and expected selector. For 403/429, reduce rate and review access policy rather than retrying aggressively. For an implausible price jump, inspect promotion, variant, pack size, currency, and parser changes. Use a unique observation or alert key if a scheduler can run twice, and ensure an error row never overwrites the last valid baseline.
How to Measure Monitoring Quality
Label a sample of target pages by hand. Extraction accuracy is correct price parses / parsed pages. Valid collection rate is accepted observations / attempted pages. Change detection precision is true changes / alerts reviewed. Freshness is time from page change to recorded observation; measure it with controlled fixtures or known update events. Duplicate alert rate is repeated alerts for the same observed change / all alerts. Review these by merchant and page template because a global average can hide one broken parser.
Read permitted product pages, extract a clearly defined price, store timestamped observations, compare against the last valid matching record, and alert only after validation.
Can a Crawl API extract prices automatically?
A Crawl API retrieves page content. You still need a site-specific parser or extraction layer to identify the correct price and product variant.
How often should prices be checked?
Choose a schedule from business freshness requirements, site policy, change frequency, and cost. Measure whether that interval catches changes in time.
How should historical product prices be stored?
Use a stable product ID, decimal amount, currency, source URL, observation time, extraction status, and any variant or region key needed to compare like with like.
Ivy Lin
Oct. 22nd 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
