XPath Selectors for Web Scraping: Syntax, Axes & Functions
XPath is a query language for navigating XML and HTML document trees using path-style expressions, and it can do three things CSS selectors fundamentally cannot: match elements by their visible text, walk upward to a parent, and move sideways to a sibling in either direction. That extra reach is what makes it the tool scrapers reach for once a target page gets too irregular for CSS alone.
⚡ Key Takeaways
- XPath can traverse in any direction — up to parents, sideways to siblings, down to descendants — while CSS selectors only move forward and downward.
- XPath can match by visible text content using functions like
text()andcontains(), something CSS has no native syntax for at all. - Relative paths (starting with
//) are far more robust than absolute paths (starting from the document root), which break the moment a single wrapper element is added anywhere above the target. - Indexing in XPath is 1-based, not 0-based —
(//li)[1]is the first match, a common source of off-by-one bugs for developers used to most programming languages. - XPath 1.0 and 2.0 cover most functionality needed for HTML parsing and are the most commonly supported versions across scraping toolsets.
- lxml, parsel, Selenium, and Playwright all support XPath directly; BeautifulSoup does not support it natively at all.
What Is an XPath Selector?
XPath, short for XML Path Language, is a query language that selects nodes from XML and HTML documents using expressions that describe a path from one point in the tree to another. Browsers expose the HTML DOM as an XML-like tree, which is exactly why XPath works for HTML even though HTML itself isn't strict, well-formed XML. It's a core part of the XSLT standard and shows up widely across XML tooling as well as most scraping stacks.
Basic Syntax
| Expression | Meaning |
|---|---|
/html | Absolute path: selects the root <html> element. Breaks easily if structure above the target changes. |
//div | Relative path: selects every <div> anywhere in the document, regardless of nesting depth. |
//div[@class='price'] | Selects <div> elements with an exact class attribute match. |
//a/@href | Selects the href attribute value from every <a> element. |
. | The current node in whatever context the expression is being evaluated. |
./.. | The parent of the current node. |
The single most important habit for durable scrapers: prefer relative paths starting with // and chained attribute predicates over absolute paths starting from the root. An absolute path breaks the instant a single wrapper <div> gets added anywhere above the target element; a relative, attribute-filtered path keeps working through most layout changes that don't touch the target itself.
Axes: Navigating in Any Direction
Axes are keywords that select nodes based on their relationship to the current node — this is XPath's defining advantage over CSS, which has no equivalent concept at all. The specification defines thirteen axes in total; the ones that come up constantly in scraping are parent::, ancestor::, following-sibling::, preceding-sibling::, and descendant::. A practical example: //h2[@id='intro']/following-sibling::p selects every <p> element that comes after a specific <h2> — a "read everything until the next heading" pattern CSS simply cannot express.
Key Functions
| Function | Purpose |
|---|---|
text() | Selects the text content directly inside an element. |
contains(@attr, 'val') | Matches an attribute or text that contains a substring, rather than requiring an exact match. |
starts-with(@attr, 'val') | Matches values beginning with a given substring. |
normalize-space() | Trims and collapses whitespace inside matched text — essential for real-world, messily-formatted HTML. |
last() / position() | Selects by position — (//li)[last()] gets the final item in a matched set. |
Remember that XPath indexing is 1-based, not 0-based: (//li)[1] selects the first matched <li>, and the parentheses around the full expression matter — //li[1] without them means something different (the first <li> child within each parent, not the first match overall).
Skip writing selectors for supported page types
Nstdata Crawl includes auto-parsing that returns clean structured data without hand-written XPath for common page structures, and full rendered HTML for custom targets when you need to write your own.
Try Nstdata Crawl →XPath vs. CSS Selectors
The practical rule: reach for CSS selectors first, since the syntax is more concise, faster in most browser engines, and covers the majority of straightforward targeting by tag, class, ID, or attribute. Reach for XPath specifically when a task needs one of its unique capabilities — matching by visible text content, selecting a parent or sibling rather than only descendants, or filtering by position within a complex, irregular structure. Many production scrapers use both selector types in the same project, choosing per field rather than committing to one language for the whole page.
Limits
XPath implementations aren't perfectly uniform across tools — some non-standard features are implementation-specific, so an expression validated in one library's XPath engine isn't guaranteed to behave identically in another. Deeply nested, positional expressions built to match a page's current exact structure are just as brittle as their CSS equivalents; XPath's extra power doesn't automatically produce a more resilient selector unless it's actually used to target stable attributes and text content rather than fragile tree positions.
Conclusion
XPath earns its place in a scraper's toolkit specifically where CSS selectors run out of reach — matching by text, walking upward or sideways through the tree, and filtering by position in irregular structures. The durability rules are the same as for any selector language: favor relative paths and stable attributes over brittle, structure-dependent positional matching, regardless of which query language is doing the matching.
For teams that want structured extraction without hand-writing selectors for every target, evaluate Nstdata Crawl against your own pages.
Further Reading
Sources
Try Nstdata Crawl for structured extraction
Auto-parsing for common page types, full rendered HTML for custom XPath targets.
Try Nstdata for Free →FAQ
Q: What can XPath do that CSS selectors can't?
Match elements by their visible text content, and navigate upward to parents or sideways to siblings — CSS selectors only move forward and downward through the tree and have no way to match on text at all.
Q: Why did my absolute XPath expression suddenly break?
Absolute paths starting from the document root break when any wrapper element is added anywhere above the target. Relative paths starting with // and filtered by attributes are far more resilient to layout changes.
Q: Is XPath indexing 0-based or 1-based?
1-based. (//li)[1] selects the first matched element, not the second — a common source of off-by-one errors for developers coming from most programming languages, which index from zero.
Q: Does BeautifulSoup support XPath?
No, not natively. BeautifulSoup only supports CSS selectors. For XPath, use lxml directly, or parsel, which supports both CSS and XPath on the same parsed document.
Q: How do I select text that contains a specific word with XPath?
Use the contains() function with text(), for example //p[contains(text(), 'important')], which selects every <p> element whose text content includes the word "important."
Was this guide helpful?
Your choice is saved on this device.
