Job Scraping: A Guide to Collecting Public Job Market Data
TL;DR
Job scraping collects public or expressly authorized job announcements and converts them into consistent records. It can support labor-market research, hiring demand analysis, and workforce planning, but a posting describes an advertisement, not an applicant or a confirmed hire.
Prefer an official jobs API or licensed feed when it covers your needs. For permitted web pages, capture only business-relevant posting fields, keep source and observation timestamps, deduplicate reposts, and remove expired records according to your retention policy. Public visibility does not mean unlimited collection or redistribution rights.
Job scraping is the software-assisted collection of job-posting data from a source such as a company careers page, a public job board, or an authorized feed. A useful record might include the employer, role title, location, employment type, salary range if published, posting date, source URL, and the date your system observed it.
Teams use the resulting data to study skills demand, hiring geography, compensation disclosures, remote-work patterns, and changes in a company’s recruiting activity. These are aggregate signals. A posting can be duplicated, left online after a role closes, or published to build a candidate pool without an immediate hire. Treat it as evidence of an advertised role, not proof of a filled position or a funded project.
For permitted public pages that need browser rendering or clean Markdown, Nstdata Crawl can provide the fetch layer; source permissions and downstream validation remain part of your workflow.
Choose the source before choosing the scraper. A published API may offer stable fields and explicit access terms; a licensed feed may offer broader coverage; a public page may be the only source for a particular employer. Compare coverage, update cadence, geography, identifiers, rate limits, historical access, and rights to store or redistribute the data.
The USAJOBS API reference documents a structured way to access federal job announcements. For labor-market trends, the Bureau of Labor Statistics’ JOLTS program publishes aggregate job openings, hires, and separations measures. The Occupational Outlook Handbook adds occupational context. Those sources answer different questions: an API listing is an individual announcement, while a statistical series is an aggregate measure.
If you use an HTML source, maintain an approved source list and record the relevant terms, collection purpose, and expected fields. Avoid pages that require a personal login or expose candidate profiles, resumes, contact details, or application status. Do not try to defeat access controls when a request is denied.
What Job Data Should You Collect?
Keep fields that support a named analysis. A compact schema might look like this:
Field
Why keep it
Quality check
source and source URL
Trace the original announcement
Canonicalize redirects and preserve the exact page link
employer and role title
Group jobs and compare demand
Retain the original text alongside normalized values
location and work arrangement
Analyze geography and remote roles
Separate city, region, country, and remote status
employment type
Distinguish full-time, contract, and internship listings
Store unknown as null rather than guessing
salary range and currency
Compare disclosed compensation
Preserve minimum, maximum, period, and source wording
posted and observed dates
Measure freshness and publication cadence
Keep source date separate from crawl time
status and expiration date
Avoid counting stale vacancies
Recheck the source and retain a closure reason
description hash
Detect edits without storing unnecessary duplicate text
Version the parsing and hashing logic
Avoid collecting information about applicants. Do not infer a person’s identity, contact details, or employment status from a job listing. Where a description contains personal information by mistake, exclude it from the downstream dataset and follow your organization’s deletion procedure.
How to Collect Public Job Listings with Nstdata Crawl
For job-market research, start with an official API or permitted data feed when one is available. For example, the USAJOBS API documentation describes its job-search API. Use crawling for public career pages when the source permits automated access. Don’t collect applicant information or content that requires a login.
Nstdata Crawl fetches a page and can return its main content as Markdown. That gives you readable source text to store or process; it does not automatically turn every career page into standardized job fields. The current Python SDK and request options are documented in the Nstdata Crawl documentation.
Keep the key in an environment variable or secrets manager; don’t commit it to source control.
2. Fetch job pages and save their Markdown
Use a curated list of public job-detail URLs. The URLs below are placeholders—replace them with pages from sources you’re permitted to crawl.
import json
import os
import re
from datetime import datetime, timezone
from nstdata_ai_crawl import( Format, NstDataClient, ScrapeRequestDto,)job_urls =["https://careers.example.com/jobs/123","https://careers.example.com/jobs/456",]deffirst_heading(markdown:str)->str|None:for line in markdown.splitlines():match= re.match(r"^\s*#{1,3}\s+(.+?)\s*#*\s*$", line)ifmatch:returnmatch.group(1).strip()returnNonedeflabeled_value(markdown:str, labels:list[str])->str|None:"""Example for pages that expose fields as 'Location: ...'."""for raw_line in markdown.splitlines(): line = re.sub(r"[*_]","", raw_line).strip() line = re.sub(r"^[-*]\s+","", line)for label in labels: prefix =f"{label}:"if line.casefold().startswith(prefix.casefold()):return line[len(prefix):].strip()orNonereturnNonedefparse_job(url:str, markdown:str)->dict:return{"source_url": url,"observed_at": datetime.now(timezone.utc).isoformat(),# Validate this candidate against the target site's page layout."title_candidate": first_heading(markdown),"employer": labeled_value(markdown,["Company","Employer"]),"location": labeled_value(markdown,["Location"]),"employment_type": labeled_value( markdown,["Employment type","Job type"]),"salary_text": labeled_value(markdown,["Salary","Compensation"]),# Preserve the original text so downstream parsing can be corrected."description_markdown": markdown,}records =[]failures =[]with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client:for url in job_urls:try: result = client.submit_scrape_task_sync( ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], timeout=60_000, onlyMainContent=True,)) markdown = result.data.get_markdown()ifnot markdown ornot markdown.strip(): failures.append({"source_url": url,"reason":"empty_markdown"})continue records.append(parse_job(url, markdown))except Exception as exc: failures.append({"source_url": url,"error_type":type(exc).__name__,})withopen("jobs.jsonl","w", encoding="utf-8")as output_file:for record in records: output_file.write(json.dumps(record, ensure_ascii=False)+"\n")withopen("job_scrape_failures.jsonl","w", encoding="utf-8")as error_file:for failure in failures: error_file.write(json.dumps(failure, ensure_ascii=False)+"\n")print(f"Saved {len(records)} records; {len(failures)} pages need review.")
The label parser is intentionally simple: it works only when a page presents fields in recognizable lines such as Location: Boston. Career sites use different layouts, so inspect the Markdown and adapt parse_job for each source. Treat the first heading as a candidate title and validate it; some sites use a generic heading such as “Careers.” Keep missing salary or employment details empty rather than inferring them.
3. Crawl a bounded careers section
If job-detail pages are linked from a careers section, Nstdata also supports asynchronous site crawling. Set limits such as maxDepth and maxPages to keep discovery within scope, then check the task status and retrieve its results using the task APIs described in the documentation. A site-wide crawl is useful for discovering linked pages; it does not remove the need to validate which URLs are job listings or parse each site’s page structure.
4. Validate and maintain the dataset
Before using the records for analysis, check that each URL is an allowed job-detail page, confirm the title and location against the source, and review failures and empty results. Deduplicate using a stable job ID from the source when available; otherwise use a normalized canonical URL. Store observed_at so you can distinguish when a listing was collected from when the employer originally published it.
This workflow produces a traceable set of public job-page content. The Markdown is the source record; structured fields are a separate, source-specific parsing step.
How Much Does Job Data Collection Cost?
Compare the full cost of three approaches: using an official API or licensed feed, maintaining your own parser, and using a managed crawling API for sources that permit automated access.
An API or feed may reduce engineering time, but coverage and reuse terms matter. A self-managed script has infrastructure costs plus the recurring work of updating selectors, handling retries, and reviewing source changes. A managed service can reduce browser and proxy operations, while your application still owns scheduling, data cleaning, storage, and source governance.
Use cost per accepted, current posting rather than cost per request:
For an example estimate, fill in your own hourly engineering rate, number of sources, fetch volume, percentage of duplicate or expired records, and storage period. If a source changes often or yields many unusable pages, the cheapest request price may not produce the lowest cost per valid posting. Treat any estimate as a model until you have measured a representative sample.
Legal, Privacy, and Data Quality Considerations
Review source terms, API policies, robots guidance, privacy rules, licensing, and your intended use before collection. Public job announcements are not an unrestricted dataset. Reuse rights can differ between viewing a page, retaining a copy, redistributing derived records, and using the data for commercial decisions.
Store only fields needed for the stated purpose. Keep a retention schedule and process for removing stale or out-of-scope records. Limit access to collected descriptions and avoid using the dataset to profile individual applicants. If the data may affect employment decisions or another high-impact decision, obtain legal and privacy review before relying on it.
Data quality needs equal care. Job titles vary across employers; salary disclosures may be annual, hourly, or absent; one role may be mirrored across several locations; and a page’s date can reflect an edit rather than the original posting. Keep raw values, document normalization rules, and report missingness instead of filling it with guesses.
How to Build a Reliable Job Market Data Pipeline
A practical recurring flow is:
Schedule collection. Choose a cadence based on source update frequency and the decision timeline. Daily checks may be enough for many market analyses.
Fetch within approved scope. Use the official API first where suitable; otherwise collect only permitted public pages.
Normalize fields. Preserve original strings and add normalized location, employment type, and compensation fields.
Deduplicate. Prefer a source ID. If unavailable, combine employer, title, location, and a normalized description fingerprint, then review ambiguous matches.
Detect stale listings. Recheck whether a posting still exists. Mark it closed or unavailable with a date rather than deleting the history immediately.
Store history. Keep observations as dated versions so analysts can distinguish a new job from an edited listing.
Measure coverage and quality. Track fetch success, valid-field rate, duplicate rate, stale-record rate, and cost per accepted posting.
A minimal posting table can contain a stable internal ID, source ID, employer, title, location fields, salary range and currency, employment type, source URL, first-seen time, last-seen time, status, content hash, and parser version. Build deletion and correction handling into the design before the first scheduled run.
Job scraping is useful when a team needs a consistent history of advertised roles and can identify sources it is allowed to use. Choose the right source, collect the minimum fields that support the analysis, preserve timestamps and provenance, and treat each record as an observation that may need review.
Job scraping is collecting job-announcement fields from public or authorized sources and organizing them for research or analysis. A posting indicates an advertised vacancy, not a completed hire.
Q: How can I collect job listings with Python?
Start with an official API or licensed feed where available. For a permitted HTML source, use an HTTP client or browser tool only within the source’s access rules, parse the required fields, and retain source URLs and timestamps.
Q: Is job scraping legal?
It depends on the source, jurisdiction, data collected, storage, and intended use. Review terms, privacy obligations, licensing, and applicable law. Public visibility alone is not permission for unrestricted reuse.
Q: How do I extract job descriptions?
Use an authorized structured source if possible. For permitted pages, extract the description while preserving the original URL and observation time, and avoid collecting applicant information or unrelated profile data.
Q: What are good sources for job market data?
Use official labor-market APIs and statistical agencies for structured listings and aggregate trends. Company career pages and licensed providers may add coverage when their access and reuse terms fit the research purpose.
Marcus Chen
Oct. 21st 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
