How to Build a We Work Remotely Scraper for Public Job Data
TL;DR
Start a We Work Remotely scraper with the official public RSS feed when recent job listings are sufficient.
Preserve the original job URL and attribute links to We Work Remotely, as its feed page requests.
Keep a listing's location restriction separate from the fact that the role is remote.
Use a bounded, authorized detail-page workflow only when the feed does not supply a required field.
Validate records before publishing them to a job board or supplying them to an AI assistant.
Nstdata Crawl is a Web Crawling API for LLM-ready data that can handle the page collection layer; job identity and eligibility rules remain your application's responsibility.
What Is a We Work Remotely Scraper Used For?
A We Work Remotely scraper converts permitted job listings into records that a hiring dashboard, research pipeline, or job-search assistant can process. A useful record connects a title and location restriction with the original listing, collection time, and source. Downloading a description without those fields makes it difficult to identify expired roles or explain why an assistant recommended a job.
The simplest implementation starts with the platform's published feed. The official We Work Remotely RSS instructions provide a general feed and category feeds, and request attribution back to the source. This is a materially different starting point from guessing selectors on a category page.
Use Nstdata Crawl when your authorized workflow also needs rendered detail pages or content from other permitted sources. Keep feed ingestion and page enrichment as separate stages so a rendering failure does not discard otherwise valid feed records.
What Do You Need Before Collecting Job Data?
You need Python, an approved use for the data, the official feed URL, a storage location, and an explicit list of required fields. The feed example below uses the Python standard library and requires no WWR account or third-party scraping subscription. Verify that your intended redistribution fits the source's requirements before turning the feed into a public product.
Experience Nstproxy Crawl - Start Your Free Trial Today
Field
Meaning
Acceptance rule
source_url
Original job detail URL
Present, HTTPS, expected source host
source_id
Feed GUID or URL
Stable within this source
title
Feed title
Nonempty; preserve original wording
published_at
Publisher's timestamp
Keep original value alongside parsed time
observed_at
Your collection timestamp
UTC timestamp generated during ingestion
region
Stated hiring restriction
Preserve missing values rather than guessing
description_html
Feed description
Treat as untrusted source content
Do not infer an employer's salary range, visa policy, or worldwide eligibility from the word “remote.” A description can contain restrictions that the title omits. Keep these fields nullable until the source actually provides supporting text.
How Does Feed Discovery Differ From Page Scraping?
Feed discovery obtains recent entries from a publisher-controlled XML document; page scraping reads a website's HTML or rendered document. Both can produce job records, but their coverage and maintenance costs differ. The feed can be a convenient discovery source without representing a complete historical archive.
During this article's research, the general WWR feed returned RSS items with fields including title, link, guid, pubDate, and description. Additional fields were present in the observed response, but a client should tolerate their absence. A later feed response can contain a different number of entries, so never hard-code the sample count as a completeness guarantee.
Use structured-data validation when mapping source fields into your job schema. For a wider system, distinguish job-market collection from eligibility classification. The former supplies evidence. The latter interprets evidence and should retain a traceable reason for every derived label.
How Do You Build the Scraper Step by Step?
Build the scraper by fetching the official feed, normalizing a small set of fields, and saving records with their source identifiers. The following methods represent different collection needs; they are alternatives or sequential stages rather than three ways to force access to the same page.
Method 1: Use RSS for recent listings
Step 1: Save the feed reader
Save the following as wwr_feed.py. The reader limits the downloaded response size, deduplicates IDs, and checks the destination host before accepting a record. It does not assume that the title can be reliably split into company and role fields.
import json
import urllib.request
import xml.etree.ElementTree as ET
from datetime import datetime, timezone
from urllib.parse import urlsplit
url ="https://weworkremotely.com/remote-jobs.rss"request = urllib.request.Request(url, headers={"User-Agent":"JobFeedReader/1.0"})with urllib.request.urlopen(request, timeout=30)as response: payload = response.read(5_000_001)iflen(payload)>5_000_000:raise RuntimeError("Feed exceeds this reader's size limit")root = ET.fromstring(payload)observed = datetime.now(timezone.utc).isoformat()records, seen =[],set()for item in root.findall("./channel/item"): link =(item.findtext("link")or"").strip() parts = urlsplit(link)if parts.scheme !="https"or parts.hostname !="weworkremotely.com":continue identity =(item.findtext("guid")or link).strip() title =(item.findtext("title")or"").strip()ifnot title or identity in seen:continue seen.add(identity) records.append({"source_id": identity,"source_url": link,"title": title,"published_at": item.findtext("pubDate"),"observed_at": observed,"region": item.findtext("region"),"description_html": item.findtext("description")})ifnot records:raise RuntimeError("No accepted feed records; inspect the response")withopen("wwr-jobs.json","w", encoding="utf-8")as output: json.dump(records, output, ensure_ascii=False, indent=2)print("accepted records:",len(records))
Step 2: Run the reader
Run python wwr_feed.py in your working directory. The same parser was exercised against the live RSS response during preparation of this article. A successful run creates a JSON array and reports a positive number of accepted records; the exact number changes as the publisher updates its feed.
Step 3: Inspect one accepted record
Open wwr-jobs.json and compare a record's title, URL, and region against its original listing. Preserve the original source link wherever you display a result. If the feed is incomplete for your purpose, document the missing field before adding another collection stage.
Method 2: Add authorized detail-page enrichment
Step 1: Select the missing field
Choose a field that the feed genuinely lacks, such as a structured employment type. Check whether the page supplies machine-readable JobPosting properties before writing a visual selector. A schema vocabulary documents field meaning; it does not establish that WWR emits every property on every page.
Step 2: Inspect an approved detail page
Use Developer Tools → Elements to compare the page's visible content with any JSON-LD script. Record the source path for the field and preserve the text used to derive it. Do not publish a selector copied from an old tutorial without checking today's markup.
[Screenshot: An approved job detail page showing the hiring-region text and the corresponding DOM or structured-data field.]
Step 3: Store enrichment separately
Join the detail result to the feed record by the source URL. Include enrichment_status, enriched_at, and a field-level evidence reference. A failed enrichment should leave a valid feed record intact and mark only the additional field unavailable.
Method 3: Add Nstdata Crawl to the collection workflow
Step 1: Obtain a current request configuration
Open the Nstdata Crawl Playground from the Crawl product page, enter an approved detail URL, choose Markdown for text processing, and inspect the generated request. Configure rendering only when the required content needs JavaScript. Use a page limit and depth limit for discovery rather than crawling the whole job board automatically.
Step 2: Connect the API client
Install nstdata-ai-crawl and set NSTDATA_API_TOKEN in your local environment. The managed example included below requires your own token and authorization for APPROVED_PAGE_URL; it was not executed against WWR with a customer credential. The SDK names are checked against the current official package.
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
token = os.environ["NSTDATA_API_TOKEN"]url = os.environ["APPROVED_PAGE_URL"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=url, formats=["markdown"], onlyMainContent=True, timeout=60000))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Page retrieval was not completed successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") Path("page.md").write_text(text, encoding="utf-8")print("Saved page.md; validate its content before downstream use")
Step 3: Accept only usable content
Confirm the task completed, check the page result, and compare the returned text with the target listing. Reject challenge pages, empty descriptions, and truncated output before passing content to a model. Record a task ID for diagnosis without recording the API token.
Where Does Nstdata Crawl Help With Job Data?
Nstdata Crawl helps when an approved job-data pipeline needs consistent page retrieval and LLM-ready text beyond a feed. It is a Web Crawling API for LLM-ready data, positioned between a URL list and your normalization code. Site discovery and recursive crawling can expand a permitted seed within explicit boundaries. JavaScript rendering can collect content missing from the initial HTML. The service does not decide whether a candidate meets a role's requirements or grant permission to republish a job description.
Output choice: Markdown supports grounded summaries, HTML supports source-aware parsing, and links support discovery. JSON, PDF, and visual artifacts serve different downstream review needs; requesting a format does not validate a job schema.
Managed proxy access: Nstdata's built-in proxy option avoids a separate proxy host, port, username, and password for a standard managed request. Proxy traffic can still affect cost; review current Crawl billing.
Bounded operation: Submit a small job first, preserve IDs and task outcomes, and retry only recoverable failures. Your application still owns deduplication, expiration, and storage.
Why Do Job Feeds Become Stale or Produce Duplicates?
Job feeds become stale when clients treat a rolling discovery window as a full inventory or conflate repeated observations with new postings. A disappearing feed entry does not prove the employer closed the role. Keep first_seen, last_seen, and a separate availability status so uncertainty remains visible.
Symptom
Likely cause
Repair
Same role appears repeatedly
Every poll creates a new row
Upsert by feed GUID or canonical source URL
“Worldwide” recommendations are wrong
Remote status substituted for location restrictions
Preserve region evidence and require explicit eligibility rules
No records after a successful HTTP response
Unexpected document, changed feed, or parser failure
Inspect content type and root element; retain the last good dataset
Summaries include navigation or instructions
Source text was accepted without validation
Restrict input to the job description and isolate untrusted content
Use the ElementTree XML interface for the demonstrated feed parsing, but keep a response-size limit and avoid accepting arbitrary user-submitted XML. Export only fields your use case needs, and apply a retention policy to stored descriptions.
How Should You Launch the Job-Data Pipeline?
Launch the pipeline with a small attributed RSS dataset, a verified field contract, and an explicit refresh policy. Compare the accepted records with the publisher before adding enrichment, and measure duplicate rate, missing-field rate, and record age. A successful download is only the beginning of a reliable job feed.
For broader collection, use bounded batch crawling to separate URL discovery from detail-page processing. Nstdata's HTML and visual outputs can support the subsequent review of extraction failures.
Q: Can I collect WWR jobs without browser automation?
Yes. The official public RSS feed can supply recent job entries without a headless browser. Check coverage and attribution requirements before using it in a product.
Q: Does the RSS feed contain all historical jobs?
The feed should be treated as a recent discovery window unless the publisher explicitly documents historical completeness. Store permitted observations prospectively if you need a time series.
Q: Can Nstdata Crawl guarantee access to every job page?
No. A managed retrieval service remains subject to target availability, access restrictions, rendering behavior, and your authorization.
Q: Should missing salary fields become zero?
No. Missing compensation should remain null or explicitly unknown. Zero incorrectly suggests a known value and distorts analysis.
Q: What proves the scraper works?
A working scraper produces valid source-linked records that agree with a manually checked sample, preserves refresh state, and reports failures without overwriting the last accepted dataset.
Lena Zhou
Nov. 9th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
