C# Web Scraping Guide: How to Collect Public Web Data
TL;DR
C# web scraping usually combines HttpClient for retrieval with a parser such as AngleSharp for HTML extraction.
Use browser automation when the required content appears only after JavaScript execution or an approved interaction.
Reuse HTTP connections, set response and time limits, and reject unexpected content before parsing business fields.
Validate record identity, field counts, numeric formats, and missing values separately from HTTP success.
A managed acquisition API can feed a .NET pipeline without requiring a native C# SDK.
Nstdata Crawl supplies LLM-ready page outputs, bounded discovery, rendering, and built-in routing; your .NET service still owns business acceptance.
How Does Web Scraping with C# Work?
Web scraping with C# works by retrieving an approved page, parsing its content, selecting the required elements, and validating the resulting records. The normal starting point is HttpClient plus an HTML parser. If the initial HTML does not contain the facts, a browser or managed rendering service may be necessary. A C# web scraping guide should make that distinction before adding automation dependencies.
C# is useful when the collector needs to live beside existing .NET APIs, background services, typed domain models, and operational tooling. Static types help express a contract, but a correctly typed amount can still belong to the wrong product. Check source identity and business meaning before writing the record downstream.
Nstdata Crawl is one option when your .NET team wants to consume acquired content rather than operate the browser and proxy layer. Keep collection and domain validation separate so either implementation can be replaced without redefining what constitutes an accepted record.
What Do You Need Before Starting?
Prepare a supported .NET SDK, a project directory, an approved test source, and an explicit record schema. The static-page method below was run with .NET SDK 10.0.404 and AngleSharp 1.10.2 against a public HTML table test page. Treat those as tested versions rather than a claim that future installations resolve the same packages.
The table contract is simple: one non-empty name and one decimal amount per row, with names unique within this demonstration. Install the SDK from Microsoft's supported distribution. Create a console project with , change into that directory, and add the tested parser with .
Experience Nstproxy Crawl - Start Your Free Trial Today
dotnet new console -n TableCollector
dotnet add package AngleSharp --version 1.10.2
For browser automation, a Playwright package and its browser binaries are separate prerequisites. For the managed method, you also need a Nstdata account, API key, Python 3.10 or later for the demonstrated worker, and approval for the target URL. No real credential should be placed in a source file, application log, or repository.
How Should You Choose Between the Three Methods?
Choose direct HTTP for useful initial HTML, a browser for observable JavaScript-dependent content, and managed acquisition when operating that infrastructure is the constraint.
Method
Useful when
You operate
Key acceptance check
HttpClient + AngleSharp
Required fields exist in returned HTML
Retrieval, parser, scheduling
Correct table and field semantics
Playwright .NET
Content requires rendering or interaction
Browser lifecycle and extraction
A meaningful readiness condition
Nstdata Crawl worker
You want an API acquisition boundary
Domain parsing and downstream workflow
Task success and usable source content
The C# versus Python comparison covers ecosystem tradeoffs. Neither changing languages nor adding a browser repairs an incorrect record definition.
How Do You Collect Data from an HTML Page?
Collect static HTML with a bounded HttpClient request and extract fields with a DOM parser. Do not parse arbitrary HTML with a regular expression or treat an empty result as a successful refresh.
Method 1: HttpClient and AngleSharp
Step 1: Add the extraction program
Replace the project's Program.cs with the following code. It requests one public demonstration page, limits the buffered response, verifies HTML, and extracts a known table. The AngleSharp parser guide documents the parser and DOM selection model.
usingSystem.Globalization;usingSystem.Text.Json;usingAngleSharp.Html.Parser;conststring url ="https://testpages.eviltester.com/pages/basics/html-tag-table/";usingvar handler =newHttpClientHandler{ AllowAutoRedirect =false};usingvar client =newHttpClient(handler){ Timeout = TimeSpan.FromSeconds(30), MaxResponseContentBufferSize =1_048_576};usingvar response =await client.GetAsync(url);response.EnsureSuccessStatusCode();if(response.Content.Headers.ContentType?.MediaType !="text/html")thrownewInvalidOperationException("Expected HTML");var html =await response.Content.ReadAsStringAsync();var document =awaitnewHtmlParser().ParseDocumentAsync(html);var table = document.QuerySelector("table#mytable")??thrownewInvalidOperationException("Expected table is missing");var records =newList<object>();var seen =newHashSet<string>();foreach(var row in table.QuerySelectorAll("tr")){var cells = row.QuerySelectorAll("td");if(cells.Length ==0)continue;if(cells.Length !=2)thrownewInvalidOperationException("Unexpected row width");var name = cells[0].TextContent.Trim();var amount =decimal.Parse(cells[1].TextContent.Trim(), CultureInfo.InvariantCulture);if(name.Length ==0||!seen.Add(name))thrownewInvalidOperationException("Empty or duplicate identity"); records.Add(new{ name, amount });}if(records.Count ==0)thrownewInvalidOperationException("No records");Console.WriteLine(JsonSerializer.Serialize(records));
Step 2: Run the collector
Run dotnet run from the project directory. The tested page returned four rows, with names Alan, Bob, Aleister, and Douglas and amounts 12, 23, 33.3, and 42. The JSON contains numeric amounts because this example serializes C# decimal values.
Step 3: Check the source contract
Compare those values with the source table before changing the selector for another page. For your own source, define the actual identifier, numeric locale, units, and missing-value policy. A currency symbol, thousands separator, or localized decimal separator needs deliberate parsing rather than an unconditional decimal.Parse call.
The demonstration creates and disposes one client for one short run. A long-running service should follow Microsoft's HttpClient lifetime guidance, including suitable connection reuse and DNS behavior. Avoid creating a new connection pool for every URL in a large job.
How Do You Handle Content That Appears After JavaScript Runs?
Handle JavaScript-dependent content by waiting for the specific state that proves the required data is ready. Browser launch and page navigation alone do not prove that the business fields have loaded.
Method 2: Playwright .NET
Step 1: Install the browser dependency
Create a separate .NET 10 console project with dotnet new console -n RenderCheck, change into its directory, and run dotnet add package Microsoft.Playwright --version 1.64.1. The Playwright .NET installation guide explains why the package and browser binaries are separate dependencies.
Step 2: Run a controlled rendering check
Use this program to prove that your installed browser can execute a delayed script and satisfy a meaningful assertion. This is a controlled HTML fixture, not a claim about access to an external dynamic website. Save the program as Program.cs, run dotnet run -- --install to invoke the package's browser installer, and then run dotnet run to execute the check.
For a real source, navigate to the approved URL with page.GotoAsync, then replace the fixture assertion with a source-specific locator and expected state. Verify an identity or required field, not merely the presence of a generic body element. Dispose contexts when work finishes and bound the number of simultaneously active pages.
If the output is empty, inspect whether the page is a login prompt, consent page, error page, or incomplete application state. The hands-on Playwright guide provides a fuller browser workflow. More waiting cannot solve a permission denial or a selector aimed at the wrong document.
How Can Nstdata Crawl Supply a .NET Data Workflow?
Nstdata Crawl can supply a .NET workflow through a collection worker that returns reviewed page artifacts. Nstdata Crawl is a Web Crawling API for LLM-ready data that can turn approved website sections into reusable inputs. Site discovery and recursive crawling support a bounded source inventory. JavaScript rendering can acquire dynamic content before conversion. Your .NET application can consume the artifacts and retain ownership of typed records, identity resolution, and business validation.
Formats for separate consumers: Markdown supports RAG and agents, HTML supports DOM parsing, links support discovery, and PDF or screenshots support inspection. JSON-facing responses are integration structures, not proof that a domain schema is correct.
Built-in proxy access: Standard managed calls inherit Nstdata routing without an additional proxy Host, Port, Username, Password, or Protocol configuration. Review proxy traffic separately from URL usage in the Crawl billing model.
Task-aware acquisition: Check completion, success, missing output, and partial content before handing a file or record batch to the .NET service. Save task identity for asynchronous work rather than submitting again after an uncertain response.
Method 3: Use a managed acquisition worker
Step 1: Configure the worker credential
Obtain your API key through Dashboard → Crawl Playground, install nstdata-ai-crawl with Python 3.10 or later, and set NSTDATA_API_TOKEN plus APPROVED_PAGE_URL. Set EXPECTED_TEXT when a required source phrase can provide a useful first acceptance check.
Step 2: Execute the collection worker
Save the following as nstdata_worker.py. This credential-dependent call needs your own key and an approved target; the tutorial does not claim an authenticated production extraction or a native C# SDK.
[Screenshot: Nstdata Crawl Playground with the approved target and requested format; keep the API key hidden.]
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
with NstDataClient(os.environ["NSTDATA_API_TOKEN"], max_response_bytes=1_048_576)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=os.environ["APPROVED_PAGE_URL"], formats=["markdown"], onlyMainContent=True, timeout=60000,))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Collection did not complete successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") expected = os.environ.get("EXPECTED_TEXT")if expected and expected notin text:raise RuntimeError("Required source text is missing") Path("accepted-page.md").write_text(text, encoding="utf-8")print("Saved accepted-page.md; validate facts before ingestion")
Step 3: Pass accepted content to .NET
Have your job runner hand accepted-page.md and its source metadata to the .NET consumer after the worker succeeds. A file is sufficient for a small local demonstration; recurring services should use an explicit queue or storage contract. If the downstream consumer requires table fields, request a suitable HTML representation and apply a tested domain parser rather than treating Markdown as a finished table.
Keep rejected outputs out of the active dataset. The batch collection guide explains why submitted pages and accepted pages should have separate counts.
What Errors Should a Production C# Scraper Handle?
A production C# scraper should distinguish transport, content, extraction, and acceptance failures. Log a non-secret request or task ID, approved URL, attempt number, stage, and reason so an operator can identify the failing boundary.
Symptom
Likely cause
Response
HTTP success with no table
Changed markup or wrong page type
Inspect the retained HTML and reject the refresh
Decimal parse exception
Locale, units, or field contamination
Normalize according to the source contract
Browser timeout
Wrong readiness condition or unavailable data
Capture a review artifact and inspect the state
Repeated duplicate rows
Pagination or identity mistake
Reconcile identities before persistence
Managed task remains incomplete
Slow or failed remote work
Query the saved task ID within a bounded polling policy
Retry transient failures with a capped policy and preserve the last accepted dataset. Do not retry an intentional denial indefinitely. For a new source, begin with one page and validate behavior before adding scheduling, parallelism, or recursive discovery.
What Should You Build First in .NET?
Build one checked record pipeline with direct HTTP before adding browsers or managed acquisition. Measure which stage actually limits correctness and maintenance, then choose the additional infrastructure that addresses it. Nstdata's alternate HTML and visual outputs can help investigate failed acquisition without changing your .NET domain contract.
No. HttpClient and an HTML parser are sufficient when the required content is present in the initial response.
Q: Does AngleSharp automatically render an arbitrary JavaScript application?
The HTML parser used here does not reproduce a full browser application. Use a suitable rendering approach when the required facts depend on JavaScript.
Q: Why use decimal for the sample amounts?
Decimal avoids binary floating-point rounding for this simple numeric contract, but parsing still needs the correct source locale and units.
Q: Does the browser example scrape a remote website?
No. It verifies installed browser execution and a delayed readiness assertion on a controlled fixture before you adapt it to an approved source.
Q: Does the managed method require a C# SDK?
No. The demonstrated worker uses the verified Python SDK and passes accepted artifacts to a .NET consumer through an explicit workflow boundary.
Ivy Lin
Nov. 12th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
