Hermes Agent: What It Is & How to Use It with Nstdata Crawl
TL;DR
Hermes Agent is Nous Research's open-source, MIT-licensed, self-hosted AI agent — a terminal and desktop app with persistent memory, a self-improving skills system, and support for Telegram, Discord, Slack, WhatsApp, Signal, and CLI.
It ships 70+ built-in tools, including web_search and web_extract, and can be extended two ways: connecting an MCP server, or writing a plugin that registers a custom tool via ctx.register_tool().
Hermes's built-in web_extract tool returns Markdown but applies a hard character-budget truncation and isn't documented to render JavaScript; its browser_navigate/browser_snapshot path can add proxy and anti-bot handling, but only through a paid Browserbase-style cloud-browser backend and a multi-step accessibility-tree interaction loop.
Wiring Nstdata Crawl in as a Hermes plugin gives the agent a single tool call that renders JavaScript, resolves through Nstdata's own proxy pool, and returns clean Markdown or JSON — without a separate browser-automation subscription.
The integration is a ~40-line Hermes plugin: a plugin.yaml manifest, a register(ctx) function, and a handler that POSTs to Nstdata Crawl's /api/v1/crawl/scrape endpoint and returns the parsed Markdown.
Nstdata Crawl bills per successful fetch (including 404/403 responses) rather than per attempt, and site-level crawls need explicit maxDepth/maxPages bounds — both matter once an agent is calling the tool autonomously instead of a human running one request at a time.
Hermes Agent is an open-source, MIT-licensed AI agent built by Nous Research, distributed as a terminal application and native desktop app for macOS and Windows. It installs with a single script (curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash on Linux/macOS/WSL2/Termux, or a PowerShell one-liner on Windows), and the installer bundles Python 3.11, Node.js, ripgrep, and ffmpeg so the agent has a working runtime out of the box. Its defining feature is a learning loop: the project's own README describes an agent that "creates skills from experience, improves them during use," and searches its own past conversation history rather than starting cold on every session.
Model access is provider-agnostic — hermes model switches between Nous Research's own Nous Portal, OpenRouter, OpenAI, or a custom endpoint without touching code — and the agent reaches users across Telegram, Discord, Slack, WhatsApp, Signal, email, or a plain CLI session. Under the hood, Hermes ships more than 70 built-in tools grouped into toolsets (browser automation, file and terminal access, web search and extraction, vision, image generation, home automation, and more), configurable per platform with the hermes tools command.
Why extend it with Nstdata Crawl
Hermes's own web tools cover the common case well but have documented edges worth knowing before you build a workflow around them. The built-in web_extract tool fetches a URL and returns Markdown, but Nous Research's own docs describe a fixed character budget on the result — roughly 15,000 characters by default, split "~75% head / 25% tail" with an explicit [TRUNCATED] footer once a page exceeds that — and the same docs point JS-heavy pages that "return little content" toward browser_navigate plus browser_snapshot instead. That browser path works, but it's a different shape of tool entirely: it opens a real browser session, returns an accessibility-tree snapshot with numbered interactive elements, and needs a CDP-capable backend (local Chrome, or a cloud option like Browserbase) to run — and proxy or anti-bot handling on that path is a feature of the paid cloud-browser tier, not something web_extract itself provides.
Nstdata Crawl fills the gap between those two tools with a third shape: one API call that renders JavaScript, routes through Nstdata's own proxy pool with browser-fingerprint handling, and returns clean Markdown, HTML, or JSON — closer to what web_extract already returns to the model, but without the truncation-by-default behavior or the JS-rendering gap, and without requiring a separate cloud-browser subscription for proxy coverage.
Take a Quick Look
When a page an agent needs to read is JS-heavy or behind bot detection, Nstdata Crawl turns it into clean Markdown or JSON in a single proxy-backed API call — no separate browser-automation backend required.
Hermes Agent installed and reachable via hermes on your PATH, with a model backend already configured through hermes model.
A Nstdata account and API key for Crawl, kept in an environment variable rather than hardcoded into any file.
Python 3.11, which Hermes's own installer already bundles.
Familiarity with editing YAML and Python — the plugin below is short, but you'll want to adapt the tool's description and parameters to your own workflow.
Install
Hermes plugins live under ~/.hermes/plugins/<plugin-name>/. Create the directory and the three files a plugin needs:
mkdir-p ~/.hermes/plugins/nstdata-crawl
cd ~/.hermes/plugins/nstdata-crawl
touch plugin.yaml __init__.py
Configure
Declare the plugin's metadata in plugin.yaml:
name: nstdata-crawl
version:"1.0"description: Fetch a URL as clean Markdown via Nstdata Crawl, with JS rendering and proxy-backed access.
Set the Nstdata API key as an environment variable rather than writing it into any plugin file — Hermes plugins run as regular Python, so os.environ is available to them the same as any script:
exportNSTDATA_API_KEY="your-api-key-here"
Real handshake
Register the tool inside __init__.py, following Hermes's documented plugin pattern — a register(ctx) function that calls ctx.register_tool() with a schema and a handler:
import json
import os
import requests
NSTDATA_API_BASE ="https://api.nstdata.io"defregister(ctx): schema ={"name":"nstdata_crawl_scrape","description":("Fetch a URL and return clean Markdown, rendering JavaScript and ""routing through Nstdata's proxy pool. Use this for pages that ""web_extract returns little content for, or that need proxy-backed access."),"parameters":{"type":"object","properties":{"url":{"type":"string","description":"The URL to fetch and convert to Markdown.",}},"required":["url"],},}defhandle_crawl_scrape(params,**kwargs):del kwargs
target_url =(params or{}).get("url","").strip()ifnot target_url:return json.dumps({"success":False,"error":"url is required"}) api_key = os.environ.get("NSTDATA_API_KEY")ifnot api_key:return json.dumps({"success":False,"error":"NSTDATA_API_KEY is not set"}) response = requests.post(f"{NSTDATA_API_BASE}/api/v1/crawl/scrape", headers={"x-api-key": api_key,"Content-Type":"application/json"}, json={"url": target_url,"formats":["markdown"],"onlyMainContent":True}, timeout=60,) response.raise_for_status() envelope = response.json()if envelope.get("err"):return json.dumps({"success":False,"error": envelope.get("msg","request failed")}) result = envelope.get("data",{})ifnot result.get("success"):return json.dumps({"success":False,"error": result.get("status","crawl did not complete")}) page = result.get("data",{})return json.dumps({"success":True,"title": page.get("metadata",{}).get("title",""),"markdown": page.get("markdown",""),}) ctx.register_tool( name="nstdata_crawl_scrape", toolset="nstdata_crawl", schema=schema, handler=handle_crawl_scrape,)
Two details matter here, both drawn from Nstdata's own API documentation and playbook: the response is nested — a top-level err/code/msg envelope wraps an inner success/status object, which itself wraps the actual data.markdown payload — and an HTTP 200 only confirms the request was received, not that the crawl succeeded, so the handler checks err and success explicitly rather than trusting the status code alone.
Attach the tool
Enable the new toolset the same way you'd enable any built-in one:
hermes tools
Select nstdata_crawl from the interactive list (or pass it directly with hermes chat --toolsets "web,nstdata_crawl"), and the model gains access to nstdata_crawl_scrape alongside its existing tools for the rest of the session.
Worked example
Here's the handler's parsing logic verified against Nstdata Crawl's documented response envelope — run against a local fixture serving that exact JSON shape, since this environment doesn't have a live Nstdata API key to call the real endpoint:
import json
import requests
defcrawl_scrape(api_base): resp = requests.get(f"{api_base}/scrape_response.json", timeout=10)# stands in for the real POST resp.raise_for_status() envelope = resp.json()if envelope.get("err"):return{"success":False,"error": envelope.get("msg","unknown error")} inner = envelope.get("data",{})ifnot inner.get("success"):return{"success":False,"error": inner.get("status","not completed")} page = inner.get("data",{})return{"success":True,"title": page.get("metadata",{}).get("title",""),"markdown": page.get("markdown","")}print(json.dumps(crawl_scrape("http://127.0.0.1:8100"), indent=2))
Output:
{"success":true,"title":"Careers at Example Corp","markdown":"# Example Careers Page\n\nWe are hiring a Senior Backend Engineer in Remote - US.\n\n[Apply here](https://example.com/jobs/8077887)"}
This matched the same nested err / data.success / data.data.markdown structure Nstdata's live API documentation describes, confirming the unwrapping logic in the plugin handler before pointing it at a real API key.
Returned output
Once the tool is enabled, a natural-language request like "read the careers page at nstdata.io and summarize open engineering roles" gives the model a nstdata_crawl_scrape call to make, and the tool returns a JSON string with success, title, and markdown fields — the same shape shown above. Hermes's own tool-calling loop reads that Markdown the same way it would read web_extract's output, so no changes are needed to how the agent reasons about the result; the difference is entirely in what the underlying fetch can handle.
Limits
The plugin above fetches a single page per call — it doesn't set maxDepth or maxPages, because those parameters only apply to Nstdata Crawl's site-level crawl endpoint (POST /api/v1/crawl), not the single-page scrape endpoint used here. If you extend this plugin to call the site-level endpoint instead, set those bounds explicitly: an unbounded site crawl launched autonomously by an agent can wander into search results, pagination, or login pages without a human in the loop to notice.
Nstdata Crawl bills per successful fetch rather than per attempt — a response that comes back with a real status code, including a 404 or 403, is billable once, since the fetch itself succeeded even though the page didn't. That distinction matters more once an agent is calling the tool on its own schedule than it does for a one-off manual request, since a bad URL fed to the agent in a loop still counts against usage. Re-verify Crawl's current subscription pricing before relying on a specific figure, since it moves, and see the Crawl API reference for the full request/response schema this plugin builds on, including the site-level crawl endpoint and its maxDepth/maxPages parameters. Nstdata's Crawl launch announcement covers the fuller feature set if this integration grows beyond single-page fetches.
The handler also assumes the Crawl API responds within its 60-second timeout; a slow or JS-heavy page can exceed that, in which case the tool should return a clear failure rather than let the request hang — production versions of this plugin should catch requests.Timeout explicitly rather than letting it propagate as an unhandled exception, since Hermes expects tool handlers to always return a JSON string.
Conclusion
Hermes Agent's built-in web tools cover most cases, but a truncated web_extract result or a browser_navigate session that needs a paid cloud-browser backend for proxy support are both real edges you'll eventually hit. A small plugin — one plugin.yaml manifest and a register(ctx) function wrapping a single HTTP call to Nstdata Crawl's scrape endpoint — closes that gap with one proxy-backed, JavaScript-rendering tool call that returns the same kind of Markdown the agent already knows how to read. Keep the API key in an environment variable, bound any site-level crawl you add later with explicit page and depth limits, and the integration slots into Hermes's existing tool-calling loop without changing how the agent reasons about its results.
FAQ
Q: What is Hermes Agent?
Hermes Agent is an open-source, MIT-licensed, self-hosted AI agent built by Nous Research. It runs as a terminal app and native desktop app, supports Telegram, Discord, Slack, WhatsApp, and Signal alongside plain CLI use, and is built around a self-improving skills system with persistent memory across sessions.
Q: Do I need to use Nous Portal to run Hermes Agent?
No. hermes model switches between backends — Nous Portal, OpenRouter, OpenAI, or a custom endpoint — without changing any code, so Nous Portal is one option among several rather than a requirement.
Q: Why not just use Hermes's built-in web_extract tool?web_extract works well for ordinary pages, but Nous Research's own docs describe a default ~15,000-character truncation on its output and note that JS-heavy pages that return little content should instead use browser_navigate/browser_snapshot. A Nstdata Crawl plugin gives the agent a single tool call that handles JavaScript rendering and proxy-backed access directly, without truncation or a separate browser-automation backend.
Q: Does this integration use Nstdata Crawl's site-level crawling, or just single pages?
The plugin in this guide calls Crawl's single-page scrape endpoint. Site-level crawling is a separate endpoint (POST /api/v1/crawl) that requires explicit maxDepth and maxPages bounds — add those bounds deliberately if you extend the plugin to crawl an entire site, since an autonomous agent has no built-in sense of when a crawl has gone far enough.
Q: Where do I put my Nstdata API key?
In an environment variable (NSTDATA_API_KEY in this guide's example), read at runtime with os.environ.get(). Never write it directly into plugin.yaml, __init__.py, or any file that might end up in version control or a shared Hermes configuration.
Q: Can Hermes Agent call Nstdata Crawl through MCP instead of a plugin?
Hermes does support connecting any MCP server via ~/.hermes/config.yaml, but Nstdata Crawl is exposed as a REST API rather than a published MCP server, so a plugin registering a custom tool — the approach in this guide — is the direct integration path today. An MCP wrapper around the same REST API would work too, if you'd rather standardize on MCP across multiple agents.
Q: What happens if the crawl request fails or times out?
The handler in this guide checks the response envelope's err and success fields explicitly and returns a JSON failure object rather than raising an exception, since Hermes expects every tool handler to return a JSON string. A production version should also catch network-level errors like requests.Timeout and requests.ConnectionError the same way, so a single failed fetch doesn't crash the agent's tool-calling loop.
Q: Does this replace Hermes's browser automation tools entirely?
No. browser_navigate and browser_snapshot are still the right choice for tasks that need to click, scroll, fill out a form, or otherwise interact with a page rather than just read it. The Nstdata Crawl plugin is a faster, cheaper path specifically for "read this page's content," which covers a large share of what an agent's web tools get used for in practice.
A practical guide to agent web data: when to use search, a Crawl API, browser automation, or direct HTTP, plus tool design, security, and evaluation. It distinguishes URL discovery from page extraction and includes permission and prompt-injection safeguards.
Lena Zhou
Oct. 20th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
