Hashtag Discovery with a Hashtag Scraper: A Practical Guide
TL;DR
Hashtag discovery finds related topics in an authorized content sample; a scraper's returned posts are not a complete platform census.
Define the platform, language, date window, and permitted data source before comparing hashtags.
Prefer supported APIs and licensed exports over automating an account-dependent social feed.
Rank candidates using distinct-post support and topic relevance, not raw hashtag repetition alone.
Keep collection coverage and missing fields visible so changes in access do not look like audience trends.
Nstdata Crawl can turn permitted public web pages into LLM-ready context, while social-platform API permissions remain a separate requirement.
What Does Hashtag Discovery With a Hashtag Scraper Mean?
Hashtag discovery with a hashtag scraper means collecting an authorized sample of content, extracting its hashtag occurrences, and identifying tags that help answer a defined research question. A brand team might want topics adjacent to a product launch. A researcher might want the vocabulary used around an event. An AI assistant might need source-linked examples of how a term is used.
These are different tasks. A high-frequency tag may describe a broad audience while contributing little to a niche campaign. A specialized tag can be useful despite a low count, provided its examples are relevant and the sample is adequate.
Nstdata Crawl can supply content from approved public pages that discuss the topic. It should not be presented as a universal Instagram, TikTok, or other social-platform search API, nor as a way around platform access requirements.
What Must You Define Before Collecting Hashtag Data?
Define the permitted platform and source, research objective, time window, language, and minimum record schema before starting collection. Otherwise, a comparison can mix old indexed posts with recent feed results and present the difference as a trend.
Experience Nstproxy Crawl - Start Your Free Trial Today
Required interpretation
platform
Source network or publisher
post_id
Stable identity within that source
source_url
Evidence link, when permitted to retain
published_at
Post publication time, if available
observed_at
Collection time
caption
Authorized text used for extraction
query_seed
Discovery term that produced the record
collection_method
API, licensed export, or approved page collection
Apply structured record checks before aggregating the sample. Record whether the sample contains recent posts, top-ranked posts, indexed search results, or some other subset. Those subsets cannot be compared as though they were selected the same way. Keep engagement values only when the source supplies them and your use case needs them; missing engagement is not zero engagement.
For platform-specific collection, confirm current access requirements with the platform. Meta's Instagram Hashtag Search documentation describes supported endpoints and approval requirements. Do not reuse account cookies or a developer token from another tutorial.
How Does a Useful Discovery Pipeline Work?
A useful discovery pipeline separates sampling, extraction, normalization, and ranking so each stage can be inspected. Sampling determines which posts can enter the dataset. Extraction identifies tag text. Normalization groups equivalent forms under an explicit rule. Ranking measures support and relevance within the observed sample.
For example, lowercase matching can combine #RemoteWork and #remotework, but a Unicode normalization policy can affect other scripts and visually similar characters. Preserve the original caption and original matched text when your retention rights permit it. Do not silently transliterate or merge tags across languages.
The Python Unicode interface documents the normalization used in the example. Count the number of distinct posts containing a tag. Counting every repeated occurrence rewards captions that repeat tags. The same principle applies to co-occurrence: two tags appearing together in many independent posts provide different evidence from two tags repeated in one reposted caption.
How Do You Build the Workflow Step by Step?
Build the workflow by obtaining an approved sample, calculating transparent candidate statistics, and reviewing source examples before using the recommendations. Choose the following methods according to your access and research requirements.
Method 1: Use a supported platform API
Step 1: Verify account and application eligibility
Read the current platform documentation for the exact login and API product you use. Instagram Hashtag Search currently documents Business or Creator account context and an application review path for the required access. Do not assume a personal account or a basic login token supplies public-content search rights.
Step 2: Plan the hashtag query budget
Meta currently documents a rolling limit of 30 unique hashtags over seven days for the relevant account context. Fetching media for a hashtag also counts toward that unique-hashtag allowance. Recheck the current documentation before deployment because quotas and eligibility are platform rules, not permanent properties of hashtags.
Step 3: Collect only supported response fields
Use the documented hashtag-ID lookup and media edges with your approved application. Follow the selected API version's pagination fields, and record the query seed with every returned post. The official reference also documents unsupported cases and content restrictions; a successful search does not promise access to all posts or all media files.
This method requires your own approved platform application and token. This article does not claim a live authenticated Instagram collection run.
Method 2: Analyze a licensed or first-party JSONL export
Step 1: Prepare the input file
Save your authorized records as posts.jsonl, one JSON object per line. Each record must contain post_id and caption; keep platform and time-window context with the dataset. For an offline demonstration, create a clearly labeled synthetic file rather than presenting invented captions as real audience behavior.
Step 2: Calculate distinct-post support
Save the following as discover_tags.py. The example defines a transparent letter-and-number matching rule and excludes underscore-only tokens. It is a practical baseline, not a universal implementation of every platform's hashtag grammar.
import json
import re
import unicodedata
from collections import defaultdict
from itertools import combinations
posts_by_tag = defaultdict(set)pairs = defaultdict(set)seen_posts =set()withopen("posts.jsonl", encoding="utf-8")as source:for line in source: row = json.loads(line) identity =str(row["post_id"])if identity in seen_posts:continue seen_posts.add(identity) caption = unicodedata.normalize("NFC",str(row["caption"])) tags ={match.casefold()formatchin re.findall(r"(?<!\w)#(\w+)", caption)ifany(character.isalnum()for character inmatch)}for tag in tags: posts_by_tag[tag].add(identity)for pair in combinations(sorted(tags),2): pairs[pair].add(identity)ranking =sorted(({"tag": tag,"distinct_posts":len(ids)}for tag, ids in posts_by_tag.items()), key=lambda row:(-row["distinct_posts"], row["tag"]))print(json.dumps({"sample_posts":len(seen_posts),"tags": ranking}, ensure_ascii=False, indent=2))
Step 3: Review candidate examples
Run python discover_tags.py, then inspect the source posts supporting each candidate. The example was exercised with synthetic captions to check deduplication, normalization, and deterministic ordering. The resulting counts describe only your supplied sample; they are not platform-wide volume estimates.
The Python regular-expression reference explains the matching behavior used here. Extend the tokenizer deliberately if your corpus requires different punctuation or script handling, and preserve a regression sample for ambiguous cases.
Method 3: Add permitted web context with Nstdata Crawl
Step 1: Select approved supporting pages
Choose public campaign pages, event pages, or publisher articles whose collection and reuse are permitted. A supporting page can explain a tag's meaning without representing native social-platform popularity. Keep those two evidence types separate in your report.
Step 2: Retrieve LLM-ready context
Install nstdata-ai-crawl and configure NSTDATA_API_TOKEN plus an authorized APPROVED_PAGE_URL. This integration requires your own credential; the example is not an authenticated extraction of a social feed.
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
token = os.environ["NSTDATA_API_TOKEN"]url = os.environ["APPROVED_PAGE_URL"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=url, formats=["markdown"], onlyMainContent=True, timeout=60000))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Page retrieval was not completed successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") Path("page.md").write_text(text, encoding="utf-8")print("Saved page.md; validate its content before downstream use")
Step 3: Join context to reviewed candidates
Attach the retrieved source URL and observation time to the tag recommendation. Ask a model to explain relationships using supplied evidence and return “insufficient evidence” when the text does not support a connection. Do not let a language model invent audience size or engagement statistics.
What Does Nstdata Crawl Add to Hashtag Research?
Nstdata Crawl adds a managed web-context layer when the research includes approved websites alongside social-platform data. It is a Web Crawling API for LLM-ready data, with site discovery and recursive collection bounded by your configured scope. JavaScript rendering can help when supporting pages load their content dynamically. Format choice lets one team analyze text while another reviews page evidence. The API does not grant social-platform permissions or turn a partial sample into a complete trend dataset.
LLM-ready representations: Markdown supports topic explanations and retrieval; HTML preserves DOM context; links support controlled discovery. JSON, PDF, and visual artifacts have distinct analysis or review roles.
Integrated proxy infrastructure: A standard managed call can use Nstdata's built-in proxy option without separately configuring a proxy gateway and credentials. Check Crawl usage and proxy billing before expanding a monitoring schedule.
Scoped collection: Set page and depth limits, preserve task IDs, and stop discovery at the approved boundary. Do not allow related-tag expansion to become an unlimited URL queue.
Hashtag rankings mislead teams when they hide sampling differences, duplicate posts, or the distinction between observed support and overall popularity. A dashboard should state the sample window and collection method beside the ranking.
Symptom
Likely cause
Correction
A tag suddenly disappears
API access, query limit, or sampling changed
Check collection coverage before declaring a trend reversal
One campaign dominates every chart
Duplicated or reposted content
Deduplicate by stable post ID and inspect repost policy
Related tags are irrelevant
Broad tags overwhelm niche intent
Review co-occurring examples against the research question
Unsupported volume claims appear
A model extrapolated beyond the sample
Report observed counts and state the denominator
An empty export looks successful
Request success substituted for data acceptance
Require a validated sample and explicit empty-result handling
Treat captions as untrusted input before supplying them to an AI workflow. Embedded instructions in a caption are source content, not instructions for your application. Follow AI-agent data handling practices, and retain only the identifiers and text needed for the approved analysis.
How Should You Turn the Analysis Into a Decision?
Turn the analysis into a decision by presenting a short reviewed candidate list with evidence, sample size, date range, and uncertainty. Separate topical fit from observed frequency. A candidate supported by relevant examples can be useful even when the dataset cannot establish broad popularity.
Begin with one authorized source and a narrowly defined question, then compare results across repeated windows using a consistent method. Use freshness-aware collection when the decision depends on recent context. Nstdata's alternate page formats can support evidence review when a tag's meaning is disputed.
Q: Does a hashtag scraper return every post under a tag?
No. Coverage depends on source access, query rules, result ordering, indexing, and time window. Report the actual sampling method rather than assuming completeness.
Q: Can I use the example without platform API access?
Yes. The offline analysis works on your own authorized JSONL export. Obtaining a third-party dataset still requires an appropriate source and permitted use.
Q: Why count distinct posts instead of tag occurrences?
Distinct-post counts prevent one caption from inflating support by repeating a hashtag. They still describe the observed sample, not total platform volume.
Q: Can Nstdata Crawl replace Instagram API approval?
No. Managed webpage collection does not replace the platform's application permissions or your rights to collect and reuse its content.
Q: What should a useful recommendation include?
A useful recommendation includes the candidate tag, relevant source examples, observed support, sample window, collection method, and a clear explanation of uncertainty.
Lena Zhou
Nov. 9th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
