Dev.to is one of the most scraper-friendly developer communities on the web, and in 2026 it remains a goldmine for content intelligence, trend analysis, and competitive research. scraping Dev.to public articles at scale is straightforward if you use the right combination of its official API, HTML fallback, and a lightweight proxy layer to handle rate limits without getting blocked.
Why scrape Dev.to in 2026
Dev.to exposes a public REST API at https://dev.to/api that requires no authentication for read operations. you can pull article metadata, tags, reaction counts, comments, and author profiles without an API key for most endpoints. the community has grown to over 1.5 million published articles, making it a serious signal source for:
- trending topics in software engineering, DevOps, and AI tooling
- author influence mapping (reactions + followers + post frequency)
- content gap analysis against your own technical blog
- training data collection for LLMs and classifiers
if you are also tracking content across platforms like Pinterest or Quora, Dev.to fits into the same pipeline with minimal adapter work because its API returns clean JSON.
The Dev.to public API: what you get for free
the three endpoints you will use most:
| endpoint | returns | rate limit (unauthenticated) |
|---|---|---|
GET /api/articles | paginated article list | ~30 req/min |
GET /api/articles/{id} | full article body (markdown) | ~30 req/min |
GET /api/tags | tag list with follower counts | ~30 req/min |
GET /api/users/{username} | author profile + stats | ~30 req/min |
GET /api/articles?username=X | all articles by author | ~30 req/min |
the articles endpoint supports page and per_page (max 1000 per page) plus filters like tag, top (N days), and state=fresh. for bulk collection, page through with per_page=30 and increment until you get an empty array back.
A minimal scraper in Python
this collector loops through paginated results and writes each article to a JSONL file. it respects rate limits with exponential backoff and logs failures without crashing the run.
import httpx, time, json, pathlib
BASE = "https://dev.to/api/articles"
OUT = pathlib.Path("devto_articles.jsonl")
def fetch_page(page: int, tag: str = "") -> list:
params = {"per_page": 30, "page": page}
if tag:
params["tag"] = tag
r = httpx.get(BASE, params=params, timeout=15)
r.raise_for_status()
return r.json()
def scrape(tag: str = "", max_pages: int = 500):
with OUT.open("a") as f:
for page in range(1, max_pages + 1):
try:
articles = fetch_page(page, tag)
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
time.sleep(60)
continue
break
if not articles:
break
for a in articles:
f.write(json.dumps(a) + "\n")
time.sleep(2)
scrape(tag="python", max_pages=200)for full article body content, make a second pass: read the JSONL, extract id fields, and hit GET /api/articles/{id} to pull the body_markdown field. keep this as a separate enrichment step so you can resume without re-fetching metadata.
Handling rate limits and IP blocks at scale
the unauthenticated limit of ~30 requests per minute is fine for small collections, but if you are running concurrent workers or pulling the full article catalog (1.5M+ articles), you will hit 429s within minutes. three options in order of complexity:
- register a Dev.to account and generate an API key (header:
api-key: YOUR_KEY). this raises your limit but does not eliminate it - run multiple authenticated keys in rotation behind a queue (useful for team pipelines)
- layer a rotating residential proxy pool in front of unauthenticated requests. this works because Dev.to’s rate limiting is IP-based for anonymous traffic
for option 3, the proxy requirement is modest — you do not need sticky sessions because each API call is stateless. datacenter IPs work fine for the JSON API; residential proxies only become necessary if you drop to HTML scraping the article pages directly. if you are comparing proxy approaches for data pipelines more broadly, the patterns covered in how to scrape ZoomInfo without an account apply here for IP rotation strategy and request throttling.
Structured data you can extract
once you have raw article JSON, the useful fields for most analytical use cases are:
title,description,body_markdown— content and search intent signalstag_list— topic classification (multi-label)public_reactions_count,comments_count,positive_reactions_count— engagement scorereading_time_minutes— content depth proxyuser.name,user.username,user.twitter_username— author identity graphpublished_at— publish velocity analysisurl— canonical link for deduplication
for NLP pipelines, body_markdown is immediately usable — no HTML parsing, no noise stripping. this is a material advantage over platforms like Medium, where the body requires extra cleaning steps after extraction.
Enrichment with HTML fallback
some fields (canonical tags, structured metadata, Open Graph data) are only in the rendered HTML. for those, scrape https://dev.to/{username}/{slug} with httpx and parse with selectolax or BeautifulSoup. Dev.to does not use heavy client-side rendering, so a simple HTTP GET returns a complete DOM with no JS execution required. this keeps your stack lean compared to Hashnode, which has more aggressively rate-limited HTML endpoints.
Storage and downstream use
for most collection sizes, JSONL to Parquet is the right output path:
import pandas as pd
df = pd.read_json("devto_articles.jsonl", lines=True)
df["published_at"] = pd.to_datetime(df["published_at"])
df.to_parquet("devto_articles.parquet", index=False)for real-time feeds or collaborative pipelines, push directly to Postgres or BigQuery. the article id is a stable integer primary key — safe to use as your deduplication anchor across incremental runs.
if you are building a content trend dashboard, aggregate by tag_list + published_at week and rank by public_reactions_count. this surfaces rising topics 2-3 weeks before they appear in mainstream tech media.
Bottom line
Dev.to’s public API is one of the cleanest scraping targets in the developer content space — JSON-native, no auth required for most reads, and well-documented. start with the paginated /api/articles endpoint, enrich with full article bodies in a second pass, and add proxy rotation only when you scale past a few thousand articles per day. DRT covers the full spectrum of content and social platform scraping for data teams that need production-grade pipelines.