Author: Xavier Fok

AutoScraper Tutorial 2026: Pattern-Based Scraping Without Selectors

AutoScraper is one of the most underrated tools in a scraper’s toolkit: give it a URL and a sample value, and it reverse-engineers the CSS/XPath patterns itself. No selector hunting, no DevTools archaeology. For engineers who scrape dozens of sites and hate maintaining brittle selector files, that’s a significant time save in 2026.

How AutoScraper Works

AutoScraper uses a training-by-example model. You point it at a page and hand it one or more example values you want to extract. Internally it fetches the HTML, finds all nodes that contain your example text, and builds a set of generalized rules that will match similar nodes across pages with the same structure.

The core loop is three lines:

from autoscraper import AutoScraper

scraper = AutoScraper()
result = scraper.build(url="https://books.toscrape.com/catalogue/page-1.html",
                       wanted_list=["A Light in the Attic", "£51.77"])
print(result)

That build() call trains the scraper. After that, scraper.get_result_similar(other_url) extracts matching data from any page with the same layout. You can serialize the trained model to JSON with scraper.save("books_scraper") and reload it later, which makes it reusable across runs without retraining.

Training, Aliases, and Multi-Target Extraction

The trickiest part of AutoScraper is that build() learns rules for all wanted values simultaneously, and the output is a flat list. If you wanted both titles and prices, the result mixes them. Use aliases and rule IDs to separate them:

scraper.build(url=url, wanted_list=["A Light in the Attic", "£51.77"])

# Assign semantic labels to rules
scraper.set_rule_aliases({"rule_id_1": "title", "rule_id_2": "price"})

# Extract into named buckets
data = scraper.get_result_exact(url, grouped=True)
# {"title": ["A Light in the Attic", ...], "price": ["£51.77", ...]}

You find rule IDs by calling scraper.get_result_exact(url, grouped=True) before setting aliases; the keys are the auto-generated rule strings. It is a bit awkward, but once mapped the model is clean and portable. For sites where one wanted value trains multiple conflicting rules, use scraper.keep_rules(["rule_id_1"]) to prune noise.

Comparing AutoScraper to Other Extraction Approaches

AutoScraper fits a specific niche. Here is how it stacks up against the approaches you are most likely already using:

Approach	Selector maintenance	JS rendering needed	Setup complexity	Best for
AutoScraper	None (learned)	No	Very low	Static HTML, repeated schemas
CSS/XPath manual	High	No	Low	Precise, stable sites
Playwright/Puppeteer/Selenium	Medium	Yes	Medium	JS-heavy SPAs
Crawlee for Python	Medium	Optional	Medium	Large crawl pipelines
LLM-based (ScrapeGraphAI)	None	Optional	Medium-High	Unstructured or varied layouts

The honest tradeoff: AutoScraper is brittle the moment a site redesigns. Learned rules are tied to HTML structure. LLM-based extractors like ScrapeGraphAI handle layout drift better but cost tokens per request. AutoScraper is free at runtime once trained.

Handling Real-World Obstacles

AutoScraper ships with requests under the hood. That means anything that blocks requests will block AutoScraper. In 2026 most anti-bot stacks fingerprint TLS and HTTP/2 negotiation, which standard requests fails badly. Your options:

Pass a custom request_args dict with headers that look like a real browser.
Replace the HTTP layer entirely by monkey-patching or subclassing and using curl-cffi or HTTPX for the fetch step.
Pre-fetch the HTML yourself (with whatever client you prefer) and pass raw HTML directly via scraper.build(html=html_string, ...).

Option 3 is the cleanest. It decouples transport from extraction:

import curl_cffi.requests as cf

resp = cf.get(url, impersonate="chrome120")
result = scraper.build(html=resp.text, wanted_list=["A Light in the Attic"])

For JS-rendered pages, render with Playwright first and pipe page.content() into AutoScraper. AutoScraper has no opinion on how the HTML arrived.

Rotating Proxies

If you are scraping at scale, pass proxies through request_args:

scraper.get_result_similar(url, request_args={
    "proxies": {"http": "http://user:pass@proxy:port",
                "https": "http://user:pass@proxy:port"}
})

This works for the training step too. Use residential or mobile proxies for sites with aggressive IP scoring.

Structuring a Production AutoScraper Pipeline

For anything beyond one-off scripts, structure your AutoScraper usage around these principles:

Train once, version the model. Save JSON model files to a /models directory in your repo. Treat them like schema files, commit them, and retrain only when a site redesigns.
Validate output shape. AutoScraper returns lists, not typed objects. Pipe results into Pydantic AI models or at minimum a plain Pydantic BaseModel to catch drift early.
Detect rule decay. If get_result_similar() returns an empty list or a list shorter than a threshold, log it and alert. That almost always means the target site changed its HTML structure.
Keep training pages cached. Store the HTML that trained each model. If you need to retrain, you can diff the new HTML against the cached version to understand exactly what changed.

A simple decay check:

results = scraper.get_result_similar(url, grouped=True)
if len(results.get("title", [])) < 5:
    raise ValueError(f"Rule decay detected for {url} -- retrain required")

Bottom Line

AutoScraper earns its place for engineers who need fast, low-maintenance extraction from stable, HTML-heavy sites and do not want to manage selector files. It is not the right tool for JS-heavy SPAs, sites that redesign frequently, or use cases where schema validation matters from the start. Pair it with a modern HTTP client for TLS bypass and Pydantic for output validation and it holds up well in production. DRT covers the full scraping stack from primitives to frameworks, so if AutoScraper hits its limits, the rest of the toolchain is one article away.

Related guides on dataresearchtools.com

May 7, 2026

Pydantic AI for Web Scraping: Type-Safe LLM Scrapers in 2026

Pydantic AI landed in late 2024 and by 2026 it’s become the go-to way to build type-safe, LLM-powered scrapers that actually return structured data instead of raw text blobs. If you’ve spent time wrestling with JSON parsing failures, hallucinated field names, or retry logic scattered across notebook cells, Pydantic AI for web scraping is worth a serious look.

What Pydantic AI Brings to Scraping Pipelines

Pydantic AI wraps LLM calls behind a typed interface. You define a Pydantic model for the data you want, pass it to the agent, and get back a validated Python object — not a string you have to parse yourself. The library handles retries, validation errors, and model switching out of the box.

For scraping this matters because the hardest part of LLM-assisted extraction isn’t prompting, it’s reliability. A scraper that works 90% of the time and silently drops 10% of records is worse than one that fails loudly. Pydantic AI’s validation layer forces the LLM to conform or retry, and when it can’t, it raises a typed exception you can catch and log.

Compare this to raw LLM calls or even Crawl4AI’s extraction mode, which gives you markdown and leaves structured parsing to you. Pydantic AI sits one layer above: you still feed it cleaned HTML or markdown, but the output contract is enforced.

Setting Up a Basic Pydantic AI Scraper

Install the stack:

pip install pydantic-ai httpx crawl4ai

Define your schema and agent:

from pydantic import BaseModel
from pydantic_ai import Agent
import httpx

class JobPosting(BaseModel):
    title: str
    company: str
    salary_range: str | None
    location: str
    remote: bool

agent = Agent(
    "openai:gpt-4o-mini",
    result_type=JobPosting,
    system_prompt="Extract the job posting details from the HTML. Return null for salary_range if not listed.",
)

async def scrape_job(url: str) -> JobPosting:
    async with httpx.AsyncClient() as client:
        html = (await client.get(url)).text
    result = await agent.run(html[:8000])  # trim to token budget
    return result.data

The result.data is a validated JobPosting instance. If the LLM returns malformed JSON or omits a required field, Pydantic AI retries up to the configured limit before raising UnexpectedModelBehavior. No silent failures.

For the HTTP layer, the choice matters more than people think. If the target site uses TLS fingerprinting, plain httpx will get blocked. Comparing httpx, curl-cffi, and niquests shows curl-cffi as the 2026 default for anti-bot targets — it’s a drop-in replacement for the client.get() call above.

When to Use LLM Extraction vs. CSS Selectors

Not every scraper should use an LLM. Here’s an honest breakdown:

Scenario	LLM extraction	CSS/XPath selectors
Schema varies per site	yes	painful
Schema is stable, high volume	overkill	preferred
Unstructured text (reviews, bios)	yes	no
Price / SKU grids	marginal	preferred
JS-rendered SPAs	pair with browser	pair with browser
Cost sensitivity	~$0.002/page (gpt-4o-mini)	~$0

The cost column is the honest check. At $0.002 per page with gpt-4o-mini, a 100K page crawl costs $200 in LLM calls alone — before proxies or infra. For stable schemas at scale, selectors win. LLM extraction is the right call when the schema is inconsistent across sources or when you’re extracting meaning from prose, not structured fields.

AutoScraper’s pattern-based approach sits between these two extremes — no selectors, no LLM costs, but it breaks on layout changes. Pydantic AI handles layout changes gracefully since it reads semantic content.

Handling JavaScript-Rendered Pages

Most 2026 targets require a browser. The standard pattern is to pair Pydantic AI with a browser layer that handles rendering, then pass the cleaned text to the agent.

Steps for a Playwright + Pydantic AI pipeline:

Launch a browser context with Playwright (stealth mode, real user-agent)
Navigate and wait for the target element or network idle
Extract innerText or the full page HTML, trimmed to token budget
Pass to the Pydantic AI agent for structured extraction
Validate result, retry on ValidationError, log failures with the raw HTML for debugging

Playwright beats Puppeteer and Selenium for this use case in 2026 because its async API integrates cleanly with Pydantic AI’s async agent interface — no thread bridging, no sync wrappers.

For teams that want a managed crawl layer instead of raw Playwright, Crawlee for Python handles request queuing, retries, and session rotation, and can pipe rendered HTML directly into a Pydantic AI extraction step. It’s a good fit when you’re crawling hundreds of pages with structured output requirements.

Model Selection and Cost Control

The main levers:

gpt-4o-mini: default choice, fast, cheap, handles well-structured HTML reliably
claude-haiku-3-5: slightly better at prose extraction, similar cost tier
gpt-4o: for complex nested schemas or ambiguous content, 10x the cost
local models (ollama): zero API cost, 3-5x slower, accuracy drops on noisy HTML

Pydantic AI lets you swap models per agent or per run, so you can route simple extractions to mini and fall back to a stronger model on retry. A practical pattern: catch UnexpectedModelBehavior on the first run with mini, then retry once with gpt-4o before logging as a permanent failure.

Keep prompts tight. Token bloat is the main cost driver. Strip