LLM-Driven Scraping Schemas: Auto-Generating Pydantic Models (2026)

—

If you’ve spent more than a few hours hand-writing Pydantic models for scraped data, you already know the problem: the moment the target site restructures its JSON or rearranges its HTML, your schema is wrong and your pipeline is silent about it. LLM-driven scraping schemas flip that contract — instead of maintaining models by hand, you let a language model infer structure from live content, generate the Pydantic class, and validate output in one pass.

Why Auto-Generated Schemas Beat Manual Ones

Hand-rolled schemas have two failure modes: they go stale when the source changes, and they never covered edge cases you didn’t anticipate. Both are expensive. A schema that silently accepts None where you expected a price integer will corrupt your dataset before you notice.

Auto-generation using an LLM gives you a third path: describe the data you want in natural language, point the model at a real sample page or API response, and get a typed model back. This is not a research project — Pydantic AI for Web Scraping: Type-Safe LLM Scrapers in 2026 covers the production-grade implementation of this pattern in detail. What this article focuses on is the schema generation step specifically: how to drive it, when to trust the output, and how to version it.

The Core Workflow

The minimal working loop looks like this:

1. Fetch a representative HTML or JSON sample from the target (one to five pages covers 90% of variance).
2. Send the sample to an LLM with a prompt asking for a Pydantic v2 model.
3. Parse the returned class definition, exec it into a namespace (sandboxed), and run it against the original sample as a smoke test.
4. If validation passes, write the model to a versioned file. If it fails, feed the validation error back to the LLM for a correction pass.
5. Pin the model hash. On every subsequent scrape run, compare live output against the pinned schema and alert on drift.

The feedback loop in step 4 is what makes this reliable. A single LLM call gets it right maybe 70% of the time; one correction pass pushes that to 95%+.

Prompt Engineering for Schema Generation

The quality of the generated schema depends almost entirely on the prompt. Vague prompts produce vague models with too many Optional fields and no validators. Specific prompts produce tight, opinionated models.

A prompt that works well in 2026:

SCHEMA_PROMPT = """
You are a Python data engineer. Given the following JSON sample from a product listing API,
generate a Pydantic v2 BaseModel class named `ProductListing`.

Rules:
- Use str, int, float, bool, list, dict -- no Any
- Mark a field Optional only if it is actually missing in the sample
- Add a @field_validator for `price` to reject negative values
- Add a model_config with str_strip_whitespace=True

Return ONLY the Python class definition, no imports, no explanation.

Sample:
{sample}
"""

Feeding this to claude-haiku-4-5 or gpt-4o-mini costs under $0.002 per schema generation and runs in under two seconds. For complex nested structures (think e-commerce product variants or job posting schemas), bump to claude-sonnet-4-6 — the improvement in field naming and validator logic is worth the 10x cost difference on a one-time generation call.

LLM-generated schemas also handle a problem that manual schemas consistently miss: heterogeneous arrays. If an API returns a features field that is sometimes a list of strings and sometimes a list of objects, the LLM will catch it in the sample and generate a Union type. You almost certainly would not have noticed.

Schema Versioning and Drift Detection

Generating the schema once and forgetting it recreates the original problem. The correct pattern is to treat the generated model as a migration artifact, not static code.

Store each schema version with a content hash and the generation timestamp. On every scrape run, validate a small probe batch (10-20 records) before committing the full crawl. If validation error rate exceeds 5%, trigger a regeneration cycle rather than crashing. This is conceptually similar to the selector-healing pattern described in Self-Healing Scrapers with LLMs: When Selectors Break (2026), extended to the data layer.

A practical schema registry is just a directory with one file per target:

schemas/
  ecommerce_product_v3.py   # hash: a1b2c3
  job_posting_v1.py         # hash: d4e5f6
  news_article_v2.py        # hash: 789abc

Your scraper loads the schema by name, validates, and writes the version tag to your data store alongside every record. When you query data later, you know exactly what shape it was validated against.

LLM Choice and Cost Tradeoffs

Not all models are equal for schema generation. Here’s how the main options compare on a benchmark of 50 real-world HTML and JSON samples (2026 prices):

For high-volume pipelines generating schemas frequently, Haiku or Flash with a correction pass is the economical default. For one-shot generation on complex nested schemas (where a second pass is expensive), Sonnet is worth it. The vision-based approach — screenshotting a rendered page and asking the LLM to infer structure visually — is covered in Headless Browser Cost Per LLM-Vision Call: 2026 Benchmarks, and costs roughly 20x more per call. Use it only when the target has no parseable HTML or API.

Integrating with Downstream Pipelines

A schema is only as useful as its integration point. The cleanest pattern is to bind your Pydantic model directly to your chunking and embedding step. If you’re building a retrieval pipeline over scraped content, the validated model fields become natural chunk boundaries — title, body, metadata — rather than arbitrary character splits. How to Scrape and Chunk Long-Form Articles for LLM Context (2026) walks through exactly this field-to-chunk mapping. And if your end goal is a RAG system, the schema layer is what lets you filter by structured fields before running vector search, which dramatically improves retrieval precision — see Building a RAG App on Scraped Documentation: 2026 Architecture for the full stack.

Key things your auto-generated schema should always include for downstream compatibility:

• A source_url: str field (inject at scrape time, not inferred)
• A scraped_at: datetime field with default_factory=datetime.utcnow
• A schema_version: str literal matching your registry hash
• Nested models for any sub-objects rather than raw dict

The dict trap is worth calling out explicitly. LLMs frequently generate metadata: dict for ambiguous nested fields. Push back in your prompt or correction pass — ask for a named sub-model. A dict field passes validation on anything and gives you nothing queryable later.

Bottom line

Auto-generating Pydantic models from live samples with an LLM is production-ready in 2026 and cuts schema maintenance overhead by a meaningful margin. Use Haiku or Flash for cost, Sonnet when structure is genuinely complex, and always run a correction pass against real validation errors before pinning a schema. Pair this with drift detection and you have a data layer that heals itself rather than breaking silently. DRT will keep covering the tooling and benchmarks as this pattern matures.

Related guides on dataresearchtools.com

• Building a RAG App on Scraped Documentation: 2026 Architecture
• How to Scrape and Chunk Long-Form Articles for LLM Context (2026)
• Self-Healing Scrapers with LLMs: When Selectors Break (2026)
• Headless Browser Cost Per LLM-Vision Call: 2026 Benchmarks
• Pillar: Pydantic AI for Web Scraping: Type-Safe LLM Scrapers in 2026